How GPUs Became the Engine of AI: Parallelism, CUDA, AlexNet and Tensor Cores

Jump to comments

Rows of black Summit supercomputer cabinets with blue lights, stretching into the distance in a large machine hall at Oak Ridge National Laboratory
The Summit supercomputer at Oak Ridge National Laboratory, which linked 27,648 NVIDIA V100 GPUs for science and AI.
Photo: Carlos Jones / Oak Ridge National Laboratory, 2018, via Wikimedia Commons (CC BY 2.0)

The chips that once painted video-game frames have become the usual engine for training large AI models. That was not an accident of branding. It followed from a simple fact: neural networks spend most of their time multiplying large matrices, and a graphics processor is a machine for doing thousands of those multiplications at once.

The short answer: A central processing unit is built to do a few complicated things quickly. A graphics processing unit is built to do the same simple math, in parallel, across thousands of tiny cores. In 2006 and 2007, NVIDIA opened that hardware to ordinary C programs through CUDA. Researchers soon found they could train much larger models. In 2012, a network called AlexNet, trained in five to six days on two GeForce GTX 580 cards, won the ImageNet contest by a wide margin and showed the field what a GPU-trained model could do. Later chips added tensor cores aimed squarely at matrix math, stacked high-bandwidth memory beside the processor, and tied groups of GPUs together with interconnects far faster than a normal PCI Express slot. One card was never enough; today’s training clusters exist because the math, the memory and the wiring all have to move together.

Why a graphics chip beats a CPU at this math

A CPU is a generalist. It has a handful of powerful cores, large caches and a lot of logic for running an operating system, jumping through irregular code and finishing a single task with as little delay as possible. That design is ideal for a web browser, a compiler or a database. It is a poor match for the inner loop of a neural network, which applies the same operation to enormous grids of numbers, over and over.

A GPU is organized the other way around. Instead of a few complex cores, it has thousands of much simpler ones, scheduled in large groups, and a memory system built to feed them a steady stream of data. The GeForce 8800 GTX, the first CUDA-capable card, already had 128 stream processors running at 1.35 GHz. The GeForce GTX 580 that trained AlexNet had 512 CUDA cores. NVIDIA’s H100 SXM accelerator, a later data-center chip, has 132 streaming multiprocessors and 16,896 32-bit CUDA cores. The point is not that any one of those cores would beat a CPU core on a messy, branching program. It is that the work of a neural network can be split into a huge number of nearly identical pieces, and the GPU can keep all of those pieces in flight at once.

Most of that work is matrix multiplication, the operation linear-algebra libraries call GEMM. A single layer of a network takes a batch of inputs, multiplies them by a weight matrix, and adds a bias. Convolution, the operation that made early image models work, can be rearranged into the same kind of multiply. The attention math inside later language models is matrix multiplication too. Graphics was already full of this pattern: transforming vertices, filtering textures and painting the lighting in a game scene are all dense linear algebra on regular grids. The hardware that made games look better turned out, almost for free, to be a superb engine for training.

There is a second constraint that matters as much as raw math: memory bandwidth, the rate at which numbers can be moved into those cores. A core that is waiting on memory is a core that is not computing. The 8800 GTX’s 384-bit GDDR3 interface delivered 86.4 gigabytes per second. The GTX 580, with GDDR5, reached 192.4 GB/s. Later AI accelerators switched to high-bandwidth memory stacked on the package, and the figures jumped into the terabytes per second. The story of GPUs in AI is as much a story of feeding the chip as of counting operations.

CUDA turns a game chip into a scientific computer

Before CUDA, using a GPU for anything but graphics meant disguising a scientific calculation as a graphics shader, a clever but brittle trick that only a small community attempted. NVIDIA changed that in two steps. On November 8, 2006, the company unveiled CUDA, short for Compute Unified Device Architecture, together with the GeForce 8800 and what it called the industry’s first C compiler for the GPU. On February 16, 2007, it released a public beta of the CUDA software kit and that compiler for Windows XP and Linux, opening the G80 architecture to ordinary developers in standard C.

A close-up photograph of the G80 silicon die from an NVIDIA GeForce 8800 GTX, showing the chip's colorful rectangular blocks of circuitry
The silicon die of NVIDIA’s G80 chip, the GeForce 8800 GTX processor that first shipped with CUDA in 2006.
Photo: Fritzchens Fritz, 2016, via Wikimedia Commons (CC0)

The 8800 GTX was a serious piece of parallel hardware for its day. NVIDIA’s own architecture brief described 128 scalar stream processors at 1.35 GHz, each able to issue a multiply-add and a multiply in the same clock, for "roughly 520 gigaflops of raw shader horsepower." The card carried 768 MB of GDDR3 memory on a 384-bit bus running at 900 MHz, for 86.4 GB/s of bandwidth. CUDA let programmers launch thousands of lightweight threads onto those processors, with on-chip shared memory so threads in a group could exchange data without a round trip to main memory.

Andy Keane, then general manager of GPU computing at NVIDIA, put the pitch in the 2007 announcement: the GPU had become "a powerful, programmable processor that is ideally suited for computationally intensive applications such as seismic processing for oil and gas exploration, computing in bioscience, and financial modeling." Machine learning was not yet the headline. The door, though, was open. Anyone who could write C could now aim a massively parallel chip at a matrix.

Researchers climb on, then AlexNet changes the field

The first widely cited demonstration that this was more than a curiosity came in 2009. Rajat Raina, Anand Madhavan and Andrew Ng at Stanford trained deep belief networks and sparse-coding models on an NVIDIA GeForce GTX 280, a 240-core card with 1 GB of memory. Their ICML paper reported that deep-belief-network learning was up to 70 times faster than a dual-core CPU implementation. A four-layer network with 100 million free parameters, they wrote, went from "several weeks" on the CPU to "around a single day" on the GPU. Their spatial convolution, including the time to copy images and filters onto the card, was more than 100 times faster than a C or MATLAB baseline.

Three years later the Toronto team of Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton entered a much larger convolutional network, later nicknamed AlexNet, in the ImageNet Large Scale Visual Recognition Challenge. ImageNet itself held more than 15 million labeled high-resolution images in roughly 22,000 categories. The contest used a 1.2 million-image training set and 1,000 classes. The network had 60 million parameters and 650,000 neurons, arranged as five convolutional layers and three fully connected layers ending in a 1,000-way classifier. On the 2010 contest test set it reached top-1 and top-5 error rates of 37.5% and 17.0%. In the 2012 contest, a variant won with a top-5 test error of 15.3%, against 26.2% for the second-best entry.

An EVGA GeForce GTX 580 graphics card with a large copper-colored heatsink and a green circuit board, photographed from above
An EVGA GeForce GTX 580 from 2010, the Fermi-architecture card used, in its 3GB version, to train AlexNet.
Photo: TheStriker, 2023, via Wikimedia Commons (CC BY-SA 4.0)

They trained it with a highly optimized GPU implementation of two-dimensional convolution, later released as cuda-convnet, on two NVIDIA GTX 580 3GB cards. A single GTX 580, the paper noted, "has only 3GB of memory, which limits the maximum size of the networks that can be trained on it." The 1.2 million training examples were enough to train a net too big for one card, so they split the kernels across two GPUs that could read and write each other’s memory directly, without going through the host. They trained for roughly 90 passes over the training set. That took five to six days.

NVIDIA’s later official comparison of the GTX 580’s GF110 chip listed 512 CUDA cores, 1,581 gigaflops of single-precision math, 192.4 GB/s of memory bandwidth and a 244-watt thermal design power, with 3.0 billion transistors on a 40-nanometer process. (The consumer card launched in 2010 at an estimated $499 with 1.5 GB of memory; the Toronto team used the 3GB version the paper specifies.) Two such cards, running for the better part of a week, were enough to beat the field. The authors were blunt about what would come next.

All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.

They were right. After 2012, the default assumption in computer vision, and soon in speech and language, was that a serious model would be trained on GPUs. The limit was no longer whether the idea could work. It was how much memory you had on the card, and how long you were willing to wait.

Tensor cores: silicon built for matrix multiply

CUDA made general-purpose math programmable on the GPU. Tensor cores made the most important of those operations a first-class piece of silicon. NVIDIA introduced them in 2017 on the Volta architecture, in the Tesla V100 data-center GPU. A tensor core is a small matrix-multiply unit. Instead of asking thousands of CUDA cores to accumulate a multiply-add one element at a time, the chip includes hardware that multiplies small tiles of matrices in a single step.

The V100 has 640 tensor cores, eight per streaming multiprocessor, plus 5,120 ordinary CUDA cores. According to NVIDIA’s Volta white paper, each tensor core performs 64 floating-point multiply-add operations per clock, so the eight cores in one multiprocessor perform 512 multiply-adds, or 1,024 individual floating-point operations, each cycle. On the V100 SXM2 module, NVIDIA rated peak tensor performance at 125 teraflops, against 15.7 teraflops of ordinary 32-bit math and 7.8 teraflops of 64-bit math. That is not a small bump. It is nearly an eightfold jump in peak throughput for the operation networks actually run, provided the software uses the mixed-precision formats those cores expect.

An NVIDIA H100 PCIe accelerator, a long dark-green circuit board with a large metal heat sink and a small NVIDIA logo, photographed on a dark surface
An NVIDIA H100 PCIe accelerator, a Hopper-generation data-center GPU used for large-scale AI training and inference.
Photo: 极客湾Geekerwan, 2023, via Wikimedia Commons (CC BY 3.0)

Later accelerators pushed the same idea further and added a caveat that matters when you read a datasheet. The A100, based on the Ampere architecture, is rated at 312 teraflops of 16-bit tensor math, or 624 teraflops with structured sparsity, a feature that can skip known zeros in a model. An 80 GB A100 SXM module carries HBM2e memory at 2,039 GB/s and has a 400-watt thermal design power; the PCIe version is listed at 1,935 GB/s and 300 watts. NVIDIA’s NVLink on A100 is rated at 600 GB/s, against 64 GB/s for PCI Express 4.0 in the same table.

The H100, on the Hopper architecture, is the cleanest illustration of how far the numbers have moved. The SXM version has 16,896 32-bit CUDA cores and 528 fourth-generation tensor cores. NVIDIA’s official specifications list 67 teraflops of 32-bit math and, with sparsity, 1,979 teraflops of 16-bit tensor math and 3,958 teraflops of 8-bit tensor math. Without sparsity, those tensor figures are half as large: 989 teraflops for 16-bit and 1,979 teraflops for 8-bit. The asterisk on the datasheet is easy to miss and easy to misuse. The dense 16-bit tensor rating is already roughly eight times the V100’s 125 teraflops, and about 625 times the GTX 580’s 1.581 teraflops of 32-bit math, though the precision, the year and the intended workload are not the same.

Memory bandwidth, HBM and the wall beyond FLOPS

Peak teraflops are a ceiling, not a promise. If the chip cannot stream weights and activations into the cores fast enough, the extra math units sit idle. That is why memory bandwidth rose at least as dramatically as compute, and why high-bandwidth memory, or HBM, became part of the AI-GPU story.

HBM stacks DRAM dies on the same package as the processor and connects them with a very wide interface, so more bits can move on each clock than a conventional graphics-card memory bus allows. The V100 used HBM2: 900 GB/s on the 16 GB configuration, and 1,134 GB/s on the 32 GB SXM2 module, at up to 95% utilization in NVIDIA’s STREAM measurements. The A100 80 GB SXM module, with HBM2e, is listed at 2,039 GB/s. The H100 SXM module was the first NVIDIA GPU to use HBM3 and is rated at 3.35 terabytes per second, with 80 GB on the package. In rough terms, bandwidth from the 8800 GTX to the H100 SXM rose from 86.4 GB/s to 3,350 GB/s, a factor of about 39, while the power envelope went from a gaming card to a 700-watt accelerator.

The other half of the memory problem is capacity. AlexNet already could not fit on one 3 GB card. Today’s large models have billions or hundreds of billions of parameters, and training also has to store optimizer state, activations and gradients. Eighty gigabytes on one H100 is a lot of memory on a single package, and it is still not enough for the largest training runs, which is why the models are split across many GPUs. Bandwidth decides how fast one chip can work. Capacity decides how many chips you need before you even start.

Power per chip, and why one GPU is never enough

The power story runs in parallel with the performance story. NVIDIA rated the GTX 580 at 244 watts. The V100 SXM2 is listed at 300 watts, the A100 SXM at 400 watts, and the H100 SXM at up to 700 watts, configurable. Those figures are thermal design power, the budget the cooling system must be designed around, not a running average, but they are the right scale. Eight H100 SXM modules in one server can draw 5.6 kilowatts for the GPUs alone, before the CPUs, memory, networking and power-supply losses are counted. A hall of such servers is why AI data centers have become a power and cooling problem, and why the industry has had to revisit how those rooms stay cool.

Power also limits how much performance you actually get. A datasheet teraflop number assumes the chip can be kept at its boost clocks, which depends on temperature, power delivery and the workload’s mix of math and memory traffic. The V100 even had an official "maximum efficiency" mode that, NVIDIA said, could deliver up to 80% of performance at half the power. The useful question for a training run is not only how many teraflops the silicon can theoretically issue, but how many useful operations you can finish per watt, per dollar of hardware and per hour of wall-clock time.

And even a fully fed, fully cooled flagship is still one chip. AlexNet’s authors hit the wall at 3 GB and split one model across two consumer cards. Everything since has been a larger version of that same compromise: the model, the batch of data, or both, have to be partitioned, and the pieces have to talk to each other.

Why training clusters need fast interconnects

Once a model no longer fits in one GPU, or once training is split across many GPUs to finish sooner, communication becomes part of the critical path. In data-parallel training, each GPU works on a different slice of the batch, then the chips must combine, or all-reduce, their gradient updates before the next step. In model-parallel training, layers or slices of a layer live on different GPUs, and activations have to move between them. If that traffic is slow, expensive math units wait.

A plain PCI Express slot is not slow in ordinary-computer terms, but it is slow compared with the memory fabric on the GPU itself. NVIDIA’s V100 datasheet lists 32 GB/s of interconnect bandwidth for the PCIe version and 300 GB/s for the SXM2 version over NVLink, the company’s dedicated GPU-to-GPU link. On the A100, NVLink is listed at 600 GB/s against 64 GB/s for PCI Express 4.0. On the H100 SXM, NVLink is listed at 900 GB/s against 128 GB/s for PCI Express 5.0. In each generation, the dedicated link is several times faster than the standard slot that would otherwise connect the card to the rest of the machine.

That still only covers GPUs that can be wired to one another inside a server, usually with NVLink and a switch such as NVSwitch. A large training run spans many servers. Those servers have to be connected by a cluster network, typically InfiniBand or a similar high-performance fabric, so that all-reduce and parameter-exchange traffic can leave the box. The topology, the number of rails and the congestion on that network decide whether a 100-GPU job actually runs at 50 times the speed of a two-GPU job, or far less.

Oak Ridge National Laboratory’s Summit supercomputer, launched in 2018, is a clear public example of the two-level design. Official OLCF specifications list 4,608 nodes, each with two IBM POWER9 CPUs and six NVIDIA V100 GPUs, for 27,648 V100s in all. Inside a node, the CPUs and GPUs share a coherent memory space and are linked with NVLink; each node has 512 GB of ordinary DDR4 memory plus 96 GB of HBM2 on the GPUs. Between nodes, Summit uses a non-blocking fat-tree of dual-rail Mellanox EDR 100-gigabit InfiniBand. The laboratory rated the machine at a peak of 200 petaflops of 64-bit math, and at 3.3 exaops of mixed-precision math for some scientific applications, with a peak power consumption of 13 megawatts. The same wiring that makes a supercomputer fast at physics also makes it fast at training: lots of GPUs, a very fast link inside the node, and a fast network between nodes.

What this adds up to

The path from a 2006 game card to a hall of accelerators is a straight line once you look at the numbers. A GPU won the AI workload because the workload is parallel matrix math, and the GPU was already a parallel matrix machine. CUDA made that machine programmable. AlexNet, trained in less than a week on two 3GB GTX 580s, proved that a GPU-trained model could beat the state of the art on a hard, public benchmark. Tensor cores then specialized the silicon for the multiply itself. High-bandwidth memory raised the rate at which those cores can be fed. Fast interconnects, inside the server and between servers, made it possible to treat many chips as one trainer when a single package ran out of memory or time.

The constraints have not gone away. They have only moved. Memory capacity still decides how a model must be split. Bandwidth still decides how much of the peak teraflop rating you actually see. Interconnects still decide whether adding GPUs speeds the job up or just adds waiting. Power and cooling now sit beside those limits, because a modern accelerator can draw several times the wattage of the card that trained AlexNet, and a cluster draws thousands of times more. The engine of AI is still the GPU. It is just no longer a single card in a workstation. It is a roomful of them, wired to act like one.

Further reading

Cover of Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville, published by MIT Press
Deep Learning (Adaptive Computation and Machine Learning series)

The standard textbook on the math and training of neural networks, including the matrix operations GPUs were built to run.

Cover of Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow by Aurélien Géron
Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow

Aurélien Géron’s practical guide to building and training models, including convolutional networks in the line that began with AlexNet.

Cover of AI Engineering: Building Applications with Foundation Models by Chip Huyen, published by O'Reilly
AI Engineering: Building Applications with Foundation Models

Chip Huyen’s guide to building on large trained models, from choosing and evaluating one to fine-tuning and serving it.

Cover of Build a Large Language Model (From Scratch) by Sebastian Raschka, published by Manning
Build a Large Language Model (From Scratch)

Sebastian Raschka walks through coding a small GPT-style model step by step, so the matrix math behind training is visible in the code.

Cover of Designing Machine Learning Systems by Chip Huyen, published by O'Reilly
Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications

A clear look at how machine-learning systems are designed for production, including data, training pipelines and the constraints of real hardware.

Aglena Best Of product picks

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top
Aglena