English NewsMonday, 16 November 2026
Latest

AlexNet: What It Is, How Deep Learning Began, and the Role CUDA Played

2026-11-16

Illustrative image: A convolutional neural network: an input image passes through convolution and pooling layers to a classification output; illustrative of the deep learning approach AlexNet popularised. Image: Irisbox, CC BY 4.0, via Wikimedia Commons.

Illustrative image: A convolutional neural network: an input image passes through convolution and pooling layers to a classification output; illustrative of the deep learning approach AlexNet popularised. Image: Irisbox, CC BY 4.0, via Wikimedia Commons.

Introduction: What Is AlexNet, and Why Does 2012 Still Matter?

In September 2012, a computer program built by three researchers at the University of Toronto did something no machine had done before. It looked at photographs it had never seen - cats, cars, mushrooms, lighthouses, dogs of 120 different breeds - and named what was in them correctly far more often than any system built by the world's largest technology companies. The program was called AlexNet. Its error rate was 15.3 percent. The second-best entry in the same competition made mistakes 26.2 percent of the time. A gap that large did not look like ordinary progress. It looked like a different kind of machine altogether, and in a sense it was. This is the story of where that machine came from: the decades-old idea of the neural network, the mathematical method called deep learning, the image database that made a fair contest possible, and a piece of graphics hardware - and the software platform called CUDA that unlocked it - without which none of it would have happened in 2012.

1. The Neural Network: An Old Idea Waiting for Its Time

The idea behind AlexNet is older than most of the computers alive when it won. In 1943, Warren McCulloch and Walter Pitts proposed a mathematical model of a brain cell - a unit that receives signals, adds them up, and fires when the total crosses a threshold. In 1958, Frank Rosenblatt turned this into the Perceptron, a machine that could learn simple visual patterns by adjusting the strength of its connections after each mistake. The excitement was enormous and short-lived. In 1969, Marvin Minsky and Seymour Papert proved that a single-layer Perceptron could not even learn some basic logical functions, and funding for neural-network research collapsed. The field entered its first long winter.

What revived it was a way to train networks with more than one layer. In 1986, David Rumelhart, Geoffrey Hinton and Ronald Williams popularised backpropagation - a method that takes a network's error and pushes it backwards through every layer, telling each connection exactly how much it contributed to the mistake. Multi-layer networks could now, in principle, learn anything. In 1989 and through the 1990s, Yann LeCun at Bell Labs built LeNet, a convolutional neural network (CNN) that read handwritten digits on bank cheques and was used commercially. In 1997, Sepp Hochreiter and Jurgen Schmidhuber introduced LSTM (Long Short-Term Memory) for sequences. The mathematics worked. What did not work was the hardware: networks deep enough to be interesting took weeks to train on the central processing units (CPUs) of the day, so the experiments stayed small, and small networks kept losing to hand-engineered alternatives. In 2006, Hinton and his colleagues showed that deep networks could be trained layer by layer and coined the modern term "deep learning". The idea had found its name. It still needed its engine.

2. The Missing Engine: The GPU and CUDA

Training a neural network is, at heart, one operation repeated billions of times: multiply large grids of numbers and add the results. A CPU is designed to do a few complicated things very quickly, one after another. A graphics processing unit (GPU) is designed for the opposite problem - colouring millions of pixels on a screen at once - so it carries hundreds of small arithmetic units that work in parallel. Pixel shading and neural-network training turn out to be the same kind of mathematics.

Researchers understood this early, but programming a graphics card to do anything other than graphics was painful. The code had to be disguised as drawing commands. The breakthrough came from NVIDIA. In 2000, Ian Buck and a small team at Stanford University had begun exploring how GPUs could run general-purpose programs; Buck later joined NVIDIA, and on 8 November 2006 the company released CUDA (Compute Unified Device Architecture) alongside the GeForce 8800 - the first C-based programming environment that let developers use the GPU's hundreds of cores directly, without pretending to draw anything. Scientists in physics, chemistry and finance adopted it within a year. In machine learning, a University of Toronto graduate student named Alex Krizhevsky began writing CUDA code to train small convolutional networks on consumer graphics cards, and in 2010 he published cuda-convnet, a fast CUDA implementation of CNN training that he made freely available. By 2012, all the pieces existed in one laboratory: Hinton's deep-learning mathematics, Krizhevsky's CUDA code, and a competition big enough to settle the argument.

3. ImageNet: The Test That Made the Contest Fair

For decades, vision researchers argued over methods without a common measuring stick. Fei-Fei Li and her collaborators changed that with ImageNet, a project begun in 2007 and released in 2009: a database that grew to more than 14 million labelled images organised into tens of thousands of categories. From 2010, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) ran an annual contest on a subset of it: 1,000 categories, about 1.2 million training images, and a headline metric called the top-5 error rate - the percentage of test images where the correct label is not among the program's five best guesses. Five guesses may sound generous, but the categories include 120 breeds of dog, and many photographs contain several objects. In 2010, the best system in the world, built on hand-crafted features and support vector machines (SVMs), got the answer into its top five only 72 percent of the time: an error rate of about 28.2 percent. In 2011 the winner improved to roughly 25.8 percent. Progress was real, but measured in single percentage points.

4. AlexNet: What the Machine Actually Was

AlexNet - built by Krizhevsky, Ilya Sutskever and Hinton, and named after its first author - was a deep convolutional neural network at a scale nobody had dared before. It contained 60 million adjustable parameters and 650,000 artificial neurons arranged in five convolutional layers followed by three fully connected layers, ending in a 1,000-way softmax that produced a probability for every category. Several choices inside it mattered as much as its size:

- ReLU (Rectified Linear Unit) neurons. Where older networks used saturating functions that slow learning to a crawl in deep models, ReLUs - which simply pass positive values through and zero out negative ones - let a network this deep train several times faster.

- Dropout. During training, half the neurons in the two largest layers were randomly switched off on every pass, forcing the network not to rely on fragile coincidences. This cut overfitting dramatically.

- Data augmentation. The 1.2 million training images were stretched into many more by cropping, flipping and slightly altering colours, so the network learned what things look like rather than memorising photographs.

- Two GPUs. The network was too large for one card, so it was split across two NVIDIA GeForce GTX 580 cards with 3 GB of memory each, each card running half the neurons, with the halves exchanging information at chosen layers.

AlexNet at a glance
AuthorsAlex Krizhevsky, Ilya Sutskever, Geoffrey Hinton (University of Toronto)
Architecture5 convolutional layers + 3 fully connected layers
Parameters60 million
Neurons650,000
Training data1.2 million ImageNet images, 1,000 categories
Hardware2 x NVIDIA GeForce GTX 580 (3 GB), CUDA
Training time5 to 6 days
ILSVRC-2012 top-5 error15.3% (winner); runner-up 26.2%

5. CUDA's Share: Why Performance Was the Winning Difference

None of the ideas inside AlexNet was a secret. Convolutional networks were 23 years old. ReLUs and dropout came from published research. What Krizhevsky and his colleagues did that nobody else had done was train a network of this size on data of this size - and that was a performance achievement before it was anything else. The GTX 580, a gaming card costing a few hundred dollars, delivered on the order of a trillion floating-point operations per second; a high-end CPU of the era delivered tens of billions. For the matrix arithmetic that dominates CNN training, that is not a small advantage. The same 90 training passes over 1.2 million images that took AlexNet five to six days on two GPUs would, on contemporary CPUs, have taken months - long enough that the experiment would simply never have been run, let alone tuned and re-run until it won.

CUDA was what made the card usable for this. Krizhevsky's cuda-convnet library implemented the convolution itself - the mathematical heart of the network - as highly tuned CUDA code, and the team released it publicly so others could repeat the feat. Just as important, the two-GPU split showed the way forward: because CUDA let the programmers control exactly how work and data moved between cards, a model that did not fit on one GPU could still be trained. That lesson - split big models across many GPUs - is now the foundation of every large AI system on Earth. AlexNet did not merely use CUDA to go faster. It proved that CUDA-class parallel computing was the thing deep learning had been waiting for since 1986.

6. The Result That Ended the Argument

When the ILSVRC-2012 results were announced at a workshop in Florence in October 2012, the room understood at once. AlexNet's top-5 error of 15.3 percent was not an edge over the runner-up's 26.2 percent; it was a gulf of nearly eleven points, after a decade in which the field's best minds had fought for single points. The Toronto team had also entered the 2010 data and beaten the earlier state of the art there too (top-5 error 17.0 percent, where the previous best had been 28.2 percent). The effect on the research community was immediate. In 2013, almost every serious entry in the competition was a deep convolutional network. By 2014, GoogLeNet had pushed the error to 6.7 percent; by 2015, Microsoft's ResNet reached 3.57 percent - better, by this measure, than the performance of a careful human labeller.

YearILSVRC winnerApproachTop-5 error
2010NEC-UIUCHand-crafted features + SVM28.2%
2011Xerox Research (XRCE)Improved features + SVM25.8%
2012SuperVision (AlexNet)Deep CNN trained on GPUs with CUDA15.3%
2013Clarifai (ZFNet)Deep CNN11.2%
2014GoogLeNetDeep CNN (22 layers)6.7%
2015ResNet (Microsoft Research)Deep CNN (152 layers)3.57%

7. What Followed

The consequences spread in every direction. Google acquired the Toronto team's small company, DNNresearch, in 2013, and deep learning moved from a research fringe to the centre of the industry: speech recognition, machine translation, medical imaging, recommendation systems and self-driving research were rebuilt on GPU-trained networks within a few years. Krizhevsky's cuda-convnet was succeeded by frameworks - Caffe, Torch, TensorFlow, PyTorch - nearly all of them built on NVIDIA's CUDA platform and its cuDNN deep-learning library, released in 2014. The hardware escalated from two gaming cards to data centres of tens of thousands of GPUs, and the models from 60 million parameters to hundreds of billions. Every one of the large AI systems in the headlines today, whatever company built it, descends from the demonstration made in 2012: that a very deep neural network, trained on a very large dataset, on parallel hardware programmed through CUDA, outperforms everything else by a margin that changes what a field believes is possible.

Conclusion

AlexNet is remembered as the moment deep learning arrived, but it is more precise to call it the moment three separate histories converged. The neural network, an idea from 1943, had spent decades waiting for a training method; backpropagation, from 1986, had spent decades waiting for computing power; and CUDA, from 2006, had been built for general science without knowing that its most important customer would be artificial intelligence. When a graduate student's gaming graphics cards met a million-image database and sixty years of mathematics, the result beat the best systems on Earth by eleven points and settled a scientific argument in a single afternoon. Everything since - every chatbot, every translated sentence, every image a machine can see - runs on the lesson of 2012: big networks, big data, and massively parallel hardware. AlexNet was where those three words first met.

More AI technology news