English NewsMonday, 16 November 2026
Latest

From Artificial Neurons to AlexNet and Beyond: The Formation of Deep Learning and Its Relationship with CUDA

2026-11-16

Illustrative image: A convolutional neural network: an input image passes through convolution and pooling layers to a classification output; illustrative of the deep learning approach AlexNet popularised. Image: Irisbox, CC BY 4.0, via Wikimedia Commons.

Illustrative image: A convolutional neural network: an input image passes through convolution and pooling layers to a classification output; illustrative of the deep learning approach AlexNet popularised. Image: Irisbox, CC BY 4.0, via Wikimedia Commons.

Abstract

The success of modern artificial intelligence did not originate from one discovery. It emerged from decades of research into artificial neurons, learning algorithms, convolutional networks and sequence models, combined with large datasets and faster computing. NVIDIA’s CUDA made its GPUs more accessible for general-purpose parallel computation. In 2012, AlexNet brought several existing advances together and produced a striking result on the ImageNet image-classification benchmark.

This article traces that history and explains how neural networks, deep learning, GPUs, CUDA and AlexNet are related. It then follows major developments in computer vision through convolutional networks, Transformers and multimodal models.

Important correction: Deep learning and computer vision did not begin with AlexNet in 2012. AlexNet was a major turning point built on work that began many decades earlier.

1. The essential relationship

Before examining the history, it helps to distinguish the components:

  • Artificial neural network: a mathematical model made of connected computational units with adjustable parameters.
  • Deep learning: methods that train models with multiple layers of representations.
  • Convolutional neural network (CNN): a neural-network architecture that uses learned filters and is particularly useful for spatial data such as images.
  • GPU: a processor designed to perform many operations in parallel.
  • CUDA: NVIDIA’s platform and programming model for general-purpose computation on its GPUs.
  • ImageNet: an image dataset and associated benchmark ecosystem.
  • AlexNet: a particular deep CNN and training system that achieved a landmark ImageNet result.

A simple way to understand their relationship is:

Research supplied the models and learning methods → data supplied examples → GPUs supplied parallel computing → CUDA helped researchers programme NVIDIA GPUs → AlexNet combined these ingredients in a breakthrough experiment.

This is a synthesis of the historical evidence, not a claim that any one ingredient was sufficient on its own.

2. 1943–1958: The earliest artificial neurons

In 1943, Warren McCulloch and Walter Pitts described an idealised mathematical model of a neuron using logic. Their work helped establish that networks of simple units could represent complex computations. It was a conceptual model, not a modern neural network trained on a large dataset or a complete simulation of the human brain. McCulloch and Pitts, 1943

In 1958, Frank Rosenblatt described the perceptron, a learning system for recognising patterns. The central idea was that a system could adjust parameters using examples instead of relying only on explicitly programmed classification rules. Rosenblatt, 1958

An artificial neuron can be expressed simply as:

output = activation(weighted inputs + bias)

The weights influence the importance of each input. The activation function introduces a transformation that helps a network represent more complex relationships. Modern networks contain many such operations arranged into layers.

3. 1980–1998: Hierarchical vision and learning hidden features

The Neocognitron

In 1980, Kunihiko Fukushima proposed the Neocognitron, a hierarchical model for visual-pattern recognition. It was an important precursor to CNNs and was designed to tolerate shifts in the position of patterns. Its original learning procedure should not be confused with the supervised backpropagation later used in AlexNet. Fukushima, 1980

Backpropagation

A key challenge was how to adjust the parameters in the hidden layers of a multilayer network. In 1986, David Rumelhart, Geoffrey Hinton and Ronald Williams published influential work showing how backpropagation could be used to learn useful internal representations. Earlier contributions to the underlying method existed; the 1986 paper should not be described as the invention of every form of backpropagation. Rumelhart, Hinton and Williams, 1986

In simplified form, training involves:

  • 1. Supplying an input and producing a prediction.
  • 2. Measuring the difference using a loss function.
  • 3. Computing how parameters contributed to that loss through backpropagation.
  • 4. Updating the parameters using an optimiser.
  • 5. Repeating the process over many examples.

A common gradient-descent update is:

new parameters = old parameters − learning rate × gradient of loss

Backpropagation computes gradients; the optimiser uses them to update parameters. These ideas are mathematical and do not depend on CUDA.

LeNet and handwritten characters

In 1989, Yann LeCun and colleagues applied backpropagation to handwritten postcode recognition. Later LeNet work demonstrated CNNs for document recognition. These systems learned visual features and showed that convolutional networks were useful well before 2012. LeCun and colleagues, 1989; LeNet research, 1998

CNNs use local receptive fields and shared learned filters. A filter is applied across different parts of an image, enabling a network to detect patterns without learning a completely separate set of parameters for every location.

Early layers may respond to simple visual patterns; later layers can combine representations into more complex features. Exactly what a unit learns depends on the data and training process.

4. Deep learning developed beyond vision

The history of deep learning is not simply a sequence of image classifiers.

Research into recurrent neural networks (RNNs) addressed sequences such as text and speech. Work in the 1990s examined why learning long-term dependencies with gradients was difficult. In 1997, Sepp Hochreiter and Jürgen Schmidhuber introduced Long Short-Term Memory (LSTM), an influential architecture for sequence learning. Bengio and colleagues; Hochreiter and Schmidhuber, 1997

In 2006, Hinton, Simon Osindero and Yee-Whye Teh proposed an efficient learning approach for deep belief networks, contributing to renewed interest in multilayer learning. This was an important milestone, but it was neither the birth of all deep networks nor the exact training approach used for AlexNet. Hinton and colleagues, 2006

5. Why GPUs mattered

A neural network performs large numbers of numerical operations. A dense layer, for example, can be represented as:

Y = XW + b

Training repeats forward calculations, gradient calculations and parameter updates across many examples. Convolutions likewise apply learned filters repeatedly across locations and channels.

Many of these operations can be organised in parallel. GPUs became valuable because they could perform large amounts of suitable numerical work concurrently. Their usefulness depends on the workload, memory transfers, implementation and hardware; a GPU is not automatically faster for every task.

The GPU does not “understand” an image. It executes the calculations specified by the software.

6. What CUDA contributed

NVIDIA introduced CUDA in 2006; CUDA 1.0 was publicly released in 2007. CUDA provides a programming platform and model through which developers can run general-purpose computations on NVIDIA GPUs. NVIDIA CUDA introduction; CUDA 1.0 release

Historically, this made GPU-based numerical computing more accessible than approaches that had to work through graphics-oriented interfaces.

CUDA did not invent neural networks, backpropagation or deep learning. It helped researchers implement and accelerate suitable calculations on NVIDIA hardware. Neural networks can also be trained on CPUs and other accelerators using other software stacks.

GPU deep learning before AlexNet

In 2009, Raina, Madhavan and Andrew Ng published research on large-scale deep learning using graphics processors. Cireșan and colleagues published GPU-based neural-network and CNN work before AlexNet, including visual classification research in 2011. Thus, AlexNet was not the first GPU-trained neural network or CNN. Raina and colleagues, 2009; Cireșan and colleagues, 2011

7. ImageNet: data and a benchmark

A powerful learning algorithm needs appropriate data and a way to evaluate progress. ImageNet, introduced in 2009, provided a large image database organised using a semantic hierarchy. The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) used defined tasks and subsets of this broader ecosystem. ImageNet original paper

The importance of ImageNet was not merely the number of images. It gave researchers a demanding shared benchmark on which different approaches could be compared.

However, strong performance on one benchmark is not proof of reliable general understanding in every real-world environment.

8. AlexNet in 2012: a watershed moment

AlexNet was developed by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton. Its paper described a CNN with approximately 60 million parameters and eight learned layers: five convolutional layers followed by three fully connected layers.

The system brought together a large labelled training set, GPU computation, ReLU activations, data augmentation, dropout and an efficient implementation. The authors report training across two NVIDIA GTX 580 GPUs with 3 GB of memory each for approximately five to six days. The network was divided between the GPUs partly because of memory limits. Original AlexNet paper

In the 2012 competition, the submitted model combination achieved 15.3% top-five error, compared with 26.2% for the next-best entry. Top-five error means the correct class was absent from the model’s five highest-ranked predictions. The competition figure should not be casually attributed to every single network.

What AlexNet did not invent

AlexNet did not invent CNNs, ReLU, dropout, backpropagation or GPU training. Its significance was in combining and implementing techniques effectively at a scale that produced a highly visible improvement. ReLU research, for example, predates AlexNet. Nair and Hinton, 2010

Alex Krizhevsky’s own research page describes CUDA-based convolutional-network software associated with this line of work. The original AlexNet predates cuDNN, so it should not be described as having been trained using that later library. Krizhevsky’s research and code; cuDNN paper, 2014

The historical lesson: AlexNet did not create deep learning. It made the power of an existing research tradition much harder for the computer-vision community to ignore.

9. Why AlexNet changed computer vision

Earlier vision systems often relied heavily on manually designed feature-extraction techniques. CNNs instead learn representations from data as part of training.

This should not be described as “machines defeating human-made algorithms”: neural-network architectures and learning procedures are also designed by people. The important shift was towards learning more of the feature representation from examples.

AlexNet’s impact came from the convergence of:

  • Earlier neural-network and CNN research.
  • Effective optimisation and training techniques.
  • Large labelled datasets and a shared benchmark.
  • Parallel GPU hardware and efficient software.
  • A result large enough to change research priorities.

10. From AlexNet to modern computer vision

The following periods are useful teaching categories, not rigid boundaries. CNNs, efficient models and Transformers continue to coexist.

2013–2014: Understanding and deepening CNNs

ZFNet: Researchers used visualisation and occlusion experiments to investigate network behaviour and modified early filters and strides. Its contribution was not merely smaller filters; it also helped researchers examine what learned networks responded to. ZFNet paper

VGG: VGG explored deep networks built largely from small 3×3 convolutions, including configurations with 16 or 19 weight layers. Its results demonstrated the value of depth under the tested conditions, not a universal rule that deeper always means better. VGG paper

GoogLeNet/Inception: Inception modules combined parallel processing paths at different scales, including dimension-reducing 1×1 convolutions. The design sought strong accuracy with manageable computation. GoogLeNet paper

2015–2017: Residual and dense connections

ResNet introduced residual mappings. In a simple shape-compatible block:

y = x + F(x)

The shortcut gives information and gradients a direct path around the learned transformation. This helps optimisation of very deep networks, but it does not guarantee that gradients never vanish or that more layers always improve results. The original paper distinguishes its degradation problem from vanishing gradients. ResNet paper

DenseNet connects each layer in a dense block to the feature maps of earlier layers through concatenation. This encourages feature reuse. Concatenation is different from the addition used in a basic residual connection, and fewer parameters do not automatically mean lower runtime or memory use. DenseNet paper

2017–2019: Efficiency for practical devices

MobileNet uses depthwise separable convolutions and configuration choices that trade accuracy against computation. Fewer arithmetic operations do not guarantee the same proportional reduction in latency on every device. MobileNet paper

EfficientNet introduced compound scaling, balancing network depth, width and input resolution. The best practical configuration still depends on the hardware and workload. EfficientNet paper

2020 onwards: Transformers, representation learning and multimodality

The Transformer architecture was introduced for sequence processing in 2017. Vision Transformer (ViT) subsequently represented an image as a sequence of learned patch embeddings and applied Transformer processing. Patch size is configurable; 16×16 is a common example, not a universal rule. Large-scale pretraining was important to its reported results. Transformer paper; ViT paper

Swin Transformer introduced a hierarchical architecture using shifted local windows, supporting tasks such as detection and segmentation. Swin paper

ConvNeXt showed that modernised CNN designs could remain competitive with contemporary Transformers. Transformers did not make convolution obsolete. ConvNeXt paper

CLIP learned aligned image and text representations using contrastive training, enabling forms of zero-shot image classification. It is not itself a general conversational multimodal assistant. CLIP paper

DINOv2 illustrates another branch: self-supervised learning of transferable visual representations. DINOv2 paper

Segment Anything (SAM) is a promptable segmentation model, while SAM 2 extends promptable segmentation to images and video. These should not be treated as interchangeable with general multimodal language models. SAM paper; SAM 2 paper

Models such as LLaVA connect visual representations with language models and instruction following, illustrating a different multimodal approach. LLaVA paper

11. CUDA after AlexNet, and the wider hardware landscape

After 2012, GPU software and hardware continued to evolve. Libraries such as cuDNN provided optimised neural-network primitives for frameworks. Later NVIDIA architectures introduced specialised matrix-computation facilities such as Tensor Cores. These were later developments and should not be projected backwards onto AlexNet’s GTX 580 hardware. cuDNN paper; NVIDIA on Tensor Cores

At the same time, AI is not restricted to CUDA. CPUs and other accelerators play important roles. For example, the original ViT research reports using TPU hardware. ViT original paper

12. Conclusion

The formation of deep learning is best understood as a long convergence, not a single invention:

Artificial neurons → learning algorithms → CNNs and sequence models → large datasets → parallel computing → AlexNet → residual networks, efficient architectures, Transformers and multimodal systems.

CUDA was a powerful enabler of research on NVIDIA GPUs. AlexNet was a landmark demonstration of what could happen when established ideas, substantial data and accessible computation came together. Neither tells the whole story alone.

Research note: This is a research-based historical review, not a report of new experiments. It uses original papers and official documentation. The Tamil translation follows separately.

More AI technology news