History of deep learning computer vision: from AlexNet to multimodal models

Illustrative image: A convolutional neural network: an input image passes through convolution and pooling layers to a classification output; illustrative of the deep learning approach AlexNet popularised. Image: Irisbox, CC BY 4.0, via Wikimedia Commons.
The history of deep learning computer vision began with AlexNet in 2012 and has developed through four main periods up to the multimodal models we use today. AlexNet proved that deep neural networks trained on GPUs can outperform older, human-designed algorithms.
1. The classic CNN period (2012–2014)
Early research focused on making networks deeper and on tuning the large filter sizes of AlexNet.
- AlexNet (2012). Introduced ReLU, Dropout and dual-GPU training with 8 layers. It proved that deep learning works well on large image collections.
- ZFNet (2013). Improved on AlexNet by reducing the 11×11 filter of the first layer to 7×7 and the stride from 4 to 2. This helped retain fine detail in images.
- VGGNet (2014). Instead of large filters, stacked small 3×3 convolution layers one after another, up to 19 layers. It showed that network depth is very important for accuracy.
- GoogLeNet / Inception (2014). Introduced Inception blocks that run 1×1, 3×3 and 5×5 filters in parallel inside a single layer. It delivered higher accuracy than VGG with far fewer parameters.
2. The very deep and residual period (2015–2017)
When networks went beyond 20 layers, gradients vanished during training (the Vanishing Gradients problem) and accuracy began to fall. Researchers solved this by changing the way information flows between layers.
- ResNet (2015). Introduced Residual Skip Connections, which let information pass directly over some layers. This made it easy to train models with 50, 101 and more than 1,000 layers.
- DenseNet (2017). Connected every layer in a block directly to all the layers that follow it. This increased the reuse of features and reduced the number of parameters.
3. The mobile and efficiency period (2017–2019)
After reaching human-level accuracy on computers, attention turned to shrinking models so that they run on smartphones and small devices.
- MobileNet (2017). Used Depthwise Separable Convolutions instead of ordinary convolutions, greatly reducing the computational cost for mobile devices.
- EfficientNet (2019). Introduced Compound Scaling, which increases the depth, width and image resolution of the network together in balance.
4. The Vision Transformer and modern period (2020–present)
Inspired by the revolution in natural language processing (NLP), researchers began to move from convolution methods to attention-based models.
- Vision Transformer / ViT (2020). Split images into 16×16 pixel patches, removed convolutions completely and processed the patches with Transformer Self-Attention.
- Swin Transformer (2021). Brought a hierarchical structure and Shifted Windows into ViT. This proved very useful for tasks such as Object Detection.
- ConvNeXt (2022). Modernised ResNet using the design details of transformers. It proved that pure CNN models can also reach accuracy equal to ViT.
- Multimodal Models (2021–present). CLIP, SAM (Segment Anything) and modern Multimodal LLMs combine images and text in a single model. This makes it possible to understand images, text and video at the same time.