A History of Artificial Intelligence and Deep Learning

Mark Andrews

Five Traditions

Five intellectual traditions

The history of AI is not one story but five intertwined traditions.
Symbolic AI: search, planning, logic, formal knowledge representation, expert systems.
Statistical machine learning: probabilistic models, kernel methods, decision trees, statistical NLP.
Neural and connectionist learning: perceptrons, multilayer networks, and modern deep learning.
Adaptive agents: reinforcement learning, control theory, robotics, game playing.
Enabling infrastructure: compute, data, benchmarks, software, and industrial investment.

One winner

All five traditions contributed, but one came to dominate the others.
Deep learning is now so thoroughly dominant that “artificial intelligence” is used to mean it.
The other traditions either operate inside deep learning systems or have been marginalised.
Understanding the present requires following the neural tradition through decades of struggle.

Before Dartmouth

McCulloch and Pitts (1943)

Warren McCulloch and Walter Pitts showed in 1943 that networks of idealised neurons could compute any logical function.
This was a mathematical result, not an engineering proposal.
It established that biological-style neural processing was computationally tractable in principle.
This paper is the true starting point of the neural network tradition, thirteen years before Dartmouth.

Hebb’s learning rule (1949)

Donald Hebb proposed in 1949 that learning results from changes in connection strength between neurons.
When two neurons fire together repeatedly, their connection is strengthened.
This provided a biological basis for the idea that networks could learn from experience.
Hebb’s rule influenced the design of learning algorithms for artificial networks for decades.

Turing (1950)

Alan Turing’s 1950 paper “Computing Machinery and Intelligence” reframed the question of machine thought.
Rather than asking whether machines can think, he proposed a behavioural criterion: can a machine converse indistinguishably from a human?
The paper anticipated and disposed of nearly every objection later raised against machine intelligence.
It set the terms of debate that the field would return to for the next seventy years.

Wiener and cybernetics (1948)

Norbert Wiener’s cybernetics examined feedback, control, and communication in both animals and machines.
Cybernetics was particularly concerned with systems that adapt continuously to their environment.
Its influence on early AI researchers was substantial.
McCarthy later coined “artificial intelligence” partly to distinguish his programme from Wiener’s.

Three views of intelligence

These pre-Dartmouth developments established three enduring views.
Intelligence as symbolic computation: manipulating representations according to formal rules.
Intelligence as adaptive control: regulating behaviour through feedback and interaction with an environment.
Intelligence as an emergent property of networks of simple processing units.
The tension between the first and third views would drive the field for fifty years.

The Dartmouth Conference

Dartmouth 1956

The term “artificial intelligence” was introduced in a 1955 proposal by McCarthy, Minsky, Rochester, and Shannon.
The Dartmouth Summer Research Project took place in 1956, running for several weeks with participants coming and going.
It did not produce a unified research programme or agreed methods.
Its importance was institutional: it gave the field a name and a shared identity.

Symbolic AI from the start

Allen Newell and Herbert Simon arrived at Dartmouth with a working programme, the Logic Theorist.
The Logic Theorist proved theorems from Whitehead and Russell’s Principia Mathematica.
They continued with the General Problem Solver, which used means-ends analysis for general reasoning.
Their physical symbol system hypothesis became the foundation of symbolic AI: intelligence is symbol manipulation.

Two traditions from the beginning

The neural network tradition was also present at Dartmouth but was not the dominant programme.
Symbolic AI was more immediately productive and better suited to the hardware available.
Two traditions were therefore established from the very beginning of the field.
This division would define the field’s internal politics for decades.

The Perceptron

Rosenblatt’s perceptron (1957)

Frank Rosenblatt introduced the perceptron in 1957 at the Cornell Aeronautical Laboratory.
The laboratory conducted military and aviation research, funded substantially by the US Office of Naval Research.
The perceptron was a simple computational neuron: inputs weighted, summed, and passed through a threshold.
Crucially, it also had a learning rule: adjust the weights when the perceptron makes an error.

The perceptron learning theorem

Rosenblatt proved that if a correct set of weights existed, the learning rule would find it in finite time.
This was a rigorous result, not just a demonstration.
For the first time, a machine could learn its own internal parameters from examples rather than following hand-coded rules.
This idea of learning from data would prove to be the defining characteristic of the winning tradition.

Public excitement and its consequences

The New York Times reported in 1958 on expectations of a machine that could walk, talk, see, and be conscious.
Rosenblatt was more measured in his technical writing, but the coverage created extravagant expectations.
Rosenblatt moved to Cornell’s Ithaca campus, where he worked until his death in a sailing accident in 1971, aged 43.
The gap between public expectation and technical reality would later make the backlash sharper.

The 1960s

Symbolic AI advances

The early 1960s were a period of genuine excitement and considerable overconfidence.
McCarthy created Lisp, which became the characteristic programming language of AI research.
Newell and Simon continued developing heuristic problem solving.
Programs could solve algebra problems, prove geometric theorems, and play checkers at a reasonable level.

Microworlds

Much research took place in restricted artificial environments called microworlds.
Weizenbaum’s ELIZA simulated a psychotherapist by pattern-matching user inputs; it had no understanding whatsoever.
Users readily attributed genuine understanding to ELIZA — Weizenbaum was disturbed, not pleased.
Winograd’s SHRDLU could converse about a simulated blocks world but could not generalise beyond it.

The optimism problem

Researchers made predictions about machine translation and general AI being solved within a decade.
The 1966 Summer Vision Project assumed that building a significant visual recognition system was a summer’s work.
These predictions proved badly wrong.
The gap between what programmes could do in constrained microworlds and what they could do in the real world was not taken seriously enough.

Minsky, Papert, and the First AI Winter

Perceptrons (1969)

In 1969 Minsky and Papert published Perceptrons, a rigorous mathematical analysis of single-layer perceptrons.
The central result: perceptrons can only solve linearly separable problems.
The exclusive-or function cannot be computed by a single-layer perceptron.
The proof was clean and the result was correct.

What the book actually proved

Minsky and Papert proved that single-layer perceptrons had fundamental limitations.
They did not prove that multilayer neural networks were incapable of solving such problems.
Multilayer networks can represent exclusive-or and many other nonlinear functions.
They noted, in passing, that multilayer networks might overcome these limitations — and then did not investigate.

The disproportionate effect

The book’s influence on funding and research was disproportionate to the actual argument.
Neural network research declined sharply through the 1970s.
The ALPAC report (1966) had already cut funding for machine translation.
The Lighthill report (1973) produced a similarly damaging reassessment in the United Kingdom.

The first AI winter

The combination of failed promises and pointed technical criticism produced the first AI winter.
Neural network research migrated to the margins, surviving mainly in psychology, cognitive science, and neuroscience.
Symbolic AI was not similarly affected and continued to be well funded.
A small persistent community continued to work on neural networks, mostly outside computer science departments.

Expert Systems

The knowledge bottleneck insight

The principal lesson drawn from 1970s AI was that general reasoning procedures are ineffective without domain knowledge.
This produced the knowledge-based or expert-system approach.
Expert systems encoded knowledge as production rules of the form “if X and Y then Z.”
An inference engine chained these rules together to answer queries.

Early expert systems

DENDRAL, from Stanford in the late 1960s, inferred molecular structures from mass spectrometry data.
MYCIN, from Stanford in the early 1970s, diagnosed bacterial infections and recommended antibiotics.
MYCIN’s performance compared favourably with human specialists in controlled studies.
These systems demonstrated that computers could perform at expert level within narrow domains.

Commercial expert systems

The 1980s saw commercial investment in expert systems at unprecedented scale.
DEC’s XCON system configured VAX computer orders and reportedly saved the company tens of millions of dollars annually.
Japan launched its Fifth Generation Computer Systems project with heavy government funding.
Specialist companies sold expert-system software, knowledge-engineering services, and Lisp machines.

The limits of knowledge engineering

Expert systems were brittle: they worked well inside their domain and failed completely outside it.
Extracting knowledge from human experts and encoding it by hand was laborious and expensive.
Large rule bases were difficult to construct, verify, and maintain.
Cheaper general-purpose computers displaced specialised Lisp machines.

The second AI winter

The expert systems bubble burst in the late 1980s.
Maintenance costs were high and systems did not deliver on broader promises.
The AI industry contracted substantially and the term “artificial intelligence” became professionally embarrassing.
A methodological shift was underway: statistical methods were beginning to replace hand-coded rules.

Connectionism Survives

Neural network research persists

After the first AI winter, neural network research did not disappear — it migrated.
It was pursued more actively in psychology, neuroscience, and cognitive science than in computer science.
The question of whether network models could explain human cognition was of direct relevance to these disciplines.
This gave the tradition an intellectual home and a different kind of legitimacy.

The connectionist programme

The San Diego group centred on Rumelhart, McClelland, and Hinton developed what they called connectionism.
Connectionism proposed that knowledge could be distributed across connection weights rather than encoded as explicit rules.
Cognitive behaviour could emerge from the collective activity of many simple processing units.
The psychological and philosophical implications were as important as the technical ones.

Hinton’s trajectory

Hinton studied experimental psychology at Cambridge and completed a PhD in AI at Edinburgh in 1978.
Edinburgh was an environment where connectionist ideas had survived the post-Minsky-Papert cold better than elsewhere.
He subsequently worked at Sussex, San Diego, Carnegie Mellon, and the University of Toronto.
Toronto became the base for his most important later work and the centre of the deep learning revival.

Key technical developments

Hopfield networks showed that networks of simple units could function as associative memories.
Boltzmann machines provided a probabilistic framework for learning in networks with hidden units.
Self-organising maps learned to represent the structure of input data without supervision.
Distributed representations offered an alternative to the local, symbolic representations of mainstream AI.

Backpropagation

The algorithm

Backpropagation computes how much each weight in a multilayer network contributed to the output error.
It propagates the error gradient backwards through the network layer by layer using the chain rule.
Paul Werbos derived the algorithm in his 1974 PhD thesis.
Rumelhart, Hinton, and Ronald Williams published the paper that reached and persuaded the field in Nature in 1986.

Why it mattered

Backpropagation allowed hidden units in a multilayer network to learn their own internal representations.
Features no longer had to be hand-engineered; the network learned them from data.
A network trained on images learned feature detectors — edges, textures, shapes — without anyone specifying them.
This was the technical answer to the limitation Minsky and Papert had identified.

Parallel Distributed Processing (1986)

The same year, Rumelhart and McClelland edited the two-volume Parallel Distributed Processing.
This became the founding document of the connectionist programme.
It presented connectionism as both an engineering approach and a theory of human cognition.
It reached researchers in psychology, linguistics, philosophy, and neuroscience as well as computer science.

NeurIPS begins (1987)

The Neural Information Processing Systems conference began in 1987.
It was founded at the intersection of neural computation, neuroscience, psychology, statistics, and computer science.
Its founding reflected the interdisciplinary character of the connectionist revival.
It would eventually become one of the most important venues in all of machine learning.

The second decline

The connectionist revival of the late 1980s did not last.
Backpropagation networks were difficult to train reliably and prone to getting stuck in local minima.
There was no good theoretical account of why they should generalise well.
By the mid-1990s, other approaches were delivering better results with cleaner theoretical foundations.

The Statistical Turn

Machine learning moves to the centre

As the expert systems bubble burst, machine learning moved from the margins to the centre of AI.
Machine learning had existed as a subfield from the beginning but had been marginal relative to the symbolic mainstream.
The internet was generating data at a scale that made statistical learning feasible in new ways.
Statistical learning theory was being strengthened: Vapnik’s work provided rigorous accounts of generalisation.

Support vector machines

Vladimir Vapnik’s support vector machine, published in its modern form in 1995, became dominant.
SVMs were theoretically principled, practically effective, and produced results competitive with neural networks.
They were more reliable to train than backpropagation networks and had better theoretical justification.
SVMs and kernel methods became the standard solution to a wide range of classification problems.

Statistical NLP

The statistical turn in natural language processing was particularly pronounced.
Hidden Markov models for speech recognition and statistical methods for parsing replaced rule-based systems.
Statistical machine translation treated translation as inference from parallel text corpora.
These methods were less linguistically interpretable but more accurate and far easier to build.

Deep Blue (1997)

IBM’s Deep Blue defeated world chess champion Garry Kasparov in 1997.
Deep Blue used brute-force tree search, examining hundreds of millions of positions per second.
It involved no learning and relied on evaluation functions partly hand-tuned by grandmasters.
It demonstrated what could be achieved with computation and domain knowledge — but not what could be learned.

The disappearance of “AI”

By the late 1990s, the term “artificial intelligence” had largely vanished from research culture.
Researchers worked on machine learning, computational linguistics, computer vision, and speech recognition as separate disciplines.
NeurIPS, founded on neural network research, had become a machine learning conference where neural networks were regarded with scepticism.
The field that had produced the first AI winter had itself been marginalised.

Neural Networks in the Wilderness

LeCun’s convolutional networks

Yann LeCun combined Fukushima’s convolutional architecture with backpropagation in the late 1980s and 1990s.
Fukushima’s Neocognitron (1980) was a hierarchical architecture inspired by the mammalian visual cortex.
LeCun’s LeNet-5 (1998) achieved excellent performance on handwritten character recognition.
It was deployed commercially for reading digits on bank cheques.

Long short-term memory (1997)

Sepp Hochreiter and Jürgen Schmidhuber published the LSTM in 1997.
Standard recurrent networks suffered from vanishing and exploding gradients over long sequences.
LSTM introduced gating mechanisms that allowed the network to maintain information across long spans of input.
It would become the standard approach to sequential data in speech and language throughout the 2000s.

The SVM era

Despite these advances, by the early 2000s, SVMs were generally regarded as more reliable than neural networks.
For most classification benchmarks, a well-tuned SVM outperformed or matched the best neural networks.
The theoretical justification for neural networks remained weak compared to statistical learning theory.
A small community continued to believe that deeper networks would eventually prove their value.

Deep Learning Re-emerges

Hinton and Salakhutdinov (2006)

In 2006, Hinton and Ruslan Salakhutdinov published a paper in Science showing deep networks could be trained effectively.
The technique was greedy layer-wise pretraining using restricted Boltzmann machines.
Each layer was trained independently before the whole network was fine-tuned with backpropagation.
This circumvented the vanishing gradient problem that had made deep networks impractical.

Bengio and the Montreal group

Yoshua Bengio at the University of Montreal developed related layer-wise methods independently.
Bengio’s group had also produced a 2003 neural language model, a genuine precursor of later language models.
Both groups demonstrated that networks with many layers could learn useful representations.
The specific pretraining methods were later found unnecessary — better initialisation and activation functions sufficed.

Why 2006 mattered

The 2006 work was historically important because it restored confidence in deep architectures.
It demonstrated that the limitations of earlier deep networks were engineering problems, not fundamental barriers.
It sparked a new wave of research into training methods, activation functions, and architectures.
The neural network tradition had been right about the broad approach; it had been waiting for the right tools.

The term “deep learning”

The term “deep learning” was coined deliberately around this time.
It distinguished the revived approach from the discredited “neural networks” label.
It emphasised the key feature: many layers of representation, each learning increasingly abstract features.
By the late 2000s, deep learning was no longer marginal, though it was not yet dominant.

AlexNet and the Turning Point

ImageNet

The ImageNet Large Scale Visual Recognition Challenge was an annual competition beginning in 2010.
Systems competed to classify over one million labelled photographs across a thousand categories.
It concentrated effort on a single measurable benchmark and made progress objectively comparable.
This kind of benchmark competition was part of the enabling infrastructure that made deep learning’s victory legible.

AlexNet (2012)

In 2012, Krizhevsky, Sutskever, and Hinton entered AlexNet in the ImageNet challenge.
It achieved an error rate roughly ten percentage points lower than the previous year’s winner.
The gap was large enough that its significance was immediately apparent to everyone in the field.
This is the moment usually identified as the beginning of the deep learning era.

What AlexNet used

GPU-based parallel training made it feasible to train a large network on a large dataset.
The ReLU activation function trained faster and more reliably than sigmoid functions.
Dropout regularisation reduced overfitting by randomly disabling units during training.
A deep convolutional architecture learned hierarchical features automatically from data.

Learned features

AlexNet did not use hand-engineered image features; it learned them from data.
The first convolutional layers learned edge detectors and colour gradients.
Later layers learned textures, object parts, and whole objects.
This automatic feature learning had been the promise of neural networks since the 1980s — now delivered at scale.

Industry response

Google acquired the AlexNet team for a reported forty-four million dollars.
DeepMind, founded in 2010, was acquired by Google in 2014 for a reported five hundred million dollars.
Investment in deep learning research scaled up rapidly in academia and industry.
The gap between deep learning and competing methods was large enough that the field rapidly converged on it.

Deep Learning Spreads

Speech recognition

Deep networks began producing improvements in speech recognition before AlexNet.
Hinton’s Toronto group collaborated with Microsoft, IBM, and Google from 2009 onwards.
By 2014, deep learning had become the standard approach to speech recognition.
The hidden Markov models that had dominated for decades were displaced within a few years.

Computer vision transformed

After AlexNet, deep learning rapidly replaced most existing methods in computer vision.
Image classification, object detection, image segmentation: deep networks outperformed everything else.
The ImageNet competition itself was discontinued in 2017 because the problem was effectively solved.
Deep learning did not just improve on previous methods; it made them obsolete.

Reinforcement learning

DeepMind’s Deep Q-Network (2015) learned to play Atari games from raw pixel input.
The same architecture, without game-specific programming, learned to play dozens of different games.
This combined deep representation learning with reinforcement learning to powerful effect.
It demonstrated that the deep learning approach generalised well beyond supervised learning.

AlphaGo (2016)

AlphaGo defeated world Go champion Lee Sedol four games to one in March 2016.
Go had been considered intractable for AI because the game tree is too large for brute-force search.
AlphaGo combined policy networks, value networks, Monte Carlo tree search, and reinforcement learning through self-play.
The contrast with Deep Blue was stark: AlphaGo learned its knowledge, Deep Blue was programmed with it.

AlphaZero

AlphaZero, trained entirely through self-play from scratch, achieved superhuman performance in Go, chess, and shogi.
It mastered all three games simultaneously using the same algorithm, different only in the rules provided.
This generalisation was the key point: a single learning system, not three specialised programmes.
Deep learning had not simply replaced search; learned networks could guide and evaluate search far more effectively.

The Transformer

Attention mechanisms

Attention mechanisms had existed before the transformer.
Bahdanau and colleagues introduced attention for recurrent networks in 2014.
Attention allowed a model to focus on different parts of the input while generating each output element.
This reduced the bottleneck of compressing an entire input sequence into a single fixed representation.

Attention is All You Need (2017)

Vaswani and colleagues at Google published the transformer architecture in 2017.
The transformer replaced recurrence entirely with self-attention.
Self-attention allows each element in a sequence to attend directly to every other element in parallel.
This made training far more parallelisable than recurrent networks, which processed sequences step by step.

Why scale became possible

Transformers could exploit modern GPU and TPU hardware far more efficiently than LSTMs.
Longer training runs on larger datasets became feasible.
The transformer was introduced for machine translation but rapidly became the standard for language modelling.
It eventually spread into computer vision, audio, scientific data, and multimodal applications.

Beyond language

Vision transformers applied self-attention to image patches, competing with and often surpassing convolutional networks.
AlphaFold, which achieved a decisive breakthrough in protein structure prediction, was built on a transformer.
Audio models, video models, and scientific AI all adopted transformer-based architectures.
The transformer proved to be a general architecture, not a specialised language tool.

Foundation Models

BERT (2018)

BERT, from Google in 2018, used bidirectional masking: predicting masked-out words seeing context from both directions.
This produced representations suited to understanding tasks requiring full-context reading.
BERT and its successors became dominant for natural-language understanding through fine-tuning.
Fine-tuning a pretrained model on a specific task required far less data and compute than training from scratch.

GPT (2018–2020)

OpenAI’s GPT (2018) showed that autoregressive pretraining produced transferable representations.
GPT-2 (2019) showed that larger models trained on broader data could perform tasks without task-specific training.
GPT-3 (2020), with 175 billion parameters, demonstrated few-shot learning: performing tasks from instructions and examples in the context alone.
Few-shot behaviour was unexpected and sparked substantial debate about what, if anything, these models understood.

Scaling laws

An important empirical finding was that performance improved predictably as model size, data, and compute increased.
Scaling laws made this relationship smooth and predictable enough to plan around.
If you trained a bigger model on more data with more compute, you would get a better model.
Scaling therefore became a systematic research programme, not just an engineering convenience.

Foundation models

Large pretrained models became known as foundation models.
A single model trained on a broad prediction task acquired capabilities that transferred across many domains.
One model could serve as the basis for many applications, adapted through fine-tuning or instruction following.
This represented a fundamental shift in how AI systems were built and deployed.

Generative AI

Generative adversarial networks (2014)

Ian Goodfellow and colleagues introduced GANs in 2014.
A generator network tries to produce examples indistinguishable from real data.
A discriminator network tries to tell real examples from generated ones.
The adversarial dynamic drives both networks toward increasingly realistic outputs.

From GANs to diffusion

GANs produced remarkable results in image synthesis through the late 2010s.
They were difficult to train and sometimes covered only part of the data distribution.
Diffusion models, which learn to reverse a process of gradually adding noise, became practically important around 2020.
By 2022, diffusion models generally surpassed GANs in image quality at the research frontier.

Image generation systems

DALL-E, Stable Diffusion, and Midjourney brought high-quality image generation to general users.
These systems combined large text encoders with diffusion models conditioned on text descriptions.
A written description could produce photorealistic images, artwork, or illustrations.
This capability would have seemed implausible to most researchers ten years earlier.

Multimodal systems

Modern generative systems do not separate language and visual capabilities.
Text encoders, transformers, vision encoders, and diffusion decoders are combined in single architectures.
Audio generation, video generation, and music generation followed similar trajectories.
The relevant object is increasingly a multimodal foundation model rather than a single-modality system.

ChatGPT and the Present

From pretrained models to assistants

Pretraining produced capable models, but they were not yet useful assistants.
Instruction tuning trained models to respond to requests in ordinary language rather than complete prompts.
Reinforcement learning from human feedback used human preference judgements to shape outputs.
These techniques transformed pretrained language models into systems people could use directly.

ChatGPT (November 2022)

OpenAI released ChatGPT on 30 November 2022.
Its underlying technology was evolutionary rather than wholly new.
The conversational interface and instruction-following behaviour made capabilities accessible to a mass audience.
ChatGPT reached one hundred million users in two months — the fastest adoption of a consumer technology in recorded history.

The current landscape

GPT-4, Claude, Gemini, and competing models established large language models as the dominant form of AI.
Earlier machine-learning systems were usually invisible components of larger products.
A conversational language model can be used directly for writing, explanation, programming, analysis, and planning.
The generality of these systems made the concept of artificial intelligence seem plausible to a mass audience again.

Current directions

Multimodal models process combinations of text, images, audio, and video.
Reasoning models use additional computation before answering, developing solution strategies rather than producing immediate outputs.
Agentic systems repeatedly plan, call tools, inspect results, and revise their actions.
Deep learning remains the common substrate, but modern AI is increasingly a larger computational system organised around a learned foundation model.

The Historical Pattern

Machine learning absorbed AI; deep learning absorbed machine learning

The broad historical movement can be compressed into a single observation.
Machine learning absorbed much of artificial intelligence.
Deep learning absorbed much of machine learning.
This does not mean earlier approaches disappeared — they operate inside deep learning systems.

What deep learning’s success required

Deep learning’s success was not purely an intellectual victory.
It required better training algorithms, large quantities of digital data, and GPU-based computing.
It also required benchmark competitions, open-source software frameworks, and unprecedented industrial investment.
The neural network tradition had been broadly right for decades before it had the infrastructure to prove it.

What was overcome

Modern language models have overcome problems many believed required explicit grammars and symbolic world models.
Tasks that seemed to demand linguistic expertise have been solved by statistical models trained on text.
Deep learning did not succeed because it proved the symbolic approach wrong in theory.
It succeeded because it worked better in practice, at scale, across a wide range of tasks.

What remains open

Language and intelligence cannot be regarded as completely solved.
Reliable factuality, causal understanding, and robust mathematical reasoning remain open problems.
Long-horizon autonomous action and stable grounding in the physical world are active research challenges.
The systems that exist are extraordinarily capable and also limited in ways that are not yet fully understood.

The central question

What has been achieved is the construction of remarkably general statistical models.
These models mediate between human language, knowledge, perception, and action.
Whether they constitute general intelligence, approximate it, or represent something fundamentally different is unresolved.
That question is the central one in the history now unfolding.