This evergreen history article uses authoritative archives and official records. Exact dates are used when documented; gradual inventions and rollouts are described as periods rather than being assigned a misleading single birthday.
Quick facts
- A language model estimates the probability of words or tokens in context.
- Statistical n-gram models dominated many speech and language systems before deep learning.
- Word embeddings represented words as vectors with learned relationships.
- The transformer architecture, published in 2017, enabled efficient large-scale sequence training.
- Modern LLMs combine pretraining, instruction tuning, human feedback, retrieval and tool use.
Statistical language modelling
Early language models counted how often words or short sequences appeared. N-gram models predicted a word from a limited history and became important in speech recognition, translation and typing assistance. They were interpretable and practical, but they struggled with long-distance context and rare phrases.
Smoothing techniques handled unseen combinations, while larger text collections improved coverage. These systems established the central idea that language could be modelled probabilistically rather than only through hand-written grammar rules.
Neural language models and embeddings
Neural networks replaced sparse count tables with learned continuous representations. Early neural language models predicted words using vector inputs, allowing similar words to share statistical strength. Word2Vec and related embedding methods made semantic relationships visible as directions in vector space.
Recurrent neural networks and long short-term memory processed sequences while maintaining a hidden state. They improved translation and speech, but training long sequences was slow and information could still fade across many steps.
Attention and the transformer
Attention mechanisms let a model compare each part of an input with other relevant parts. In 2017 the transformer architecture removed recurrence from the core sequence model and relied on self-attention. Because many tokens could be processed in parallel, researchers could train much larger models on modern hardware.
Transformers rapidly became dominant in language tasks. BERT, introduced in 2018, learned bidirectional contextual representations and improved search and classification. Generative transformer models scaled autoregressive prediction, producing coherent passages and adapting to new tasks from prompts.
Scale, instruction tuning and public assistants
GPT-3 in 2020 demonstrated strong few-shot behaviour at unprecedented scale. Later systems were tuned to follow instructions and refined with human feedback. Conversational interfaces made the models accessible to people without machine-learning expertise.
Scale produced broad capability but not guaranteed truth. LLMs predict plausible sequences and can invent citations, misread instructions or reproduce bias. Retrieval systems connect them to external documents, while tools allow calculation, browsing, coding and database actions.
From language model to agent platform
Modern systems combine models with memory, planning loops, permissions and software tools. This creates the possibility of agents that complete multi-step tasks, but it also introduces prompt injection, data leakage and unintended-action risks.
The future of LLMs may include smaller specialised models, multimodal inputs, efficient sparse architectures and stronger verification. The most useful system may not be the largest model alone, but a carefully designed combination of model, trusted data, tools, human review and security boundaries.
Common misconceptions
- An LLM does not store every training document as a searchable database.
- More parameters do not automatically produce factual accuracy or safe behaviour.
- The transformer and GPT are related but not identical concepts.
- A model’s fluent language should not be confused with verified understanding or consciousness.
Timeline: key years and locations
| Year | Location | Event | Why it mattered |
|---|---|---|---|
| 1948–1950s | United States | Information theory and probabilistic language research develop | Provided mathematical foundations for sequence prediction. |
| 1980s–1990s | Global speech laboratories | N-gram models dominate practical systems | Made statistical language modelling central to speech and translation. |
| 2003 | Montreal, Canada | A neural probabilistic language model gains influence | Showed how learned distributed representations improve prediction. |
| 2013 | Google research, United States | Word2Vec popularises efficient embeddings | Made semantic vector relationships widely usable. |
| 2017 | Google research, United States | Transformer architecture is published | Enabled parallel training of large sequence models. |
| 2018 | Google research, United States | BERT is introduced | Advanced contextual understanding in search and classification. |
| 2020 | San Francisco, United States | GPT-3 demonstrates large-scale few-shot learning | Accelerated the foundation-model era. |
| 2022 onward | Global | Instruction-tuned chat systems reach mass audiences | Turned LLMs into general conversational and tool-using interfaces. |
Frequently asked questions
What makes a language model “large”?
There is no universal threshold. The term usually refers to models with many parameters trained on broad data and significant computing resources.
Why do LLMs hallucinate?
Their objective is to generate likely sequences, not automatically verify every statement against a trusted source.
Are all LLMs transformers?
Most modern large language models use transformer-based architectures, but language modelling as a field predates transformers and can use other designs.