What is an LLM

A large language model is a neural network trained to predict the next token of text. That single, simple objective — applied to trillions of tokens with billions of parameters — produces systems that can write, summarise, translate, reason through problems, and write code. This lesson explains what an LLM is, how it is built and trained, and where its abilities and its failures come from.

Last updated: 01 Nov, 2026

Definitions: from AI to LLMs

These terms are often used interchangeably, but each is a subset of the one before it.

┌──────────────────────── Artificial Intelligence ────────────────────────┐
│  rules engines, search, planning, ...                                    │
│  ┌─────────────────────── Machine Learning ──────────────────────────┐   │
│  │  decision trees, linear models, clustering, ...                    │   │
│  │  ┌───────────────────── Deep Learning ─────────────────────────┐   │   │
│  │  │  CNNs for images, speech models, ...                         │   │   │
│  │  │  ┌──────────── Large Language Models ────────────┐           │   │   │
│  │  │  │  GPT, Claude, Gemini, Llama, Mistral, Qwen ... │           │   │   │
│  │  │  └────────────────────────────────────────────────┘           │   │   │
│  │  └──────────────────────────────────────────────────────────────┘   │   │
│  └────────────────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────────┘

What makes a language model “large”?

A key discovery of the last decade is that scaling these three together makes models predictably better — and unlocks abilities such as following instructions, translating between languages they were never explicitly taught to translate, and writing working code.

A language model in 20 lines of Python

To demystify the idea, here is the simplest possible language model: a bigram model. It counts which word follows which in some training text, then predicts the next word with probabilities. Real LLMs replace the counting with a giant neural network and words with tokens, but the job is identical: given what came before, how likely is each possible next piece?

from collections import Counter, defaultdict

corpus = """the model reads the prompt . the model predicts the next token .
the model repeats . the user reads the answer ."""

words = corpus.split()
follows = defaultdict(Counter)
for current, nxt in zip(words, words[1:]):
    follows[current][nxt] += 1          # "training": count what follows what


def next_word_probs(word: str) -> dict[str, float]:
    counts = follows[word]
    total = sum(counts.values())
    return {w: round(c / total, 2) for w, c in counts.most_common()}


print(next_word_probs("the"))
print(next_word_probs("model"))

This toy model already shows the essential behaviour: after the, it believes model is most likely. It has no understanding — only statistics of what tends to follow what. An LLM learns vastly richer statistics over long contexts, which is enough to capture grammar, facts, styles, and even step-by-step reasoning patterns.

The transformer, at a high level

You do not need to implement a transformer to build with LLMs, but knowing its four stages explains a lot of behaviour:

1. Tokenisation and embedding. Text is split into tokens, and each token id is mapped to a vector of numbers (an embedding) that represents it. Position information is added so the model knows word order.
2. Self-attention. In each layer, every token computes how much to “pay attention” to each previous token. In “The trophy did not fit in the suitcase because it was too big”, attention helps it connect to trophy.
3. Feed-forward layers. After attention, each token's vector passes through a small neural network. These layers are thought to store much of the model's factual and procedural knowledge. Attention + feed-forward together form one transformer block, and models stack dozens of them.
4. Output head. The final vector for the last position is converted into a score (logit) for every token in the vocabulary — the model's prediction of what comes next.

How an LLM is trained

Prompt:  "Write a haiku about vectors."

Base model continues the text:
  "Write a haiku about vectors. Write a haiku about matrices.
   Write a haiku about ..."             <- it saw lists of prompts in training

Chat model answers the request:
  "Arrows in the dark /
   pointing where the meanings live /
   close means similar"

What LLMs are good at

• Transforming text: summarising, rewriting, translating, changing tone, extracting fields into JSON.
• Generating drafts: emails, documentation, test cases, marketing copy, code.
• Reasoning over provided context: answering questions about a document you include in the prompt (the idea behind RAG).
• Classification and routing without training a separate model: sentiment, intent, topic, priority.
• Coding: writing, explaining, and fixing code in most languages.

What LLMs are bad at (and why)

Discriminative vs generative models

LLMs are generative, but they are frequently used for discriminative tasks too — asking “Is this review positive or negative? Answer with one word.” turns a generator into a classifier.

The model landscape

LLM vs the product built on it

ChatGPT, Claude.ai, and Gemini are products. The LLM inside them only predicts tokens. The product adds a system prompt, conversation memory, web search and other tools, file uploads, safety filters, and a user interface. When you build with the API, you are responsible for those layers — which is exactly what the rest of this course and the RAG and Agents courses teach.