Skip to content
All apps

LLM training stages

About this app

The same prompt through pretraining, SFT, RLHF/DPO and LoRA: the knowledge comes from pretraining, the later stages mainly shift which answer is likely.

How a language model is trained

A large language model passes through several training stages with the same architecture; only the weights change. In pretraining it learns to predict the next token on trillions of tokens of web text, books and code. Afterwards it knows language and facts but only continues a text. Supervised fine-tuning (SFT) on instruction–answer pairs teaches it the answer format. Preference optimisation with RLHF or DPO shifts probability towards helpful and harmless answers. Optionally, LoRA adapts the style to a domain with small low-rank adapters ΔW = B·A while the base weights stay frozen.

What the app shows

A timeline runs from random initialisation through pretraining, SFT and RLHF/DPO to LoRA. For four prompts, such as “What is the capital of Australia?”, the app shows the probabilities of the first answer tokens at every stage and a sampled answer. After pretraining the question is often followed only by another quiz question; an answer comes after SFT. A data panel gives orders of magnitude for an open model with around 70 billion parameters, from roughly 15 trillion tokens in pretraining to about 10 million for LoRA.

Things to try

Drag the timeline through all stages and watch the bar for “Canberra”: it is already high after pretraining, and the later stages mainly change which continuation is likely. Set the temperature and resample answers, or click a bar to see exactly that continuation. Switch the prompt, for instance to “How do I pick a lock?”, and compare how the preference stage shifts the behaviour. The weight-change display shows how many parameters each stage trains.

Frequently asked questions

How is a language model like ChatGPT trained?
First the model is pretrained on very large amounts of text to predict the next token. It then learns to respond to instructions through supervised fine-tuning on examples written by people. Finally it is aligned with preference methods such as RLHF or DPO towards answers people prefer.
What is the difference between pretraining and fine-tuning?
In pretraining the model learns language and world knowledge from raw text, which takes most of the data and compute. Fine-tuning uses far less, carefully selected data and changes the weights only slightly. It mainly teaches the model a behaviour, such as an answer format or a style.
Does RLHF teach a language model new knowledge?
Essentially no. The knowledge comes from pretraining. RLHF and DPO shift which of the possible answers is likely, changing the model’s style, helpfulness and safety behaviour.
What can I try in this app?
You send one of four prompts through all training stages and see the probabilities and a sampled answer at each stage. You set the temperature, resample answers and compare data volumes, compute and trained parameters across the stages.

Subject: Neural networks

More from Neural networks

A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps