Modern AI chatbots are trained in three broad stages. Pretraining teaches a large language model to predict the next token across trillions of tokens of text, which gives it grammar, facts and patterns but not the habit of answering questions helpfully. Supervised fine-tuning then trains it on thousands of example conversations written by people, so it learns the shape of a good reply. Finally, preference training, most famously reinforcement learning from human feedback (RLHF), has people rank several candidate answers; a reward model learns those preferences and the chatbot is tuned toward answers people rate higher. OpenAI's 2022 InstructGPT paper found labelers preferred answers from a 1.3-billion-parameter model trained this way over the raw 175-billion-parameter GPT-3. Variants such as Anthropic's Constitutional AI replace some human rankings with AI feedback guided by written principles. Training shapes behavior but doesn't guarantee accuracy, which is why chatbots can still state wrong things confidently.
Reading time
— 4 min
Updated
— Sep 30, 2026
Fact-reviewed
— Sep 30, 2026
Key Takeaways
Key Takeaways
1A chatbot is trained in stages: pretraining on huge amounts of text, then fine-tuning on example conversations, then tuning on human preferences.
2Pretraining gives the model its knowledge and fluency. The later stages give it the habit of being a helpful assistant.
3Preference training makes answers sound better to people, which is not the same as making them always true.
The concept
Think of training a new customer-service hire. First they spend years just reading: books, websites, manuals, forums. They pick up how language works and a huge number of facts, but nobody has told them what the job is. Next, a manager sits with them and shows worked examples: "When a customer asks this, a good answer looks like that." Finally, the manager watches them handle real questions, compares two or three of their replies, and says which one was better, again and again, until good habits stick. AI chatbots go through the same three stages. The reading is called pretraining, the worked examples are fine-tuning, and the "this one was better" feedback is preference training. The chatbot you talk to is the result of all three.
You type 'What is the capital of Japan?' into a base model that has only been pretrained. What might it plausibly produce?
Worked examples
Example 1: One ranking round (baseline case)
A prompt comes in: "Explain compound interest to a 12-year-old." The fine-tuned model writes four candidate answers. Answer A is accurate but full of jargon. Answer B uses a piggy-bank analogy and a clear number example. Answer C is friendly but gets the math wrong. Answer D is one sentence long. A labeler ranks them B, A, D, C. Each ranking turns into several pairwise comparisons (B beats A, B beats D, and so on). Across tens of thousands of prompts, the reward model learns that clear, correct, age-appropriate answers win, and the chatbot is tuned toward writing more like B.
Example 2: Why smaller can beat bigger (edge case)
In the InstructGPT study, the 175-billion-parameter GPT-3 knew more raw information than the 1.3-billion-parameter model, which is more than 100 times smaller. Yet labelers preferred the small model's answers, because it had been through fine-tuning and preference training and actually did what was asked. The lesson is that the later stages don't add much knowledge. They change how the model uses what it already has.
Example 3: Reading a chatbot's confident mistake (real-world use)
You ask a chatbot for a court case supporting an argument, and it names one with a plausible title, year and citation. It sounds exactly like a good answer, because preference training rewarded answers that look complete and confident. But no rater checked that particular citation, and the model generates plausible text rather than retrieving verified records. Knowing how it was trained tells you what to do: treat names, numbers and citations as leads to check in an original source, not as facts. See what AI hallucination actually means.
Quick check
Human raters in RLHF are shown two answers. One is short and correct. The other is long, polished and contains a subtle factual error they don't notice. What's the risk?
How it works (visual)
The three stages of training a chatbot
Common mistakes
Common Mistakes
✕
Assuming the chatbot learns from your conversation in real time.
→ The model's parameters are fixed when you use it. Some companies may use chats to train future versions, depending on their settings and policies, so check the privacy controls.
✕
Believing a helpful, polite tone means the answer was checked.
→ Tone comes from preference training. Verify facts, figures and citations in a primary source.
✕
Thinking bigger models are always better assistants.
→ Fine-tuning and preference training matter a great deal. A smaller, well-tuned model can be preferred over a larger raw one.
Common misconception
“AI chatbots are programmed with rules for what to say.”
Almost none of a chatbot's behavior is written as rules. Its knowledge comes from predicting text during pretraining, and its assistant manners come from training on examples and human rankings. That's why behavior can be inconsistent: the model learned tendencies from data, not a rulebook. Developers can add filters and instructions on top, but the core behavior is learned.
What to do next
What to do next
Treat chatbot answers as a well-read first draft: check names, numbers and citations in an original source.
Look in your chatbot's settings for whether your conversations can be used to train future models, and choose deliberately.
Ask for sources and then open them. A citation you haven't opened isn't a verified citation.
When comparing AI tools, test them on questions you already know the answers to.
FAQ
FAQ
Related terms
Related terms
Pretraining
The first and most expensive training stage, where a language model learns to predict the next token across a very large body of text.
Base model
A model that has only been pretrained. It continues text well but doesn't reliably follow instructions or behave like an assistant.
Supervised fine-tuning (SFT)
Further training on example prompts paired with high-quality answers written or approved by people, so the model learns the shape of a good reply.
Reward model
A separate model trained on human rankings of answers, which scores new answers by how much people would likely prefer them.
RLHF
Reinforcement learning from human feedback: tuning a model to produce answers that a reward model, built from human preferences, scores highly.
Constitutional AI
An Anthropic method where the model critiques and revises answers against a written list of principles, and AI-generated preference labels replace some human ones.
This entry was researched from public sources and drafted with AI-assisted tools, then edited — errors are still possible. Spot one, or want a topic covered? Read our disclaimer.
KnowThis