AI library

Source · paper

Training language models to follow instructions with human feedback

Author
Ouyang et al. (OpenAI)
Published
4 March 2022
Added
4 October 2026
Original
https://arxiv.org/abs/2203.02155

Summary

Describes InstructGPT: a pre-trained model fine-tuned on human demonstrations, then optimised with reinforcement learning from human feedback (RLHF) using a reward model trained on human rankings of outputs.

Key points

  • Three stages: supervised fine-tuning, reward model training, RL optimisation (PPO).
  • A 1.3B InstructGPT model was preferred by raters over the 175B base GPT-3.
  • Alignment training improves helpfulness and truthfulness at small cost to benchmarks.

Why it matters

Established the post-training recipe behind chat assistants; later methods (RLAIF, constitutional training, DPO, RL with verifiable rewards) build on or replace parts of it.