Technical

Reinforcement Learning from Human Feedback (RLHF)

A training technique where a reward model trained on human preference data is used to fine-tune an LLM via reinforcement learning, aligning it with human values.

Explained at five levels

Level 1

Teaching the AI to be nicer and more helpful by having people tell it "good answer!" or "bad answer!" over and over.

Level 2

A way to make AI better by having humans rate its answers — thumbs up or thumbs down — so it learns what people actually want.

Level 3

A training technique where a reward model trained on human preference data is used to fine-tune an LLM via reinforcement learning, aligning it with human values.

Level 4

An alignment method that trains a reward model from pairwise human preferences, then optimizes the language model policy via PPO or DPO to maximize the learned reward while maintaining output diversity.

Level 5

A preference-based alignment technique: first training a Bradley-Terry reward model on human comparison data, then optimizing the LLM policy via proximal policy optimization with a KL-divergence penalty against the SFT reference — increasingly supplanted by direct preference optimization.

Definitions are educational summaries. Terminology can vary by source and context.

Sources