← All explainers
An explainer
LLM
Post-training
Cover & TOC
Foundations & framing
  • 01 What is post-training?
  • 02 Probability, policies & gradients
  • 03 From next-token to behavior
  • 04 The alignment problem
Instruction tuning & supervised fine-tuning
  • 05 Instruction tuning is born
  • 06 The SFT stage in practice
  • 07 Synthetic & self-generated data
RLHF & the preference era
  • 08 Learning from human preferences
  • 09 Optimizing against the reward
  • 10 RLHF scales to language
  • 11 Reward models
  • 12 RLAIF & Constitutional AI
RL fundamentals & PPO
  • 13 Policy gradients & REINFORCE
  • 14 Value, advantage, baselines
  • 15 TRPO to PPO
  • 16 PPO for RLHF in practice
Offline & direct preference optimization
  • 17 Reward hacking & over-optimization
  • 18 Direct Preference Optimization
  • 19 The DPO zoo
  • 20 Rejection-sampling alignment
RLVR & the reasoning era
  • 21 Bootstrapping reasoning
  • 22 Process vs outcome rewards
  • 23 Inference scaling & o1
  • 24 RL from verifiable rewards
  • 25 GRPO & DeepSeek-R1
Modern algorithms, agentic RL & the frontier
  • 26 GRPO refinements
  • 27 Scaling open post-training
  • 28 Agentic & tool-use RL
  • 29 On-policy distillation
  • 30 RL scaling laws
  • 31 Generative reward models
  • 32 Recap
§ Glossary
Section 16

PPO for RLHF in practice

The KL-to-reference penalty and the loop

Premium section

Unlock “LLM Post-training, from the ground up”

This section is part of the paid explainer. Buy it once for lifetime access to all 32 sections. Sections 01, 02, 03 are free to read.

Sign in to buy · $8

New here? Create an account

← Previous · §15
TRPO to PPO
Next · §17 →
Reward hacking & over-optimization
© 2026 Knowlific Inc. All rights reserved.
Terms of Service Privacy Policy