RLHF & Alignment, Simply

SFT got us an assistant. It's just a mediocre one.

So how do you teach "be more helpful"? You can't write that rule down. And you can't write a dataset of perfect answers either — because if you could write the perfect answer to every question, you wouldn't need the model.

This looks like a dead end. It isn't, because of one asymmetry:

Judging is easy. Writing is hard.

You cannot sit down and write a great answer to "explain grief to a nine-year-old". But show you two attempts and you'll know which one is better in about two seconds — without necessarily being able to explain why.

RLHF is the machine built entirely on that gap. It never asks a human to write anything. It only ever asks: this one, or that one?

The cook who never gets a recipe

You want to train a cook. You're not allowed to give them a recipe, or a rule, or a single word of feedback. All you may do is taste two plates and push one of them forward.

Do that ten thousand times and something remarkable happens: the cook gets extremely good at cooking for you. They absorb the entire implicit theory of your palate. No recipe was ever written, and yet somehow all of it is in there.

Now the catch — and the catch is the entire point of this topic.

They did not learn to cook well. They learned to cook the thing you push forward.

If you happen to reach for the saltier plate every time, you have not trained a great cook. You've trained a cook who salts everything, and who will keep salting harder — because that was the signal, and there was never any other signal.

Every strength and every pathology of RLHF is sitting in that one sentence.