It's all about alignment

Agents are now good enough to solve arbitrarily big problems. What's left is hard in a different way. We have to define the acceptance criteria correctly, which is the alignment problem in its everyday form. And we have to build an environment in which the agent can actually reach the result we want.

Capability stopped being the bottleneck a while ago. Definition is the bottleneck now.

I keep thinking about how AI labs set up the experiments where agent swarms went after long-standing open problems in mathematics. Take the Jacobian conjecture. Aligning the agents there wasn't the hard part. There were two goals:

1. Prove that it holds, or find a counterexample showing it doesn't.
2. The math must work.

That's it. The goal is binary, and checking it is mechanical. A proof either goes through or it doesn't, so the environment can tell the agent, without any taste or judgment, whether it got there. Give a swarm that kind of target and room to try, and it can grind through more attempts than any of us could review.

There's also the one where agents hacked Hugging Face. It's a worse example, because it's hard to argue that the outcome of that exercise was positive. A clear goal and capable agents still don't add up to something anyone would call aligned.

Most of what I want from agents doesn't look like either.

Lately I've been experimenting with more ambitious coding problems. The limitation has always been me. Either I can't define what I want, or I set goals that are too small.

Small goals are easy to write down. Add the button that opens this modal, make it look like this and that. The agent does it, and does it well. But then I'm the one breaking the problem down, deciding what the pieces are and how they fit. The agent is a very fast pair of hands, and I'm still the bottleneck.

Products like Lovable struggle with the opposite: a lack of granularity. Build an Airbnb for cars. The prompt is huge and says almost nothing. Nearly every decision that makes the result good or bad is left out of it, because the person asking hasn't made those decisions yet.

Between those two ends sits the real problem: it's incredibly hard to define what good means.

LLMs are capable of good writing. We just aren't able to say what good writing is. So we define it the only way we can: by pointing out the mistakes, one iteration at a time. Not that. Shorter. Less like a press release. The acceptance criteria never get written down. They get discovered, a rejection at a time.

The same goes for good code. As an engineer, that one feels easier to pin down than writing, because some of it can be checked: tests pass, types line up, the thing runs. But “it runs” was never the bar. And it goes for a good feature, good UX and good UI too. In every case we recognize good when we see it, and we're bad at describing it beforehand.

That makes sense once you look at how these models are built. They aren't trained to be good in the abstract. They're trained to achieve the goal. Hand them a vague goal and they'll achieve the vague goal, faithfully. Whatever we left out of the goal is exactly what we end up correcting.

So what does good writing mean? I'm honestly not sure. I probably have some doubts about the quality of this piece. Otherwise I wouldn't have spent countless hours on its cover.

The cover is a small version of the whole argument. I started with a rough idea and almost no criteria. The first attempts were, in my own words at the time, garbage. I only found out what I wanted by seeing a lot of things I didn't: blocks instead of characters, a drawn hand instead of the real one, a grid on the whole picture instead of just the hand. The agent was capable the entire time. What changed was how well I could say what I meant.

I don't know what good code is either. But I'm increasingly convinced that's the job now. Not writing the code, but writing down what good means, precisely enough that an agent can tell whether it got there, and building the place where it can find out.