Agent Harnesses Are Quietly Hard

People talk about building agents as if the hard part is picking the right model and wiring up a few tools.

This is not my experience.

The quietly hard part is the harness around the model. The thing that decides what goes into context, when tools are available, how state is carried forward, what gets remembered, what gets retried, what gets blocked, and what gets logged.

Each part looks manageable on its own. That is what makes it deceptive. The problems start when all of them need to work together at the same time.

Your context has to be relevant but not too large. Your tools have to be clear but not too broad. Your state has to persist but not leak. Your guards have to be strict but not break the experience. Your retries have to recover failures without creating duplicates or loops.

None of this is flashy. Most of it does not show up in a demo. This is where agent systems actually succeed or fail.

That is why so many agent prototypes look impressive for five minutes and then fall apart in production. The model is only one moving part. The harness is what makes the system hold together.

And for that to happen, a surprising number of small things need to be perfectly aligned. Like Pokémon cards, you have to collect them all to have a chance at doing something reliably useful.