Evaluation Beats Prompt Cleverness

I think prompt advice gets treated with a little too much mysticism sometimes. Maybe more than sometimes.

Yes, clearer prompts help. No doubt about it. If you tell a coding assistant the language, framework, constraints, and expected output, you usually get a better first draft than if you just throw “fix this” into the void.

But I would still take a mediocre prompt plus strong evaluation over a beautifully engineered prompt plus weak review. Because the real risk is not that the model misunderstood your “vibe”.

The real risk is that the output looked plausible enough to slip through without proper resistance. That is how you end up with fabricated APIs, subtle bugs, insecure defaults, and code that technically exists but does not really belong in the system.

quote

“Prompting is useful. Evaluation is decisive.” - me, humbly.

Check the docs. Run the tests. Read the diff carefully. Ask whether the output matches the architecture, not just whether it compiles. Treat generated code like external input that now needs to earn its place.

The better your evaluation loop gets, the less you need prompt wizardry to feel safe.

I go deeper on that workflow in Using AI Coding Tools Responsibly.