Honestly? For a while, we had visibility, not validation. Different things, and conflating them is the whole story.
We could watch every run our agents made. Pull up a case, see what went in, see what came out. What we couldn't do was say whether the answer was right, which step went wrong when it wasn't, or whether this month was any better than last. That's a black box with a receipt stapled to it.
This is the story of how we closed that gap, and why I think the way you grade AI now matters more than which model you pick.
What We Were Missing
Our tracing was doing real work. It captured what went into every run and what came out. What it didn't capture was the middle: the individual model calls and tool steps where the agent actually does its reasoning. So when an answer came back wrong, we couldn't tell whether one step broke, or whether every step was fine and the logic connecting them wasn't.
Validation, meanwhile, was a person. Somebody would ship a prompt change, read a handful of traces, and decide it looked better. We hit the same three problems over and over. Nobody had written down what correct meant for each step, so every reviewer used their own private standard. The traces people read were the ones that looked off, but the failure that actually hurts is the plausible, confident, well-written wrong answer, which renders beautifully in a trace viewer and gets scrolled right past. And nothing was saved, so "is this better than last month?" had no answer beyond a feeling.
The part that bothered me most was that the people who knew the right answer were already producing it. Our underwriters review, correct, and override what our agents produce every single day. That's ground truth, generated for free as a byproduct of the work, and none of it was being joined back to what the agent had said about the same case. That wasn't negligence. It's normal for a fast-moving AI product, and the tracing we already had is what made the next step possible.
Seatbelts for Sports Cars
I came to this from traditional machine learning, where nobody ships a model without a held-out evaluation. That isn't controversial. It's just data science: build a dataset, label it, look at where you're wrong, iterate. Agents somehow got a pass on all of that, and I wasn't willing to give them one.
Agents are sports cars. Evals are the seatbelt.
The way I explain it: agents are sports cars, and evals are the seatbelt. A seatbelt doesn't steer, and it doesn't tune the engine. It just keeps you from eating the steering wheel when something goes wrong. If your stance is that you don't need evals because you're careful, you're confusing confidence with safety.
For anyone who isn't an engineer, the practical version is simpler. If I swap the model under one of our agents for a newer one, I need to know we got better or stayed the same, and that nothing regressed. Without a baseline, even an obvious upgrade is a guess. And reviewing by hand doesn't scale. A person reading traces after every change sees a small sample, and everything else ships unseen.
Most of the cost of skipping evals shows up where nobody's counting. Quality slips quietly, a step starts dropping a field, and it just becomes the new normal. Every wrong answer lands on an expert who has to catch it and redo it, and that never appears in an engineering budget. The biggest cost is the decisions you can't make. Should we upgrade the model? Which step needs work? Can we lean on this agent for more? Without a number, the answer to that last one defaults to no, indefinitely.
What Echo Does
Echo is our evaluation system. It breaks every run into its individual steps and grades each one with checks written for that step, so when an answer is wrong, we can point to where it went wrong. Every grade also has to explain itself. An LLM spitting out a number is pointless. What does 60% even mean? Our LLM judges are part of the evaluation, not the evaluation itself. One of our checks, for example, has an LLM break a response into individual claims that can be verified, then grades each claim against the truth. If the agent got three out of five right, that's 60%, and it's math, not a model guessing. Then we can look at the two claims that failed and find the pattern behind them.
Production is only half of it. Before a change ships, whether it's a new prompt, a new step, or a different model, we run it against a fixed set of real cases where we already know the right answer, because our experts decided them. That's our golden dataset, and it grows every time we find a new way to fail. Every failure case is a free bug report: we collect it, name it, and add it to the set. A change has to hold the baseline steady or raise it before it goes out. Once it's live, Echo keeps grading the same steps on real traffic, so anything the dataset didn't anticipate shows up by the next monitoring run instead of whenever someone happens to notice.
We've built more than 30 evaluation metrics, purpose-built for our agents, and not one of them came off the shelf. The test for adding one is a question: what do you keep checking by hand? The test for removing one matters just as much. We retired a citation check after it sat at 100% for months, and we never bothered with generic hallucination or toxicity scores on internal agents. A metric that never moves is a metric nobody opens.
The obvious objection is that this is just AI grading AI. I agree that no model should be the only judge of its own work. When Athena prepares an underwriting recommendation with one family of models, Echo can grade it with a model from a different vendor, so the judge doesn't share the same blind spots. On the evaluations that matter most, several judges grade independently. When they agree, our confidence goes up. When they disagree, a person looks at it.
That costs more, and it's on purpose. Be inexpensive when generating recommendations. Be expensive when evaluating them. Generation happens on every quote and has to be fast, while evaluation is where confidence gets built. Every judge also gets calibrated against what our experts actually decided, because LLM judges are useful, but they're also liars with confidence.
"ok I'll just do it"
The clearest lesson came from Abby, our product configuration agent, which helps our product team build and ship changes.
In 16 of the 42 sessions we graded, 38%, Abby told someone something like "I've updated the handler and the tests should pass now," and there was no commit and no pull request anywhere in the session. Every accuracy metric passed those turns. The claims were coherent and based on files she'd actually read. There was nothing factually wrong to catch.
What the transcripts showed was people asking again. Not angrily:
"did that actually get committed"
"I don't see it"
"ok I'll just do it"
That last line is the expensive one. Nobody complained, because technically nothing was wrong. People just quietly stopped trusting the agent and did the work themselves. On a pass/fail grade, that session counts as a success. In reality, it's a total loss. So we added a reliability check that compares the claim against reality: is there an actual commit or pull request? "Did it say it finished" and "did it finish" are two different columns now.
Two more things came out of that work. First, experts don't get emotional, they get brief. Our sentiment judge was reading "no, wrong path" and all-caps corrections as neutral, when an expert redirecting an agent three times in one session is about the strongest frustration signal you'll ever get. Second, check that a problem exists before you fix it. Not a single session scored positive, so I loosened the judge to catch more. It caught none, and it softened five genuinely frustrated sessions to neutral. There weren't any delighted sessions in that data to find. I reverted it and shipped nothing.
How Athena Earned Our Underwriters' Trust
Athena, our AI underwriting support agent, didn't start out trusted. Like a new underwriter, she spent a long stretch learning next to experienced ones: three underwriters, three months, roughly 3,000 cases. Our underwriters were the decision-makers then, and they still are. Athena analyzed submissions, made recommendations, and explained her reasoning, and a person reviewed every one.
That period showed us where her reasoning matched our underwriters' and where it didn't, and it taught her MGT underwriting rather than just insurance. A general model understands insurance concepts. It doesn't know our appetite, our referral process, or the quirks of our products. She also didn't learn automatically from every case. A person turned experience into guidance first, so a one-off never quietly became policy.
We started with our most rule-based product, where "correct" is hardest to argue about, and every improvement had to hold our evaluation baseline steady or raise it. When the evals showed her recommendations matching our underwriters' decisions, the answer to "can we trust her advice?" finally changed from no to yes. Brian and Connor wrote about what that looks like from the underwriter's chair.
My favorite example is a referral trigger that kept firing on cases our underwriters approved basically every time. No single case looked wrong, so grading cases one at a time would never have caught it. Only the aggregate showed it. Echo surfaced the pattern, a person reviewed it, and it became guidance. Now Athena knows that.
The underwriters didn't go anywhere, and they still make the decisions.
The underwriters didn't go anywhere, and they still make the decisions. What changed is how much they trust her advice. Instead of rebuilding every recommendation from scratch, they spend their time on the exceptions and edge cases where judgment matters.
A Loop With a Person in It
The hardest part of building Echo was closing the loop. A score nobody acts on is an expensive opinion.
Every week, Echo groups the grades into patterns and drafts candidate lessons from them. Those run through a set of checks and then land in front of a person in a weekly digest. Nothing adopts itself. Approved lessons reach the agent two ways: as guidance it can look up, and as base rates for specific referral flags, drawn from what our underwriters actually decided. Then the new production traffic gets graded again.
Every one of those checks exists because something went wrong without it. The one I'd warn other teams about: the model behind our relevance check got retired, and the check quietly fell back to simple keyword matching. Effectively everything passed. The pipeline looked healthy, and it was accepting everything. A gate that fails open is worse than no gate, because it launders noise as reviewed. Now it fails loudly.
Grounding a feedback loop in the agent's own output is how you build a very confident echo chamber.
Keeping a person in that loop is deliberate, and so is grounding it in what humans decided rather than what the agent thought. Grounding a feedback loop in the agent's own output is how you build a very confident echo chamber. Ours has a reference point outside itself, and that's the difference between a system that learns and one that just agrees with itself more and more over time.
What Evals Can't Tell Us
Echo can only grade the evidence it has, and that has limits worth saying out loud.
Human agreement is one of our strongest signals, but it isn't ground truth. When Athena's recommendation matches an underwriter's decision, Echo can conclude that she aligned with human judgment. It can't conclude that the human decision was objectively correct. Underwriting involves real judgment, and experts don't always agree.
Some answers also take years. Whether an underwriting call was right can show up long after, when a claim comes in or doesn't, and sometimes the human decision lands a week after the agent's recommendation. In those cases, I'd rather report "we don't know yet" than manufacture a label to fill the column. It's also why Echo grades the reasoning and not just the outcome. A right answer reached for the wrong reason is a coin flip that hasn't landed yet.
So we still audit by hand, and we keep calibrating. Evals aren't one-and-done. They're an investment, and shipping a new dashboard is not the same as shipping reliability.
Measure Before You Need To
When someone tells me they'll add evals once things stabilize, my answer is usually the same: you can delay sophisticated evaluation, but you shouldn't delay measurable evaluation. AI systems get harder to evaluate as they grow. Early on, one engineer can read every response. At thousands of users and dozens of workflows, that's gone, and if you weren't measuring from the start, you've lost the baseline you'll wish you had. You can add a metric later. You can never go back and measure last quarter.
A weekend prototype doesn't need a framework. The line is when people start relying on the system. You don't need a tool to start, either. Take 30 real examples, put them in a spreadsheet, and write down what "good" means. If you can't do it by hand for 30 examples, you're not ready to automate it for 30,000.
If I had to pick what I'm proudest of, it's one word: trust. At my last company, a ten-person evaluation team did the job Echo does today.
Two years ago, the big question in AI was whether these models could do useful work. That's mostly settled. The hard part now is building AI whose advice organizations can trust, and proving that trust every day. Everyone has access to the same models. The advantage is everything you build around them.
The first chapter of AI was about intelligence. The next one is about trust. And trust isn't something you prompt into a model. It's something you build into the system.
Figures as of August 2026.




