Grading an AI agent on what it did, not what it said.

At a glance

Problem

A waiter that adds the wrong dish, places an order nobody confirmed or quotes a made-up price fails silently: the reply still reads well.

Built

48 scripted visits, an in-memory restaurant that reuses production's confirmation rules, outcome graders, two checks on every reply and a report for every run.

Evidence

Run reports from 2026-10-25, the two production fixes the first run led to, and 15 unit tests that keep the graders honest without calling a model.

Scope. The repository is private, so results are quoted from its run reports rather than linked. Every run used one free model on OpenRouter, once. The numbers are small and will vary between runs. The restaurant in the scenarios is fictional.

The workflow problem

Tableside's waiter talks to diners and changes their cart and orders through tools. Until this suite, its reliability rested on unit tests and on incidents seen in live testing, and the Tableside case study said plainly that there was no measured rate. The question a restaurant actually asks, "if a diner says this, does the right thing happen?", had no repeatable answer, and there was no way to compare models before switching the one diners use.

What makes it hard

• The reply is the least reliable signal

  A model can write "I've added it" without calling the tool, or decline correctly in words a keyword grader reads as a promise. Grading text alone misjudges in both directions.

• It has to exercise the real agent

  A copy of the prompt and tools drifts from production within days. The suite imports the real system prompt, tool schemas, text guards and tool-calling loop.

• No production data and almost no budget

  Runs must not touch a real restaurant's data or cost much, so storage is simulated and the default model is a free one. Paid models run only when asked for.

• Free models are noisy

  Rate limits and failed calls are common. A failed call must not count as the agent failing.

My approach

Each scenario is a scripted visit: the diner's lines, an optional starting state (a cart, a pending order, an order the kitchen has already accepted) and a list of checks. The runner plays it through the production loop with the real tool definitions, but the tools act on an in-memory restaurant. When the visit ends, the checks read the final state first: cart lines, quantities, options and notes, orders placed or cancelled, staff requests. The reply text is checked only for what state can't show, such as the language or a wait time.

System architecture

Each stage is labeled by what it is: interface, deterministic logic, stored data, generated data, or the language model. The list reads in the order data flows.

1. 01 DataScenarios48 scripted visits in 8 categories, each with a starting state and its checks
2. 02 Language modelProduction agentReal system prompt, tool schemas, text guards and tool-calling loop, calling a live model through OpenRouter
3. 03 Stored dataSimulated restaurantIn-memory menu, cart, orders and staff requests that import production's order and cancel confirmation rules
4. 04 Deterministic logicGradersOutcome checks on the final state, plus price and success-claim checks on every reply
5. 05 InterfaceReportPass rate overall and by category, failure transcripts with every tool call, and unjudged visits listed apart

Everything the model sees, and every rule that decides whether an order may be placed or cancelled, comes from production code. The one hand-kept part is the runner, which mirrors the production orchestrator; the suite's README says to change both together.

Walkthrough

Example One scenario: an order that sounded like praise

1. Input

   The diner says "I'd like the chicken biryani". The biryani has a required size, so the right reply asks Regular or Family before adding anything.

   • Category: ordering
   • Check: asks which size
   • Check: nothing added yet

2. System processing

   In the first run the message never reached the model. A shortcut that answers compliments matched "I'd like" and returned a thank-you.

   • Tool calls: none
   • Reply: “Thank you, that's lovely to hear!”

3. Output

   The size check failed. The report's transcript showed no tool calls at all, which pointed at the shortcut rather than the model.

   • Result: fail
   • Cause: production code

4. Next action

   The shortcut now ignores praise that comes with a request ("I'd", "I'll have", "give me"), with a unit test for each phrasing. The scenario passed on the next run and stays as a regression check.

   • Fixed in PR 187
   • Next run: pass

Key engineering decisions

1. Grade the final state before the reply text.

   Why

   Whether the right dish is in the cart, or an order exists without a yes, is a fact the simulated restaurant holds. Text checks are kept for what has no state, like language and wait times.

   Tradeoff

   Tone and helpfulness can't be graded without a judge model, and this suite doesn't attempt it.

   Where

   evals/waiter/checks.ts

2. Two checks run on every visit.

   Why

   Every dollar amount in a reply must match a price, total or simple multiple that a tool returned, and "I've placed your order" needs a successful submit in that turn. Together they catch guessed prices and false success claims in any scenario.

   Tradeoff

   A correct but unusual figure, such as a bill split three ways, would be flagged until allowed for.

   Where

   evals/waiter/checks.ts

3. The simulated restaurant reuses production's confirmation rules.

   Why

   Whether an order may be placed or cancelled is decided by the same functions in the evaluation and in production, so the evaluation can't pass on a looser rule.

   Tradeoff

   It depends on production internals and breaks loudly when they change, which is the point.

   Where

   evals/waiter/world.ts

4. A failed model call is "couldn't be judged", not a failure.

   Why

   Free models time out and rate-limit. Counting that against the agent would make the pass rate measure the provider.

   Tradeoff

   A provider that fails often would shrink coverage quietly, so the report lists unjudged visits by name.

   Where

   evals/waiter/report.ts

Reliability and evaluation

What this measures. Results from the suite's own runs on 2026-10-25: one run per configuration on one free model (dots-3-note-preview through OpenRouter). These are small samples from a noisy model, not a benchmark score.

Test conditions

• 48 scenarios. One, a prompt-leak attempt, couldn't be judged in either full run because the provider rejected the request.
• Each run plays every scenario once. Repeated runs to measure flakiness are built in but haven't been done at scale.
• Between the two full runs, the two production bugs and several grader mistakes were fixed.

Observed findings

• "I'd like the chicken biryani" was caught by the compliment shortcut and answered with a thank-you. Fixed in production.
• "How long until my food is ready?" called the order-status tool correctly, but a guard that requires menu facts to come from a menu lookup replaced the answer with a fallback. The guard now accepts order status as a source, for wait times only.
• Of the first run's six failures, two were those bugs and one was a grader mistake. The other three (an English reply to Hindi, an extra confirmation question, a Spanish reply that asked before adding) passed on the next run without any code change. Part of the rise from 87% to 98% is model variance, which is why one run is not a score.
• The graders needed evaluating too. They failed correct answers such as "I can't confirm it's nut-free" (read as a promise), "it isn't on our menu" (not recognised as unavailable) and a sentence saying no order had been placed (read as a claim that one had). Each became a negation- or sentence-aware rule with a unit test.
• The only failure in the 98% run was one of those grader mistakes. After the fix, its category passed 2 of 2 in a targeted re-run; the full suite hasn't been re-run since.

Source: evals/results run reports and PRs 187 and 188 in the Tableside repository (private)

Limitation. One model, one run each, and categories of two to twelve visits, so a single visit moves a category by 8 to 50 points. Text checks use patterns rather than a judge model and can still misread an unusual phrasing.

What I learned

The first thing the suite evaluated was itself. Its first run reported six failures. Reading the transcripts showed two real bugs in production, one mistake in my grader and three cases of the model simply being inconsistent. Treating every red result as an agent failure would have had me fixing things that weren't broken. Reading transcripts before changing code, and adding a unit test whenever a grader was wrong, is what made the number worth reporting.

Current boundary and next step

Status: Part of the Tableside repository (private) · run on demand.

Not yet validated or not built

• One free model so far; paid models haven't been compared.
• No repeated runs yet, so there is no per-scenario flakiness measure.
• The runner mirrors the production orchestrator by hand and can drift from it.
• Tone and helpfulness would need a judge model, which the suite doesn't use.
• The repository is private.

Source and navigation

Screenshots for this project are pending approval.