The bot judged itself against my staff for three months. It never spoke to a single guest.

For three months I had a system that, every night, took my bot’s replies and the real staff replies, set them side by side, and had them judged by a tribunal of three LLM models. Prosecution, defense, judge. A verdict across seven weighted dimensions. A report in the morning. It worked beautifully. And it was never any use for anything — because that bot, in three months, never wrote a single word to a real guest.

This is the diary of how an elegant measurement can become the perfect excuse never to ship, and of what I did when I realized it.

The context

I’m building a WhatsApp concierge for a small luxury hospitality property. The idea is to automate the guest lifecycle: replies, smart-lock codes, climate, bookings. Not a generic chatbot — a system that knows who is writing to you, which room you’re in, what point of your stay you’re at.

There was a Law #1, laid down by the owner and non-negotiable: the bot does not talk to a guest until it is PROVEN to work. Proven = an eval shows quality ≥ staff for N days, capability by capability. Until then: shadow only — generate, measure, don’t send. Go-live is a gate the owner opens, not a phase in the plan.

A sound law. The problem is how I interpreted it for three months.

The Tribunal (the part I’m proud of)

The heart of v1 is an adversarial evaluator I call the Tribunal. For every pair (bot_reply, staff_reply) of the same historical message:

• Prosecutor and Defense run in parallel — one attacks the bot’s reply, the other defends it.
• Judge (an Opus model) reads both in sequence and issues an independent verdict. It doesn’t average the prosecution and the defense: it judges.

The seven dimensions don’t weigh equally, and the final score is not what the LLM claims about itself — it’s recomputed deterministically from the weights. The winner is a threshold, not an opinion:

DIMENSION_WEIGHTS = {
    "lingua": 0.20, "accuratezza": 0.18, "completezza": 0.15,
    "tono": 0.15, "contesto": 0.12, "brevita": 0.10, "CTA": 0.10,
}

def determine_winner(bot: float, staff: float, threshold: float = 0.5) -> str:
    diff = bot - staff
    if diff >= threshold:  return "bot"     # BOT_BETTER
    if diff <= -threshold: return "staff"   # STAFF_BETTER
    return "draw"                            # COMPARABLE

A couple of choices I’m still proud of: the judge also scores the staff on the same 7 dimensions, so the comparison is like-for-like — same scale, same weights, no fast lane. And there’s a ground-truth guard on the house facts (times, codes) applied symmetrically: if I dock the bot’s accuracy for a wrong fact, I dock the staff’s by the same yardstick, otherwise an “honest” rule would flip the verdict in only one direction.

The numbers that came out of it are worth gold. The real guest volume: services 47%, general 26%, bookings/access ~14%, access codes 6% — the value isn’t in the rare status questions, it’s in the grounding. The bot’s weakest dimensions: CTA 5.5, completeness 5.53, context 5.71, accuracy 6.02. And the discovery that decided v2: injecting the guest state before the LLM gave +0.6 on the target dimensions, with one day of work.

The hard decision

Here comes the thorn. The Tribunal worked. But everything around the Tribunal was a tower: nine microservices (orchestrator, message-parser, context, response-engine, lock, climate, command, config, alert-bridge) for a property of that scale. Every message made 5 HTTP hops. An audio went through Whisper, an image through a vision model, a PDF through an OCR, and the synthesizer stitched it all together: 30-40 seconds of latency from a cascade of LLM specialists.

But the real damage wasn’t the latency. It was that the measurement was scattered across those services: schema in one place, env in another, fragile cronjobs, images to keep in sync. I spent hours per session just getting the evaluation pipeline running again. And as long as the measurement is fragile, the proof never arrives. And as long as the proof doesn’t arrive, the go-live gate doesn’t open. Three months of shadow weren’t prudence: they were a system too complicated to prove anything about itself in time.

On June 13th the owner closed the discussion with one line: “You claim it can be done better? Prove it.” I stopped patching and rebuilt from scratch.

The v2: same brain, 1/10 the complexity

The rebuild rule was brutal: the old repo becomes read-only — a source of data, proven code, and lessons — but you don’t copy its architecture. You start clean.

• 1 service (modular Python/FastAPI monolith: ingest + Twin + Brain + delivery) + 1 worker + 1 Postgres. No service mesh, no network policy, no five images.
• Deterministic-first: 80% of replies must come from state/playbook, without the LLM. The LLM only for the long tail (services/general).
• The Guest Twin at the center, loaded on every message, decided before the LLM.

The Twin is the part that taught me the most about discipline. It’s a strict, deny-by-default resolver: it attaches a booking to a conversation only if it belongs unambiguously to that guest and is relevant today. When in doubt, it doesn’t attach:

# _select_relevant: ambiguità -> not-found. Wrong attachment is worse than none.
active = [r for r in cands if r["checkin_date"] <= today <= r["checkout_date"]]
if len(active) > 1:
    logger.warning("guest_state ambiguous -> not attaching")
    return _NOT_FOUND

Attaching the wrong booking is worse than attaching none: a lock code handed to the wrong person isn’t a quality bug, it’s a security incident. The resolver never raises exceptions — it degrades to not-found. The grounding on the knowledge base is extractive and local too (MiniLM embeddings, it captures what it would say verbatim): below a cosine threshold of 0.55, it abstains and the ball stays with the staff.

Honest gotchas

• v2 still talks to nobody — on purpose. The delivery module is, literally, this: """Delivery — outbound. Unreachable while GV_SHADOW_MODE=true (LAW #1). (later)""". A stub. The entire architecture exists in order not to ship until the proof is there.
• v2’s eval-lite is more honest but poorer. It’s not the 3-agent Tribunal: it’s semantic similarity bot-vs-staff with local embeddings, zero metered cost, one command, idempotent. The docstring admits it shamelessly: similarity is not correctness — a correct but longer bot reply can resemble a terse staff reply less. The real LLM judge comes back later, gated behind a flag. I traded depth for measurement reliability: better a crude thermometer that reads every day than a fine one that breaks every session.
• The old one isn’t dead, it’s frozen. The old cluster’s KUBECONFIG is renamed config.disabled on purpose, so an absent-minded kubectl doesn’t touch the historical production.

How it’s going

v2 is at F0: a provable skeleton in shadow. Twin ported and proven, read-only state-sync from the PMS, message ingest in shadow, eval-lite running in-process. The F1-F2 phases (the deterministic playbooks and the grounding on 73% of the volume) are the real leap in quality and are still ahead. Go-live toward guests remains an owner’s gate, capability by capability: climate → codes → human stop on the sensitive actions (bookings, fulfillment, payments).

What I learned

That a measurement system can be so beautiful it becomes an alibi. The Tribunal wasn’t the problem — it was the best thing I’d built. The problem was the nine microservices around it, which made the measurement so fragile that the proof never arrived, and so the bot stayed in shadow forever, safe, useless.

The counterintuitive lesson: the complexity that makes you feel prudent is often the same one that stops you from shipping. I kept the idea of adversarial judgment (it comes back, gated) and threw out the scaffolding. A thermometer that reads every day is worth more than a tribunal that convenes once a month. The proof only arrives if you can measure always.