The 59%: why I stopped letting the LLM do everything

Take five steps with an LLM, each one 90% reliable. How much comes through intact? 0.9^5 = 0.59. Fifty-nine percent. In practice: a five-step agentic pipeline that’s “almost always right” fails two times out of five. That number — which isn’t mine, it’s trivial arithmetic — is why I spent a weekend building a scaffold that takes work away from the LLM instead of giving it more.

The idea

agno-agents is a personal scaffold for multi-agent automations built on the Agno framework (v2.5+). But the core isn’t Agno: it’s an architectural rule I imposed on myself, written down in black and white in the project’s CLAUDE.md. Three layers, separated responsibilities:

• Layer 1 — Directive (what to do). SOPs in Markdown inside directives/. They’re natural-language instructions, written the way you’d give them to a mid-level employee: goals, inputs, tools to use, expected outputs, edge cases.
• Layer 2 — Orchestration (deciding). This is the LLM. Its only job is intelligent routing: it reads the directive, calls the right scripts in the right order, handles errors, asks for clarification. It doesn’t do the dirty work.
• Layer 3 — Execution (doing). Deterministic Python scripts in execution/. API calls, I/O, database. Testable, fast, repeatable.

The sentence I keep pinned up: you don’t scrape the site yourself — you read directives/scrape_website.md, prepare inputs and outputs, then launch execution/scrape_single_site.py. The LLM isn’t the bricklayer, it’s the foreman.

How it actually works

On top of these three “philosophical” layers runs a second, more concrete pattern: a meta-team that builds other teams. There’s an Architect Team (Tier 1) of 4 agents — Analyzer, Designer, Soul Writer, Builder — that takes a prompt like “I need a team that writes Instagram posts for hotels” and spits out a deployable project: a SOUL file, a YAML manifest, and the DB registration. Then the Task Team (Tier 2) actually executes.

The piece I like best is how the nested teams are wired in team_factory.py. A sub-team of workers who collaborate, and a sub-team of reviewers who instead must not collaborate:

# Workers: share context, build on each other
work_team = Team(
    name=f"{manifest.project_id}-workers",
    mode="coordinate",
    members=worker_agents,
    share_member_interactions=True,   # <-- they see each other's work
    ...
)

# Reviewers: broadcast, each blind to the others
review_team = Team(
    name=f"{manifest.project_id}-reviewers",
    mode="broadcast",
    members=reviewer_agents,
    share_member_interactions=False,  # <-- no groupthink
    ...
)

That share_member_interactions=False on the reviewers is a deliberate decision: if the reviewers see each other’s scores, they converge. I wanted independent opinions, so each reviewer sees only the original output, gives a score 0-100, and the leader synthesizes. If even a single one is below threshold, the work goes back. The threshold isn’t a magic number but depends on the domain: 70 for marketing, 75 for research, 80 for code, 85 for legal/compliance. The higher the stakes, the meaner the gate.

Each agent is defined by a SOUL — a YAML validated by a Pydantic model (SoulConfig) that becomes the agent’s personality and constraints:

name: quality-reviewer
display_name: The Hawk-Eye
role: Quality Reviewer
communication_style: blunt
is_reviewer: true
review_score_threshold: 80
review_criteria:
  - "Accuracy: are the facts correct and cited?"
  - "Clarity: is the writing clear and structured?"
constraints:
  - "Approving work without specific, actionable feedback."

Underneath is the rest: 3 providers (openai gpt-4o/mini, anthropic sonnet-4/haiku-4.5, ollama local), a TOOL_REGISTRY of 12 tools resolved dynamically by name, SQLite for per-project memory, and a FastAPI dashboard for the final human approval. About 3,300 lines of Python in execution/ alone.

The hard call: two human-in-the-loop gates

The temptation, when you build agentic stuff, is auto-approval. “The reviewers said 88/100, ship it”. I chose the opposite: no output is complete until a human signs it off. Two gates in sequence — automatic peer review, then human_approval_tool which serializes the entire deliverable to JSON and queues it on the dashboard. And if the work fails 3+ rounds of review, the leader stops digging in and calls notify_human_tool with urgency high. No infinite loops burning tokens in silence.

Is it slow? Yes. But the whole point of the scaffold is that I don’t trust the LLM for the deterministic things, and I don’t trust the LLM reviewers for the final OK either. Consistency is guaranteed by Python; the final judgment is guaranteed by me.

Honest gotchas

A project is credible when it admits where the README and reality diverge. Here they diverge in three places:

• data/projects/ is empty. The scaffold knows how to generate and run a team, but I haven’t yet committed an end-to-end project that was born from the Architect and produced an approved deliverable. The meta-team is code + documentation, not yet a demo that runs.
• Three “mirror” files, but one doesn’t exist. At the top of CLAUDE.md it says “mirrored across CLAUDE.md, AGENTS.md, and GEMINI.md”. CLAUDE.md and AGENTS.md are byte-identical — GEMINI.md isn’t there. Documentation that describes an intention, not the state.
• The key that leads nowhere. The .env exposes GOOGLE_API_KEY, but the provider registry in config.py only knows openai, anthropic, ollama. Google is wired into the environment but not into the code.
• Infrastructure bonus: there are complete k8s manifests (Postgres, ingress, networkpolicy) for a system that hasn’t yet run a single real project. I scaffolded the deploy before the product — the classic mistake.

There’s also a conceptual tension I keep in mind: the “three layers” of the philosophy (directive/orchestration/execution) and the “two tiers” of the runtime (Architect → Task Team) are two different maps of the same territory. Convenient for me, potentially confusing for anyone else who opens the repo.

How it’s going

POC, honestly. A single commit, “full implementation”, which is more a dense scaffold than a product. The pieces are there and they stand up: config, security (path validation, secret redaction with regex, guards against path traversal), team factory, SOUL loader, dashboard. What’s missing is the mileage: running it enough to discover where it actually breaks, and then self-anneal — which is the part of the CLAUDE.md where I promise myself to update the directives every time I learn something. So far I’ve only learned by writing it.

What I learned

• The 59% is the thesis, not a detail. Every decision in the scaffold — SOPs in Markdown, deterministic scripts, multiple gates — exists to move steps out of the “probabilistic” column and into the “deterministic” one. The fewer steps you entrust to the LLM, the higher that number climbs back toward 100%.
• Separation of concerns is easier to write than to respect. It’s tempting-easy to let the LLM do “just a little scraping” in orchestration. The discipline is saying no and writing the script.
• Reviewer independence is a boolean flag. share_member_interactions=False is one line; the groupthink avoided is worth much more.
• Scaffolding the deploy before having a real output is vanity. My k8s manifests are more mature than my empty data/projects/ folder. The next useful line of code isn’t in Kubernetes: it’s the first project that’s born, runs, and gets approved for real.