The IT department that argues with itself: three layers and a human in the loop

There’s a precise moment when a single operator realizes they’re too small for enterprise tools and too big to handle everything by hand: when, facing a “slow OSD” alert, they have to decide alone whether to buy a disk, move a workload or do nothing — and they have no one to argue with before deciding. IT-Department is born from there: an IT department made of AI agents that rebut each other, so the human receives an already-debated decision instead of a raw alarm.

The idea: not a script, an org chart

The system manages a hybrid self-hosted Kubernetes cluster — workers and Ceph storage on bare metal, control-plane on Hetzner VMs and services inside and outside the cluster. But the point isn’t what it manages — it’s how it thinks. Instead of a single omniscient agent, I modeled six teams with distinct mandates:

• NOC — eyes and ears: metrics, health checks, first response.
• SOC/CERT — security: CVEs, hardening, certificates, incident response.
• SRE — capacity planning, Ceph, DR, drafting of technical tickets.
• Architecture Board — the decision table: plays devil’s advocate on every proposal.
• Procurement — hardware market scouting.
• Project Office — migrations and change management.

The rule that holds it all together is a single one: the teams feed and contradict each other. If the NOC says “we need more RAM”, the SRE can retort “no, the bottleneck is the disks, look at Ceph’s latencies”. The Board weighs both positions. The human decides, informed.

Architecture: three layers, one clean separation

The backbone is a three-layer model, repeated obsessively in every instruction file:

• Layer 1 — Directive (what to do): SOPs in Markdown in directives/. They’re natural-language instructions, the way you’d give them to a mid-level employee. Each team has its own.
• Layer 2 — Orchestration (deciding): the LLM. It reads the directive, calls the right tools in the right order, handles errors. It’s the glue between intent and execution.
• Layer 3 — Execution (doing): deterministic Python scripts in execution/. APIs, parsing, kubectl, files. Reliable, testable, fast.

The rationale is written in black and white in the CLAUDE.md, and it’s the sentence that justifies the whole structure:

If you do everything yourself, errors multiply. 90% accuracy per step = 59% success over 5 steps. The solution is to push complexity into deterministic code.

Translated: the LLM shouldn’t do the work, it should decide which deterministic script to invoke. A collect_ceph_status.py that parses ceph osd perf doesn’t get it wrong one time in ten the way a model “looking at” the output would. Intelligence is added on top of the deterministic data, it doesn’t replace it.

How a human gets woken up: two levels and a state machine

Here lies the heart. Not everything deserves to wake the operator. The orchestrator distinguishes two classes of action:

• L1 — autonomous: routine maintenance the system does on its own. A real example from the code: if the Velero namespace has more than 10 Succeeded pods, it cleans them up without asking.
• L2 — requires authorization: anything destructive or costly. Here the system doesn’t act: it prepares and flags.

And there’s a choice that says everything about the philosophy — pods in CrashLoopBackOff are detected and logged, never restarted automatically:

if "CrashLoopBackOff" in line:
    # log only, non restart automatico
    crashloop_pods.append({"namespace": ..., "name": ..., "restarts": restarts})

A pod that crashes 2400 times isn’t a restart problem: it’s a symptom. Restarting it automatically would hide the disease. Better an L2 alert.

But the piece I’m most proud of is the decision state machine (decision_manager.py). Every Board proposal becomes a JSON file with an explicit lifecycle:

NEW → NOTIFIED → SEEN → ACCEPTED/REJECTED → IMPLEMENTED
NOTIFIED → (5 giorni senza SEEN) → EXPIRED → re-analisi → NEW

The transitions are validated upfront — you can’t jump from NEW to IMPLEMENTED:

valid_transitions = {
    "NEW":      ["NOTIFIED"],
    "NOTIFIED": ["SEEN", "EXPIRED"],
    "SEEN":     ["ACCEPTED", "REJECTED"],
    "ACCEPTED": ["IMPLEMENTED"],
    "REJECTED": [], "IMPLEMENTED": [],
    "EXPIRED":  ["NEW"],
}

The elegant detail is EXPIRED → NEW. If a decision sits for 5 days without the human looking at it, it doesn’t expire in silence: it gets re-analyzed with fresh data and re-proposed, with a refresh_count that grows. The system insists, but never decides in your place. It’s the opposite of alert fatigue: instead of 200 identical notifications, one decision that comes back more informed.

The honest gotcha: the debate isn’t (yet) code

Here’s the part a README would sell differently. The documentation says: “Orchestration: AGNO framework for multi-agent teams with roles and collaboration”. I ran grep -rn agno on the whole repo. Zero imports. Zero Agent(), zero Team().

The multi-agent framework, today, is aspirational. The debate between NOC and SRE doesn’t happen between two Python processes exchanging messages: it happens inside one LLM which, reading the various teams’ directives as a persona, plays both roles and produces a structured Decision Record. The “confrontation” is a prompt-orchestration pattern, not an architecture of concurrent agents.

Does it work? Yes, surprisingly well — because the real substance is in the deterministic scripts (those exist and run) and in the state machine (that exists and validates). The AGNO layer is the icing that will formalize the roles, but the cake already holds up without it. It’s exactly the kind of README/reality drift that must be said out loud: the value isn’t in the cited framework, it’s in the fact that the data is reliable and the human stays in the loop.

How it’s going: genuinely operational

It’s not a slideware POC. Phases 0–4 are complete and the department produces real output. A recent cycle turned up, among other things:

• NOC/SRE: 7 CRITICAL + 8 WARNING across 7 nodes; a node with Ceph storage ~19× slower than another (root cause: HDD under I/O stress, not CPU/RAM); a control plane at 2 vCPUs saturated at 100%; a worker with 283 pods, 55% of the cluster.
• Capacity: ~120 PVCs, ~5TB used out of ~43TB (12%) — no storage panic, the bottleneck is latency, not space.
• SOC: 6 CRITICAL and 124 WARNING (SSL flexible, no fail2ban on the bare metal, dozens of DNS records with the origin IP exposed), against TLS certificates all valid.

Every finding flowed into one of the 4 P0/P1 Decision Records that wait — patiently, with their refresh_count — for my decision. Continuous automation (collection agent via cron every 5 minutes, alerting) is the next incremental phase.

What I learned

1. Reliability doesn’t come from the LLM, it comes from where you take it out. Every time I moved logic from the model’s reasoning to a deterministic script, the system became more predictable. The model should choose, not compute.
2. The debate matters more than the answer. Forcing the system to always produce a “do nothing” option, a devil’s advocate and a dissenting opinion changed the quality of the recommendations more than any “be smart” prompt.
3. Letting a decision expire is a feature, not a bug. EXPIRED → NEW is the difference between a system that torments you and one that reminds you of things at the right moment, with updated data.
4. Write the drift out loud. Saying “AGNO is documented but not yet implemented” costs a moment of pride and buys all the credibility of the rest.

The system thinks, rebuts itself, and prepares. But the last word stays where it belongs: with the operator.

args:
- --smartctl.device=/dev/sda
- --smartctl.device=/dev/sdb
- --no-smartctl.scan        # never touch the rbd/nbd devices
- --smartctl.interval=300s