Kochab: building trust into code — the star that never deletes

Kochab is the star at the tip of the Little Dipper — the one that points north, still while everything rotates around it. The name is a promise: a fixed point watching over your servers. But a tool that runs with privileges across all your machines is only useful if you trust it. This diary is about how trust gets written into code.

The problem

Anyone running a handful of servers lives in an awkward zone: too small for enterprise tools, too big to manage by hand. Security is the first thing to slip — permissive SSH keys, a firewall left open “temporarily” two years ago, DKIM/SPF/DMARC never sorted out.

Architecture: three pieces, one clean separation

• Agent (Go, a ~9.7 MB static binary, a 4.6 MB .deb): runs on every host under systemd. Outbound only — no local DB, no UI, no listening port. The go.mod has a single direct dependency. Minimal attack surface.
• Platform (Go): API + out-of-band workers. It owns the inventory, the audit engine, task signing. PostgreSQL is the single source of truth.
• App: a React 19 / Vite 8 / Tailwind 4 PWA that draws the fleet as a constellation.

Why the agent calls home, and not the other way around

I ruled out SSH-exec and listening agents. The agent does HTTP long-polling, inspired by Cloudflare tunnels — “the daemon calls home, never the other way around”:

• GET /v1/tasks: 204 = nothing to do, reconnect; 200 = a task; 410 Gone = node decommissioned, the agent stops polling forever.
• Client timeout 90s, deliberately longer than the server-side 60s wait.
• Minimum TLS 1.3, authentication required via HMAC-SHA256 over the body.
• Loop poll → verify signature → execute → report, exponential backoff 2s→60s.

Tasks are signed with Ed25519 by the platform and verified by the agent before executing. The canonical message is identical on both sides:

// "taskID|taskType|hex(sha256(payload))|RFC3339(timestamp)"
msg := fmt.Sprintf("%s|%s|%x|%s", taskID, taskType,
    sha256.Sum256(payload), timestamp.UTC().Format(time.RFC3339))
sig := ed25519.Sign(privKey, msg)

A subtlety that cost me a bug: the signed timestamp has to match the saved created_at down to the second, or verification fails.

The audit engine

Ten checks in the initial seed, each mapped to CIS and NIST: key-only SSH (CIS-5.3.4 / AC-7), default-deny firewall, TLS ≥1.2, shadow permissions, fail2ban, unexposed services, DKIM/SPF/DMARC, backup present, Docker without host-network. Each rule is a Go file that self-registers: a new rule = a new file, no hardwired switch.

type CheckFunc func(ctx context.Context) (passed bool, context map[string]any, err error)

Preconditions (a missing binary) are modeled as pass-with-context, not errors, so the report stays sensible across heterogeneous hosts. The SSH check is the most refined: it concatenates sshd_config + the .d/*.conf files in their effective order, probes every configured port to figure out whether it’s publicly exposed, and feeds those signals into the severity calculation.

Diary-level honesty: there’s still no LLM in the shipped code. It’s all deterministic. The AI is designed to sit outside the execution loop — it will generate new audit rules that are then “crystallized” into that same deterministic form and approved by a human. The reasoning: “90% accuracy per step = 59% success over 5 steps”. So the logic lives in the code, the LLM at most writes rules, and never touches a live host.

”Never deletes” — guaranteed four ways

This is the part I’m proudest of, because it’s trust made mechanical:

1. Nodes are only soft-deleted. DELETE /v1/nodes/{id} validates the reason against an allowlist and does an UPDATE removed_at = now(), never a SQL DELETE.
2. A CI guard enforces the filter. A script fails the build if a query over nodes lacks removed_at IS NULL. It exists because four real “zombie node” bugs (a removed node coming back to life) had slipped through.
3. The audit trail is physically immutable, in Postgres, not in code:

CREATE FUNCTION prevent_audit_trail_mutation() RETURNS trigger AS $$
BEGIN
  RAISE EXCEPTION 'audit_trail is append-only: % operations are forbidden', TG_OP;
END; $$ LANGUAGE plpgsql;
CREATE TRIGGER audit_trail_immutable_delete BEFORE DELETE ON audit_trail
  FOR EACH ROW EXECUTE FUNCTION prevent_audit_trail_mutation();

4. Findings get resolved, not deleted. And the fixes are never executed by the system: they’re rendered as copy-paste shell for a human.

There’s also an asymmetry I like: the “disable SSH passwords” fix is fail-closed — it stays hidden unless a preflight proves a working key already exists (so it can’t lock you out). Every other rule is fail-open, so a flaky probe never hides a legitimate fix.

The constellation

The fleet is a star field. Two pure, tested functions: one places the nodes in a deterministic radial layout, the other maps (state, severity, last heartbeat) → visual state, where reachability beats severity (an unreachable node is offline, period). Accessibility out of the box: 44×44px targets, aria-label in Italian, activation with Enter/Space, animation disabled with prefers-reduced-motion. And a transparency view that shows the operator exactly what the agent transmitted — because we only collect metadata, never log contents.

Honest gotchas

• README/reality drift: the docs say “Watermill for events”, the code does Postgres LISTEN/NOTIFY + TanStack polling. Watermill is deferred post-MVP. Better to say so.
• Rules in dual representation (Go + SQL seed) create a mismatch risk, mitigated only by a grep in CI. A maintenance tax accepted to keep the CIS metadata in the DB.

How it’s going

I’m sprinting toward the MVP. The three pieces compile, the images are at release-candidate, and I’m testing it on my real machines — the best way to know whether a guardian star works is to have it watch over something that, if it goes down, ruins my day.

Code: github.com/vcnngr/kochab-agent · install via get.kochab.ai. Agent Apache-2.0, platform BSL.