The AI's advice doesn't count: what counts is the guardrail that rejects it

p=none is the most widespread DMARC record in the world, and it’s also the most useless: it declares “check me” and then enforces nothing. Anyone can still send mail in your name. The problem is that the next step — quarantine, then reject — is the one that sends your legitimate mail to spam if you get it wrong, and it’s not a one-minute rollback: you see the damage days later, in the aggregate reports. I built a small platform to take that step on a personal mail cluster (two nodes, a handful of domains) in an observable, reversible way. And the most interesting part wasn’t the LLM advisor that recommends the policies. It was designing it so that its opinion wouldn’t matter.

The idea

A DMARC domain should always have a clear state: who legitimately sends in its name, how much of that mail is aligned (SPF/DKIM coming back with the right domain), and what the next safe move is. The platform collects DMARC aggregate reports into Postgres, classifies every source IP, computes a readiness score from 0 to 100, and produces a policy recommendation. No automatic DNS changes: the machine advises, the human applies by hand. This isn’t laziness, it’s the main safety guarantee.

How it actually works

The score is a deterministic weighted sum, not a black box:

score = round(
    40 * clamp(pct_aligned_30d)                 # allineamento è tutto
    + 15 * int(dkim_present_for_primary_sources)
    + 15 * int(spf_within_10_lookups)           # SPF entro 10 lookup DNS
    + 10 * days_ratio                           # osservazione / 14 giorni
    + 10 * (1 - clamp(unknown_sources_ratio))
    + 10 * (1 - clamp(spoofing_likely_volume_ratio))
)

Forty points out of a hundred come from alignment: if most of the mail doesn’t come back aligned, you’re not ready for anything, full stop. From that score comes a monotone, cautious progression — never the direct jump that kills deliverability:

p=none
p=quarantine; pct=10 → 25 → 50 → 100
p=reject; pct=10 → 50 → 100

With hard thresholds in code: reject only above a score of 90, quarantine only above 60, and in any case at least 14 days of observation before asking for pct=100. The source classification is just as brutal and readable: over 500 messages from an unaligned IP with an unknown ASN → spoofing_likely; local cluster IPs → trusted; fewer than 50 messages → unknown_low_volume. Rules, not vibes.

The catch: the advisor I made irrelevant

Then I added the LLM. The temptation was to feed it the domain dossier and do what it says. I did the opposite. The advisor has a provider chain — a Claude proxy, then OpenAI, then a fallback to deterministic rules — but the key point is that every path, including the model’s success, goes through the same guardrail function:

def validate_recommendation(payload, readiness_score, has_blocking_unknown=False):
    if payload.get("recommended_policy") not in ALLOWED_POLICIES:
        raise ValueError("invalid recommended_policy")
    if policy == "reject" and readiness_score < 90:
        raise ValueError("reject blocked by readiness guardrail")
    if policy == "quarantine" and readiness_score < 60:
        raise ValueError("quarantine blocked by readiness guardrail (<60)")
    if has_blocking_unknown and policy in {"quarantine", "reject"}:
        raise ValueError("enforcement blocked by unknown sources")
    ...

If the model recommends reject on a domain with a score of 74, the guardrail rejects it with a ValueError, and that provider is treated as failed: it moves on to the next one in the chain. The system prompt tells the model the same rules (“don’t recommend reject below 90”), but I don’t trust the prompt: I trust the raise. The model is a source of hypotheses; the authority is the deterministic code that validates them.

The counter-intuitive discovery came with the fallback. The rules fallback is designed to “never fail” — it must always return something safe to the dashboard. But the fallback derives its recommendation from the score… and the score can drop below the threshold mid-window, while the dossier it’s evaluating contained a higher one. Result: the deterministic rule can recommend an enforcement that the guardrail then rejects. Two pieces of my own logic disagreeing. The solution isn’t to force them to match, it’s to degrade explicitly:

try:
    validated = validate_recommendation(recommendation, readiness_score, has_blocking_unknown)
except ValueError as exc:
    validated = {
        "recommended_policy": "none",          # in caso di dubbio, non fare niente
        "recommended_pct": 100,
        "rationale": f"safe fallback: {exc}",
        "risk_level": "high",
        "confidence": 0.4,
        "generated_by": "rules:safe-fallback",
    }

When even the fallback gets rejected, the answer isn’t an exception that makes the advisor unavailable: it’s “go back to p=none”, which is always safe. Better a dashboard that honestly says “I don’t advise moving” than one that breaks. On top of that: an SHA256 cache on the canonical dossier (TTL 6h) so as not to burn quota, and a per-provider circuit breaker (3 failures → parked for 300s) to avoid cascades of timeouts.

The rest is dry-run everywhere

The 0.4.0 rollout came in waves (A→G): DNS scan, Rspamd snapshot, XSD validator for the reports, an outbound send gate, DKIM rotation tooling, Prometheus metrics, CI. The golden rule written at the top of the plan: no data deleted, no email lost. Every worker is read-only outside the DB and idempotent — scan-dns inserts a row only when the DNS value has changed compared to the last snapshot, and logs the drift; it never touches DNS. The retention job runs in dry-run by default and only counts until I review the numbers by hand. And apply_dns_change in production always does only this:

def apply_dns_change(*, dns_auto_apply, domain, new_value):
    if not dns_auto_apply:
        raise DnsApplyBlocked("DNS auto-apply disabled until Phase 4 approval workflow")

Auto-apply is wired OFF. The “change the DNS by itself” feature has never been enabled, and in the plan it won’t be without an explicit approval.

Honest gotchas

• Doc/reality drift, in both directions. The test count wobbles: one doc says “157 passed”, the rollout runbook declares “168”, and today the repo has 225 test_ functions across 28 files. The numbers in changelogs age faster than the code. And the README undersells itself: it says “local MVP scaffold, workers in dry-run”, but the internal reports already show domains at p=quarantine; pct=100 in monitored production, with Gmail confirming dkim=pass / spf=pass / dmarc=pass. Reality is further ahead than its storefront.
• The quarantine guardrail came later. In the first version only reject had a threshold; a provider could prescribe quarantine on a domain with practically zero data. I added the threshold at 60 when I realized that “less than reject” doesn’t mean “safe”.
• The circuit breaker is in-memory. On pod restart it starts clean. For a low-frequency advisor that’s perfectly fine; if this became a high-volume service, that state would need to be shared.

How it’s going

Shipped, and in use. A small group of domains is at p=quarantine; pct=100 with positive external evidence; others are in “watch”, waiting for more volume before being promoted. The LLM advisor is queued behind the other gates: until then it’s the rules chain that responds, which is exactly the point — it works without a model. No DNS change has been applied automatically. Not one.

What I learned

When you put an LLM in an operational path, the right question isn’t “how good is the model”, it’s “what happens when it’s wrong, and who stops it”. The best answer I found is to make every source — model, rule, cache — converge on the same deterministic guardrail, and to design failure as a first-class state: if everything collapses, the recommendation is “don’t move”. In the mail domain this translates into an almost boring rule: safety isn’t in the advice, it’s in the raise that can reject it. And the fact that even my deterministic logic can contradict the guardrail reminded me that “deterministic” doesn’t mean “always in agreement with itself over time”.