2194 claims, all stuck at 0.30: teaching a graph what matters

I opened the KB panel one morning and the graph was a flat prairie. 2194 claims ingested from four YouTube channels and two RSS feeds, all with the exact same number next to them: confidence 0.30, status staging. Not a single node promoted to verified. Ever. The system aggregated content like a diligent collector, but had not the faintest idea what was news and what was noise. This diary is about that stuck number, and what it takes to make it move.

The idea

The goal is trivial to state and brutal to build: replace the manual work of following dozens of creators. An agent that monitors sources (YouTube, Discord, RSS/Google Alerts), extracts atomic claims, dedups them inside a live vector knowledge base, and every morning pushes to WhatsApp the 3 pieces of news that really matter — with a link to the original source. Not a summarizer: a graph that grows, where the same story from three sources becomes one consolidated entry, not three duplicates.

Architecture: four nodes in a row

The backbone is a LangGraph pipeline (StateGraph) that runs a typed state through five stages:

extract_claims → triage_claims → kb_operations → validate_claims → END

• Extract (gpt-4o): breaks a transcript into typed atomic claims — factual, opinion, prediction, metric. Discards greetings and filler.
• Triage: for each claim it embeds (text-embedding-3-large), searches the top-5 neighbors in Qdrant, and decides NOVEL / CONFIRM / CONTRADICT.
• KB operations: creates a new node, or enriches an existing node (confidence bump), or flags a contradiction. First it goes through dedup: exact SHA-256 hash + semantic similarity ≥ 0.95.
• Validate: external cross-reference via web search, only for novel or contradictory claims.
• Synthesis + Distribution: clustering by topic, an Italian LLM that writes the entries, then WhatsApp via Evolution API.

The orchestration conditions the jumps with real edge functions — if extraction produces no claims, it goes straight to END; it validates only if there’s at least one NOVEL or CONTRADICT. All asynchronous, with Celery/Redis upstream and Postgres as the source of truth for dedup.

The backbone: why nothing ever rose above 0.30

The bug wasn’t a crash. It was worse: all green, all working, useless result. I pulled out the calculator and reconstructed the chain.

A node is born at KB_INITIAL_CONFIDENCE = 0.3. It rises only when it’s confirmed, and the boost shrinks with each confirmation:

def calculate_confirmation_boost(self, current_confidence, source_count):
    base_boost = 0.15               # confirmation_boost_initial
    boost = base_boost * (0.7 ** max(0, source_count - 1))
    return min(boost, 1.0 - current_confidence)

With the promotion threshold at 0.7, going from staging to verified takes ~5 independent confirmations: 0.30 → 0.45 → 0.55 → 0.63 → 0.68 → 0.72. Doable. Except the confirmations were zero. confirmations_count stayed glued to 0 on all 2194 nodes.

Why? Triage decided like this:

if top_score >= self._confirm_threshold:      # 0.85
    return CONFIRM
if top_score >= self._contradict_threshold:    # 0.70 → LLM alignment check
    ...
return NOVEL

TRIAGE_CONFIRM_THRESHOLD = 0.85 is too high for text-embedding-3-large. Two sentences saying the same thing with different words — “Nvidia posted 58.3 billion in revenue” vs “Nvidia’s quarterly revenue hits 58B” — typically land at 0.6–0.8 cosine, not 0.85+. Above 0.95, dedup kicks in and the claim gets thrown out as an exact duplicate. The result is a narrow, nearly empty confirmation band: almost everything fell below 0.85 → classified NOVEL → new node at 0.30 → flat prairie. The graph wasn’t growing in depth, it was only widening.

There was also a fossil: in the CONFIRM path the triage still logs confidence_bump_deferred_to_story_3_5, complete with the comment # Placeholder: Story 3.5 will implement.... In reality the actual bump is already done downstream by enrich_node — that log is a dead crumb from when the logic was broken. Textbook README/reality drift: it cost me half an hour looking for a bug where there wasn’t one.

Teaching what a confirmation is worth

Lowering the confirmation threshold (0.85 today, too high) would unblock the flow. But here’s the project’s counterintuitive lesson: not all confirmations are worth the same. If two “sources” confirming the same story are the same author, or cite the exact same URL, that’s not convergence — it’s an echo. Counting them as two independent votes inflates the confidence with hot air.

So the boost passes through a source-diversity analyzer, all heuristic (no LLM, for cost):

same_author_factor    = 0.1   # stesso autore → l'eco vale un decimo
shared_origin_factor   = 0.3   # citano lo stesso URL
derivative_factor      = 0.2   # testi troppo simili (coseno ≥ 0.85)
# indipendenti → 1.0

effective_bump = base_delta * diversity_score

Two different creators reaching the same conclusion from different paths are worth a full bump. The same guy repeating the story across three channels is worth a tenth. This is where the graph starts to understand something: it tells three megaphones pointed at the same press release apart from three heads genuinely converging.

Honest gotchas

• Three thresholds on the same metric. Dedup 0.95, confirm 0.85, cluster 0.62 — all cosine on the same embedding space. Every time I touch one I have to recheck they don’t fight. The “useful” band for a confirmation is literally 0.72–0.95, and figuring out how populated it is requires measuring the real distribution over the 2194 claims, not guessing.
• The importance score is written, not built. The refocus plan specifies a weighted formula — w1·confirmations + w2·source_authority + w3·recency + w4·cluster_size — complete with a migration ALTER TABLE kb_nodes ADD COLUMN importance_score. But a grep in src/ and migrations/ finds the word importance nowhere. The column doesn’t exist yet. For now the digest orders clusters by size (clusters.sort(key=lambda c: len(c["nodes"]), reverse=True)), a poor proxy: “more sources = more important”. Honestly: the real notion of relevance is still on paper.
• Synthesis wastes. The builder generates target * 3 = 45 LLM calls to produce 15 filtered entries — ~$0.20–0.30/day thrown away generating candidates then discarded. The fix (skip single-node clusters with claims < 60 characters) is planned but not yet active.
• WhatsApp distribution is configured but silent. The publisher code is there, Evolution API is wired up. I haven’t pulled the trigger yet: I’d rather fix signal/noise before spamming notifications.

On the Discord self-bot

One of the sources is a Discord connector that reads messages from private servers and forwards them to the platform’s internal webhook. It’s deliberately read-only — no sending, no reactions, no presence change — with randomized rate limiting (30–90 s) and reconnection backoff. Let me say it plainly: automating a Discord user account is a gray area with respect to the ToS, and I treat it as such. I only use it on servers I’m already part of, the process is stateless and conservative so as not to resemble a scraper. Token and webhook URL live only in .env, never in the repo. It’s not a feature to brag about: it’s a compromise to be aware of.

How it’s going

Solid POC, not yet in serious production. The pipeline runs end-to-end: ingestion (4 YouTube routed via proxy, 2 RSS), extraction, KB on Qdrant, triage, clustering, Italian synthesis. The “graph that learns” part is half unblocked: the echo/convergence analysis is in code; lowering the confirmation threshold, the importance score, and selective distribution are the next pieces. Definition of “done”: I get 3 pieces of news/day ordered by relevance, with links, and I can search the KB for “what have I read about X”. Still a few boxes missing.

What I learned

That the hard part of a knowledge graph isn’t ingesting, it’s judging. Extracting claims is a solved problem; deciding that this claim matters more than that one, and that this confirmation is real convergence and not an echo, is 90% of the value and 90% of the effort. A system that says “everything is new, everything is worth 0.30” is technically perfect and practically mute. Confidence isn’t a number that climbs on its own: it’s a model of what you consider evidence. Until I wrote it out for the graph — thresholds tuned on the real embedding distribution, not on optimism — it stayed a flat prairie, green and useless.