Relation F1 at 0.38: when the lesson wasn't the model, but the data

This is the project I can tell the least about in detail, and that’s fine: the domain stays out. But I can explain the technical lesson in full, and it holds for anyone working with LLMs. In one line: before you change the model, look at the data.

The task

Long, dirty text, in Italian → a knowledge graph in JSON. Each document is split into segments; the target of each is an object with this shape:

• entities: list of {name, type} — typed nodes with a canonical name
• relations: list of {subject, predicate, object, evidence_text} — typed directed edges, plus the textual quote that proves them

With a hard constraint imposed in the schema: subject and object must be exactly a name present in entities — the graph must be internally coherent. The vocabulary is closed: a fixed enum of entity types and predicates.

The model and the fine-tuning

I start from a dense base model of ~27B, and do LoRA/SFT (PEFT + TRL). No training from scratch: I update 0.58% of the parameters.

mods7 = ["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"]
target = r".*language_model.*\.(" + "|".join(mods7) + r")$"  # salta la vision tower
LoraConfig(r=32, lora_alpha=64, lora_dropout=0.1,
           target_modules=target, bias="none", task_type="CAUSAL_LM")
# bf16, grad-checkpointing, lr=2e-4 cosine, micro_bs=1 x grad_accum=8,
# cutoff 16384 token, ~4 epoche, best su eval_loss + early stopping

The serving is vLLM with multi-LoRA: the base stays resident once, the adapters attach as named modules — multiple fine-tunings share the same base. It runs on on-prem HPC (2 GPUs, H200 class, Slurm), bound only to loopback, tracked in MLflow — where only aggregates end up, never raw text or generations.

The turning point (which is the whole project)

I trained v1, froze it and put it in production. It worked. And then I made the classic mistake: thinking that to improve I needed a bigger model.

Wrong. Looking at where it went wrong, the picture was clear:

• Entities: F1 ~0.68-0.71, decent.
• Relations: F1 ~0.38-0.41, weak.

And the weakness was concentrated in the rare tail: predicates with fewer than 500 examples scored ~0.00, with a clear correlation between rarity and F1. The root cause, once I found it, was embarrassing and instructive: the rich schema had been added after the corpus was already annotated. Documents that were textbook examples of a new relation extracted ~0 of it, because what was missing wasn’t the text — it was the annotation layer.

So the lever wasn’t “more documents” (tried: yields ~0). It was re-annotating the existing text under the new schema.

v2 is data engineering, not model engineering

Three moves, all quantified:

1. Schema widening — new predicates enumerated from the data, never invented.
2. Fine re-annotation — same text, sharper labels (remap the volume of a coarse catch-all onto the precise directional children).
3. Tail densification — previously empty predicates filled with delta re-annotation on ~1,800 candidate chunks.

And a discipline I really liked: learnable vs measurable as a split constraint. The test set is chosen by a deterministic greedy selector that must cover every priority predicate with ≥30 test instances (measurability) without dropping any predicate below 50 in training (learnability). Where the two objectives conflict, the predicate is declared “learnable but not measurable on this split” — kept and trained, excluded from the metric. Not thrown away.

Measuring honestly

• Micro P/R/F1 separate for entities and relations, with greedy one-to-one matching.
• Alias-aware scoring: gives credit if the form matches any known variant of the gold entity, not just the canonical name — separates real errors from mere naming variance (relation F1 0.38→0.41).
• Two audits beyond F1: I sample the false positives and check whether both endpoints and the quote actually appear in the chunk (≈47% of the “errors” were correct extractions missing from the gold — incompleteness of the annotation, not of the model); and I verify that the evidence_text of the true positives is verbatim in the text (~87.5%, an anti-hallucination check).

What I learned

The base was already large, and its entities were strong; scaling it up wouldn’t have manufactured relations that the gold didn’t contain. The audits said it plainly: a big slice of the “false positives” were correct extractions absent from the annotation. It’s the same pattern as the public record (TACRED→Re-TACRED, +16 F1 just by cleaning the gold, architecture unchanged).

Not very exciting to put on a slide. But it’s what moved the needle: in an extraction project, the product isn’t the model — it’s the curated dataset behind it. The model is replaceable. A well-built gold standard is not.

For confidentiality, the domain and contents stay out of the diary. I tell the method, not the subject matter.