Detecting PII before the model sees it

No detector I have tested catches everything a masking layer needs. Two registered experiments measured the same frozen data with two different systems, and each missed a different category by a wide margin. The numbers, not an impression, say which one and by how much.

Key takeaways

• A small open PII model (GLiNER2-PII) misses most surnames, recall 0.291, while reaching near-perfect email and phone detection, 1.000 and 0.995, on the same frozen sample.
• An ensemble of standard open components, pattern recognisers plus a multilingual NER model, fixes surnames, recall 0.970, but its phone recall drops to 0.400 on the same data.
• Neither configuration dominates: the choice is which failure mode is cheaper for a given pipeline, not which system is better.
• Detection before the prompt is not the whole surface. Recent work on chain-of-thought leakage finds reasoning traces expose personal data too, a channel no pre-prompt detector touches.
• Full tables for both experiments sit in a public repository, not only in this summary.

What did the two registered experiments actually measure?

Both runs used the same frozen sample, 4,000 public examples from ai4privacy/pii-masking-openpii-1m, split evenly between English and Bulgarian, with pre-registered kill-gates fixed before either system saw the data. The method behind that setup, why the thresholds are written down first, is the subject of why I pre-register a kill-gate for every experiment.

The first run scored GLiNER2-PII alone, a 0.3 billion parameter model, against nineteen entity types, and its registered name-recall gate fired: a name threshold set at 0.90 was missed by a wide margin. The second run scored an ensemble built from standard open components, Presidio-style orchestration, a multilingual NER model, and pattern recognisers for structured formats, on the identical sample, and its registered phone-precision gate fired in the opposite direction. Both runs used the same CPU-class hardware budget, and both experiments, with full result tables, are published in a public repository, so the entity-by-entity numbers below can be checked line by line rather than taken on trust.

Where does each detector actually fail?

Names and phone numbers trade places between the two systems. GLiNER2-PII catches given names at 0.850 recall but surnames at only 0.291, a gap wide enough that most surnames in free text pass through unmasked. The ensemble closes that gap almost entirely, surname recall 0.970, because a dedicated NER model trained for entity extraction handles multi-token names better than a compact multi-purpose one.

The same ensemble then loses on phones. GLiNER2-PII detects phone numbers at 0.995 recall; the ensemble drops to 0.400, and the deeper problem is precision, not recall: registered phone predictions land on credit card, ID and social security numbers almost as often as on real phones. Adding a dedicated phone recogniser did not add coverage, it added confusion between numeric formats.

One category defeats both. Honorific titles score 0.131 recall for the small model and 0.000 for the ensemble, though the second number needs a caveat: no recogniser in the ensemble targets titles at all, so it is an unattempted category, not a measured failure, while the small model at least tries and mostly misses. Framed either way, a title currently reaches the model unmasked in both configurations.

Does masking before the prompt cover the whole surface?

Everything above happens before a prompt is sent, which is where a masking layer intervenes, as I described in why the masking layer matters more than the model. A 2026 study on chain-of-thought leakage argues that this is not the only surface. The authors measured PII exposure across eleven entity types and multiple model families, comparing direct prompting against reasoning-heavy chain-of-thought prompting, and found that reasoning consistently increases leakage for high-risk categories; the effect of a larger reasoning budget varies by model family, sometimes worsening exposure and sometimes reducing it. Four inference-time defences were tested, and none worked universally across models and settings.

The relevance to masking is structural, not measured by me: a pre-prompt detector controls what enters the model, but if any real value legitimately reaches it, because a task genuinely needs a customer's real name, the model's own reasoning trace becomes a second channel. If that trace is logged or displayed anywhere, it can carry personal data no pre-prompt filter ever inspected. I have not measured this on my own pipeline yet; the honest state of that question is "unmeasured", which is itself a finding worth naming rather than skipping past.

What is missing from this picture?

Every number above describes this dataset's synthetic formats, not any live traffic, the same caveat that applies to every entity table in the public repository. Checksum-validated ID recognisers could not be recall-tested at all here, because randomly generated synthetic digits fail checksum validation by design, so their near-zero measured recall reflects the data, not a real weakness in the recognisers. Neither run measured latency or cost trade-offs, only detection quality, so a deployment decision still needs its own budget conversation on top of these numbers. And the chain-of-thought leakage risk above is a literature pointer, not a registered experiment; it names the next candidate for the same discipline, a frozen plan and a threshold before any code runs, rather than a conclusion I can currently defend with my own numbers.

FAQ

Which detector should I actually deploy, the small model or the ensemble?

It depends on which false negative costs more in your pipeline. Free text heavy with personal names argues for the ensemble; phone-heavy structured fields argue for the small model or a dedicated regex layer. Many pipelines end up routing by field type rather than picking one system for everything.

Does adding more detection components always improve results?

No. Phone precision fell from near-certain to worse than a coin flip once the ensemble replaced the single model, because more recognisers created more chances to mislabel a card or ID number as a phone. Additional layers change what fails, not whether something does.

Is chain-of-thought leakage a risk if users never see the reasoning trace?

The cited study measures leakage inside the trace itself, not only in the final answer shown to a user. A hidden trace that still reaches a log file or an internal tool carries the same exposure; hidden from the user is not the same as not exposed at all.

Where are the full entity-by-entity numbers?

In the public repository linked from why I pre-register a kill-gate for every experiment: frozen plans and complete result tables for both experiments, with redactions marked in place where a line held non-public detail.

References

• Urchade Zaratiana, Ash Lewis, George Hurn-Maloney. GLiNER2-PII: A Multilingual Model for Personally Identifiable Information Extraction, 2026. arxiv.org/abs/2605.09973
• Pritesh Jha. Fine-Tuning Over Architectural Complexity: Broad-Coverage PII Detection on PIIBench with DeBERTa, 2026. arxiv.org/abs/2605.25816
• Patrick Ahrend, Tobias Eder, Xiyang Yang, Zhiyi Pan, Georg Groh. Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs, 2026. arxiv.org/abs/2603.05618