Why I pre-register a kill-gate for every experiment

A kill-gate is a threshold written down before an experiment runs: if the result crosses it, the idea dies. The writing down happens first, because after the numbers arrive, every threshold becomes negotiable. Last week two of my gates fired and one refused to answer, which taught me more than a success would have.

Key takeaways

• A kill-gate names, in advance, the number at which I abandon my own hypothesis; the plan file is frozen before the first run and never rewritten afterwards.
• Gates exist because results negotiate: search-time contamination alone can inflate agent benchmark scores by up to 4%, and LLM-judge scores now come with conformal prediction intervals because point values wobble.
• Of the three verdicts a gate can return, "cannot be pronounced" is the most valuable one: it indicts the instrument instead of the hypothesis.
• The axis you are tempted to leave out is the one to register: my phone-detection gate only told the truth because it measured precision as well as recall.

What is a kill-gate?

A kill-gate is one sentence in a plan file: a hypothesis, a metric, a number, and the action taken if the number is crossed. The file is dated and frozen before any code runs; corrections are appended as a dated journal, never edited in place. Everything else about the experiment may be humble, but this part is a contract.

The contract matters because evaluation without pre-set rules drifts toward the answer its author wants. The drift is documented at every scale. A 2026 study of deep research agents found search-time contamination, where the agent's own web searches touch benchmark material, across all six public benchmarks tested, inflating scores by up to 4% without anyone cheating deliberately. Evaluations that use a model as judge carry enough score variance that researchers now wrap them in conformal prediction intervals rather than trust a single number. If public leaderboards with reviewers need this much guarding, a solo practitioner grading his own idea needs more. The gate does not make me objective; it makes my bias arrive too early to help me.

What did the gates do in a real experiment?

Registration came first: before evaluating a small PII detection model (GLiNER2-PII, 0.3B parameters) on 4,000 public examples from the ai4privacy/pii-masking-openpii-1m dataset on Hugging Face, I froze a gate saying that name recall below 0.90 on English kills the claim "a small open model suffices out of the box". The result was 0.850 for given names and 0.291 for surnames. The gate fired, the claim is dead, and I am not allowed to describe that model as sufficient in any future article.

The surname failure dictated a second experiment: whether an ensemble of standard open components, pattern recognisers orchestrated with a multilingual NER model, recovers what the single model misses. On the same frozen sample, surname recall rose to 0.970, so the ensemble genuinely fixes the single model's blind spot. Its phone detection then returned recall of 0.400 against the small model's 0.995, and the registered precision axis came back at 0.36 against a 0.99 bar; both numbers describe this dataset's synthetic formats, not any live traffic. Two experiments, two dead claims, and a conclusion neither run would have volunteered on its own: the two shapes of detector are blind to different things. Numbers I registered thresholds for became findings; numbers I merely observed stayed observations.

Why is the gate that refused to answer the most useful?

One registered gate targeted a hypothesis I raised in why the masking layer matters more than the model: that Cyrillic text weakens detection. If English and Bulgarian name recall differ by under 5 points, it dies. Before pronouncing it, I ran a validity check on the data, and the check ended the question: 100% of the 3,410 name values in the Bulgarian subset turned out to be Latin-script. The dataset's language tag promises Bulgarian, and the surrounding text is Bulgarian, but every person in it has a Latin-alphabet name. The gate cannot be pronounced, because the instrument does not contain the thing the hypothesis is about.

Without a registered gate I would have computed the difference, found it under 2 points, and written that Cyrillic is not a problem: numeric, plausible and wrong. The registered wording forced the prior question of whether this data can answer me at all. An unpronounceable gate returns "unmeasured", which is an honest state that a spreadsheet will never volunteer on its own.

How do I write one?

Four rules cover what I currently know. First, one sentence per gate: hypothesis, metric, number, action; a gate that needs a paragraph is hiding a decision. Second, choose the metric by asymmetry of cost: in PII masking a false negative leaks personal data while a false positive slightly degrades an answer, so recall guards the gates for leakage. Third, register the axis you are tempted to omit. My phone gate began as recall-only; the operational concern is really false positives, an order reference flagged as a phone number, so a precision axis with a 0.99 bar went in before the run. That axis returned 0.36 and caught the actual failure. Recall alone would have shown a softer problem and let the dangerous one pass as a green light. Fourth, keep a dated journal: amendments before results are method, amendments after results are negotiation, and the file format should make the difference visible to any reader.

FAQ

Is a kill-gate the same as a statistical hypothesis test?

No, and it does not need to be. A kill-gate is a commitment device about timing: the threshold exists before the data does. A significance test can serve as the metric inside a gate when samples are small, but the protection comes from the pre-registration, not from the formula.

What happens when a result lands just under the gate?

The idea dies, and that is the entire point. A new gate may be registered for the next experiment, with a written reason; this one does not move. The sentence "0.88 is basically 0.90" is precisely the sentence kill-gates exist to kill, and it always arrives dressed as reasonableness.

Do experiments whose gates fire get published?

Yes, with the same weight as successes. A fired gate is a result: it names a claim nobody should repeat and the number that killed it. The frozen plans live in a public repository, so a reader can check what was promised before the numbers existed.

References

• Yongjie Wang, Xinyue Zhang, Kunhong Yao, Zhiwei Zeng, Kaisong Song, Jun Lin, Zhiqi Shen. Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation, 2026. arxiv.org/abs/2606.05241
• Huanxin Sheng, Xinyi Liu, Hangfeng He, Jieyu Zhao, Jian Kang. Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction, 2025. arxiv.org/abs/2509.18658