What EWC buys you that flat L2 does not

A flat L2 penalty can match EWC's exact Task A retention number on Permuted MNIST: 97.63% against 97.63%. It does this by collapsing Task B accuracy to 18%, against EWC's 97%. The same headline number hides two very different costs.

Key takeaways

• A registered gate asked whether flat, uniform L2 regularisation matches EWC's Fisher-weighted penalty at comparable cost. The cheapest tested setting landed within 0.86 points of EWC's Task A accuracy, inside the registered 3-point bar - the gate fired.
• No tested setting met the registered "fair budget" of costing at most 1 point of Task B accuracy. The cheapest one still cost 2.70 points, over six times what EWC paid.
• At a stronger setting, flat L2 hits EWC's exact Task A number, 97.63%, while Task B accuracy collapses to 18.42% against EWC's 97.43%.
• Fisher weighting is not required to match EWC's retention number in this setup. It is required to match it without wrecking the model's ability to learn the new task.

What did the registered gate actually test?

An earlier experiment found that Elastic Weight Consolidation (EWC, Kirkpatrick et al., 2016) recovers 98.0% of the accuracy a small network loses to catastrophic forgetting on Permuted MNIST. EWC works by penalising drift away from Task-A-optimal parameters, weighted by a diagonal Fisher information estimate: parameters that mattered more to Task A get pulled back harder. That estimate costs something to compute, 1,000 samples and a forward-backward pass through each. The open question this experiment registered: does the Fisher weighting itself earn that cost, or would any quadratic pull-back toward the old parameters, uniform and uninformed, do just as well?

The registered kill-gate compared a flat L2 penalty, same shape as EWC's but with every parameter weighted equally, against EWC's actual result. To keep the comparison honest, the plan fixed a rule before running anything: search a range of L2 strengths, and report the one whose cost to Task B accuracy sits closest to, but under, 1 percentage point, matching EWC's own 0.41-point cost. If Task A accuracy at that point comes within 3 points of EWC's, the claim that Fisher weighting matters would be rejected.

Where did the gate fire, and what did that hide?

The registered gate fired, literally. Across six tested strengths (0.01 to 1000, log scale), the cheapest one, lambda=0.01, reached 96.77% Task A accuracy, 0.86 points under EWC's 97.63%, comfortably inside the 3-point bar. By the letter of the pre-registered rule, the hypothesis that Fisher weighting provides a measurable advantage is rejected.

But the fair-budget rule that gate was supposed to respect was never actually satisfied. That same lambda=0.01 setting cost 2.70 points of Task B accuracy (97.84% down to 95.14%), not the 1 point the plan called "fair." Every cheaper option would have cost less on Task B but also recovered less of Task A; every stronger option recovered more of Task A at a steeper Task B cost. No point in the sweep lived inside the budget the plan defined as comparable to EWC. fair_pick_in_budget: false sits in the raw metrics file precisely so this gets reported rather than smoothed over.

What does the lambda=10 result reveal?

One point in the sweep makes the trade-off impossible to miss. At lambda=10, flat L2 reaches 97.63% Task A accuracy, identical to EWC to two decimal places. Read as a single number, EWC and flat L2 look interchangeable. Task B accuracy at that same setting is 18.42%, against EWC's 97.43%. The network did not learn to protect what mattered on Task A while staying free to absorb Task B. It nearly stopped learning Task B at all, and freezing a network is a cheap way to protect old knowledge from any source.

This is the shape a naive comparison misses: pick one number, retention on the old task, and two very different mechanisms can look tied. Add the number nobody asked for and the picture reverses completely. It also explains why the sweep never found a point inside the fair budget: Task A accuracy rises smoothly toward EWC's figure as lambda climbs, but Task B accuracy falls off a cliff well before Task A catches up. There is no gear between "too weak to matter" and "strong enough to freeze the network," at least at this granularity of search.

So does Fisher weighting matter?

Both, depending on which question is being asked. If the question is "can a simpler method hit EWC's retention number," this experiment says yes, at least in this mild Permuted MNIST setting, with enough uniform pressure. If the question is "can a simpler method hit that number while leaving the network able to learn the next task," the answer across every tested setting is no. Fisher weighting is not a hard requirement for the headline number. It is the reason EWC can post that number while still being useful for anything after it.

The distinction matters beyond this one benchmark. Any report that states only "method X preserved Y% of old-task accuracy" is stating half a result. The other half, what the method spent to get there, decides whether it is a usable continual-learning method or a network that has quietly stopped learning. A retention number without its corresponding new-task cost is not a false claim, but it is an incomplete one, and this experiment is a case where the missing half changes the conclusion entirely.

FAQ

Does this mean EWC is unnecessary, and flat L2 works just as well?

Not for the actual goal of continual learning. Flat L2 can copy EWC's Task A number, but only by paying far more of the model's capacity to learn new tasks, an option that defeats the point of learning continually in the first place. The two methods are not interchangeable once both accuracy numbers are read together.

Why sweep several L2 strengths instead of tuning one directly to match EWC?

Tuning a single value to match a known target after seeing the result is exactly the hindsight pre-registration exists to prevent. The sweep and the fair-budget rule were both fixed before any run, so the reported pick is the honest output of that rule, not a value chosen because it looked good afterward.

What prompted this specific comparison?

A correspondent, following up on the earlier EWC experiment, asked directly whether a flat L2 baseline had been tested. It had not. Answering from intuition would have been easy and probably wrong in an interesting way, given how the lambda=10 result turned out, so the honest option was to register a plan and run it rather than guess.

What would make this comparison more convincing?

A finer sweep near the two extremes that bracket the trade-off, more task pairs than one, and ideally a network trained from scratch by the same author, since this Permuted MNIST setup is a mild benchmark where forgetting itself was measured, not catastrophic, in the original experiment.

References

• James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, Raia Hadsell. Overcoming catastrophic forgetting in neural networks, 2016. arxiv.org/abs/1612.00796