LoRA on a 16 GB card: what fits and what does not

A 7B model is the honest ceiling for LoRA fine-tuning on a 16 GB consumer card, not because anything bigger throws an error, but because a 14B model silently spills into system memory and runs 50 times slower without ever reporting a failure.

Key takeaways

• 7B is the real ceiling on an RTX 5070 Ti at LoRA rank 16, sequence length 512, batch size 1: 11,897.9 MiB peak, well inside the card's 16,303 MiB.
• 14B completed without a raised exception in both tested conditions, but peak VRAM read 22,189.8 MiB (136% of the card) and 17,470.9 MiB (107%) - both over the physical limit, through Windows' silent system-memory fallback.
• The cost of that fallback is severe: 21.7 seconds per training step at 14B against 0.4 seconds at 7B, roughly 51 times slower, with no error message anywhere.
• Gradient checkpointing is close to free at 7B and below - same or less time, meaningfully less memory - but does not rescue 14B: it leaves the model still over the limit and makes the step 59% slower, not faster.
• The registered hypothesis that checkpointing moves the ceiling up a tier is rejected; the wording of the first kill-gate also needed correcting, because "did not throw an out-of-memory error" turned out to be the wrong proxy for "fits".

Why doesn't "no error" mean "it fits"?

I pre-registered two kill-gates on an RTX 5070 Ti, 16,303 MiB of physical VRAM: whether a 14B Qwen2.5-Instruct model, loaded in 4-bit with a LoRA adapter, completes one real training step without a raised CUDA out-of-memory exception, and whether gradient checkpointing moves that ceiling up by at least one tested size, following the same discipline described in why I pre-register a kill-gate for every experiment. Neither 14B run threw an exception. By the literal wording of the first gate, it fired - 14B "fits". But torch.cuda.max_memory_allocated() reported 22,189.8 MiB without checkpointing, 5,887 MiB above the card's physical limit, and the step took 21.7 seconds instead of the sub-second range every smaller model showed. Windows ships a driver-level system-memory fallback policy that lets CUDA allocations succeed past a card's physical limit by quietly paging into ordinary RAM instead of raising an error, and nothing in the console said so. The gate measured the wrong thing: "did not throw an exception" is not the same claim as "fits in 16 GB", and only comparing the allocator figure against the card's actual VRAM, plus noticing the 50x time cliff, exposed the gap.

What did the clean numbers actually look like?

Three of the four sizes behaved exactly as expected. Without checkpointing, 1.5B peaked at 4,000.7 MiB, 3B at 6,749.0 MiB, 7B at 11,897.9 MiB - all comfortably under 16,303 MiB, all completing a forward-backward-optimizer step in under half a second. 7B used 73% of the card's VRAM at rank 16, sequence length 512, batch size 1, with real room to spare. Full numbers for all eight configurations sit in the frozen plan and results in the public experiment repository. Enabling gradient checkpointing lowered every one of those three figures - 2,855.5 MiB, 4,490.8 MiB and 9,709.9 MiB respectively - at essentially no time cost: 7B ran in the same 0.425 seconds either way, 3B cost 25% more time for a third less memory. The 1.5B timing is excluded from this comparison because it ran first in the script and likely carries a one-time CUDA initialisation cost. The honest ceiling for this exact setup, Qwen2.5-Instruct, 4-bit base, LoRA rank 16, one sequence of 512 tokens, batch size 1, is 7B. Push any of those variables up and the real number moves down, not up.

Does gradient checkpointing rescue the 14B case?

No. With checkpointing enabled, 14B still reported 17,470.9 MiB, 1,168 MiB over the card's physical limit - smaller than the 5,887 MiB overflow without checkpointing, but still spilling into system memory, not fitting inside it. The step time went the wrong direction: 34.5 seconds, 59% slower than the 21.7 seconds without checkpointing. Checkpointing buys memory by deleting intermediate activations during the forward pass and recomputing them during backward, and that recompute cost is normally hidden by the time saved from avoiding memory pressure, which is exactly why it helped 7B and below for free. Once the card is already paging to system RAM, the saved memory only shrinks an already-severe penalty instead of removing it, and the extra computation adds straight on top. The registered hypothesis, that checkpointing moves the ceiling up by at least one tested tier, is rejected. On this card, for this model family, checkpointing is a real, close-to-free lever below the ceiling, not a way past it.

FAQ

Does this mean a 16 GB card can never run a 14B model?

Not for training, no, not with this exact setup. A smaller rank, a shorter sequence, a smaller LoRA target set, or a more aggressive quantisation scheme could plausibly bring a 14B model under the limit; none of those were tested here. For inference rather than training, the memory math is different again and was not measured in this experiment.

Why trust torch's reported VRAM figure at all, if it can be wrong?

It is not wrong about what it allocates - torch.cuda.max_memory_allocated() reports real tensor memory the CUDA allocator granted, honestly. What it does not tell you is whether that memory was physically on the card or transparently paged to system RAM by the driver. Comparing the figure against the card's known physical limit, rather than trusting the absence of an exception, is what catches the difference.

Why only test rank 16 and batch size 1?

Those are close to the smallest realistic defaults most people reach for first. Testing a single point cleanly, rather than sweeping many configurations in one pass, keeps the registered claim specific and checkable. A larger rank or batch size sweep is a legitimate follow-up, not something this experiment's kill-gates covered.

Where do the numbers come from?

Every figure in this piece is read directly from results.json in the registered experiment, not estimated or rounded from memory. The frozen plan, the full experiment code, and the raw per-run output all sit in the public repository linked above, so the eight numbers in the table can be checked against the source instead of taken on trust.

References

• Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, 2021. arxiv.org/abs/2106.09685