Abstract
Recent work has identified a counterintuitive phenomenon termed “Hyperfitting”, where fine-tuning Large Language Models (LLMs) to near-zero training loss on small datasets surprisingly enhances open-ended generation quality and mitigates repetition in greedy decoding. While effective, the underlying mechanism remains poorly understood, with the extremely low-entropy output distributions suggesting a potential equivalence to simple temperature scaling. In this work, we demonstrate that this phenomenon is fundamentally distinct from distribution sharpening; entropy-matched control experiments reveal that temperature scaling fails to replicate the diversity gains of hyperfitting. Furthermore, we falsify the hypothesis of static vocabulary reweighting, showing through ablation studies that hyperfitting relies on a dynamic, context-dependent rank reordering mechanism. Layer-wise analysis localizes this effect to a “Terminal Expansion” in the final transformer block, where a substantial geometric expansion of the feature space (ΔDim ≈ +80.8) facilitates the promotion of deep-tail tokens. Additionally, we introduce Late-Stage LoRA, a targeted fine-tuning strategy that updates only the final 5 layers, yielding robust generation with minimal parameter updates.
Hyperfitting does not merely sharpen the output distribution — it learns to dynamically reorder token ranks. We localize this behaviour to a Terminal Expansion in the final transformer block and exploit it with Late-Stage LoRA, which adapts only the last 5 layers and cuts trainable parameters by roughly 78–83%.
Key Contributions
The Entropy–Quality Paradox
Even when matched to the same low-entropy regime (T* ≈ 0.59), the original model stays trapped in repetitive loops (TTR 0.40), while the hyperfitted model reaches TTR 0.68. Identical confidence, very different text — so it is not temperature scaling.
Falsification of Static Bias
Injecting a learned static logit bias for “promoted” tokens only hurts — even tiny perturbations worsen repetition, and large ones cause mode collapse. The reordering is therefore dynamic and context-dependent, not a global vocabulary re-weighting.
Mechanistic Localization
Early layers preserve linguistic features; the final block executes a substantial geometric expansion (ΔDim ≈ +80.8, L2 22.0→81.6) that “unfolds” the hidden state and surfaces deep-tail tokens.
Late-Stage LoRA
Guided by this localization, adapting only the final 5 layers reproduces the rank-reordering dynamics, reducing trainable parameters by ~78–83% while matching (and on deeper models, beating) full-network fine-tuning.
It’s Not Temperature Scaling
Temperature scaling is rank-preserving: lowering T sharpens the distribution but never changes which token wins. We tune T so the original model exactly matches the hyperfitted entropy and ask whether diversity is recovered. It is not.
Figure 3. Entropy matching fails to recover diversity. With temperature tuned so the original model matches the hyperfitted entropy (T* ≈ 0.59), TTR barely moves (0.397 vs. 0.400 at T = 1.0), whereas the hyperfitted model reaches 0.684 — a 71% relative gain in lexical diversity at identical predictive confidence.
Where Do the Winning Tokens Come From?
We trace every token the hyperfitted model greedily selects back to its original rank. The reordering has a tripartite structure: a large linguistic anchor, local exploration, and a critical slice of deep-tail promotions that temperature scaling can never reach.
Figure 4. The provenance of hyperfitted winners. Of the tokens greedily selected by the hyperfitted model, 60.9% were already Top-1 (the linguistic anchor), 26.2% are promoted from ranks 2–10, and 12.9% originate from the deep tail (rank > 10) — some from rank > 200. Because temperature scaling preserves ranks, these deep-tail promotions are theoretically impossible under it.
The Loss–Rank–Quality Triad
Hyperfitting drives the training loss toward zero, and validation perplexity explodes — the classic signature of overfitting. Yet generation quality keeps improving. Likelihood and quality decouple, and the rank statistics explain why.
Figure 2. The Loss–Rank–Quality triad in hyperfitting. (left) Training loss falls while validation perplexity diverges. (middle) Despite this, the Type-Token Ratio (TTR) rises and plateaus near 0.7. (right) The selected Top-1 tokens increasingly originate from low original ranks (growing yellow/red areas), visualizing the rank-reordering that yields diverse generation despite the perplexity expansion.
Mechanistic Localization: The Terminal Expansion
Where does this transformation happen? A layer-wise representational analysis shows a sharply bipartite topology: a stable, linguistic early region and an explosive geometric expansion in the very last block.
Figure 5. Late-stage geometric expansion (TinyLlama-1.1B). (A) Early layers preserve linguistic features (high cosine similarity); the structural shift concentrates at the end as a Terminal Explosion in L2 distance (22.0→81.6). (B) Effective-dimensionality change: intermediate layers compress (negative Δ) before a large expansion (ΔDim ≈ +80.8) in the final block that provides the representational capacity to separate long-tail tokens.
Quantitative Dissection
At matched entropy (the difference is not statistically significant, p = 0.73), the hyperfitted model achieves far higher diversity and far lower repetition, and its global rank ordering departs strongly from the original (Spearman ρ ≈ 0.43). Temperature scaling, in contrast, keeps ρ ≈ 1.0.
| Metric | Original (T = 1.0) |
Original (T = 0.59) |
Hyperfitted |
|---|---|---|---|
| Type-Token Ratio (TTR) ↑ | 0.400 | 0.397 | 0.684 |
| Bigram Repetition ↓ | 0.592 | 0.604 | 0.140 |
| Trigram Repetition ↓ | 0.536 | 0.548 | 0.069 |
| Top-1 Agreement (vs. T=1.0) ↓ | 1.000 | 0.997 | 0.570 |
| Spearman Rank Corr. ρ (vs. T=1.0) ↓ | 1.000 | 0.998 | 0.430 |
| Prediction Entropy (nats) | 2.083 | 0.875 | 0.862 |
Mean over n = 30 sequences of 256 tokens. All diversity/ranking differences between the hyperfitted model and the entropy-matched baseline are significant (p < 0.001); the entropy difference is not (p = 0.73). Source: Table 1 of the paper.
Mechanism-Inspired Intervention: Late-Stage LoRA
If the generative capability is localized to the terminal block, full-network fine-tuning should be unnecessary. Late-Stage LoRA restricts adaptation to the final transformer blocks — and it works.
Figure 6. Optimization dynamics: Full vs. Late-Stage LoRA (TinyLlama-1.1B). Freezing the first 18 layers and adapting only the final 5, Late-Stage LoRA follows a closely matched trajectory and converges to an indistinguishable terminal loss (L ≈ 0.066). The optimization capacity for hyperfitting is concentrated in the terminal layers.
TinyLlama-1.1B · adapt layers 18–22
| Variant | TTR ↑ | BiRep ↓ | Top-1 Agree ↓ |
|---|---|---|---|
| Original | 0.400 | 0.592 | 1.000 |
| Hyperfitting (full) | 0.684 | 0.140 | 0.570 |
| Full LoRA | 0.508 | 0.331 | 0.523 |
| Late-Stage LoRA | 0.469 | 0.345 | 0.517 |
Qwen2.5-1.5B · adapt layers 24–28
| Variant | TTR ↑ | BiRep ↓ | Top-1 Agree ↓ |
|---|---|---|---|
| Original | 0.315 | 0.662 | 1.000 |
| Hyperfitting (full) | 0.434 | 0.652 | 0.545 |
| Full LoRA | 0.575 | 0.248 | 0.469 |
| Late-Stage LoRA | 0.591 | 0.213 | 0.459 |
Late-Stage LoRA reduces trainable parameters by ~78.3% (TinyLlama) and ~82.7% (Qwen2.5-1.5B). On the deeper Qwen architecture it outperforms full LoRA, and an LLM-as-judge evaluation gives it a 57.3% win rate, driven by a +16.1 pt coherence advantage. On a single RTX 4090 it reaches the useful checkpoint range about 2× faster than full supervised fine-tuning.
Late-Stage LoRA Generalizes Across Architectures
Across base and instruction-tuned models, Late-Stage LoRA tracks Full LoRA and converges to a comparable low-loss regime. (Drag or use the arrows to browse.)
Qwen2.5-1.5B — Full vs. Late-Stage LoRA training loss.
Gemma-2-2B — Full vs. Late-Stage LoRA training loss.
LLaMA-3.2-3B — Full vs. Late-Stage LoRA training loss.
LLaMA-3.1-8B-Instruct — Full vs. Late-Stage LoRA training loss.
Qwen2.5-7B-Instruct — Full vs. Late-Stage LoRA training loss.
The Terminal Expansion Is Universal
The same late-stage effective-dimensional expansion appears across architectures: early layers stay stable, and the final block expands to surface diverse tokens.
Qwen2.5-1.5B — layer-wise localization & terminal expansion.
Gemma-2-2B — layer-wise localization & terminal expansion.
LLaMA-3.2-3B — layer-wise localization & terminal expansion.
BibTeX
@article{li2026beyond,
title={Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion},
author={Li, Meimingwei and Ding, Yuanhao and Arias, Esteban Garces and Heumann, Christian},
journal={arXiv preprint arXiv:2605.22579},
year={2026}
}