Weaken to Strengthen

Overview · 1 min

The paper in one minute.

Animated from the talk slides. No sound.

TL;DRSelf-distillation usually builds a stronger self-teacher by giving the model privileged information, such as reference solutions or environment feedback. WISD reverses the pair: the original model is the stronger member, and a copy weakened by low-bit quantization or magnitude pruning is the weaker one. Their token-level log-probability contrast supervises the model on its own rollouts, from unlabeled prompts alone. Across five reasoning models and five math and code benchmarks, it improves average accuracy by 4.0 points with quantization and 5.5 with pruning.

+5.5ptsAverage accuracypruning · 25 model–benchmark settings

+4.0ptsAverage accuracyquantization · 25 model–benchmark settings

70.5%HMMT 2025Qwen3.5-4B · from 57.7%, pruning

56.7%LiveCodeBench v6Qwen3.5-4B · from 39.1%, pruning

23/ 25Settings improvepruning; 21 / 25 with quantization

0Labels neededno solutions, feedback or external teacher

Bar charts of math and code accuracy for five reasoning models. On every model, at least one WISD variant is above the base model; the largest gains are on Qwen3.5-4B, from 66.3% to 75.3% on math and from 40.4% to 60.3% on code.

Figure 1 of the paper. Math averages AIME 2024, AIME 2025 and HMMT 2025; code averages LiveCodeBench v5 and v6.

Abstract

Self-distillation offers a way to improve reasoning models when a suitable external teacher is unavailable. However, common approaches rely on privileged information to construct a stronger self-teacher, limiting their use when only unlabeled prompts are available. In this paper, we introduce Weakening-Induced On-Policy Self-Distillation (WISD), which instead constructs a weaker counterpart of the original model and uses the contrast between the two as a training signal without requiring privileged information. We explore two ways to construct the weaker model: low-bit quantization and magnitude pruning. We evaluate WISD across diverse architectures and scales on mathematical and coding reasoning tasks. Across these benchmarks, quantization and pruning improve average accuracy over the original models by 4.0 and 5.5 percentage points, respectively, achieving state-of-the-art performance even without privileged information. Notably, pruning raises Qwen3.5-4B's accuracy from 57.7% to 70.5% on HMMT 2025 and from 39.1% to 56.7% on LiveCodeBench v6. Quantization raises Qwen3.6-35B-A3B's accuracy from 68.5% to 74.6% and from 61.8% to 69.5% on the same benchmarks, respectively. These results establish WISD as a novel and effective self-distillation framework for large reasoning models.

01 · Method

Don't strengthen the teacher. Weaken a copy.

A pair of models carries a training signal in the difference between them, even when neither is stronger than the student. Prior self-distillation builds the stronger member with extra context. WISD keeps the model itself as the stronger member and builds the weaker one from its own weights.

Prompts

Unlabeled promptsno solutions, no feedback

Roll out

original π₀

Own rolloutsfour per prompt, sampled once and cached

Weaken

3-bit round-to-nearestprune |w| ≤ max / 6

Weaker copy π_wtraining-free, no calibration data

Contrast

Token signal r_woriginal vs weakened log-probability, scored once

DistillStudent onlyinitialized from π₀, trained offline, used alone

1

Reverse the pair

Viewed as a pair, prior self-distillation sets the model itself as the weaker member π⁻ and the model with privileged context c as the stronger one π⁺. WISD sets π⁺ = π₀, the original model, and π⁻ = π_w, its weakened copy.

\[ r_w(a\mid s)=\log\pi_0(a\mid s)-\log\pi_w(a\mid s) \]

2

Weaken without data

Per group of 128 input columns with maximum M_G: round each weight to b = 3 bits, or drop weights at or below τM_G with τ = 1/6 and restore each row's norm. Embeddings, heads, norms and routers are untouched; only the frozen scorer is weakened.

\[ \widehat W_{ij}=\Delta_G\,\mathrm{clip}\!\Big(\mathrm{round}\big(\tfrac{W_{ij}}{\Delta_G}\big),-q_b,q_b\Big),\quad \Delta_G=\tfrac{M_G}{q_b} \]

3

Distill the contrast

Following the offline DOPD procedure of Lightning Weave, the contrast tilts the original model into an explicit target, and the student minimizes its KL to it. Tokens the original prefers over its weakened copy gain probability.

\[ q_w(a\mid s)\propto\pi_0(a\mid s)\exp\!\big(r_w(a\mid s)/\alpha\big) \]\[ \mathcal L(\theta)=\alpha\,\mathbb E_{s\sim d_{\pi_0}}\big[D_{\mathrm{KL}}(\pi_\theta\,\|\,q_w)\big] \]

4

A small, label-free budget

Per domain: 3,200 unlabeled prompts, four responses each, 12,800 cached trajectories and 200 optimizer updates. The frozen pair scores the cache once, so no scoring model is live during training.

Method overview. (a) Prior self-distillation strengthens the model into a self-teacher with privileged information c. (b) WISD keeps the original model as the stronger member, builds a weaker copy by quantization or pruning, and distills their token-level contrast on the model's own rollouts into the student.

Figure 2 of the paper. (a) Prior self-distillation strengthens the model with privileged information. (b) WISD weakens a copy instead; the frozen pair scores the same rollouts, and its token-level contrast supervises the student.

02 · Main results

Every model gets better, from its own weights.

Five reasoning models from 4B to 35B, dense and mixture-of-experts. Pruning improves every model; quantization improves four of five and is level on Qwen3.6-27B.

OriginalWISD (quantization)WISD (pruning)

Per-benchmark accuracy (%)

Avg@32 for AIME 2024, AIME 2025 and HMMT 2025; Avg@4 for LiveCodeBench v5 and v6. Bold marks the best value per column. Grey rows are published results of self-distillation with privileged information (OPSD, SDPO, PW-OPSD, LOPD), each in its source paper's evaluation setting, so treat them as a reference. † taken from Lightning Weave. From Table 2 of the paper.

03 · Ablation

Weaken moderately: not too little, not too much.

Too little weakening gives too little contrast. Too much destroys the weak model's reasoning, and noise drowns the signal. Both constructions peak in between, and the best settings also shorten responses.

Bits b3

Qwen3.5-4B on HMMT 2025, Avg@32 over 960 responses per setting. Dashed line: the original model (57.7%, 30.0K tokens). Defaults: b = 3 and τ = 1/6. From Figure 3 of the paper.

04 · Analysis

The gain comes from weakening, and lands on reasoning steps.

Other weakenings and simple baselines

Five of seven weakenings help on both tasks. Rewarding the model's own confidence, directly or against a temperature-softened copy, stays within 0.6 points of the original.

Qwen3-4B, change against the original (73.4% AIME 2024, 53.0% LCB v5). Wanda and SparseGPT use 2:4 sparsity. From Table 3 of the paper.

Where the training signal goes

explore + verify

Share of tokensSignal · quantizationSignal · pruning

Exploring solution paths gets 22% of the signal from 14.8% of tokens, and verification 11% from 8.1%. Routine implementation gets less.

64 Qwen3-4B math trajectories labelled with the ThinkARM taxonomy; share of the initial logit-gradient L1 magnitude. From Table 4 of the paper.

Two original-model trajectories coloured by how WISD changes each token's target probability. Quantization reinforces a digit in a paint-coverage calculation and the word not in the conclusion that ten gallons are insufficient; pruning reinforces the intermediate zero used to exclude an invalid root.

Token-level effects. Blue and orange mark increased and decreased target probability. Quantization reinforces the coverage calculation and the conclusion that ten gallons are insufficient (a); pruning reinforces the zero used to exclude an invalid root (b).

Citation

BibTeX

@article{wu2026wisd,
  title={Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models},
  author={Wu, Yecheng and Han, Song and Cai, Han},
  journal={arXiv preprint arXiv:2610.38681},
  year={2026}
}