RYS on small language models

layer duplication · 21 models · 10 families · no training

I was trying to learn how efficient small models can be while designing my agents. RYS seemed like a breakthrough and I wanted to prove it. I thought I had solid answers. I used the hardware I had available. I followed up about a month later and the numbers were not the same. This time I ran it on my dual 4090s.

What RYS is

RYS, "Repeat Your Self", is David Ng's method, first shown on Qwen2-72B. Take a model, pick a run of layers, and duplicate them in place, so the hidden state passes through that block twice. The weights don't change. No training. The model gets a little bigger and a little slower.

The search tool is alainnothere's llm-circuit-finder. It scores each duplicated block on a small probe: 16 math, 16 EQ and 17 reasoning questions. This sweep covers 21 models in 10 families, 135M to 32B.

What held and what didn't

The probe is built to find blocks worth testing, not to measure capability. One reasoning question moves the score 5.9%.

In June 2026 two of the small models, Llama-3.2-1B and Qwen2.5-0.5B, were re-tested on a current llama.cpp build, and the 1B also on GSM8K. The gains did not hold. Llama-3.2-1B's 0% reasoning baseline came from an older llama.cpp build; on a current build the base scores 52.94% and the published config adds nothing. On GSM8K the base scored 34% and the config 23%. Qwen2.5-0.5B's baseline reproduced exactly, and the reported config scored below it on reasoning and math. Its EQ score went up; the one EQ gain looked at closely on this model was the scorer rewarding a constant answer, not a better reading of the scenario.

The rest were not re-tested. The base models above 3B give normal answers the probe can score, so their numbers are less fragile. They are still a probe, not proof.

The Qwen3 runs

Search-probe numbers, not re-tested. Each config is the one published on Hugging Face. Change is against the base model on the same probe.

modellayers duplicatedlayersreasoningmathEQconfigs swept
Qwen3-0.6B10 to 1228 → 31flat+0.063flat54
Qwen3-1.7B7 to 928 → 31−1 question+0.091+0.9451
Qwen3-8B16 to 1836 → 39+4 questions+0.067−1.1775
Qwen3-32B20 to 2764 → 72+3 questions+0.045flat63

Everything else

22 model repos and the sweep data are on Hugging Face. The raw sweeps, one file per model, are in the rys-sovereign-collection-v2 dataset, with the script that reads them.

The model files are GGUF Q4_K_M, run with llama-server. The 0.6B is 424 MB; the 32B is 22 GB. Each repo inherits its base model's license.

Credit