Summary: Hundreds of teams have fine-tuned the same small open model, Qwen3.5-0.8B. We read one layer of every fine-tune we could reach, 254 distinct ones, without downloading any of them in full. The typical fine-tune changes that layer by 3.5%, and different fine-tunes change it in unrelated directions. We then tested whether those parts could be combined into a wider, more capable model with no training. They can’t: an eight-model union three times the size of the base scored exactly like a plain weight average, which first-order theory predicts for changes this small. For anyone buying or building on fine-tuned models, the practical lesson is that most fine-tunes are small edits to the same model, and the only reliable way to choose between them is a test on your own tasks.
The question
Every fine-tune of a base model starts from the same weights, so neuron 1,234 in layer 12 means the same thing in all of them. That makes them combinable. The standard way to combine them is a weight merge: average the weight matrices and keep the model the same size. Averaging fine-tunes of one model is well studied, from model soups [1] to task arithmetic [2] and TIES-Merging [3].
We wanted to know whether the same parts could build a wider model instead. In each layer, place the MLP neurons of K fine-tunes side by side (K times the width), scale the output projection by 1/K, and keep attention shared. That model computes the average of the K fine-tunes’ MLP outputs at every layer, rather than the output of their averaged weights. Then measure which neurons in the pool actually fire and prune back to any width: in principle, a model where each neuron is the most useful version of itself.
There is a good reason to expect it to fail. When fine-tunes differ from their base by only a few percent, an MLP is close to linear in that difference, so the average of the outputs and the output of the averaged weights differ only by a second-order term. Wortsman and colleagues made the same point when they compared weight averages with output ensembles [1]. The experiment tests how close “close” is, and whether anything in a real population of fine-tunes breaks the approximation.
A census of the fine-tunes
Hugging Face lists 621 full fine-tunes of Qwen3.5-0.8B. For each one we read the config and a 256 KB slice of layer 12’s MLP down projection by HTTP range request, and compared it with the base. Most of the population can be measured this way in minutes.
| Fine-tunes of Qwen3.5-0.8B | Count |
|---|---|
| Listed on Hugging Face | 621 |
| With safetensors weights the scan could read | 317 |
| Same shape as the base, weights actually changed | 281 |
| Distinct after dropping re-uploads | 254 |
The rest are mislabeled (embedding models, draft models, an image generator), unchanged re-uploads, or repositories with missing files.
How much a fine-tune changes the model. Relative to the base’s weight norm, a fine-tune changes layer 12’s down projection by 0.3% at the 10th percentile, 3.5% at the median, and 14.7% at the 90th. A handful sit at 100%, which are broken uploads.
How alike the changes are. The cosine between two fine-tunes’ weight changes has a median of 0.001 and a 99th percentile of 0.073. Only 0.3% of pairs exceed 0.5, and nearly all of those are the uncensored (“abliterated”) versions, whose changes point the same way. That fits published work showing refusal is controlled by a single direction in the model [4]. Covering half the variety among the 254 changes needs 86 independent directions, so there is no small set of shared “skill directions” at the weight level.
On paper, that looks like 254 independent parts. The next measurement shows why it isn’t.
What we built
We chose eight fine-tunes trained for different jobs, all with real weight changes, and fetched only their MLP tensors (about 530 MB each). Attention, embeddings and norms came from the base in every variant.
| Part | Trained for | Change to layer-12 MLP |
|---|---|---|
| Luth-2-0.8B | French general instruct | 12.5% |
| cheon-cypher-0.8b-v1 | Text to Cypher queries | 10.8% |
| Holo-3.1-0.8B | Computer-use agent | 6.7% |
| chess-reasoner-0.8b | Chess through tool calls | 5.6% |
| finch-0.1-0.8B | Travel QA with citations | 5.1% |
| Huihui-Qwen3.5-0.8B-abliterated | Refusals removed | 4.6% |
| Apodex-1.0-0.8B-SFT | Deep-research agent | 3.2% |
| Qwen3.5-0.8B-ToolCalling-SFT | Tool calling | 2.4% |
From these we built five variants, each an ordinary GGUF that stock llama.cpp loads. merge8 is the control: the average of the eight MLP weight matrices, at the base’s width. union8 places all eight neuron sets side by side, eight times the MLP width and 2.6B parameters in all. best1 keeps, for each base neuron, the most active of its eight versions, and top2 keeps the two most active. Two further variants, selecting the most active neurons across the whole union, were built but not scored, for the reason below. “Active” means a neuron’s mean squared activation over a 200,000-token calibration mix, times the squared norm of its output weights.
Results
The union scores exactly like the merge, and every pruned variant is worse than both. Perplexity is on held-out text (lower is better); the task columns are accuracy with thinking turned off and a 4,096-token limit.
| Model | Width | Prose ppl | Code ppl | Chat ppl | MATH-500 | GSM8K | HumanEval | Tool calls |
|---|---|---|---|---|---|---|---|---|
| base | 1× | 15.02 | 3.342 | 6.339 | 39% | 49% | 24% | 60% |
| merge8 | 1× | 15.12 | 3.439 | 6.125 | 42% | 60% | 16% | 55% |
| union8 | 8× | 15.12 | 3.440 | 6.125 | 46% | 60% | 16% | 60% |
| best1 | 1× | 17.89 | 4.209 | 7.126 | 16% | 56% | 8% | 35% |
| top2 | 2× | 16.00 | 3.732 | 6.527 | 27% | 50% | 13% | 45% |
MATH-500 and GSM8K used 100 problems each, HumanEval all 164, and the tool-call set 20, so it moves in 5-point steps. Each task score comes from a single sampled run at temperature 0.6, so differences under about 5 points are noise, including the gap between merge8 and union8 on MATH-500.
Three results stand out. union8 and merge8 are the same model: their perplexities agree to three decimals on all three corpora, so eight times the width buys nothing. The merge itself is a trade, not a gain: it helps GSM8K (49% to 60%) and chat text, and hurts HumanEval (24% to 16%) and code. Seven of the eight parts were trained on agents, chat or French, and the average leans toward them, as weight merges do. And pruning by activity is harmful: best1 and top2 lose on every column, and best1’s math replies loop 13% of the time against 1% for the base.
Why the union equals the merge
The eight versions of any neuron fire almost identically. Measured on union8 over the calibration mix, the most active version of a neuron is on average only 1.06 times as active as the second most active, and each fine-tune’s neurons carry an equal share of the total activity, 12.0% to 13.1% each. That is the first-order prediction made visible: a 3.5% change in any direction leaves a neuron computing what it computed before. The weight changes being unrelated (cosine 0.001) never meant the fine-tunes computed different things.
The same measurement explains why selecting neurons by activity fails. With all eight versions within a few percent of each other, “most active” is decided by which fine-tune moved its weights furthest, not by any specialization:
| Part | Change to layer-12 MLP | Neurons where its version is most active |
|---|---|---|
| Luth-2 | 12.5% | 49.8% |
| cheon-cypher | 10.8% | 29.9% |
| chess-reasoner | 5.6% | 11.5% |
| ToolCalling-SFT | 2.4% | 2.6% |
| finch | 5.1% | 2.5% |
| Apodex | 3.2% | 2.4% |
| Holo-3.1 | 6.7% | 0.9% |
| abliterated | 4.6% | 0.4% |
best1 is therefore mostly two fine-tunes’ neurons, with the rest of each layer drawn from models that were never trained together. Mixing neurons from different fine-tunes inside one layer breaks the coordination between them, and the model degrades on everything.
LoRA adapters are smaller changes, not larger ones
Adapters seemed like a better source of distinct parts: they are trained on narrower jobs and concentrate their change in a few directions. The same base has 680 adapters listed. 470 are LoRAs that touch the MLP, and we downloaded and measured the 234 with at least 20 downloads (185 distinct).
The premise did not hold. Adapters move the MLP less than full fine-tunes, and their changes overlap far more:
| Change to layer-12 down projection | 185 LoRA adapters | 254 full fine-tunes |
|---|---|---|
| 10th percentile | 0.45% | 0.27% |
| Median | 0.52% | 3.5% |
| 90th percentile | 1.8% | 14.7% |
| Pairs whose changes point the same way (cosine above 0.5) | 17.1% | 0.3% |
Rank barely matters: the ten largest changes include rank 16, 32 and 64 adapters, and many rank-64 adapters sit near the bottom. Training length and learning rate decide the size of the change, and most adapters were trained briefly. Only a few adapters stand out: the eight largest move the layer by 7–10%, two to three times a typical full fine-tune, while the rest sit well below it. As a population, adapters offer smaller and more alike parts to build from, not larger or more distinct ones.
What has been tried before
Weight averaging of fine-tunes from one base is a mature line of work [1][2][3], and merging tools such as MergeKit make it routine, including “passthrough” merges that stack layers from different models into a new size [5]. Our union construction is new as far as we know, but it is equivalent to averaging each layer’s outputs, which is why its outcome could be predicted.
The more promising direction, routing each token to the right fine-tune instead of averaging them, has also been explored. Branch-Train-MiX turns separately trained experts’ feed-forward layers into a mixture of experts with a learned router [6], and Arrow routing selects among a library of LoRA adapters for each input without retraining [7]. Our results add one constraint for anyone building that kind of router: activations cannot tell the experts apart, so the router has to be trained on loss.
What this means if you are buying or building
Most fine-tunes are small edits to the same model. The median one in this population changed a layer by 3.5%, and its neurons still do what the base’s neurons did. That is usually a good thing, since the base’s general ability survives, but it means a fine-tune’s name or description says little about how different it is. Measure it on your own tasks before relying on it, as we argued in Match the model to the task.
Combining fine-tunes is a trade, not a free gain. A merge moves a model toward the average of its parts, and here that gained grade-school math and lost code. If one capability matters more than the others, a single fine-tune chosen by test, or a routed system, will beat an average of everything available.
And the census itself is cheap and reusable. Reading one layer slice over HTTP gives the size and direction of every fine-tune’s change in minutes, without downloading a model. Repeating it on a larger base, where fine-tunes may diverge further, is the obvious next step.
Method
The base was Qwen3.5-0.8B, the instruct release, as Unsloth’s BF16 GGUF: 24 layers, hidden size 1,024, MLP width 3,584. Variants were assembled in NumPy and written with gguf-py, copying every non-MLP tensor and all metadata from the base. A LoRA part is the base MLP plus (alpha / r) · B · A for each projection the adapter touches. Activity was measured with llama-imatrix (llama.cpp build 12061) over 400 chunks of 512 tokens from WikiText-2, Flask source and 1,000 Alpaca examples. Perplexity used llama-perplexity at context 2,048 on WikiText-2 test, the requests library source and 400 further Alpaca examples. Tasks were served by llama-server with the model’s own chat template, thinking disabled, temperature 0.6, top-p 0.95 and top-k 20. In thinking mode the base ran out of its 4,096 tokens on nearly every MATH-500 and GSM8K problem, which is why the comparison uses non-thinking mode. Everything ran on one RTX PRO 6000 Blackwell; building a variant takes about a minute and scoring one about five.
Sources
[1] Wortsman et al., Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, ICML 2022
[2] Ilharco et al., Editing Models with Task Arithmetic, ICLR 2023
[3] Yadav et al., TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023
[4] Arditi et al., Refusal in Language Models Is Mediated by a Single Direction, NeurIPS 2024
[5] Goddard et al., Arcee’s MergeKit: A Toolkit for Merging Large Language Models, EMNLP 2024 Industry Track
[6] Sukhbaatar et al., Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM, 2024
[7] Ostapenko et al., Towards Modular LLMs by Building and Reusing a Library of LoRAs, ICML 2024
