← All insights

A laptop-sized AI system that classifies insurance risk as well as Claude Sonnet and Kimi K3.1, at a fraction of the cost

November 5, 2026 · 13 min read AI Engineering Architecture

The job

NixerComp is a workers’ compensation program built for staffing firms. Staffing firms place temporary workers with other businesses, and those workers need workers’ comp coverage like any employee. NixerComp works with more than 70 staffing firms nationwide, through their insurance brokers, and handles the audits and claims that come with the coverage. We built the portal its team uses to take and review submissions; that work is in our NixerComp case study.

Every submission names a host employer, a job and a state, and every one needs a class code: the entry in that state’s manual that describes the work. The code sets the rate, so a wrong one misprices the policy and surfaces later as an audit adjustment. Speed matters too. A staffing firm can’t place the worker until the code is settled, and NixerComp advertises an average class-code turnaround of about an hour, against the days traditional carriers take. Getting codes right quickly is a large part of what it sells.

Most cases are routine. The hard ones are where the right code is not the employer’s governing code, the one that describes its main business. A clerical worker placed at a roofing contractor is one example. Delaware has its own classification system, and remote workers follow their own rules. In our test set, 130 of the 270 cases are hard in this sense.

Today the team asks Claude, in the Claude app, on a subscription of about $500 a month. That version of Claude has no access to the manuals or to the team’s history. It answers from what it already knows, plus web search.

What moved the score

We tested every model the same way. Each one got the same tools and the same instructions: search the state manuals (with synonyms) and read any entry, look up how the team decided similar cases before, apply the Delaware and remote-worker rules, and a nudge to decide after five lookups. We call this setup “manuals+”. Each model then classified the same 270 cases, which none of them had seen in training.

The first result matters most to anyone buying AI. The same Claude models scored far apart with and without the tools.

Access to the manuals and the team’s history lifted Sonnet by 30 points and Opus by 15. Training lifted a small open model by 28. The difference between that small model and the best frontier model, both with the same tools, was seven. On this job, what the model could look up mattered more than how large it was.

How we built it

The model, which we named Phrase, is a LoRA adapter on Qwen3.5-9B, an open base model. The adapter trains 116 million of the model’s 9.5 billion parameters, 1.2%. That kind of small edit is typical of fine-tuning, as we found when we measured 254 fine-tunes of one model. The skill is in the data it learns from.

The teaching data came from other models, solving real training cases with the same tools. We ran three open-weight models as teachers, because their licenses allow training on their output: GLM 5.3 Flash on all 4,833 training cases, then DeepSeek V4 Pro on the cases GLM missed, then Kimi K3.1 Pro on what was left. Their correct, fully worked conversations, lookups included, became 3,011 examples of how to solve a case. None of the test or development cases were used.

The hardest cases were written by hand. 180 training cases defeated every teacher. We reviewed each one and wrote 113 of them out the way an underwriter would reason, with real tool output at every step. The other 69 could not be supported by the manual or the case text at all, so the recorded answer was probably wrong, and we dropped them.

We taught it less from memory. The first version, Phrase 2, learned from 5,271 short examples that paired a job with its code. Phrase 2.1 keeps 2,861, because too many of them teach a model to answer from memory instead of looking the code up.

That gives 7,251 rows, and look-it-up conversations make up about 95% of the text. Training took 2 hours 50 minutes on one GPU. We saved four checkpoints and chose one on a separate development set of 145 older cases, by a rule fixed before any test run. The checkpoint at 1.5 passes over the data scored best there (67.6%, against 62.1% for Phrase 2). A full second pass scored worse, so more training on this data would not help.

The results

Every model below had the same tools. The filled square is how often its first pick was right. The open square is how often the right code was among its top three picks, which matters because an underwriter reviews the suggestions.

Is the gap real? With 270 cases, a point or two either way is noise. So for each pair of models we counted only the cases where one was right and the other wrong, and asked whether the split could be chance.

The models Phrase 2.1 ties use far more computation for each word they produce. Most of the open models are mixtures of experts: GLM 5.3 Flash uses 18 billion parameters per word, DeepSeek V4 Pro 49 billion of 1.6 trillion, and Kimi K3.1 Pro 104 billion of 2.8 trillion. Anthropic does not publish sizes for Claude. Phrase 2.1 uses 9.5 billion.

The gain from Phrase 2 is on the hard cases. Where the right code differs from the governing code, Phrase 2.1 is right 57.7% of the time against 50.8%. On the easy cases the two are level. The hard cases are what the hand-written examples were for.

It also looks things up better. Phrase 2.1 checks the team’s past decisions on about half the cases. Phrase 2 never did. It makes fewer lookups (3.3 a case against 5.1) and reads less than half as much text (about 14,000 tokens a case against 33,000). Fewer memorized answers made it use the tools more, and more efficiently.

Where it falls short

Its backup picks are weaker. The right code is in Phrase 2.1’s top three 81–83% of the time. Every Claude model is at 89–95%. When an underwriter scans the suggestions, that gap matters more than the first-pick tie. Fixing it is one of the next steps below.

Rare codes didn’t move. On the 19 test cases whose codes appear fewer than five times in training, Phrase 2.1 is right on nine, the same as Phrase 2.

Opus is still clearly better. Seven points, and significant. If the only goal were the highest score, the answer would be Opus with our tools.

The labels aren’t perfect. We dropped 38% of the hardest training cases as unsupported by the manual, and some test labels are disputed and going back to managers for review. No model will reach 100% on this test, and part of every model’s error is in the answer key.

Small enough for a desk

A model’s weights are normally stored at 16 bits each. A 4-bit copy stores them in about four to five, so the file shrinks from 16.7 GB to 5.2 GB. We ran that copy through the same 270 cases twice. It scored 73.3% and 70.4%, against 72.2% and 70.7% at full precision: +0.4 points, with a 95% range of −2.2 to +3.0. Case by case it was a coin flip. Its one cost was three answers in 540 naming a code that isn’t in that state’s manual.

A 5.2 GB model runs on ordinary hardware: any laptop or desktop with a graphics card that has 8 GB of memory, or a Mac with 16 GB. Many of those cost well under $1,000, and many offices already have one. The machine changes the speed, not the answers. For extra speed it can run on a desktop with a 24 GB card, which costs about $1,000 new. The next section prices each option.

One setting matters to whoever hosts it. The model often searches the manual and checks the team’s history in the same step. llama.cpp, the engine behind Ollama and most low-cost hosting, keeps only the first of those requests unless parallel tool calls are switched on. Without that setting, in a check on 12 cases, the model got its team-history results in none of them. Nothing failed visibly; the model was just quietly worse.

What it costs

NixerComp classifies about 1,000 submissions a month, so the comparison is a monthly one.

Almost all of that range is the price of the machine. Running it costs very little: at about 1,000 cases a month and a few seconds each, the model works for a couple of hours a month and sits idle the rest of the time. On office hours (weekdays 9 to 5, at $0.17 per kWh) a Mac mini costs well under $1 a month in electricity, and a desktop with a graphics card about $2 to $3:

Whichever machine runs it, the cost is the same at 100 cases a month or 10,000. On the $1,000 desktop we estimate about 600 cases an hour from published benchmarks for that class of graphics card, which we will confirm on the machine before buying. A smaller card is slower, but even several times slower is far more capacity than the portal needs.

Against today’s setup, the answer is clear. Phrase 2.1 scores about eight points above Opus in the app and 30 above Sonnet, for less than a tenth of the subscription. If it takes over the classifying the subscription is used for, even the $1,000 desktop pays for itself in about two months, a used machine in under one, and a computer NixerComp already owns saves money from the first month. If only part of the subscription goes to classifying, payback takes longer: about four months at $250 a month, about 11 at $100.

Against Claude with our tools, the answer is closer. Haiku with the same tools costs about $2 a month for the same first-pick score and better backup picks. Opus costs about $64 for seven more points. On a computer NixerComp already owns, Phrase costs about the same as Haiku; on a new one, more. So the case for Phrase is not price. It is that the cost is flat, the case data never leaves the building, and no provider can change the model’s behavior or retire it. For an insurer those can matter more than a few points. It is a choice about control, and NixerComp is making it knowing that Haiku is the other good option.

What this means if you’re deciding

Build the tools before you choose the model. The manual search, the team-history lookup and the state rules lifted every model we tested, by up to 30 points. They are ordinary software, they lift whichever model sits behind them, and they carry over when you change models.

Test on your own cases before believing any claim, ours included. 270 held-out cases and a paired comparison were enough to tell a seven-point gap from a tie. Without that test, nobody would have known that the Claude app, working without the tools, was getting 42% of cases right with Sonnet and 63% with Opus. We made the same argument in Match the model to the task, and this project is what it looks like in practice.

A small model is now a real option for a narrow job. It took one LoRA, about 7,000 training rows and an afternoon of training to match commercial models on this task. It does not match the best one, and most of the work was in the data, not the training run.

What’s next

• Cleaner data from real submissions. Since November 4 the portal has recorded the manager’s reasoning behind each code, including the codes they ruled out. With the reasoning attached, a wrong label is much easier to spot. About 300 of those decisions will become a new test set.
• Phrase 2.2, a full fine-tune. Once about 1,000 more of those decisions have built up, we plan to train Phrase 2.2 as a full fine-tune: all 9.5 billion parameters, instead of the 1.2% a LoRA adapter changes. The model stays the same size, so its 4-bit copy should still be about 5.2 GB and run on the same machines. We will check that the 4-bit copy loses nothing, as we did for Phrase 2.1, and measure it against Phrase 2.1 on the same cases before it replaces anything.
• Better backup picks. The teacher runs produced 371 pairs where one model was right and another wrong on the same case. Training on those pairs teaches the model which near-miss codes to rank below the right one, the weak spot in its top three.
• Search fixes. Better synonyms and a “works remotely” field on the submission. Like the tools themselves, these help every model.
• A hosting trial on the chosen machine before buying it, to confirm its speed.

Method

All models used the manuals+ setup described above, at medium reasoning effort for Claude and the other API models, and with thinking off for Phrase and the untuned base. API models from other providers ran on hosts with zero data retention. Test cases (270) and development cases (145) were never used for training or for teacher runs. Phrase 2.1 is a fresh rank-64 LoRA on Qwen3.5-9B, not stacked on Phrase 2, trained for two epochs (968 steps) with checkpoints every half epoch. The 4-bit version is the chosen checkpoint merged into the base, converted to GGUF and quantized to Q4_K_M, served by llama.cpp with parallel tool calls on. The full-precision runs used vLLM. Each Phrase 2.1 configuration was run twice with different sampling seeds; the two full-precision runs agreed on right or wrong for 246 of 270 cases. Significance compares the cases where two models disagreed. API costs are actual usage at list prices; for models billed through OpenRouter, list prices times 1.45, the ratio between what was billed and the estimate. The two Claude scores without tools come from a separate run with only the model’s own knowledge and web search, the closest stand-in we have for the Claude app. The manual text, the client’s case data and API keys stayed on our own machine throughout.