Should a Fine-tuned 8B Model Replace the Production Extractor?
By Stanley · Founder, Piaopiao Travel · 2026-10-28
Question
Each chat turn in Piaopiao Travel starts by extracting the user's message into fixed JSON: seven trip slots, an intent, and whether a draft can be produced now. On 200 held-out samples, the incumbent model, glm-5-2, failed the JSON gate 15% of the time (12% on a re-run two months later), and a failed gate drops that turn's structured information. Keep the vendor model, switch vendors, or fine-tune an open-weight model on our own data?
Basis
Decision rules, written before the results.
- Same samples, same gate: valid JSON ≥99% and intent ≥95%.
- Quantization may cost at most 2 points on JSON or intent.
- End-to-end dialogue gate: over 20 scripted conversations the mean score must reach 4.70, and "draft first" may not drop below a same-day baseline.
- A feature flag keeps the extractor off by default. On failure or timeout, the turn is byte-identical to the flag being off. The extractor never produces visa facts.
Training data. Labels were produced by a commercial model and audited by two judge models from different families: the first scored all 52,097 examples; the second blind-reviewed a stratified 10,000, put label errors in the kept set at about 0.8%, and rescued 2,980 examples the first judge had wrongly dropped. Final set: 51,620 (46,419 train, 5,201 validation). Qwen3-8B was fine-tuned with QLoRA for one epoch on a single GPU, 27.4 hours; validation loss fell steadily from 0.138 to 0.107.
Comparison. Same 200 samples and gate for every row. Baseline quality from 2026-09-01, fine-tuned model from 2026-10-28. Latency measured from outside the serving region; API models at concurrency 4, the fine-tuned model single request. USD at ¥7.1.
Findings
| Model | Valid JSON | Intent | Ready | All slots, literal / semantic | Latency p50 / p95 | Cost per call | Ownership |
|---|---|---|---|---|---|---|---|
| glm-5-2 (incumbent) | 85.0% | 81.5% | 82.0% | 39.0 / 82.0 | 3.82 / 6.33 s | $0.00077 | Vendor-hosted |
| doubao-seed-2.0-lite | 95.0% | 93.0% | 93.5% | 35.5 / 83.0 | 3.89 / 5.99 s | $0.00009 | Vendor-hosted |
| deepseek-v4-flash | 94.5% | 90.5% | 91.5% | 36.0 / — | not measured | ≈$0.00032 | Vendor-hosted; endpoint closed by vendor by Oct 28 |
| Fine-tuned Qwen3-8B (4-bit) | 100.0% | 98.5% | 100.0% | 56.0 / 86.5 | 1.83 / 3.05 s | ≈$0.000012, electricity only | Own weights; runs on a single GPU |
"Ready" means correctly judging whether a draft can be produced. The semantic column is an approximate match and is sensitive to the sampling seed (86.5 and 90.0 on two runs of the 4-bit model).
Against the rules:
- Same-sample gate: only the fine-tuned model passed. One API option, deepseek-v4-flash, also had its endpoint closed by the vendor mid-project, which is why ownership is a column.
- Quantization: the 4-bit build lost 0 points on JSON and 1.0 on intent while cutting model size by about two-thirds, so it stayed.
- Dialogue gate: extractor runs scored 4.96 and 4.90 against same-day baselines of 4.79 and 4.78; "draft first" scored 5.00 and 4.85 against 4.60 and 4.45.
- Fallback: with a 3.5 s timeout, 20.8% (concurrency 1) to 37.5% (concurrency 4) of turns still timed out and fell back to the flag-off path.
Decision
The candidate passed offline evaluation and the dialogue gate. It is not rolled out. The fallback rate is too high, and production concurrency would push it higher.
Next, in order:
- Fix a fallback ceiling (none was set in advance), then cut latency: faster inference or hardware, capped output, keeping the model loaded, handling queueing.
- Add a check that every non-null slot value is grounded in the user's text.
- Re-run the dialogue gate with the original judge and the rewrite layer.
- Once a public, authenticated endpoint exists, run a 3-day canary in development before discussing production. Production hosting is undecided.
Limits
- Latency flatters the fine-tuned model. It was measured from outside the serving region: API calls carry about 0.2–0.5 s of cross-region round trip, while the fine-tuned model ran next to the measuring client. Inside the dialogue gate, its single-request latency was slower (p50 2.45 s, p95 3.99 s), for reasons not yet known.
- Cost excludes depreciation. Hardware depreciation over 36 months adds a fixed monthly cost; break-even vs. the vendor model is roughly 47,000 calls a month. At current volume, far below that, cost is not the reason to switch; quality and ownership are.
- "Zero hallucinated slots" is not claimed. The gate shows defaults never overwrote filled slots, not that every value came from the user's words.
- The judge changed. With the original judge unavailable, deepseek-v4-pro scored the gate, so the 4.70 bar is not the original yardstick; only same-day arms compare directly. Drafts were scored before the rewrite layer.
- In-distribution advantage. Validation samples come from the same distillation pipeline as the training data, so part of the 17-point literal lead is learned label wording. On the semantic measure the lead is 3.5–4.5 points, and the benchmark is near saturation.
Source: evaluation logs kept in the project repository (distillation, training and integration records, 2026-08-23 to 10-28, and the same-sample benchmark runs).