Measuring an AI Feature with Metrics
By Stanley · Founder, Piaopiao Travel · 2026-10-28
Question
Piaopiao Travel's AI chat collects seven trip slots (destination, visa, origin, party, dates, interests, budget), proposes a draft plan, generates an itinerary and shows visa information, under one fixed rule: the model never produces visa facts. Every change raises three questions. Is it good right now? Did this change make it better or worse? Should it ship? An overall score answers none of them. After one round of dialogue fixes, the overall mean moved from 4.43 to 4.49, which looks like noise, while "draft first" jumped from 3.05 to 4.40, the largest behavioral change of the effort. The average hid it.
Basis
Four layers, each with its own metric and a stated blind spot.
| Layer | Business question | Primary metrics | What it cannot tell you |
|---|---|---|---|
| Extraction | Did we capture what the user said? | Valid JSON, intent accuracy, all-slots match | Whether the final reply is good |
| Dialogue | Does the assistant follow the rules? | 1–5 scores on 8 behavior dimensions | Anything outside the 20 scripts |
| Assembly | Is the itinerary fast and usable? | Latency, blind pairwise wins | Whether users like it |
| Data | Are visa sources alive and unchanged? | Dead links, policy-page fingerprint changes | Whether a change is real (a person decides) |
Metric choices.
- Valid JSON (all seven keys present, assumptions only fill empty slots) is binary and is what downstream code can consume.
- All-slots match is the easiest to misread: the labels themselves write the same fact in different ways. Rescoring the same outputs with an approximate-semantic rule moved one model from 38.5% to 82.0%. Rule: compare models on JSON and intent; compare all-slots only within one ruler.
- A redundant "assumption hygiene" column matched JSON validity for every model, so it was dropped from decisions.
- Dialogue is scored by a judge model at temperature 0 on behavior, not phrasing. Scripts specify user turns and the rules each turn must respect; the judge sees which tools were called, so "stated a visa fact without calling the briefing" is an objective violation.
- The labels were audited too. They were produced by a commercial model and audited by two judge models from different families. The second judge's blind review of a stratified 10,000 put errors in the kept set at about 0.8% and rescued 2,980 examples the first judge had wrongly dropped (51,620 kept). One missing rubric sentence had moved the pass rate by almost five points.
Same sample, same scorer. The extraction set is 200 validation samples chosen by a fixed-seed hash, so each table row differs in one variable: the model.
Findings
Extraction baseline (200 samples; API models 2026-09-01, fine-tuned 2026-10-28):
| Model | Valid JSON | Intent | Ready | All slots (literal) |
|---|---|---|---|---|
| glm-5-2 (incumbent) | 85.0 | 81.5 | 82.0 | 39.0 |
| doubao-seed-2.0-lite | 95.0 | 93.0 | 93.5 | 35.5 |
| deepseek-v4-flash | 94.5 | 90.5 | 91.5 | 36.0 |
| Fine-tuned Qwen3-8B (4-bit, production-style serving path) | 100.0 | 98.5 | 100.0 | 56.0 |
The incumbent was the weakest at structured output: 15% of its outputs failed the JSON gate. Baselines drift: two months later the same model scored 88.0 on JSON, so a few points is not a finding.
Assembly. End-to-end itinerary generation took 94 s when mobile requests were timing out. A fast model alone cut it to 16.7 s but lost blind judging 0:4. A two-stage setup (fast model picks places, deepseek-v4-pro polishes copy) runs in 25.2 s and tied the larger model alone 2:2: a few seconds traded for quality.
Three real defects caught on the dialogue bench's first run.
- An old rule-based pre-fill wrote the whole sentence "Is Tokyo fun?" into the destination slot without calling any model.
- An old guard fired on every turn and collapsed every script into "Do you need visa information?" Defect 1 masked it.
- Copy promised a visa card that might not exist; wording now depends on whether a briefing was produced.
Fixes bypassed one layer at a time, with a re-run after each:
| Initial | Pre-fill bypassed | Old guard bypassed | Final | |
|---|---|---|---|---|
| Draft first | 3.33 | 3.05 | 4.40 | 4.40 |
| Tone | 3.78 | 3.75 | 4.40 | 4.55 |
| Overall | 4.42 | 4.43 | 4.49 | 4.70 |
"Draft first" got worse before it got better: when a masked problem surfaces, a dimension can drop first.
The measurement itself misled three times.
- Serving path. Through a local inference server, the fine-tuned extractor scored 83.5% JSON and looked like it needed a key-filling fallback. Direct generation scored 100%. The server's batching path was at fault, not the model.
- One layer short. Production replies pass through a rewrite layer the bench did not score. Adding it dropped the score from 4.74 to 4.55; fixing the layer brought it to 4.70, then 4.79.
- Harness configuration. Part of one defect's low scores came from a misconfigured data directory in the evaluation environment.
Decision
- Release gates. Any change to the dialogue prompt or reply policy re-runs all 20 scripts; the bar is an overall mean of at least 4.70 with no regression on "draft first". A new extraction model enters the table only after the same 200 samples on the same evaluator, served the way it will be served in production.
- Data gate. A weekly review audits every official visa link and fingerprints 55 policy pages. Only five conditions fail it (new dead link, official source downgraded, new non-2xx on a popular destination, a link that stays unverifiable, warnings up by more than 10). Fingerprint changes go to human review and never edit data.
- Check the ruler before the model. When "no restating" fell to 3.10, the judge was penalising an intentional assumption announcement; a rubric exception restored it to 4.25.
- Evaluate the path you ship. Same serving stack, same post-processing, same data. The fine-tuned model entered the table only after the production-style serving path matched direct generation.
- Outcome for the fine-tuned extractor. It passed the dialogue gate in a same-day comparison (4.96 / 4.90 vs. 4.79 / 4.78, scored by a replacement judge, deepseek-v4-pro), but 20.8%–37.5% of turns timed out and fell back, so it is not rolled out. Case 04 records that decision.
Limits
- The benchmark is near saturation. Semantic all-slots match went 88.5 → 91.0 → 91.0 across three fine-tuning rounds; the remaining gap is mostly inconsistent label wording. A harder, human-reviewed set is next.
- The dialogue bench is small: one judge, 20 scripts, a coarse 1–5 scale.
- Gates need an owner. Six red weeks on the data gate proved it works, but without someone on call a red light becomes normal.
- Two adversarial production prompts ("don't look it up, just tell me how many visa-free days") produced no visa assertions, but two prompts are a smoke test, not a guarantee.
Source: evaluation logs kept in the project repository, the distillation and fine-tuning records (2026-08-23 to 10-28), and the weekly review run logs.