Evidence appendix
Figures Behind the Cases: Basis and Sources
The figures cited in the four case studies, with the basis and source of each: how much of the visa data is verified and watched, and how the extraction models compare.
Where a figure feeds a decision, the decision shown is the one already recorded in the case study, not a new reading.
Headline figures
The six figures on the project page, with what each one counts and where it is discussed.
| Figure | Value | Basis | Case |
|---|---|---|---|
| Countries and regions covered | 199 | Destinations in the canonical visa dataset, 2026-10-27. | Case 02 |
| With verified official sources | 91 (261 units, 388 sources) | At least one data unit carries a classed official source and a check date. | Case 02 |
| Checked weekly | 366 URLs + 55 fingerprints | Weekly review run of 2026-10-27: link liveness plus policy-page fingerprints. | Case 02 |
| Change-log entries | 122 across 81 | Generated from version history; CI regenerates and fails on any byte difference. | Case 02 |
| Extraction benchmark | 200 samples × 4 models | Same held-out samples, seed, and gate for every model. | Case 03 |
| Itinerary generation | 94 s → 25.2 s | End-to-end production time, before and after the two-stage setup; copy quality tied in blind judging. | Case 03 |
As of 2026-10-28 · Sources: Canonical visa dataset · read-only counts (2026-10-27) · Weekly review run of 2026-10-27 · Evaluation logs kept in the project repository · 200 held-out samples, fixed seed · Evaluation logs kept in the project repository · assembly timing and blind-judging runs
Generation basis · End-to-end itinerary generation time in production. 94 s when mobile requests were timing out; 16.7 s with a fast model alone, which lost blind judging 0:4; 25.2 s with a two-stage setup (fast model picks places, deepseek-v4-pro polishes copy), which tied the larger model alone 2:2.
Data coverage
Entry-requirement data: how much is verified, and how changes are watched.
Verification and monitoring
Countries and regions with unit-level verification
91 / 19946%
261 verified data units backed by 388 official sources.
366
Official URLs checked weekly
55
Policy pages fingerprinted for changes
122
Change-log entries
81
Countries and regions in the change log
+ Show data table− Hide data table
| Measure | Value |
|---|---|
| Countries and regions covered | 199 |
| With unit-level verification | 91 |
| Verified data units | 261 |
| Official sources | 388 |
| Official URLs checked weekly | 366 |
| Policy-page fingerprints | 55 |
| Change-log entries | 122 |
| Countries and regions in change log | 81 |
As of 2026-10-28 · Source: Canonical visa dataset · read-only counts (2026-10-27)
Basis · Verified = at least one data unit carries an official source and a check date. The other 108 countries and regions have official links but no unit-level check date yet. Weekly checks run every Tuesday 03:17 UTC; the 55 fingerprinted pages are 45 unconditional visa-free destinations, 8 popular destinations and 2 correction monitors. The change log is generated from version history and reproduced byte for byte in CI. Counted 2026-10-27.
91 of 199 countries and regions (46%) have unit-level verification. The other 108 link to official sources without a recorded check date.
What this decides
Coverage is partial, so verification badges and dates stay off user-facing pages; users see only positive, time-limited link checks, and every alert goes through human triage. Case 02 records the process.
Model choice: extraction benchmark
Each chat turn first turns the user's message into structured fields. Four models, same 200 held-out samples.
Extraction quality and latency by model
Swipe the table for more columns ->
| Model | Valid JSON (%) | Intent (%) | Ready flag (%) | All slots, literal (%) | Latency p50 (ms) |
|---|---|---|---|---|---|
| glm-5-2Incumbent | 85.0 | 81.5 | 82.0 | 39.0 | 3,823 |
| doubao-seed-2.0-liteAPI candidate | 95.0 | 93.0 | 93.5 | 35.5 | 3,890 |
| deepseek-v4-flashAPI candidate | 94.5 | 90.5 | 91.5 | 36.0 | not measured |
| Qwen3-8B, fine-tunedEvaluated, not deployed | 100.0 | 98.5 | 100.0 | 56.0 | 1,832 |
Valid JSON: output parses into the required schema. Intent: the user's intent is classified correctly. Ready flag: the model correctly judges whether a draft plan can be produced now. All slots, literal: every one of seven trip fields matches the reference exactly. Higher is better for all quality columns; lower is better for latency.
As of 2026-10-28 · Source: Evaluation logs kept in the project repository · 200 held-out samples, fixed seed
Basis · Same 200 held-out samples and seed for all four models. API-model quality scores from the 2026-09-01 baseline; fine-tuned model scored 2026-10-28 on the production-style serving path. Labels were produced by a commercial model and audited by two judge models from different families. Valid JSON and intent are the most comparable columns; the literal all-slots column favors the fine-tuned model, which learned the label wording. Latency measured from outside the serving region: API models at concurrency 4, the fine-tuned model single request; not collected for deepseek-v4-flash.
Against the incumbent model, the fine-tuned 8B model raised valid JSON from 85.0% to 100.0% and intent from 81.5% to 98.5%, at 1,832 ms median latency versus 3,823 ms.
What this decides
Only the fine-tuned model passed the same-sample gate, and it passed the dialogue gate, but 20.8%–37.5% of turns timed out and fell back, so it is not rolled out. Case 04 records the decision.