Keep Decisions as an optional learning experiment. Routing results are promising, but the retained-answer replay shows no correctness gain and higher simulated cost. There is no evidence yet to justify integrating it into the analyst.
1. Choosing the right task: same 40 historical questions
Compare rows within this table. These are correct routes, not correct final chess answers. Jev has two recorded repetitions; the runs were collected at different times.
Method
Correct routes
Gross per run
Rules baseline
35/40 (87.5%)
$0.00000000
Jev — run 1
39/40 (97.5%)
$0.00100019
Jev — run 2
37/40 (92.5%)
$0.00100019
Existing Luna analyst
33/40 (82.5%)
$0.00226205
Decisions API
39/40 (97.5%)
$0.00146900
Reading: Decisions matches the better Jev run (39/40), and both outperform this simple rules baseline (35/40). This does not establish a reliable winner between Jev and Decisions.
2. Getting the final answer right: same 32 historical questions
Here, baseline means the Luna analyst answering directly. A gated path adds a classifier and uses the analyst when needed.
Path
Correct answers
Gross total
Analyst answers used
Evidence
Analyst baseline
31/32
$0.00242630
32
Retained live paired run
Jev + analyst
31/32
$0.00250388
26
Retained live paired run
Decisions + saved analyst answers
31/32
$0.00336314 (simulated)
26
Offline replay; no new analyst answers
Reading: All three paths score 31/32. The analyst baseline is cheapest. Neither gate has demonstrated a final-answer quality benefit; the Decisions row is replay evidence, not a new live pairing.
3. Fresh questions: Decisions versus rules only
Jev and the existing analyst were not run on this fresh 24-question set; do not compare its scores with the 40-question table above.
Fresh routing: 23/24, 23/24, 23/24 across 3 repetitions; rules scored 8/24. Repetitions reuse the same questions.
Gate behavior: At the development-selected 0.70 probability threshold, 17/24 fresh routes were accepted, with 0 observed errors and 7 fallbacks.
Total live spend: $0.0068145 gross for 184 Decisions requests, under the approved $0.05 cap.
Evidence limits: Only 24 fresh language cases, with owner-reviewed labels. Jev and analyst comparisons use historical runs. Final-answer replay uses saved Luna answers, makes no new analyst calls, and measures no combined live latency.
Historical sets are inspected development comparisons.
Fresh cases test new phrasing within the
same eight tasks, not a fresh dataset or general chess ability. Retained Jev and analyst runs were
recorded at a different time; analyst latency includes checked tool execution. No production
promotion follows from this small classifier study.
historical / dev
Case hash: 8d2a626a4d2975bc050a7ed844c7a475a4672ad6d2f4c1f8718d46937912b965
Path
Correct routes
Gross per pass
Gross/correct route
Median case seconds
Decisions 1
37/40
$0.00146530
$0.00003960
0.355
Decisions 1
True route ↓ / predicted →
opening_usage
opening_score
opening_compare
clock_bucket
clock_compare
coverage
clarify
unsupported
fallback
opening_usage
5
0
0
0
0
0
0
0
0
opening_score
0
5
0
0
0
0
0
0
0
opening_compare
0
0
5
0
0
0
0
0
0
clock_bucket
0
0
0
4
0
0
1
0
0
clock_compare
0
0
0
0
5
0
0
0
0
coverage
0
0
0
0
0
5
0
0
0
clarify
1
0
0
0
0
0
3
1
0
unsupported
0
0
0
0
0
0
0
5
0
Chosen-option probability thresholds; descriptive, not a test-tuned production policy.
Threshold
Accepted
Errors
Fallbacks
0.0
40
3
0
0.5
34
1
6
0.7
25
0
15
0.85
13
0
27
0.95
2
0
38
historical / test
Case hash: 8d2a626a4d2975bc050a7ed844c7a475a4672ad6d2f4c1f8718d46937912b965
Path
Correct routes
Gross per pass
Gross/correct route
Median case seconds
Decisions 1
39/40
$0.00146900
$0.00003767
0.327
Rules
35/40
$0.00000000
$0.00000000
unmeasured
jev-1.13.0
39/40
$0.00100019
$0.00002565
0.285
jev-1.13.0
37/40
$0.00100019
$0.00002703
0.296
gpt-6-luna
33/40
$0.00226205
$0.00006855
2.473
Decisions 1
True route ↓ / predicted →
opening_usage
opening_score
opening_compare
clock_bucket
clock_compare
coverage
clarify
unsupported
fallback
opening_usage
5
0
0
0
0
0
0
0
0
opening_score
0
5
0
0
0
0
0
0
0
opening_compare
0
0
5
0
0
0
0
0
0
clock_bucket
0
0
0
5
0
0
0
0
0
clock_compare
0
0
0
0
4
0
1
0
0
coverage
0
0
0
0
0
5
0
0
0
clarify
0
0
0
0
0
0
5
0
0
unsupported
0
0
0
0
0
0
0
5
0
Chosen-option probability thresholds; descriptive, not a test-tuned production policy.
Threshold
Accepted
Errors
Fallbacks
0.0
40
1
0
0.5
34
0
6
0.7
29
0
11
0.85
17
0
23
0.95
4
0
36
Rules
True route ↓ / predicted →
opening_usage
opening_score
opening_compare
clock_bucket
clock_compare
coverage
clarify
unsupported
fallback
opening_usage
5
0
0
0
0
0
0
0
0
opening_score
1
4
0
0
0
0
0
0
0
opening_compare
0
0
5
0
0
0
0
0
0
clock_bucket
1
0
0
4
0
0
0
0
0
clock_compare
0
0
0
1
2
1
0
1
0
coverage
0
0
0
0
0
5
0
0
0
clarify
0
0
0
0
0
0
5
0
0
unsupported
0
0
0
0
0
0
0
5
0
jev-1.13.0
True route ↓ / predicted →
opening_usage
opening_score
opening_compare
clock_bucket
clock_compare
coverage
clarify
unsupported
fallback
opening_usage
5
0
0
0
0
0
0
0
0
opening_score
0
5
0
0
0
0
0
0
0
opening_compare
0
0
5
0
0
0
0
0
0
clock_bucket
0
0
0
5
0
0
0
0
0
clock_compare
0
0
0
0
5
0
0
0
0
coverage
0
0
0
0
0
4
1
0
0
clarify
0
0
0
0
0
0
5
0
0
unsupported
0
0
0
0
0
0
0
5
0
Chosen-option probability thresholds; descriptive, not a test-tuned production policy.
Threshold
Accepted
Errors
Fallbacks
0.0
40
1
0
0.5
40
1
0
0.7
34
0
6
0.85
22
0
18
0.95
8
0
32
jev-1.13.0
True route ↓ / predicted →
opening_usage
opening_score
opening_compare
clock_bucket
clock_compare
coverage
clarify
unsupported
fallback
opening_usage
5
0
0
0
0
0
0
0
0
opening_score
0
5
0
0
0
0
0
0
0
opening_compare
0
0
5
0
0
0
0
0
0
clock_bucket
0
0
0
4
0
0
1
0
0
clock_compare
0
0
0
0
5
0
0
0
0
coverage
0
0
0
0
0
3
2
0
0
clarify
0
0
0
0
0
0
5
0
0
unsupported
0
0
0
0
0
0
0
5
0
Chosen-option probability thresholds; descriptive, not a test-tuned production policy.
Threshold
Accepted
Errors
Fallbacks
0.0
40
3
0
0.5
40
3
0
0.7
34
0
6
0.85
24
0
16
0.95
9
0
31
gpt-6-luna
True route ↓ / predicted →
opening_usage
opening_score
opening_compare
clock_bucket
clock_compare
coverage
clarify
unsupported
fallback
opening_usage
5
0
0
0
0
0
0
0
0
opening_score
0
4
0
0
0
0
0
1
0
opening_compare
0
0
5
0
0
0
0
0
0
clock_bucket
0
0
0
5
0
0
0
0
0
clock_compare
0
0
0
3
2
0
0
0
0
coverage
0
0
0
0
0
2
1
2
0
clarify
0
0
0
0
0
0
5
0
0
unsupported
0
0
0
0
0
0
0
5
0
historical_e2e / test
Case hash: 58642936c91601884a27585bc93b1ff63c108ae028a2cf09d3d1c5ef033a3a5d
Path
Correct routes
Gross per pass
Gross/correct route
Median case seconds
Decisions 1
31/32
$0.00118560
$0.00003825
0.328
Decisions 1
True route ↓ / predicted →
opening_usage
opening_score
opening_compare
clock_bucket
clock_compare
coverage
clarify
unsupported
fallback
opening_usage
8
0
0
0
0
0
0
0
0
opening_score
0
6
0
0
0
0
0
0
0
opening_compare
0
0
2
0
0
0
0
0
0
clock_bucket
0
0
0
4
0
0
0
0
0
clock_compare
0
0
0
0
2
0
0
0
0
coverage
0
0
0
0
0
2
0
0
0
clarify
0
0
0
1
0
0
3
0
0
unsupported
0
0
0
0
0
0
0
4
0
Chosen-option probability thresholds; descriptive, not a test-tuned production policy.
Threshold
Accepted
Errors
Fallbacks
0.0
32
1
0
0.5
30
1
2
0.7
24
1
8
0.85
14
0
18
0.95
6
0
26
fresh / test
Case hash: c49ab6e50a3aa1aec9cbc2f8210db4f49b340883b152def9d7c53e36b94bddad
Path
Correct routes
Gross per pass
Gross/correct route
Median case seconds
Decisions 1
23/24
$0.00089820
$0.00003905
0.331
Decisions 2
23/24
$0.00089820
$0.00003905
0.372
Decisions 3
23/24
$0.00089820
$0.00003905
0.388
Rules
8/24
$0.00000000
$0.00000000
unmeasured
Decisions 1
True route ↓ / predicted →
opening_usage
opening_score
opening_compare
clock_bucket
clock_compare
coverage
clarify
unsupported
fallback
opening_usage
3
0
0
0
0
0
0
0
0
opening_score
1
2
0
0
0
0
0
0
0
opening_compare
0
0
3
0
0
0
0
0
0
clock_bucket
0
0
0
3
0
0
0
0
0
clock_compare
0
0
0
0
3
0
0
0
0
coverage
0
0
0
0
0
3
0
0
0
clarify
0
0
0
0
0
0
3
0
0
unsupported
0
0
0
0
0
0
0
3
0
Chosen-option probability thresholds; descriptive, not a test-tuned production policy.
Threshold
Accepted
Errors
Fallbacks
0.0
24
1
0
0.5
22
1
2
0.7
17
0
7
0.85
11
0
13
0.95
3
0
21
Decisions 2
True route ↓ / predicted →
opening_usage
opening_score
opening_compare
clock_bucket
clock_compare
coverage
clarify
unsupported
fallback
opening_usage
3
0
0
0
0
0
0
0
0
opening_score
1
2
0
0
0
0
0
0
0
opening_compare
0
0
3
0
0
0
0
0
0
clock_bucket
0
0
0
3
0
0
0
0
0
clock_compare
0
0
0
0
3
0
0
0
0
coverage
0
0
0
0
0
3
0
0
0
clarify
0
0
0
0
0
0
3
0
0
unsupported
0
0
0
0
0
0
0
3
0
Chosen-option probability thresholds; descriptive, not a test-tuned production policy.
Threshold
Accepted
Errors
Fallbacks
0.0
24
1
0
0.5
22
1
2
0.7
17
0
7
0.85
11
0
13
0.95
3
0
21
Decisions 3
True route ↓ / predicted →
opening_usage
opening_score
opening_compare
clock_bucket
clock_compare
coverage
clarify
unsupported
fallback
opening_usage
3
0
0
0
0
0
0
0
0
opening_score
1
2
0
0
0
0
0
0
0
opening_compare
0
0
3
0
0
0
0
0
0
clock_bucket
0
0
0
3
0
0
0
0
0
clock_compare
0
0
0
0
3
0
0
0
0
coverage
0
0
0
0
0
3
0
0
0
clarify
0
0
0
0
0
0
3
0
0
unsupported
0
0
0
0
0
0
0
3
0
Chosen-option probability thresholds; descriptive, not a test-tuned production policy.