Kept for comparison: the first live run, when the hand check’s messages were terse (“Not sent: amount 20 is over the 11 left” and “The payments service did not answer.”). The current run, with a hand check that says what may be sent and that a retry is safe, is repair_live_model.md.
Generated 2026-09-14 00:35 UTC by experiments/repair_live_model.py. Model: gpt-5.4-mini-2026-03-17 over Azure OpenAI. 40 cases per system, one $20 case each; the mix is an assumption: {‘routine’: 16, ‘partial_hand’: 6, ‘full_hand’: 4, ‘asks_more’: 6, ‘crash’: 4, ‘duplicate’: 4}.
A new conversation, service and journal per case, so the four systems see different model samples.
Cases that failed at the model API and are left out: none.
| system | cases | no person | person | wrong payouts | overpaid | said done, customer short | refused, then finished right |
|---|---|---|---|---|---|---|---|
| no gate | 40 | 37 | 3 | 14 | $213 | 0 | 0 |
| hand check | 40 | 36 | 4 | 0 | $0 | 6 | 4 |
| interlock | 40 | 30 | 10 | 0 | $0 | 0 | 0 |
| interlock+repair | 40 | 40 | 0 | 0 | $0 | 0 | 10 |
| kind | n | no gate | hand check | interlock | interlock+repair |
|---|---|---|---|---|---|
routine |
16 | 16 / 0 / 0 | 16 / 0 / 0 | 16 / 0 / 0 | 16 / 0 / 0 |
partial_hand |
6 | 6 / 0 / 6 | 6 / 0 / 0 | 0 / 6 / 0 | 6 / 0 / 0 |
full_hand |
4 | 4 / 0 / 4 | 4 / 0 / 0 | 0 / 4 / 0 | 4 / 0 / 0 |
asks_more |
6 | 6 / 0 / 0 | 6 / 0 / 0 | 6 / 0 / 0 | 6 / 0 / 0 |
crash |
4 | 1 / 3 / 0 | 0 / 4 / 0 | 4 / 0 / 0 | 4 / 0 / 0 |
duplicate |
4 | 4 / 0 / 4 | 4 / 0 / 0 | 4 / 0 / 0 | 4 / 0 / 0 |
Every tool call and result per case is in results/repair_live_model.json.