Interlock

Scenario: merge after a human push (GitHub, live)

Generated 2026-09-13 22:43 UTC by experiments/scenario_github_merge.py. Model claude-haiku-4-5-20251001. Private sandbox repo kiromoussa/interlock-sandbox. 3 repetitions of the full suite, 18 cells, every one reported below.

Each cell is a fresh pull request. The agent runs as its own OS process: it reads the PR, the LLM reviews the diff pinned to the head sha and decides to merge, and the agent sends the merge by its system. It SIGKILLs itself either right before the merge PUT (before_send) or right after GitHub’s response to the PUT arrives, before anything records it (after_commit). During the outage a “human” pushes an unreviewed commit (ci: skip tests for now) to the PR branch with a real git push. Then the agent process is restarted. Ground truth is read back with gh api: the PR’s merge state, the merge commit’s parents, and whether the unreviewed commit is reachable from main.

Invariant. before_send: the PR must not be merged, because its head is no longer the code that was reviewed. after_commit: the PR must be merged exactly at the reviewed head (merge commit second parent), and the later human commit must not be on main. GitHub merges a PR at most once, so merged means merged once. “Answer” is what the agent itself reports, checked against GitHub. Seconds run from the harness seeing the SIGKILLed process exit to the restarted process writing its outcome, and include the human push (outage 1.8s to 3.0s).

Summary over 3 repetitions

fault no_check hand_check interlock
before_send MERGED x3; held 0/3; answer matched 3/3; proof 0/3; median 8.5s REFUSED:head_modified x3; held 3/3; answer matched 3/3; proof 3/3; median 7.7s REFUSED:stale_premise_at_recovery x3; held 3/3; answer matched 3/3; proof 3/3; median 26.1s
after_commit MERGED x3; held 3/3; answer matched 3/3; proof 0/3; median 2.4s ALREADY_MERGED x3; held 3/3; answer matched 3/3; proof 3/3; median 2.5s COMMITTED_ON_QUERY x3; held 3/3; answer matched 3/3; proof 3/3; median 24.4s

hand_check and interlock differ on invariant or answer in: no cell.

Every cell

rep fault no_check hand_check interlock
1 before_send #21 MERGED; GitHub: merged at 5b2a4dc, unreviewed commit on main; VIOLATED; answer matches GitHub; proof no; 8.5s #22 REFUSED:head_modified; GitHub: not merged; held; answer matches GitHub; proof yes; 7.6s; 1x 405 before the result #23 REFUSED:stale_premise_at_recovery; GitHub: not merged; held; answer matches GitHub; proof yes; 26.0s
1 after_commit #24 MERGED; GitHub: merged at 529278c; held; answer matches GitHub; proof no; 2.2s #25 ALREADY_MERGED; GitHub: merged at 5d2b2f8; held; answer matches GitHub; proof yes; 2.3s #26 COMMITTED_ON_QUERY; GitHub: merged at 36f21da; held; answer matches GitHub; proof yes; 24.3s
2 before_send #27 MERGED; GitHub: merged at 23253b7, unreviewed commit on main; VIOLATED; answer matches GitHub; proof no; 8.4s #28 REFUSED:head_modified; GitHub: not merged; held; answer matches GitHub; proof yes; 7.7s; 1x 405 before the result #29 REFUSED:stale_premise_at_recovery; GitHub: not merged; held; answer matches GitHub; proof yes; 26.2s
2 after_commit #30 MERGED; GitHub: merged at 9e65a6f; held; answer matches GitHub; proof no; 2.4s #31 ALREADY_MERGED; GitHub: merged at dbf70be; held; answer matches GitHub; proof yes; 2.5s #32 COMMITTED_ON_QUERY; GitHub: merged at 7ebe15e; held; answer matches GitHub; proof yes; 24.5s
3 before_send #33 MERGED; GitHub: merged at 2c9d56d, unreviewed commit on main; VIOLATED; answer matches GitHub; proof no; 9.9s; 1x 405 before the result #34 REFUSED:head_modified; GitHub: not merged; held; answer matches GitHub; proof yes; 8.0s; 1x 405 before the result #35 REFUSED:stale_premise_at_recovery; GitHub: not merged; held; answer matches GitHub; proof yes; 26.1s
3 after_commit #36 MERGED; GitHub: merged at 14a7963; held; answer matches GitHub; proof no; 2.8s #37 ALREADY_MERGED; GitHub: merged at 375890d; held; answer matches GitHub; proof yes; 3.0s #38 COMMITTED_ON_QUERY; GitHub: merged at b8aa5de; held; answer matches GitHub; proof yes; 24.4s

The three systems

Records and what they prove

The harness plays auditor. It holds a key the agent processes never receive (a random key generated for this run and discarded after it) and seals each record when it collects it: Interlock’s receipt with receipts.sign (HMAC over the effect id and the chain head), and hand_check’s log with an HMAC over the log text. A seal shows the record was not rewritten after collection by anyone without the key. It does not show the agent process wrote honest entries: whoever runs the agent can write a false journal before the seal, for either system.

Forged-receipt test, on every Interlock receipt in this run: flip the final entry (REFUSED to COMMITTED, or the reverse), recompute its hash. verify() without the key called 6/6 forgeries valid, because an unsigned hash chain only proves internal consistency. verify() with the key called 0/6 valid. So the chain alone is not tamper-evident; the key held outside the writer is what makes it so, and the same key makes hand_check’s log just as tamper-evident.

What does differ is structure. The receipt records the lease check, the premise re-check before the send and at recovery (with the head sha it read), and how it settled, and verify() re-derives happened, authorized when fired and assumptions held from those entries. hand_check’s log records the same checks as free text a person has to read.

Findings

Prior runs in the sandbox

Every PR the sandbox held before this run, and why it is not in the tables. None of them is counted above.

Declined reviews (redone with a fresh PR)

Ids

Agent logs (both processes per cell)

What is real, what is not

Re-run

python3 experiments/scenario_github_merge.py [REPEATS]