Interlock

11. Quality plan: what the competition does, what to build, what we may say

Written 2026-09-13 from the live competitor runs in results/competitors/, the public-source review in results/competitors/closed_and_protocols.md, the weakness audit in docs/11-weakness-audit.md, three design proposals and three independent judgments of them. It changes no code. Every number below names where it was measured; anything not measured says so.

The starting point, stated plainly. On live services, a fair hand-written check (an idempotency key, a lookup, and a premise re-read, 6 to 10 lines) ties Interlock on every outcome it was run against, and settles faster after every crash. Interlock wins only against each framework’s idiomatic pattern, and on parts of the record. The plan below is aimed at removing the measured losses and at the one lead on record quality that could be measured next. It does not aim at an outcome win over a careful engineer, because none of the proposed changes produce one.

1. Competitor table

1.1 Run live in this project

All rows: Stripe test mode, one $20 refund approved on a $100 payment, real SIGKILL of a separate OS process, ground truth from Stripe’s refund list re-read after the last process exited. “Settle” is crash to settled, median. Interlock arms used claim_ttl 40s and unsigned receipts.

System What it checks, and when Record Where it beats Interlock (measured) Where Interlock beat it (measured)
DBOS Transact 2.31.1 (MIT), Postgres 17 Nothing about the world by default. Replays recorded steps when a process with the same executor_id restarts. cancel_workflow stops a workflow. Datasource transactions run exactly once. workflow_status plus one row per completed step. No row for the killed attempt. No hash or signature: forged step outputs returned by its own API 21/21. The $40 double refund was recorded as SUCCESS. Settle 2.9 to 5.6s against 41.2 to 42.2s. Shared $30 cap held natively 20/20 + 20/20 in 35 lines, where Interlock’s core held 25/40. Revocation by cancel_workflow held 3/3 at 2.9s with no re-check code. DBOS plus a 10-line re-check tied Interlock 9/9. Idiomatic DBOS: hand refund during the outage gave $40 in 2 refunds 3/3; a revocation recorded only in the approval row still refunded 3/3. A process with a different executor_id left the workflow PENDING 3/3 until someone called resume_workflow (Conductor waits 60s by default, per docs, not run).
LangGraph 1.2.11 (MIT), PostgresSaver Re-runs the node on resume. interrupt plus Command(resume) for the approval, idempotency key from thread id. Revocation by graph.update_state. Checkpoint history holds the interrupt payload, the resume value (approver), the revocation as its own checkpoint, and the refund id. No trace of the killed attempt in 0 of 87 histories. Edited plaintext checkpoints read back silently; EncryptedSerializer caught a flipped byte only in encrypted blobs. Settle 1.6 to 5.6s against 40.8 to 42.4s. Revocation via update_state held 3/3 with no gate. LangGraph Store held the cap 20/20 (Store put is an unconditional upsert, race window not timed). A 19-line checked node tied Interlock 29/29 on Stripe end state. Idiomatic LangGraph: hand refund during the outage gave $40 3/3. All 49 Interlock receipts matched Stripe, and the 29 recovered effects record the re-check values. Noted, a loss for neither: the docs’ Command(goto) routing plus update_state silently dropped the pending node; the measured graph used a static edge.
Google ADK 2.9.0 (Apache-2.0), require_confirmation, SqliteSessionService Confirmation bound to the exact call arguments (source: request_confirmation.py); the pause survives process death. Call-id idempotency key. Session rows with confirmation, arguments and one tool response. No entry for the killed attempt. An edited sessions.db loaded without error. Settle 5.0 to 10.2s against 45.1 to 46.3s. ToolConfirmation(confirmed=False) as the resume message stopped the refund 2/2 natively. Hand refund during the outage: $40 2/2. Revocation recorded outside ADK: refunded 2/2. app: state for a shared cap: 0/6 held with no crash, 0/6 after_commit, ledger disagreed with Stripe 12/18. A 9-line hand re-check held 6/6 and an flock arm held 18/18, both faster than Interlock.
OpenAI Agents SDK 0.22.2 (MIT), via LiteLLM to Claude needs_approval pause, serializable RunState, tool_call_id stable across reloads. Idempotency is left to the tool. Approvals stored as call-id lists with no approver or time (0/87). The killed send is missing in 0/60 crashed-agent records. Edited RunState copies loaded 87/87. Settle 5.5 to 19.5s against 43.0 to 45.8s. The reload, approve, run cycle honored a revocation 3/3 in 32 lines. An flock cap inside the tool held 9/9 in 43 lines, where the Interlock arm needed 93. Without a key the SDK paid twice 3/3. Hand refund during the outage: $40 3/3. A sticky approved RunState refunded under a revoked approval 3/3 (Interlock combined with sticky resume was not run). No state shared between runs: cap 0/10.
open-multi-agent 1.19.0 (MIT, TypeScript) Durable tool approval with a SHA-256 requestHash over tool name, input, agent, task and call id. Optional run store with a heartbeat lease (default 60s). RunLedger.cancel. Approval row with reviewer and hash. Journal shows every attempt of a tool call. Not chained or signed: tail drop, a deleted tool result and edits with recomputed hashes all passed verifyRun. Settle 1.4 and 1.5s without the run store, 5.4 and 5.5s with a 5s heartbeat lease, against 44.0s (with the default 60s lease OMA took 60.7 and 60.9s). Cancel during the outage settled in 1.6 and 1.8s against 43.6s. Heartbeat leases tell a slow sender from a dead one (read from code, not run). Hand refund during the outage: $40 2/2 (OMA plus 9 lines tied). No revocation of a recorded decision without the run store: refunded 2/2. FileStore cap with compareAndSet: 2/5, no crash.
A fair hand-written check (per harness) Premise re-read and lookup inside the tool, at send and on retry. User-written status strings. No record of the killed attempt, no chain. Tied Interlock on outcomes everywhere it ran; settled faster in every harness (e2e: 15s median against 43s). Nothing on outcomes. Interlock’s record holds DISPATCHED before the kill and the re-check values; the hand check’s does not.
Interlock (main) for reference Lease and premises at submit; lease and premises again at recovery before any resend; tiered resolution (retry under the provider key, lookup, or AMBIGUOUS). Hash chain per effect with DISPATCHED written before the send. Unsigned in every live run; a rebuilt chain passed verify 6/6 (github_merge), 9/9 (DBOS harness), 25/25 (SDK harness); only an outside key rejected it.    

1.2 Public sources only, not run

System What it checks, and when Record Ahead of Interlock (read, not measured) Interlock ahead (read, not measured)
Salus (salus-ai 0.3.7, MIT SDK, hosted control plane) Policy decision before the call, including whether arguments are grounded in fresh evidence. Brokered execution with a single-use grant. decision_events in hosted Postgres; SDK source says no hash chain; no source says signed. Argument grounding, brokered credentials, repair hints, shadow mode, Slack approvals, Rewind compensation after commit, multi-host state. After a crash the SDK raises on an ambiguous replay instead of resolving it; the default key uses a random per-instance run_id, so it does not survive a restart; CumulativeLimit is session-scoped only. A real Stripe run looks possible and has not been done.
Bifrost (Apache-2.0; audit logs Enterprise) Gateway; tool calls are suggestions and the app owns approval and recovery. HMAC-signed audit of administrative activity, not tool effects. Signed audit events with archival; enterprise governance. No documented fire-time premise re-check, durable intent, or crash recovery for tool effects.
IBM ContextForge 1.0.10 (Apache-2.0) Plugins at tool_pre_invoke; a retry policy can re-send. audit_trails for CRUD, no hash or signature fields. RBAC, OIDC, OTel, SIEM export, clustering. No idempotency or lookup for tool effects; no shared cap.
LiteLLM PR #38241 (open) Agent 365 / Defender allow or block before an MCP call. Not an approval PR (corrects kiro-research.md 3.3). Spend log with guardrail status; Microsoft audit attributes the call to the user. Per-user Entra attribution, threat detection on arguments. No idempotency, premise, revocation window or cap.
HumanLayer 0.7.9 (deprecated) A person approves each call; a re-run creates a new call id. Approval records in the vendor cloud. None beyond human review. Duplicates, revocations and changed facts resolved mechanically without a second review.
AP2 v0.2.0 (Apache-2.0) Signed mandates; stateless verifier (this project’s probe accepted one presentation twice). Signed SD-JWT mandate, ES256 receipt if issued. Cryptographic signatures. No refund mandate, no revocation, no replay protection in v0.2.
ACP and Stripe Shared Payment Tokens Enforced by the receiving service; idempotent POSTs, 409 in-flight, 422 conflict; SPT revocation with a webhook. Service-side objects; order adjustments[] show refunds and disputes. Enforcement a crashed client cannot bypass; post-commit data a chargeback watch needs. Refunds are out of scope of delegate_payment; no premise re-check; per-token allowance only.

1.3 What the table says

Interlock’s measured advantages are narrow and real: it held the hand-refund-during-outage and revoked-outside-the-framework cells that every idiomatic arm violated, and its record shows the killed attempt and the re-check values. Its measured disadvantages are also real: 8 to 10 times slower to settle after a crash, a shared cap that needed scenario code, and a record that is not tamper-evident without a key held outside the writer. Nothing in the compared set watches the money after commit, except Salus Rewind and ACP adjustments, neither run here.

2. Core changes, in build order

Selection rule: keep a change if it removes a measured loss, closes a measured correctness hole, or makes the one record lead measurable; cut it otherwise (section 4). Invariants ship as assert-based tests next to each change, not as a model-checking gate.

2.0 Step 0: harness-only work that decides the rules (no core edit, can start now)

Writable today under experiments/competitor_*.py and results/competitors/.

  1. Finish competitor_receipt_proof.py. No receipt_proof.{json,txt} exists; the earlier run wrote no cell output. Every anchor and reconcile claim waits on it.
  2. Quiesce cell (new competitor_quiesce.py). The flock probe (flock_claim.txt) used a synchronous in-process file target, so it cannot show the gap a judge found: a lock freed by the kernel proves the process is dead, not that its request has stopped. The kernel still flushes the socket, and the server may still be processing. Target: a local HTTP server that accepts a request, waits 2s, then applies. SIGKILL the sender right after the request bytes are written. Arms: flock with zero-wait takeover, and flock plus the quiesce rule (2.1). Prediction, not measured: 2 effects for the first, 1 for the second. Add a fork cell: a child forked while the lock is held keeps the effect pinned as live.
  3. Live Stripe mid-POST kill with the FlockClaims mixin from competitor_flock_claim.py wrapping the journal, recovery started under 1s. Question: what does a concurrent same-key POST return, and does recovery treat it as not final? Today StripeClient.request raises StripeError("409 ..."), _resend marks it interlock_sent, and recover() reports UNRESOLVED and keeps the claim. That is safe if it holds live; it has not been observed.

2.1 Interfaces agreed before parallel work

Builders work on disjoint files against this contract, written down first:

2.2 Step 1: parallel builds (disjoint files)

Builder Owns Change
A interlock/journal.py C1 kernel-lock claims, C2 budget inside dispatch, holder()
B interlock/gate.py C1 quiesce rule, C3 inline recover on submit, over_limit mapping, anchor pass-through, IN_FLIGHT detail
C interlock/receipts.py, new interlock/reconcile_stripe.py C5 honest field names, C6 keyless reconcile
D interlock/targets/stripe_api.py C6 anchor metadata, send_timeout, distinct in-progress error
E interlock/easy.py C4 expect=, kept return values, budget= / send_timeout= / anchor pass-through
F interlock/integrations/adk.py, interlock/mcp_proxy.py, new interlock/why.py C7 refusal sentence
G experiments/competitor_*.py, results/competitors/** Every live measurement in 2.3

These edits touch interlock/, which other work owns and this session may not edit. Each lands only with the owners’ approval. Readers outside interlock/ that C5 changes are listed under C5 for their owners.

C1. Kernel-lock claims with a quiesce rule (builders A and B, about 30 lines)

C2. Shared budget reserved inside dispatch (builders A, B, E; about 20 lines)

C3. Recover inline on submit (builder B, about 10 lines)

C4. Expected-value premises and kept return values in easy.py (builder E, about 6 lines)

C5. Honest receipt fields (builder C, 2 lines in receipts.py, plus readers by their owners)

C6. Service-anchored receipts and keyless reconcile (builders B, C, D; about 50 lines)

C7. One refusal sentence for the model (builder F, about 20 lines)

2.3 Step 2: measure step 1

Builder G runs every experiment above against the merged code, same models, reps and Stripe test mode, and writes results/competitors/*.md with raw per-cell rows. No claim in section 3.2 moves to 3.1 before its row exists.

2.4 Step 3: settlement, evidence only (after C6’s run lands)

2.5 Step 4, optional: attributed “done by someone else”

In the Stripe target only: a refund on the payment whose metadata names this case refuses the agent’s refund as REFUSED:done_by_others with that refund id. It can only narrow sends. It needs support tooling to stamp the case id; the harnesses’ hand refunds carried none, so it changes nothing measured until that exists, and it does not fix the unrelated_refund_during_outage SHORT row.

3. Claim matrix

3.1 Claims we can make today

Each is measured, with its source.

3.2 Claims we can make only after the named run passes

Claim Needs
After a SIGKILL on the same host, Interlock settles non-stale cells within the fair checked arm’s spread (a tie on time, not a win). C1 runs in e2e, ADK, DBOS, LangGraph and SDK harnesses.
Stale cells settle within send_timeout plus 2s, and no “not found” lookup is trusted while a request may still land. C1 quiesce cell and live runs at 30s and 10s.
Zero-wait takeover never double-applies a request still in flight at the server. Step 0 quiesce cell (local) plus live mid-POST kill.
A live but hung sender is never taken over, where the TTL rule double-applies on a no-dedup target. Parity with OMA’s heartbeat lease, a lead over Interlock’s current core. C1 hung-sender cell on a lookup-only live target.
The core holds a shared cap 40/40 under SIGKILL with no subclass and names the holder. A tie with hand_lock and DBOS on money. C2 run.
Workers need no polling loop; Temporal uses 2 attempts per cell. C3 run.
easy.py as shipped holds the calendar matrix 7/7. A tie with hand_check. C4 run.
Unsigned receipts no longer report tamper evidence. C5 merged with its readers.
Anyone can check an Interlock refund receipt against Stripe without trusting the writer, and a rebuilt chain is rejected. Among DBOS, LangGraph, ADK, the OpenAI Agents SDK and OMA as run here, no other record can be checked this way. A lead on record quality, not on outcomes, Stripe only, within 30 days. C6 receipt_proof run including probes I and E.
The model explains refusals correctly at least 9 times in 10 and does not retry. C7 run.
The receipt reflects final money after a failed refund or a chargeback. A tie with a hand check given the same webhook. Step 3 run.
Interlock answers AMBIGUOUS where the idiomatic hand check sends a second email after the 25-hour key window. A real 25-hour run; today it is read from code.

3.3 Claims we should never make

4. What to cut