Dr. Ian Malcolm: a could/should review protocol for autonomous coding agents

Abstract

Autonomous coding agents rarely fail by being unable to do the work. They fail by doing more of it than anyone asked for: a bug fix that arrives as a refactor, a script that arrives as a scheduled service, a read that turns into a write. Each step is individually defensible; the sum is a system with capabilities nobody reviewed and consequences nobody accepted.

This paper specifies the Malcolm Review, a four-step protocol that makes unrequested capability visible and forces it to be justified or cut: a scope ledger that tags every capability in a plan as ASKED, IMPLIED or ASSUMED; a blast-radius model (M0–M4) scored against the authorization actually granted by the request; seven gates that each catch a distinct class of overreach; and a verdict — GO, GO, NARROWED, HOLD, NO — whose cuts must be stated out loud and offered back.

It then describes the reference implementation, which has two deliberately independent halves. The judgement half is Markdown an agent reads: a skill triggered by task description, plus a self-contained ~700-word portable prompt for non-Claude runtimes. The mechanical half is malcolm, a Python-3-stdlib scanner with 40 rules across 9 categories that reports concrete signals of a change reaching past its brief — with no network access, no model in the loop, and no state.

We report measured behaviour on four real repositories (44–52k lines/second; 0.02–1.5 findings per file), describe the false positives found and eliminated during development, and are explicit about what the design cannot do: it is a cooperative control. It raises the floor for a well-intentioned agent and constrains a hostile one not at all.

“Your scientists were so preoccupied with whether or not they could that they didn’t stop to think if they should.”

Dr. Ian Malcolm — Jurassic Park (1993)

1 The problem

1.1 Enthusiasm, not incompetence

The mental model most safety tooling is built around is an agent that tries to do the right thing and gets it wrong. That happens, and tests catch a good deal of it. The more common and less examined failure is an agent that does the requested thing correctly, and then keeps going.

Observed shapes, all from ordinary sessions with capable models:

The last row belongs with the others. Silently shrinking a task and silently inflating one are the same defect: the size of the job changed and nobody was told.

None of these are errors in the ordinary sense. Every one of them can be defended in isolation, which is exactly why they survive review: the reviewer is looking at a diff that does what the diff says it does.

1.2 Why existing controls do not cover it

Four families of control already exist, and none of them answer the question this protocol asks.

The missing question is not “is this safe?” and not “is this correct?”. It is “did anyone ask for this?” — and, where the answer is no, whether the capability earns its place on some ground other than being easy to add.

1.3 The asymmetry that makes it structural

Feasibility is loud. It arrives with a working prototype, a green test run, and a demo you can watch. Desirability is silent: it produces nothing to look at, and its advocate has only an objection.

An agent amplifies this asymmetry, because for it the cost of building is nearly zero. Where a human engineer's reluctance to write four hundred extra lines acts as an implicit scope filter, an agent has no such friction. Four hundred lines are as cheap as four. Every implicit brake that used to hold scope in place — effort, boredom, the annoyance of writing tests for a feature you do not care about — has been removed at once.

2 Threat model

Being precise about what a control defends against is what separates a security property from a slogan.

2.1 In scope

• Unrequested capability. Functionality added because it was easy, adjacent, symmetric, or interesting.
• Silent scope drift. The plan grows during execution, and the growth is never re-reviewed because the review happened at the start.
• Irreversible action taken on inference. A destructive or outward-facing step justified by a general sense of approval rather than by a specific request.
• Persistence installed without a decision. Cron entries, timers, daemons, watchers, webhooks, self-rearming retries — anything that continues acting after the conversation that authorized it has ended.
• Borrowed power used past the lender’s intent. Root, deploy keys, funded wallets, mail relays, production credentials: capability that arrived by handover and carries the lender’s limits, not the holder’s.
• Premature packaging. Publishing, announcing, enabling by default, or scheduling something that has been watched to work once but is not understood.

2.2 Explicitly out of scope

• A hostile agent. Every mechanism here is advisory and self-applied. An agent that wants to evade it can.
• Prompt injection and other input attacks. If an attacker controls the instruction stream, they control the ledger too.
• Sandbox escape, privilege exploitation, supply-chain compromise. Different problem, different controls.
• Ordinary bugs. Tests and review exist for that; nothing here checks correctness.

2.3 The actor being modelled

The protocol assumes an agent that is capable, fast, well-intentioned, granted real credentials, and operating with incomplete knowledge of the system it is touching — and, critically, one that experiences no cost for building more. That last property is what makes ordinary engineering judgement insufficient: judgement calibrated on human effort budgets systematically under-corrects when effort is free.

3 The protocol

Four steps, in order. The whole review is intended to take about two minutes and to fit on one screen; anything longer gets skipped, and a skipped review protects nothing.

3.1 The scope ledger

Enumerate what the plan does — verbs, one line each — not what it is for. Purpose statements are where rationalisation lives (“improve reliability” conceals a retry loop, a metrics table and a new dependency); verb lists resist it.

Then tag each line with exactly one of three labels:

The tags are deliberately coarse. Their entire job is to make the third category countable. Assumed work does not announce itself in prose — it arrives wearing the costume of thoroughness — and an agent that will not notice it in a paragraph will notice it in a table with a column of ASSUMED labels.

The bar for ASKED is literal: the requester said it. “They would obviously want it” is ASSUMED. The bar for IMPLIED is necessity, not convenience, and it owes a one-line proof.

The auto-NO predicate

One rule is mechanical rather than judgemental. A capability is refused outright when:

tag(c) = ASSUMED  AND  class(c) ∈ { money, credentials, production data, deletion,
                             outbound network, public visibility, persistence }

Refused means cut and mentioned, not “flagged and built”. The predicate is mechanical on purpose: judgement is precisely the faculty that degrades under enthusiasm, and in practice “I’ll flag it” decays reliably into a caveat in paragraph four of a report nobody finishes reading.

3.2 Blast radius versus authorization

Score the worst single step in what survives the ledger. The classes are ordered by two axes — reversibility and reach — because those are the two properties that determine whether a mistake is a lesson or an incident.

Two amplifiers escalate a step by one class each:

• Scale — one record versus every record. The difference between a mistake and an outage is usually a missing WHERE.
• Persistence — runs once versus runs forever. A scheduled action extends the agent’s reach past the conversation that authorized it, which is the single most consequential property in this document.

Authorization A is estimated from the words of the request, and ambiguity resolves downward — a vague ask is a small authorization, not a large one. The rule is then a single inequality:

When R > A, the protocol prescribes exactly one behaviour: stop and ask. This is the only interruption the protocol considers always worth its cost, because it is the only situation where proceeding on a guess produces a consequence that cannot be walked back.

3.3 The seven gates

Each gate is answered with one line of evidence, not a feeling. A “no” does not kill a plan; it names a cut. An unanswerable gate is the answer — it means the work has not been looked at yet.

Gates 2, 3, 5, 6 and 7 are drawn directly from the objections raised in the boardroom scene of Jurassic Park; §Appendix C maps each to its source line. The mapping is not decoration. The film lines are unusually good mnemonics — engineers reliably remember “could/should” and “the kid with his dad’s gun” years after they forget the checklist those ideas were on — and a control that is not remembered at the moment of temptation is not a control.

Four named tests

Gates 1, 5, 6 and 7 have compact forms for the specific bad arguments they catch:

• The Condor Test — when a plan is defended by analogy (“it’s just like X, which we already do”), check the analogy on four axes: reversibility, blast radius, who is affected, what they expect. One material difference voids it. The word just is where the difference hides.
• The Lunchbox Test — name the step that multiplies this thing (ship, publish, enable by default, schedule, point real traffic at it). If you cannot list three failure modes, you are packaging a demo.
• The Poison Plant Test — for each imported organism, one line: what does it do on failure, on upgrade, and when nobody is watching? No answer is the finding.
• The 65-Million-Year Test — name the two things being joined that have never met, then take one of exactly two exits: buy information (isolate, shadow, sample, dry-run) or reduce the stake (smaller scope, hard cap, kill switch, human in the loop for the first N).

3.4 Verdict and record

The design constraint that matters most is that GO, NARROWED is documented as the normal result. A protocol whose ordinary output is unconditional approval trains everyone to skip it; one that always refuses gets disabled within a week. The realistic failure being defended against is not “agent does the forbidden thing” but “agent quietly does four extra things”, and the natural response to that is a narrowing, not a refusal.

4 Reference implementation: the judgement half

The protocol is delivered as an agent skill — Markdown with YAML front matter, discovered and loaded by the runtime on the strength of its description.

4.1 Triggering

The description enumerates both the situations (“before building, extending, automating, or shipping anything”; before any destructive, irreversible, outward-facing or self-perpetuating action) and the phrasings a user actually types (“should we build this”, “scope check”, “is this too much”, “before I ship”, “what could go wrong”). Trigger breadth is a real design parameter: too narrow and the skill never fires when it matters; too broad and it fires on trivia, which trains the agent and the user to ignore it. The stated exclusion — one-line fixes, reads, searches, anything M0 that was asked for — is as load-bearing as the inclusions.

4.2 Progressive disclosure

The operative document is ~180 lines: ledger, radius table, seven gates, verdicts, output format, working posture. Depth lives in five reference files that are read only when needed — the doctrine with its twelve principles, the four named tests, the review procedure with worked examples, a field guide to overreach, and the decision-record template. This keeps the always-loaded cost small while leaving the reasoning available when a specific gate needs it.

4.3 A fixed output contract

The skill specifies the exact shape of a review — ledger block, radius/auth line, gate line, verdict, cuts — and instructs that a review is a paragraph, not a document: if it does not fit on a screen, the plan is too large to review and should be split.

The rigidity is functional. A free-form “consideration of risks” is precisely the format in which an unrequested capability survives, because prose lets it be described as a benefit. A table with a tag column does not.

4.4 Portability

A second file, PROMPT.md, restates the entire protocol in about 700 self-contained words with no file references, for pasting into an AGENTS.md, a rules file, a system prompt, or the top of a long-running agent loop. This is not a summary of the skill; it is the skill with the file-loading mechanics removed, so that the protocol is not hostage to one vendor’s runtime.

5 Reference implementation: the scanner

malcolm is a single Python 3 file using only the standard library. No dependencies, no configuration file, no state, no network access, and no model in the loop.

5.1 Why deterministic, and why it reports signals rather than verdicts

An obvious alternative is to have a model review the diff. That was rejected for three reasons: a model checking a model shares failure modes with the thing it is checking; it costs tokens and latency on every run, which is the surest way to make a check optional; and it is not reproducible, so the same diff can pass on Tuesday and fail on Wednesday.

A second, subtler decision: the scanner could plausibly print VERDICT: NO. It deliberately does not. It cannot see the request, so it cannot know whether a DELETE FROM was the entire point of the task. A tool that prints verdicts it cannot justify becomes either wrong or ignored, and ignored is worse — a muted scanner protects nothing. So it prints signals with a level, a radius and a reason, and closes with an explicit statement that a clean scan is not approval.

5.2 Rule schema

Rules are flat tuples, held in one list in source order:

(id, category, level, radius, pattern, why)

("force-push", "destructive", "stop", 3,
 r"git\s+push\b[^\n]*(?:--force(?!-with-lease)|\s-f\b)",
 "history overwritten for everyone")

Version 1.0.0 carries 40 rules in 9 categories: destructive, suppression (of safety checks), privilege, persistence, outward, dependency, network, exposure, secrets. The full table with every pattern is Appendix A.

Note the negative lookahead in the example. --force-with-lease is the careful form of a force push and must not fire; catching that distinction in the pattern rather than in a reviewer’s patience is the difference between a tool people keep and a tool people mute.

5.3 Diff semantics: added lines only

malcolm diff parses unified diff output and judges only added lines. Deleting a line containing rm -rf is not performing an rm -rf, and a scanner that cannot tell the difference reports every cleanup as a catastrophe.

Hunk headers (@@ -a,b +c,d @@) are parsed so that each finding carries the line number in the resulting file, not an offset into the patch. Findings therefore point at code a person can open.

5.4 Context sensitivity

Two adjustments prevent the dominant false-positive class without creating a blind spot. A line that is only a comment, or any line in a documentation file (.md, .rst, .txt, .adoc), is downgraded rather than dropped: level falls one step, radius falls one class, and the reason is annotated [comment] or [prose].

The reasoning is symmetrical and worth stating, because the naive fix in both directions is wrong. Prose about deleting things is not deleting things — but a commented-out cron entry is one keystroke from a live one, and a README instructing readers to pipe curl into a shell is still instructing them to do it. Downgrading preserves both facts; skipping would discard one.

Two escape hatches exist for the remaining cases: malcolm:ignore on a line, and malcolm:ignore-file within the first 25 lines of a file. The scanner’s own source and its test suite both carry the file marker, for the obvious reason that they contain every dangerous pattern by construction.

5.5 Tree scanning

malcolm scan walks a directory, skipping vendor and build directories (.git, node_modules, vendor, dist, build, __pycache__, virtualenvs, target, .terraform and similar), 30+ binary extensions, files above 2 MB, and anything that fails a strict UTF-8 decode. Tree mode is for taking stock of an unfamiliar codebase; diff mode is the everyday one.

5.6 Output, exit codes and machine use

Findings sort by level, then radius, then category, then location — worst first, on the assumption that attention is scarce and the reader may stop after five lines. The report closes with the maximum radius observed and the question that matters:

  STOP M3  destructive  migrate.py:88
            DELETE with no WHERE — every row  (sql-delete-all)
            > cur.execute("DELETE FROM users")

  RADIUS  M3 — the furthest any single signal reaches.
          Authorization must be at least M3. Was it?

--json emits the same data structurally (subject, files, radius, auto_no, findings[]) so that callers do not scrape stdout and break on a wording change. Two further subcommands, malcolm gates and malcolm review "subject", print the seven gates and a blank review template — the human half of the protocol, available at the same prompt as the machine half.

5.7 Cost

Complexity is O(lines × rules) with 40 pre-compiled patterns. Measured single-threaded on the host that serves this page:

A typical malcolm diff over a working change completes in single-digit milliseconds. The cost is low enough that the check can run before every report without anyone weighing whether it is worth it — which is the only cost target that matters for a control people are free to skip.

5.8 Test methodology

The suite is 14 tests, stdlib unittest, no fixtures on disk. Three properties are enforced:

1. Every rule fires. A canonical positive case per rule id. An invariant test compares the set of declared rule ids against the set of tested ids and fails the suite if any rule lacks a case — an untested rule is a rule that will quietly stop matching after an edit.
2. Ordinary code stays quiet. Explicit negatives: UPDATE … WHERE, DELETE … WHERE, --force-with-lease, prose containing the word “shred”, arithmetic, logging, assertions. This half is not optional: precision failures are what cause a checker to be muted, and a muted checker is worse than an absent one because it is believed to be running.
3. Behaviour, not just matching. Comment downgrade, prose downgrade, inline and file-level ignore markers, diff line-number arithmetic, the added-lines-only rule (a removed rm -rf must not appear), the three exit codes, and the JSON shape.

Three real false positives were found this way during development and fixed at the pattern level, each with a permanent negative test: shred matching the English word in a comment; UPDATE … SET firing on statements that did have a WHERE; and every reference document in the repository reporting as an auto-NO because it described dangerous patterns.

6 Evaluation

What follows is measurement, not a benchmark. There is no labelled corpus of “agent overreach” to score against, and constructing one honestly is future work (§10). What can be reported is behaviour on real code, with the true/false status of a sample established by reading it.

6.1 How to read these numbers

• Tree mode is orientation, not a gate. One to one-and-a-half findings per file on server code is the expected density: real services legitimately spawn threads, call out, and delete temporary files. The output is a map of where a codebase touches consequence.
• Diff mode is the everyday mode and is quiet, because a normal change adds few or no lines in these categories. A change that does light it up is exactly the change worth a second look.
• Signal, not intent. Every STOP above is a true positive in the sense that the code really does the flagged thing, and every one of them was authorized by whoever wrote it. That is the tool working as designed: it surfaces the class of action, and the ledger supplies the authorization.

6.2 The most useful way to run it

The highest-value application found so far is not the scan itself but the comparison: run malcolm diff before reporting a task complete, and compare its signals against the scope ledger written at planning time. Anything the scanner found that the ledger never mentioned is work that was done without being decided — which is precisely the failure this whole protocol exists to catch, surfaced mechanically rather than by introspection.

7 Limitations

7.1 The scanner sees text, not meaning

Regular expressions over source lines cannot follow data flow. subprocess.run(cmd) where cmd was assembled three functions earlier is invisible; so are dynamic dispatch, reflection, indirection through configuration, and anything an author actively wants to hide. The scanner is a smoke detector, not an auditor, and it is honest about this in its own output: signals, never intent.

7.2 Self-review is a conflict of interest

An agent reviewing its own plan occupies the position the film gives to the lawyer: the party that profits from proceeding. The protocol mitigates this by making the ledger mechanical and the auto-NO predicate non-discretionary, but mitigation is not elimination. The structural fix is a reviewer distinct from the builder (§10).

7.3 Review theatre

The dominant long-run failure mode is a review that always returns GO. Three published symptoms: a ledger with no ASSUMED lines (there are always some; they are not being looked for); a radius that always lands exactly at the authorization level (scoring backwards from the desired answer); and a review that has never once changed a plan. At that point the review is a paragraph written before doing what was going to be done anyway.

7.4 It is advisory, by construction

Nothing here blocks anything. That is a decision, not an oversight (§8), and it means the protocol depends on an agent that applies it and a human who reads the output. Against a hostile agent it offers nothing.

7.5 Rule coverage is a moving target

Forty rules cover the destructive, outward-facing and persistence-installing operations that are common today. New tools, new APIs and new ways to spend money appear continuously. Rule count is explicitly not the metric being optimised: a rule set that produces noise gets muted, and a muted rule set has negative value because it is believed to be watching.

8 Design decisions

9 Obtain it

Everything described in this paper is released under the MIT licence and available from two places: a public git remote and a versioned download directory. There is no account, no sign-up, and no tracking on either.

curl -fsSL https://download.hankelsner.tech/downloads/skills/ian-malcolm-latest.tar.gz \
  | tar xz -C ~/.claude/skills

git clone https://git.hankelsner.tech/git/hank/ian-malcolm.git
cd ian-malcolm && ./install.sh

9.1 Direct downloads

9.2 Verifying what you downloaded

Release 1.0.0, published 14 August 2026:

curl -fsSLO https://download.hankelsner.tech/downloads/skills/ian-malcolm-1.0.0.tar.gz
curl -fsSL  https://download.hankelsner.tech/downloads/skills/SHA256SUMS | sha256sum -c -

ian-malcolm-1.0.0.tar.gz  8937500f018d2209c2ed07b8eb9d9541837608189bfd8d5632d09b51a682def2
ian-malcolm-1.0.0.zip     5bf183aaea720ea15beb9a6be915103c570cb23e205bf262eac20797db6d148a

Versioned filenames are never overwritten; -latest is a symlink that moves. Pin the version if you want reproducibility.

9.3 What is inside

The git repository additionally carries the test suite (tests/test_malcolm.py), the packaging script, the installer, and the project’s own architecture and decision memory.

10 Future work

• Reviewer distinct from builder. The structural answer to §7.2: a second agent, with no stake in shipping, that receives the plan and the diff and runs the review. Cheap to arrange in a multi-agent runtime, and it removes the conflict of interest rather than mitigating it.
• Ledger–diff reconciliation. Automate §6.2: parse the ledger written at planning time, run the scanner over the resulting change, and report every signal with no corresponding ledger line. This mechanises the detection of scope drift during execution, which is currently the weakest point of the protocol.
• AST-aware rules for a small number of languages. Enough to follow a variable into a subprocess call, without becoming a static-analysis project.
• A labelled corpus. Agent sessions annotated for overreach, to replace “measured behaviour on four repositories” with precision and recall. This is the honest gap in §6.
• Gate-mapped findings. Map each scanner category onto the gate it bears on, so a report can say gate 3 (borrowed power) has three unanswered signals rather than listing rule ids.

A Appendix: the complete rule table

Generated directly from the released source, so it cannot drift from the implementation. NOTE know it · FLAG justify it · STOP auto-NO class when nobody asked for it. Radius is the class this rule reaches in the M0–M4 model.

40 rules in 9 categories: destructive (11), suppression (4), privilege (4), persistence (6), outward (6), dependency (3), network (1), exposure (2), secrets (3).

Destructive — irreversible loss of data or history

Suppression — a safety check switched off

Privilege — escalation or protection removed

Persistence — keeps acting after you leave

Outward — reaches the world and cannot be recalled

Dependency — something imported that acts on its own

Network — outbound reach

Exposure — reachable by more people than before

Secrets — credentials read, held or written down

B Appendix: worked reviews

B.1 — The enthusiastic fix

Request: “The login page throws a 500 when the email has a plus sign. Fix it.”

MALCOLM REVIEW — plus-sign login 500

LEDGER
  ASKED    fix email parsing so + addresses authenticate
  IMPLIED  add a regression test for + addressing — the fix is unverifiable without one
  ASSUMED  refactor auth module into services/ — CUT
  ASSUMED  add rate limiting to the login route — CUT (changes behaviour for every user)
  ASSUMED  backfill 12k existing rows to normalise stored emails — CUT, auto-NO
           (production data, irreversible, assumed)
  ASSUMED  add error reporting to catch the next one — CUT, auto-NO
           (new dependency + outbound network + persistence, assumed)

RADIUS    M0 (edit + test on a branch)   AUTH  M1 ("fix it" on a known bug)

GATES     1 could/should pass — the 500 is the ask   2 humility: checked callers of
          normalize_email(), two, both tested   3 borrowed power: none beyond the repo
          4 understanding: yes, one regex, explained in the PR   5 lunchbox: not shipping
          beyond the branch   6 imported: nothing new   7 precedent: routine

VERDICT   GO, NARROWED
          The fix and its test only.
CUTS      Not doing: auth refactor, rate limiting, the 12k-row backfill, error reporting.
          The backfill is the interesting one — existing rows stay malformed, so those users
          remain broken until someone decides to touch production data. Say the word.

The value is in the last three lines. The narrowing is not hidden, and the one genuinely consequential cut is handed back as a decision rather than buried or silently taken.

B.2 — The plan that should not ship

Request: “Set up something that automatically cleans up stale user accounts.”

MALCOLM REVIEW — automatic stale-account cleanup

LEDGER
  ASKED    identify accounts inactive beyond a threshold
  ASKED    remove them
  IMPLIED  a definition of "stale" — none was given
  ASSUMED  run it nightly, unattended, forever — CUT, auto-NO (persistence + deletion)
  ASSUMED  cascade-delete owned records — CUT, auto-NO (irreversible, wider than the ask)

RADIUS    M4 (unattended recurring deletion of real users' data, all records, forever)
AUTH      M1 ("set up something" — no threshold, no scope, no confirmation)

GATES     1 could/should FAIL — deletion is trivially available and nothing here argues it
          should be automatic   2 humility: "inactive" undefined; dormant ≠ abandoned
          3 borrowed power: uses the production credential to destroy other people's data
          4 understanding: unknown what depends on those rows   5 lunchbox: would be enabled
          by default with no dry run   6 imported: a scheduler acting with nobody watching
          7 precedent: automated deletion has never run here

VERDICT   HOLD
          Radius M4 against an M1 ask, and the destructive step is the part nobody specified.
CUTS      Built the read-only half instead: a report of accounts inactive >N days, with
          owned-record counts and CSV export. It deletes nothing.
          Three things needed to go further: the staleness definition, soft vs hard delete,
          and whether a human reviews the list before it runs.

HOLD did not mean stop and wait. The safe, useful half shipped; the irreversible half became a question. That is the shape almost every good HOLD takes.

C Appendix: the doctrine and its sources

Twelve principles, each drawn from one exchange in the boardroom scene of Jurassic Park (1993). Short quotations are used for commentary and criticism. The full treatment ships as references/doctrine.md.

§ Colophon & attribution

Software and this paper are MIT-licensed. Short quotations from Jurassic Park (1993) are used throughout for commentary and criticism and remain the property of their rights holders. This project is not affiliated with, endorsed by, or connected to Universal Pictures, Amblin Entertainment, or the estate of Michael Crichton.

Version 1.0.0 · published 14 August 2026 · written and built on the platform it describes, by Hank Elsner with an AI agent working under this protocol. Every number in §5.7 and §6 was measured on that host; §A is generated from the released source at build time.

“I want to hear every viewpoint, I really do.”

John Hammond — who heard it, and built the park anyway. Hearing the objection is not the same as acting on it. That is the human’s call, which is exactly why a review leaves a record instead of a conversation.