Code Execution
Scope
Giving a model an interpreter converts it from something that predicts an answer into something that computes one. Sandboxing and Isolation covers where that code is allowed to run and what it can reach. This page covers the loop it runs inside: the model writes code, the code executes, the result comes back, and the model revises. Most of the engineering difficulty is not in the boundary but in that return path, because what comes back is rarely just a number.
The Execution Loop
Five steps, each with a way of quietly going wrong. The loop is a specialisation of the verification loop in Agentic Loops, with the interpreter as the evaluator.
The model emits a snippet against an environment it cannot see. It is guessing at installed packages, available data, file paths and API shapes, using a training-time picture of the world.
Goes wrong when: The environment is described vaguely or not at all, so the model invents imports and paths that do not exist and burns a full iteration discovering it.
The snippet runs inside the boundary with a wall-clock timeout and hard resource caps. Execution is the only step here that is not probabilistic.
Goes wrong when: No timeout, so a loop or a large allocation holds the slot until something else notices.
Return value, stdout, stderr, exit code, and any files or images written. These are separate channels and a model shown only one of them is working blind.
Goes wrong when: Only stdout is captured, so a result assigned to a variable and never printed looks to the model like the code did nothing.
A dataframe print or a stack trace can be enormous. Something must decide what the model sees, and that decision determines whether the next attempt is informed or a guess.
Goes wrong when: Silent truncation from the wrong end, cutting the exception type off the bottom of a traceback and leaving only the frames.
The model reads the outcome and either fixes the code, moves on, or gives up. This is where the budget lives and where a loop without a stop condition becomes expensive.
Goes wrong when: No detection of repeated identical failures, so the same fix is attempted until the iteration cap is hit.
Execution Models
Whether the interpreter keeps state between calls changes the whole interaction. A stateful kernel lets the model build on what it just computed. A fresh process makes every call reproducible and every call expensive.
| Model | State | Suits | Cost of the Choice | Reproducible |
|---|---|---|---|---|
| One-shot script | None. Fresh process each call | Self-contained computation, untrusted input, multi-tenant | Cold start per call, and all context must be re-established | Yes |
| Stateful kernel | Variables persist across calls | Exploratory analysis where each step builds on the last | One poisoned cell contaminates everything after it | Order-dependent |
| Notebook style | Cells plus an execution order | Work a human will later read and rerun | Out-of-order execution makes the visible document a lie | Only if rerun clean |
| Persistent workspace | Filesystem survives between sessions | Long projects with expensive setup or large local data | Accumulated state nobody can account for | No |
Getting Data In and Results Out
The part that is never specified until it breaks. An interpreter returns far more than a value, and each channel needs a decision about size, encoding, and what the model is actually shown.
The expression result. Structured, typed and usually small. The channel most worth privileging when it exists.
Decide: Whether to serialise rich objects or render them, and what happens when the value is not serialisable.
Print statements, progress bars, library chatter. High volume and low signal density, and the channel models most often rely on.
Decide: A byte cap, and whether to truncate from the head, the tail, or the middle. Progress bars alone can fill a context window.
Tracebacks and warnings. Small, dense and the single most useful thing to return in full.
Decide: Return it complete before truncating anything else. Deprecation noise is worth filtering; exceptions never are.
Charts, CSVs, models, reports. The actual deliverable in most analysis tasks, and invisible unless the harness looks for them.
Decide: How files are discovered, size limits, and whether they reach the user directly or only the model.
A rendered chart is the output for a large share of data work. Whether the model can see it depends on the model being multimodal.
Decide: Whether to feed images back for self-correction, or pass them straight through to the user unexamined.
The reverse direction. Uploaded files, query results and prior artefacts have to arrive somewhere the code can reach.
Decide: Mount path convention, size limits, and how the model is told what exists without pasting the data into context.
The Error Feedback Loop
A traceback is the highest-quality feedback signal an agent ever gets. It is precise, it is machine-generated, and it is not a judgement call. Getting it back into context well is worth more than a better model.
| Signal | What to Return | Why |
|---|---|---|
| Exception and traceback | The whole thing, exception type first, deepest frame included | The type and the failing line are the two things that determine the fix |
| Missing package | The import error plus what is actually installed | Otherwise the model guesses a second package name and fails again |
| Timeout | That it timed out, the limit, and any partial output | A silent kill is indistinguishable from a crash, and prompts the wrong fix |
| Resource kill | The limit that was hit, named explicitly | Memory limits demand a different strategy, not a retry of the same code |
| Empty output | An explicit statement that nothing was emitted | Silence reads as success and the model moves on with nothing |
| Repeated identical failure | A stop, not another attempt | Two identical tracebacks mean the model lacks the information to fix it |
The loop needs a stop condition as much as a retry. Repeated failures on the same line, or a fix that reintroduces an earlier error, mean escalation rather than another attempt. See Judge and Escalation for the verdict structure and Agentic Loops for oscillation detection.
Failure Modes
The model imports a package that does not exist, or calls a function removed several versions ago. With unrestricted installation this becomes a supply chain problem, because an attacker can register the plausible name the model keeps inventing.
Fix: Pin an allowlisted set of packages, state them in the prompt, and never let the sandbox install from a public registry at runtime.
A large result is cut to fit the context and the informative part is discarded. Tracebacks lose their exception type, dataframes lose the row that mattered, and the model debugs a fiction.
Fix: Truncate the middle rather than either end, label the cut explicitly, and give stderr priority over stdout in the budget.
In a stateful kernel a variable defined three steps ago silently satisfies a name the current code should have failed on. The snippet appears to work and will not work anywhere else.
Fix: Validate against a clean run before accepting a result as final. Treat kernel state as scratch, not as the answer.
After repeated failures the model stops solving and starts asserting, reporting plausible numbers it never computed. The most dangerous mode, because the output is well formatted and entirely fictional.
Fix: Require every reported figure to be traceable to a captured execution result. Treat any claim without a matching run as unverified.
Each iteration carries the whole accumulated history plus a fresh traceback. A ten-attempt debugging session can cost more than the entire rest of the task.
Fix: Cap iterations, compress earlier attempts to their lesson rather than their transcript, and stop on repeated identical failures.
Unseeded randomness, wall-clock reads and network calls make the same code produce different answers, so a result cannot be reproduced when someone asks how it was reached.
Fix: Pin seeds and freeze the clock where reproducibility matters, and record the exact code alongside its output.
When Code Execution Is the Wrong Tool
Generated code is the most general tool available and therefore the easiest to reach for when something narrower would be correct, cheaper and verifiable.
| Task | Why Code Is Tempting | Use Instead |
|---|---|---|
| Exact arithmetic or unit conversion | It is trivially expressible as one line of code | A deterministic tool, with no interpreter to contain |
| Constraint or scheduling problems | The model can write a brute-force search that works on small inputs | A solver, called directly |
| Querying data that lives in a database | Loading it into a dataframe feels like fewer moving parts | A scoped query tool, so the data never enters the sandbox |
| Validating structure or schema | Writing an ad hoc check is faster than defining a schema | Constrained generation against a declared schema |
| Anything touching production systems | The credentials are right there and the script is short | A gated action with its own authorisation |
Related: Sandboxing and Isolation for the containment this assumes, Applied Neuro-Symbolic for the case where the executed artefact is a solver call rather than a script, Agentic Errors for the broader failure taxonomy, and Hallucinations and Grounding for fabricated APIs and packages.