stretto

Appearance

Changelog ​

The stretto CLI and stretto-proxy are the product. The library crates (stretto-trace, stretto-oracle, stretto-model, stretto-report) make no stability promise before 1.0. The file formats carry versions of their own (docs/formats.md).

Unreleased ​

Two deep-dive films, and the explainer on the home page ​

  • The site's home page opens on the explainer video, full width under the hero's words, with a play button and a "Watch the explainer" action. The two deep dives sit under it, each playing in place. Every player on the site starts with its captions track off, since the films set their captions in the picture; the player's captions control still turns it on.
  • A film on the math and the probabilistic programs inside stretto (brand/media/math.mp4), on the research page beside the paper. It covers the event a lookup is decided on, Proposition 1, the rule and its live costs, the back-off counts, alpha's posterior as a fugue program, the bindings, and how the estimate held up: calibration, the threshold sweep, the share of the ceiling, the live tasks and learning from few sessions. Its formulas are typeset, each symbol in one color throughout, each term named where it sits, and its derivations are worked in place. Where a value moves (q, alpha, the bindings tried, the threshold), the numbers in the formula beside it follow. Its charts draw the published rows of the reach round, and alpha's posterior is fugue's own run on the console's test fixtures (brand/video/math/data/). Every number has its scope on screen.
  • A tour of the console (brand/media/console.mp4), on the console's guide page. It is made from the real console over its test fixtures: brand/video/console/capture.sh serves them, and capture.mjs drives the console as a user does and saves each shot.
  • The films open on their titles: the explainer's and the console film's wordless openings are gone. The explainer's stays as the README's teaser.
  • The films share one stage (brand/video/stage/) and one renderer, node brand/video/film.mjs FILM. narrate.py voices any film on it, compares numbers digit by digit, and pins "Dirichlet" and "nats" in its lexicon.

The explainer video, remade ​

  • The explainer video (brand/media/explainer.mp4) is a new film: ten chapters, one diagram each, on a still stage, from the proxy's own output to the live results and how to start. It adds a chapter on the console, shown from its own screens. It shows the real thing: the reads a result carries as the proxy writes them, τ²-bench retail's tools, the counts of the flow in docs/examples/, and the commands of the docs. The line being spoken is set at the foot of the frame, each word lit as it is said. The poster and the README's teaser come from it.
  • The voice-over of both videos and of the interactive explainer is Kokoro-82M's af_heart, run locally (brand/video/narrate.py), in place of Chatterbox. Each line is checked by speech recognition as before. The videos have no sound effects now: the voice, the music bed under it, and its motif in and out.
  • brand/video/PACING.md records the pacing the film follows, measured from Auracle's films: one breath a sentence, one idea a line, and silence written into the script.

The paper in two columns, with more figures ​

  • The paper's PDF (paper/latex/) is set in two columns, with the abstract boxed, the propositions boxed, run-in heads in the accent color and running heads. A table or figure spans both columns only when it is too wide for one, and the note under a table travels with it.
  • Six new figures: how a lookup spares an LLM turn and the event the speculator estimates (Figure 1), a read's expected value against its probability of use at the counted costs of retail, airline and telecom (Figure 2), what decides each turn across the benchmarks (Figure 4), the share of the ceiling each speculator takes in replay (Figure 8), the live comparisons of the speculator and of a prompt for parallel calls, with their intervals (Figure 10), and the cascade on solo telecom (Figure 11). scripts/paper_diagrams.py draws them, from numbers written as the paper writes them, and fails if the paper's text (its figures' captions aside) does not hold each one in its sentence or table row. The earlier figures are renumbered.
  • scripts/paper_figures.py sets the ceilings in two panels side by side and the reliability diagrams narrower, for the two-column page.

The cold start on more agents ​

  • scripts/cold_draws.py run learns three flows from random draws of an agent's first sessions (the habit alone, the draw's own arbiter, a shipped arbiter) and replays each on the agent's test episodes; report gives each flow's spread across draws, how many draws save under half of what every training task gives, and the shipped arbiter's paired difference (#9).
  • Measured (results): on Qwen3.5-397B, Claude Sonnet 4.5 and Claude Opus 4.5, twenty draws of 5 and 10 sessions each. From five sessions the habit alone saved under half in 16 of 120 draws; a shipped arbiter lifted 13 of the 17 narrow draws above half, at no more than 3.3 points of the median. Jev's 39,541 new answers are published.

Flows and guards together ​

  • pilot/run_episode.py --arm reach-guards serves a flow on the reach decider with the guards in the proxy, the design's arm E (#8). pilot/run_paired.py --model runs another agent model, and --flow-oracle mock spares Jev's key for flows that ask no model.
  • Fixed: the guards, and the confirmation judge's check of the records a write names, read the lookups a flow appended to a result as reads the agent was shown (stretto_trace::mcp::appended). When the server behind the proxy runs the flow, as the pilots' tau2_mcp.py does, its lookups reach the proxy inside the agent's own result. The guards read that result whole, so a user id with lookups after it authenticated nobody, and they refused writes the policy allows.
  • Measured (results): glm-5.3-flash, served a flow learned from Claude Opus 4.5's, Claude Sonnet 4.5's and Gemini 3 Pro's training episodes with the guards, took 20.8% fewer LLM turns on the pilots' twenty tasks, 33.3% in retail and 10.1% in airline. Passes held at 18 of 20, and the guards refused one write, which the tool refuses too.

Tickets no trace showed ​

  • scripts/telecom_workflow.py --unseen N runs the compiled procedure on tickets of τ²-bench's full telecom set whose faults no base task combines, N of them drawn with --seed, or all 2,171 (#38). pilot/run_procedure.py --unseen N --seed S runs the same draw through stretto-procedure. pilot/tau2_mcp.py and run_episode.py find a task in the domain's full split when its default tasks lack it, so a solo agent and the cascade's hand-backs run on those tickets too.
  • Measured (results): the procedure passed 84.3% of the 2,171 tickets with no model, the same tickets for a customer renamed throughout, and handed back 334, all failures. stretto-procedure matched the script on 100 of them. Claude Haiku 4.5 resolved 19 of 20 hand-backs, and passed 8 of 20 tickets alone.

τ²-bench's own reward ​

  • pilot/run_episode.py records τ²-bench's own reward beside the database check: reward_basis in result.json, the product of every check in the task's reward basis (#12). A task whose basis counts natural-language assertions needs a judge. --judge claude:MODEL has a Claude model judge them through claude-agent.sh, with τ²-bench's prompt and rule (pilot/judge.py), and result.json lists the assertions and the verdicts.
  • pilot/rescore.py --scoring basis --judge claude:MODEL rescores recorded episodes the same way, and scores solo episodes in solo mode. scripts/basis_report.py combines several judgings of the same episodes by majority. pilot/analyze_paired.py pairs --basis (and pool) and pilot/analyze_trials.py --basis report passes under every check, from either one's rows.
  • pilot/run_paired.py pairs any two arms, each serving its own flow (--arms ARM[=FLOW] ARM[=FLOW]), on some of the test tasks (--tasks), and passes --judge on.
  • Rescored (results): all 1,101 published τ²-bench episodes, the assertions judged three times by Claude Haiku 4.5. 18 lose their pass, 17 on one retail task, as often with a flow as without, and no comparison changes direction.

Surprise ​

  • stretto learn --surprise Q gives a flow a surprise gate (#17, RFC-001 §3.6). After each call where the flow decides, the gate scores the agent's next step by its surprise under the habit, −ln p. A step the flow does not offer counts as handing back, and the flow's own lookups (the proxy's stretto-N calls) are not the agent's. Once --surprise-window of the agent's steps in a row (default 5) average more than the gate's threshold, the flow hands back after every call for the rest of the session, with the reason the session surprised the flow.
    • The threshold is the Q-quantile of the most surprising run of steps in each successful training session, each scored by a habit learned without its task (the arbiter's folds).
    • The gate is stored in the flow (surprise), which makes it format 2. flow-show shows it, flow-diff lists a change to it, and stretto stage learns it again at its quantile, keeping one set by hand.
    • serve and flow-serve --surprise off|NATS, and stretto-proxy --flow-surprise off|NATS, serve a flow without its gate or with another threshold.
    • The audit and stretto drift score a flow's probabilities whatever its gate says.
  • Replayed on GLM-5's and Claude 3.7 Sonnet's τ²-bench test episodes, retail and airline, at five quantiles over four windows (results). At the quantiles a deployment would choose, 0.8 to 0.99, turns saved and detours moved by at most one in 62 of 64 runs. Where the gate tripped, the flow had already made 90% or more of its lookups. No setting paid in more than one of the four agent and domain pairs, so the default stays off. scripts/surprise_study.py replays it.
  • pilot/tau2_mcp.py and trace_env.py give the flow's lookups the proxy's ids (stretto-1, …), and pilot/check_flow.py and run_episode.py take --flow-surprise.
  • In the libraries: stretto_report::surprise; Flow::surprise, with_surprise and with_surprise_override.
  • In the console (#36):
    • A server serves its flow with the gate the flow stores, with none, or at another threshold, which its configuration for each host passes to the proxy (--flow-surprise). The form says what the chosen flow's own gate does.
    • A flow's summary carries its gate (surprise), and a session's says whether it tripped (surprised).
    • The sessions list marks a session that surprised its flow, and the session's page says after which call the gate tripped and what it measured.
    • In the libraries: surprise::SURPRISED, how a hand-back for surprise begins its reason; init::Served::surprise.

Staged flows ​

  • A deployment's flow comes in two parts: the committed flow the proxy serves, and a staged flow that learns as sessions arrive (#35, staged flows).
    • stretto stage --flow NAME.flow.json --sessions DIR learns the staged flow, NAME.staged.flow.json, from every session so far, as learn --habit-only would. It keeps the committed flow's arbiter, promotion and thresholds. It takes --otel spans and --results too, and --half-life.
    • Before it learns from a new session, it scores the session with the committed flow and with the staged flow as it was, as each is served. The report compares the two on the last --window sessions, out of sample for both, per site, with intervals, and lists what committing would change, as flow-diff does. Its state is NAME.stage.json.
    • stretto flow-commit makes the staged flow the committed one, written whole. Every committed version is kept in NAME.history/, with what changed and the comparison the commit rested on. A committed flow changed by other means is kept as a version first.
    • stretto flow-rollback restores an earlier version as a new one, and stretto flow-log lists them.
  • stretto promote no longer counts a served flow's lookups as the agent's. In a session a flow served, the proxy's lookups (stretto-1, stretto-2, …) are in it, and the agent had no reason to make them again. A lookup the proxy had made now counts as served, neither used nor a detour, and the share and its bound rest on the rest. A promotion records them per site (served). The proxy's lookups are no longer decision points of their own, either: they are the chain after the agent's call.
  • stretto doctor and the console leave committed flows' histories out of their flows.
  • Replayed along a stream (results): on GLM-5's retail test episodes, 20 at a time, committing the staged flow after every batch found 21% more of the agent's lookups and made 2.2 times the detours. Committing only when the staged flow netted more on the batch took most of the gain for fewer detours. In airline, and after a new agent took over, relearning gained nothing, and the drift alarm found the new agent a median of 8 sessions in. scripts/staged_study.py replays it. stretto stage's report now says that a lookup counts after each of the agent's calls, as stretto promote counts it.
  • In the console (#36), a flow's Staged tab shows the staged flow beside it. It has the comparison site by site, with the share's interval, what committing would change, and every version with the evidence it rested on. It commits the staged flow, with a note, and rolls back to any version.
    • A stage job learns the staged flow. The flows list marks staged flows, and committed flows with staged changes waiting.
    • The API: GET /api/flows/:key/stage, POST /api/flows/:key/commit and POST /api/flows/:key/rollback, and stage among the jobs. A flow's summary says its part in staged learning (stage).
    • stretto_report::stage::candidate checks a commit before it writes. stage::Refused marks the commits and rollbacks that do not fit the files, which the console answers with 409.
  • In the libraries: stretto_report::stage, promote::score_as_served, promote::Tally::served and detours, flow::SiteRecord::served; stretto_trace::mcp::is_flow_lookup.

Drift ​

  • stretto drift --flow F --sessions DIR watches for the agent changing under a flow (RFC-001 §3.3 and §4, #18).
    • It scores the sessions in the order they ran, as audit does, for surprise, disagreement and the share of the agent's steps at sites the flow does not know.
    • Bayesian online change-point detection (Adams and MacKay, 2007) watches the three together. It sounds an alarm when a change within the last --window sessions is more likely than --threshold, and names the sites whose surprise rose most across the change.
    • The alarm needs at least ten sessions before a change, and three after it, so one odd session, or two retries of one task, is no change.
    • It exits with 1 while the alarm sounds after the last session, and 2 on an error. It also takes τ²-bench results (--results, --tau2), which come before any sessions.
    • On four τ²-bench agents, it found agents that fit a flow clearly worse, or called tools it never saw, a median of 7.5 to 10 sessions after they took over. On one agent's own sessions in random orders it sounded in 5 runs of 160 (results).
  • stretto learn --half-life N forgets old sessions: one N sessions older than the newest counts half in the habit's counts, and in reach's. Relearned with a 100-episode half-life, a retail flow fit a new agent better than one learned from the new agent's episodes alone (results).
  • stretto-proxy records why it leaves a lookup the flow chose to the agent, in the flow log ("withheld") and once per tool on stderr. The reasons are:
    • the server no longer lists the tool;
    • it marks the tool as a write;
    • it no longer marks the tool read-only, for a flow learned from its listing (new);
    • the tool's input changed;
    • --flow-tools does not grant it.
  • stretto-mcp-demo --world retail takes --hide TOOL and --hint TOOL=false|none, to stand in for a server that changed since a flow was learned.
  • In the libraries: stretto_report::drift, Flow::trained_on, phase0::Config::half_life; in stretto_model, EncodedEpisode::weight and with_weight, BackoffModel::observe_weighted and count.
  • In the console (#36), a drift job runs stretto drift on a flow's sessions, from the flow's page or the jobs page.
    • stretto drift exits with 1 while its alarm sounds. The job records that as an alarm (alarm on the job), not a failure, and the jobs list shows it as one.
    • When a flow's last drift job sounded its alarm, the flow's page says so at the top, with how many sessions ago the change likeliest came, where the flow's surprise rose most, any tools training never saw, and a link to the report. The overview's health lists it too.
    • The API: drift among the jobs, and a flow's last drift run in its detail (drift).

LLM turns and promotion ​

  • stretto infers an agent's LLM turns from timing as well as overlap. A call the agent sends within 500 ms of its turn's last response, with nothing said in between, joins that turn, unless it passes a value that first appeared in what the turn returned. The proxy's own lookups join a turn only by overlap, as before. Claude Code runs a turn's calls as the model streams them, so a call could reach the proxy after an earlier call of its turn had returned, and stretto counted it as a new turn (results, #40).

    • This changes the turns that learn, promote, audit and the console read from a proxy's log.
    • A script that sends its next call within 500 ms of a result it does not use now reads as one turn. The quickstart and the walkthrough wait 0.6 s between calls.

  • stretto-proxy decides at the site of the call whose result came back, and leaves the calls of its turn still on their way to the agent. Before, it decided at its turn's last call, and could look up a call the agent had already sent.

  • stretto promote scores a flow as the proxy serves it:

    • as each result comes back;
    • along chains of lookups, up to --per-call (8, as the proxy's --flow-per-call), and only through the sites it promotes;
    • with a lookup used only when the agent makes it in a later LLM turn.

    On AgentDojo's Slack it no longer promotes sites whose lookups the agents had already asked for.

  • In the libraries: stretto_trace::mcp::episode_sent, mcp::SAME_TURN_MS and Episode::after_call. In stretto_report, flow::PER_CALL, promote::promote_served, and promote::score over promote::Recorded sessions with a promote::Serving; promote::one_call_per_turn is gone. Flow::next_explored takes the calls still on their way.

stretto-proxy ​

  • With --commit, the proxy tells the agent when to use stretto_commit: a sentence after the server's own instructions in its answer to initialize, which hosts such as Claude Code put in the system prompt (#37).
  • stretto-proxy --listen ADDR serves hosts over MCP's Streamable HTTP transport (revision 2025-11-25) in place of stdio, for hosts that connect to servers by URL (#80). The endpoint is /mcp, and the proxy prints its URL.
    • Each MCP session gets a server of its own, with its own log and its own run of the flow, the guards and the judge, as one stdio proxy has. The Stretto-Session header on initialize names its host session.
    • A request is answered on an event stream, or as one JSON body; the server's own messages go on the host's GET stream. DELETE ends a session, and so do its server's exiting and --listen-idle minutes of quiet.
    • Without --listen-token-file, only loopback, checked by the Host header, and pages on other sites are refused by their Origin. With it, every request needs Authorization: Bearer <token>.
  • {session} in --context, --flow-log and --confirm-log stands for the host session, else the session's own id: a file per session.
  • In the library: listen, listen_on and Listen; retain and session_path.

OpenTelemetry spans ​

  • stretto learns from the OpenTelemetry GenAI spans an agent framework exports, with no proxy recording its sessions (#23, OpenTelemetry spans).
    • stretto_trace::otel reads OTLP JSON exports, as the Collector's file exporter writes them, and makes an episode of each trace. Each inference span is one LLM turn with its token usage, and the execute_tool spans after it are that turn's calls and results, several at once being parallel calls. The user's messages and the assistant's text come from the opt-in message content.
    • It reads the names of the GenAI conventions of semantic-conventions 1.38.0, the first to define a tool call's arguments and result; a flow learned from spans records the version in its provenance (conventions).
    • stretto learn --otel FILE --manifest M learns a flow from them, with the tool kinds from the manifest, since spans do not carry them. A call whose arguments the spans did not capture has null arguments: the flow learns which lookups follow which calls but leaves those lookups to the agent, and learn says so.
    • audit, promote and drift take --otel in place of --sessions.
  • docs/examples/pydantic-ai-shop.otlp.jsonl is a Pydantic AI agent's export for eight customers of the demo shop, with content capture on, recorded by scripts/otel_fixture.py with a scripted model; docs/examples/shop.manifest.json gives its tools' kinds.

Several servers ​

  • An agent with several MCP servers leaves one log per server in each session; stretto now learns from them as one session (#22, several servers).
    • Each proxy records the host session it ran in, in the log header (host_session): STRETTO_SESSION when the host's environment sets it, else the host's process, host-<pid> (on Unix). stretto-proxy --server-name names the server (server_name); by default a server goes by --domain, else the name it gives itself.
    • stretto learn, promote, audit and drift merge the logs of one host session that ran at the same time into one session: their lines in time order, each tool named after its server (docs::search), the conversation once, and LLM turns inferred across the servers. They say so on stderr. A host session in which two servers of one name ran at once, such as a harness's sessions run from one process, stays apart, and they say that too.
    • stretto-proxy, serving a flow learned so, looks up only its own server's tools, by their own names, and leaves a lookup on another server to the agent. A proxy whose server's name is none of the flow's says so, and looks nothing up.
    • stretto redact counts a host session's logs as one session when it decides which values few sessions share.
  • In the libraries: stretto_trace::mcp::merge with Sessions, session_groups, server_name, qualify, server_of, unqualified and qualify_episode; LogHeader::host_session and server_name; stretto_proxy::Config::host_session and server_name. A flow learned across servers still decides at every site, and looks up only the tools of the server whose call returned.

The console ​

  • stretto-console, a new crate and binary: the server of stretto's management plane, a local web app over the data directory (~/.stretto, --data or $STRETTO_HOME). Its JSON API lists and shows the recorded sessions call by call, with each of the flow's lookups and the decision that made it; the flows, as stretto flow-show reviews them and as stretto flow-diff compares them; a registry of the MCP servers stretto fronts (servers.json), with the host configuration stretto init prints for each and a live test of the connection; and jobs that run the stretto CLI (learn, promote, audit, redact, doctor). It requires a token (a cookie, or a bearer), a header on every change, and a loopback Host under --no-auth; --read-only changes nothing (its README). The UI is console/; its API types are generated from the Rust.

  • The console's UI (console/, Vue 3), which stretto-console embeds:

    • an overview of what stretto recorded and learned, by domain, with its health checks;
    • the servers, with their form, the configuration for each host and the connection test;
    • each session as a timeline: the conversation, the agent's calls, and the lookups the flow read ahead after each call, with their probabilities against the threshold;
    • each flow's graph at a threshold of your choosing, its sites, bindings and tools, its review, and a comparison with another flow;
    • the jobs, run from the page or again from an earlier job's parameters, with their output as it comes.

    It has light and dark themes, works from a phone's width up, and has a command palette (Ctrl-K or ⌘K). It follows the data directory as it changes. make console builds it and runs the console.

  • The console ships with the rest. The release archives, install.sh, install.ps1 and the Homebrew formula install stretto-console, with its UI built in; the installers still install a release from before it. A second image, ghcr.io/alexnodeland/stretto-console (the Dockerfile's console target), runs it on port 8080 with a health check, and compose.yaml runs it over your ~/.stretto. docs/console.md is its guide.

  • The console's jobs can be cancelled, from the job's page or with POST /api/jobs/:id/cancel: a queued job never runs, and a running one's stretto is killed. A job that ends so is cancelled.

  • stretto-console stops on Ctrl-C or SIGTERM while a browser has the console open. Before, the page's stream of live updates kept it serving until the page was closed, and a container's stop waited for its timeout.

  • stretto-console --open opens the URL with $BROWSER when it is set.

  • stretto init --upstream URL [--upstream-header NAME=VAR] configures the proxy in front of a Streamable HTTP server.

  • A server in the console sets what the proxy does on writes, and how long it keeps what it records (#36). Under Writes and retention: the policy guards, the confirmation judge, logging or enforcing, with the file the host appends the conversation to, stretto_commit, and a number of days. Its configuration for each host passes them to stretto-proxy (--guards, --confirm-judge with --context, --commit, --retain-days).

    • stretto init takes the same options.
    • The guards need a domain that has them, retail or airline, and the judge needs the guards. The console and init refuse a setup that the proxy would refuse when it starts.

  • A server in the console can keep its flow to some of its lookups, chosen from the flow's own or typed for a flow the console does not have (#36). Its configuration for each host passes them to stretto-proxy (--flow-tools). A flow's summary names the tools of its lookups (lookup_tools). In the libraries: init::Served::tools.

  • stretto init --listen ADDR [--listen-token-file FILE], and a server's How hosts connect in the console, set up one proxy that every host connects to by URL, over Streamable HTTP, rather than each starting its own over stdio (#80).

    • Each host's configuration names the proxy's URL: claude mcp add --transport http for Claude Code, url for Cursor, "type": "http" for VS Code. With a token file, it sends the token as a bearer: Claude Code's from the file, the others' from STRETTO_PROXY_TOKEN.
    • The next steps start the proxy, and each change after that restarts it rather than replacing the host's configuration.
    • Claude Desktop, which starts its servers as commands, still starts a proxy of its own over stdio, and its placement says why.
    • Without a token file, both refuse an address beyond loopback, as the proxy does; both refuse port 0, which hosts could not find.
    • In the libraries: init::Listening, Setup::listen, init::placement and init::TOKEN_VAR; init::next_steps takes the host.

  • stretto doctor --data DIR checks a data directory other than ~/.stretto; the console's doctor job checks the one it serves.

  • stretto_report::review::view returns a flow's review as data, which flow-show renders from; review::diff's result lists its changes by heading and serializes.

Documentation ​

  • The paper's PDF is linked wherever the paper is: the paper's page, the research index, the home page, the sidebar and the README (#74). The site serves it at /paper/stretto.pdf, copied from paper/latex/stretto.pdf when it builds.
  • The documentation site is at stretto.alexnodeland.com. Links to the old address, alexnodeland.github.io/stretto, redirect there.
  • The documentation site builds for the address GitHub Pages gives it (SITE_URL, from the Pages workflow), so its base, canonical links and sitemap follow a custom domain. A link from the repository's Markdown to the site itself stays on the site: the roadmap's and the changelog's links to the documentation site opened the research notebook before.

Development ​

  • A Makefile with the everyday commands (make help). make ci runs what CI's check job runs; make coverage, make msrv, make bless, make quickstart, make walkthrough, make site, make docker and make docker-console run the rest.
  • CI also runs the doctests, builds the API docs with rustdoc warnings as errors, and checks the workspace on Rust 1.88, the rust-version, with the committed lockfile. A coverage workflow measures line coverage with cargo-llvm-cov; its LCOV report is an artifact, and goes to Codecov when a CODECOV_TOKEN secret is set. It fails under the Makefile's COVERAGE_MIN, which is 100%: every line of the workspace runs in a test. It also fails when a line a pull request adds is not run by any test (scripts/patch_coverage.py).
  • Every stretto command runs in its tests in-process, on a miniature τ²-bench checkout and a shop's recorded sessions (crates/stretto-report/src/cli_tests.rs), with a test for each way it fails, so coverage counts the whole CLI.
  • Every line of stretto-report runs in a test. stretto_report::cli_doc::check_page_with checks a page as check_page does, taking whether to rewrite it as an argument instead of from STRETTO_BLESS, and shadow::OracleKind::name gives an oracle's name as --oracle takes it.
  • Every line of stretto-trace, stretto-model and stretto-oracle runs in a test, with or without a Jev key in the environment. The Jev client is tested against a server on a local port. stretto_model::projection::project and project_design take the model as &dyn Predictor, and a failed write to a ReplayCache names the answer's file rather than its temporary copy.
  • Every line of stretto-proxy runs in a test, its helper binaries among them. stretto-mcp-demo and stretto-procedure run in their tests' own process, against scripted stdio servers and over Streamable HTTP. The proxy's engine is tested one line at a time: each way a flow hands back, commits, the guards and the confirmation judge, requests past their patience, a server that ends mid-request, and a side that stops reading. A scripted HTTP server covers what an upstream can get wrong. stretto_report::shadow::ShadowConfig::build_with is public, so a caller can pass the Jev client in.
  • Every line of stretto-console's library runs in a test, whether or not the UI is built and whatever the environment holds. The console reads the variables it checks, and PATH, once as it starts (Config::env), so its tests set them. The probe takes its patience and the variables as arguments (probe::probe_within), and is tested against scripted stdio and HTTP servers that page their tools, ask questions of their own or get something wrong. The job runner is tested with a stretto that cannot start, a log that cannot be opened, a read-only console and a child that outlives a cancel; it no longer holds its own queue, so it ends with the console's state. api::console_dir and ApiError's conversion from anyhow::Error, which nothing used, are gone.
  • Every line of the stretto-console binary runs in a test too. Its command line runs in the tests' own process, given the environment it reads and a way to stop it; its healthcheck runs against scripted servers.
  • A Claude Code setup: CLAUDE.md, which imports AGENTS.md, and .claude/, with permission rules, a hook that runs rustfmt on each edited Rust file, a session-start hook for Claude Code on the web, four skills (check, results, release, steward) and two subagents (reviewer, claims-checker).
  • A dev container (.devcontainer/) and VS Code settings (.vscode/).
  • A debug build of stretto-console serves the UI in console/dist even when the UI was built after it (make ci, then make console). Before, such a build served only the page that says how to build the UI.
  • CI's ui job checks the console's UI: Prettier, ESLint, vue-tsc, Vitest and the build (make ui-check). It also runs Playwright in Chromium on a mock API and on the console serving its test fixtures (make e2e), and fails on any error in the browser's console. On the mock API it also runs axe-core's WCAG 2.1 A and AA rules on every page in both themes, and on the palette, a dialog, the phone layout, first run and sign-in. make types regenerates the API's TypeScript, and CI's check job fails when it has drifted.

0.1.0 (2026-09-27) ​

The first release: RFC-001's design, built and measured, with release binaries, installers, a container image and a documentation site. docs/results has every result, and the working paper puts them together. The development log below lists every change on the way here.

Install and first run ​

  • Each release carries archives for five targets (Linux x86_64 and aarch64, macOS on Apple silicon and Intel, Windows x64) with their SHA256SUMS, install.sh and install.ps1, which check an archive's checksum before installing it, and a Homebrew formula. A container image, ghcr.io/alexnodeland/stretto, is published for amd64 and arm64; it runs as a non-root user, with /data as its home (install).
  • stretto init --host claude-code|claude-desktop|cursor|vscode -- <server…> writes an MCP host's configuration for a server behind the proxy, and prints the steps to a served flow.
  • stretto doctor checks the installation: the binaries on PATH, ~/.stretto, whether a key is set (never its value), and the flows and sessions. stretto completions <shell> prints shell completions.
  • examples/quickstart/ records, learns, reviews and serves a flow on stretto-mcp-demo, with no key, in under a second; CI runs it.
  • Building from source needs Rust 1.88.

Measure an agent (stretto phase0) ​

  • How compressible an agent's behavior is, from τ²-bench's published trajectories, with no key: a hierarchical Dirichlet habit of the next tool call, its concentration fitted with fugue, coverage and agreement on held-out tasks, transfer between agents, macro-tool headroom and argument provenance.
  • With --oracle, System-One questions at every held-out decision (Phase 0b): v1, over every tool, and v2, read-only lookups with a state slice, a stop question, yes/no predicates (data/predicates-v2.json) and an arbiter that weighs the answers against the habit. Answers go through a replay cache, and published answer bundles replay without a key (export-answers, import-answers). stretto ask answers questions of your own through the same cache. --domain telecom measures τ²-bench's third domain.

Flows ​

  • stretto compile writes a flow (the flow IR, stretto_flow: 1) from τ²-bench results. stretto learn learns one from sessions the proxy recorded: with the habit alone (--habit-only), with an arbiter fitted on held-out sessions, or with a shipped arbiter (--arbiter-from). A flow's lookups bind their arguments from earlier results, including results that list values one per line.
  • stretto serve and stretto flow-serve answer a flow's decisions over a local port. stretto export-arbiter writes a flow's arbiter to its own file (stretto_arbiter: 1), and stretto fit-arbiter fits one on the decision logs of several compiles. data/arbiters/ ships two, fitted on four agents' retail and airline decisions; they carry between those two domains, not to one unlike both.
  • stretto audit scores recorded sessions under a flow, run as a fugue program: agreement, calibration and surprise per site. A flow without an arbiter is audited with its habit.
  • stretto promote keeps a flow to the sites where its lookups were the agent's own, scored on recorded sessions or τ²-bench results; a promoted flow hands back after every other call. stretto-proxy --flow-shadow records the sessions to promote on: the flow decides and logs, but makes no lookups.
  • stretto flow-show renders a flow for review: the tools it may call, the lookups it may make after each call and where their arguments come from, and how it decides. stretto flow-diff lists what changed between two flows for a pull request, and exits with 1 when a change needs review (reviewing flows).
  • Deciders. A flow decides with its arbiter where it has one, and otherwise with reach: the chance that the agent makes a lookup before its next write, counted from traces, times its binding's, against the threshold. It asks no model and needs no key. serve, flow-serve, audit, promote, init and stretto-proxy --flow-decider take habit, reach or arbiter.
  • Bindings. A lookup's arguments come from earlier results, tried in the order the agent used them at the site. learn --constants learns an argument the agent always passed with one value, never a string a user wrote with a digit or an @; a list argument takes every value at the path of an earlier result; a search the agent always narrows is not made bare; and a pick of a record other than the one the customer named is scored by how often the agent made it.
  • Flow format 2 holds per-site thresholds, constants, lists and those counts; this release reads formats 1 and 2. A flow learned from recorded sessions pins each tool's input contract, and the proxy makes no lookup of a tool whose server now lists another.
  • stretto evaluate estimates what another rule would have done from a flow's exploring decisions (--flow-explore), directly and by IPS, self-normalized IPS and doubly robust estimates. stretto refine keeps candidate predicates that raise the arbiter's held-out likelihood. stretto search runs NSGA-II over each site's threshold and the decider.

Compiled procedures (stretto-procedure) ​

  • Where no user speaks, a whole workflow, writes included, compiled once from agents' traces is a file stretto runs: the procedure IR (stretto_procedure: 1, formats), with a decision tree per site over the run's state, identifiers bound again at run time from where they came from, a guard against repeats, and the checks of each outcome a ticket may state.
  • stretto-procedure --procedure FILE --ticket TEXT -- SERVER… runs it against an MCP server with no model and prints its calls, its check of the ticket's outcome, and its verdict: resolved, transferred, or hand_back when the check failed and the ticket should go to a model.

Checks on writes ​

  • stretto guards: typed policy checks for τ²-bench's retail and airline domains, audited against the published trajectories.
  • stretto confirm: a System-One model judges whether the customer confirmed each write, with an optional second question. stretto match asks which record a customer means.

stretto-proxy ​

  • A stdio MCP proxy that forwards byte for byte and records sessions (stretto_mcp_log: 2). --upstream URL wraps a Streamable HTTP server instead of a command, with headers from the environment that are never logged.
  • Active mode: --flow runs a flow behind the agent's calls and appends its lookups to the result. --guards refuses writes a policy check fails. --confirm-judge log|enforce adds the confirmation judge, and --confirm-second-shadow asks its second question but only logs the answer. --commit adds a tool for confirmed writes in one call. --context reads the conversation the host writes.
  • Jev's key can come from a file: TYPESAFE_API_KEY_FILE names it when TYPESAFE_API_KEY is not set.
  • stretto-mcp-demo, a tiny server for trying it.
  • --retain-days N deletes, at start, the logs and cached answers older than N days.
  • Each run of a flow after one of the agent's calls is a fugue program (program.rs, RFC-001 §3.2):
    • The sites. A decision site decide#i comes before each lookup, and the flow's arbiter decides it. An outcome site outcome#i comes after, and takes the server's answer and scores it.
    • The program is in the flow. The flow IR holds it as program, in fugue's serializable program format (RFC-001 §3.5), and it is checked when the flow loads. A flow written without one runs the standard run, which draws exactly what the Rust it replaced drew. flow-show prints the program, and flow-diff lists a change to it as needing review.
    • Its distributions. The flow registers two, Decide and Outcome, from its own statistics: a flat Dirichlet's predictive over what the agent did next at a site, and a flat Beta's over how often a tool succeeded, from fugue's conjugate helpers. They carry their site as metadata (WithMeta).
    • Three interpreters. The proxy runs the program with fugue's run_async. PriorHandler simulates a flow with it, and ScoreGivenTrace scores a recorded run.
    • The flow log. Decision lines carry their address, and each run ends with a run line holding its trace and its surprise.
  • --flow-tools TOOLS names the only tools a flow may call on its own, and --flow-explore EPSILON explores at read-only sites for counterfactual evaluation.

Privacy ​

  • stretto redact writes a pseudonymized copy of recorded sessions: a value fewer than --keep-shared sessions contain becomes a salted hash, the same wherever it appears, and --hash-field hashes named fields however many sessions share them. A flow learned from the copy matches one learned from the originals, on synthetic sessions, 40 real ones and 5 with their conversation (privacy).
  • stretto learn --sessions and redact skip the confirmation logs the proxy writes beside the sessions; they failed on them before.

Live runs and research tooling ​

  • The pilot harness runs a Claude model as the agent or the customer (run_episode.py --agent-cli claude, --customer-cli claude --customer-model, through pilot/claude-agent.sh), and Z.ai credits count the GLM side only. The guards arm takes the confirmation judge (--confirm-judge, --confirm-second, --confirm-second-shadow), with Jev's key handed to the proxy in a file, and --label names an arm's directory.
  • pilot/run_paired.py runs every test task of a domain in both arms, reusing a pilot's pairs, under a Z.ai credit budget that it checks against a ledger before each episode. pilot/analyze_paired.py reports on the result: paired pass rates with a bootstrap interval and McNemar's test, turns and tokens saved, detours and their token cost, and a pooled estimate across domains (the paired run). Its arms command compares any arms run on the same tasks (the cold start, live).
  • pilot/run_episode.py --solo runs τ²-bench's no-user mode live, and --handback has the agent take over from a compiled procedure. --batch-reads adds Anthropic's sample prompt for parallel tool calls to the agent's system prompt, the prompting baseline.
  • pilot/run_trials.py runs a paired design with Claude models as agent and customer, resumably, within a Claude Code subscription's five-hour and seven-day windows; pilot/analyze_trials.py reports pass^k, turns, tokens, cost and time per arm and compares the arms, and pilot/analyze_recorded.py compares arms recorded at different times.
  • pilot/bench/ runs AgentDojo and BFCL tasks live, their tools served over MCP by the benchmarks' own code and each episode scored by the benchmark's check.
  • scripts/ holds every analysis behind the results: the ceilings by where arguments come from (ceiling.py), the costs and the threshold (costs.py, per_decision.py, priced.py), learning curves, repeated reads, the replays of six more benchmarks from their published trajectories (scripts/bench/), and the paper's figures.

Documentation ​

  • The walkthrough runs the whole loop on the official MCP filesystem server, and CI runs it.
  • Reviewing flows, with example diffs of real flows that a test keeps current.
  • Privacy: what each file holds, what is sent to the System-One model, retention and redaction.
  • The file formats, the CLI reference (generated from the code, kept current by CI), the design and the roadmap.
  • The working paper, published as the research notebook at stretto.alexnodeland.com/notebook.
  • The documentation site (website/, VitePress): a guide, integrations, the reference, the research and the community pages.
  • The brand kit: the mark, color tokens, an interactive explainer, and the explainer and walkthrough videos, narrated and captioned.
  • The working paper as an arXiv-ready LaTeX manuscript (paper/latex/), built from the Markdown.
  • CONTRIBUTING.md, SECURITY.md, CODE_OF_CONDUCT.md, CITATION.cff, and issue and pull request templates.

Release ​

  • Install a release with install.sh, install.ps1, Homebrew or the container image (install), or build the tag: cargo install --locked --git https://github.com/alexnodeland/stretto --tag v0.1.0 stretto-proxy stretto-report. It is not on crates.io: it depends on fugue's program feature, which is not in a fugue release yet.
  • The shipped arbiters stay in data/arbiters/; take them from a checkout or from the tag.
  • A version tag, or the release workflow run by hand, makes a GitHub release with that version's section of this file as its notes (releasing).

Development log before 0.1.0 ​

Every change made before the first release, newest first, as it was logged.

  • Install and first run.

    • Each release now carries:
      • archives for five targets (Linux x86_64 and aarch64, macOS on Apple silicon and Intel, Windows x64), with SHA256SUMS;
      • install.sh and install.ps1, which check the archive's checksum before installing;
      • a Homebrew formula.
    • A container image, ghcr.io/alexnodeland/stretto, is published on each version tag, for amd64 and arm64. It runs as a non-root user, with /data as its home.
    • stretto init --host claude-code|claude-desktop|cursor|vscode -- <server…> writes an MCP host's configuration for a server behind the proxy, and prints the next steps.
    • stretto doctor checks the installation: the binaries on PATH, ~/.stretto, whether a key is set (never its value), and the flows and sessions. stretto completions <shell> prints shell completions.
    • examples/quickstart/ records, learns, reviews and serves a flow on stretto-mcp-demo, with no key, in under a second, and runs in CI. docs/install.md and docs/releasing.md cover installing and cutting a release.
    • Building from source needs Rust 1.88.
    • stretto runs on a thread with an 8 MB stack. On Windows, whose main thread has 1 MB, a debug build overflowed it parsing the command line. CI now builds on macOS and Windows and runs each binary's --version.

  • Live runs on other benchmarks (pilot/bench/).

    • bench_mcp.py serves an AgentDojo or BFCL task's tools over MCP, running each call with the benchmark's own code.
    • run_bench_episode.py runs one episode in Claude Code and scores it with the benchmark's own check. It stops with a clear error when the task's server never started.
    • run_bench_paired.py runs every held-out task in both arms, resumably. It runs every task's first arm and then every task's second, so that two arms sharing a prompt prefix do not share the prompt cache.
    • run_bench_paired.py --split train runs the training tasks, to record an agent's own sessions for stretto promote.
    • live_to_tau2.py writes a live run's episodes as τ²-bench results, so the same episodes can be replayed from the record. Replayed there, the no-flow episodes projected 11 saved turns, against 10 saved live where the agent used a lookup. They found 3 of the 52 live detours, because the record cannot follow a chain through tools the agent never called (pilot/trace_env.py now says so).
    • analyze_bench.py pairs the arms by task, each arm known by its label, so a promoted flow can run as an arm of its own. It reports:
      • bootstrap intervals over tasks, with a task's pairs for all models drawn together;
      • an exact sign test;
      • rows pooled over models;
      • named sets of domains (--set);
      • the pairs in which the flow made no lookup, which measure run-to-run variation.

    Results for GLM-5.3 and Claude Haiku 4.5, 244 episodes (results):

    • In AgentDojo's Slack and travel suites, 10.1% fewer LLM turns (5.8% to 14.0%), with passes unchanged.
    • Over all of AgentDojo, 6.0% fewer, where the replay projected 7.2%.
    • No effect in BFCL, whose replay projected 1.6%.
    • Promotion, live on Slack: scored on each agent's own training sessions, it removed every detour and the saved turns with them. One site served both kinds of task, and only the request tells them apart. It also kept lookups of calls the agent had already asked for in the same turn (#40).

  • A flow without an arbiter decides with reach by default, the chance of each lookup before the agent's next write.

    • stretto-proxy serves such a flow with reach unless --flow-decider names another. Before, it refused one without --flow-decider habit.
    • stretto promote scores it with reach too, and stretto init writes --flow-decider reach and passes --decider reach to the promote step it prints.
    • stretto flow-show and flow-diff show what it does at each site with reach.
    • A flow learned before flows held reach's counts is served with the habit, as before. stretto audit still scores a flow without an arbiter with the habit, the model of the agent's next step.
    • The quickstart and the walkthrough serve with the default, and save as many calls as with the habit.
    • With reach, a hand-back now names the likeliest lookup under the threshold and its probability, instead of "no lookup to make".

  • stretto audit and stretto flow-show name the reach decider when a flow is served with it, instead of the arbiter.

  • The documentation site (website/, VitePress), deployed to GitHub Pages: a guide, integrations, the reference, the research and the community pages. The research notebook moves to /notebook/.

  • The brand kit (brand/): the mark, the wordmark and lockups, color tokens with a contrast check, messaging, the social card, an interactive explainer, and the launch and walkthrough videos with their sources. The videos are narrated, with WebVTT captions, and the explainer reads each step aloud when its sound is on. The voice is Kokoro-82M, an open text-to-speech model run locally (brand/video/narrate.py).

  • CONTRIBUTING.md, SECURITY.md, CODE_OF_CONDUCT.md, CITATION.cff, issue and pull request templates, and .editorconfig.

  • DTap-Bench's legal, finance and research domains (200, 200 and 160 tasks) are counted for their ceilings and replayed separately from the six (scripts/bench/replay.sh dtap-more, tables.py's dtap_more_by_source): reads are 39–77% of their turns and the ceiling 1.4–14.9%; in legal the use-before-write speculator saves 4.7% of turns against 3.9% with the agents' own runs (+0.8 points, 0.7–1.0) and ties with flows from all seven agents, and its scores there are better calibrated (an expected calibration error of 0.009 against 0.042). Its lead is each agent's own reading ahead, which pooled counts average away; weighing the others as a prior (the agent's own runs plus 100 of theirs) keeps it, at 5.6% of turns against 4.7%; over the domains with reads to take, the prior gains in legal and CRM and costs 0.5–0.8 points of the own flows' savings in the other four. In finance and research the two tie. Its browser and macOS domains (34 and 30 tasks) are counted only: where calls act on a screen, almost nothing is in the ceiling. dtap_to_tau2.py reads a browse (finance's browse_stock) and leaves the Claude Agent SDK's Agent tool out of reads and writes, as Task; no published label changes.

  • scripts/ceiling.py counts, per episode, whether every value of every call, writes included, came from a result, a constant or what the user wrote, with none the agent composed: 4–37% of the episodes in the benchmarks without a simulated user, and at most 8% with none from the user's words either, so compiling a procedure once there needs a model to read the request, and most often one to write. Figure 2's per-agent counts of the fifth ceiling are now the single-pick ones its summary and text report (64 rows had come from an experiment that walked the model's ranking).

  • Repeated reads, counted for every replayed agent (scripts/remade.py --results): with no write between, agents make again 0.8% of their reads on τ²-bench, 1.0% on τ-bench, 0.9% on BFCL, 1.5% on AgentDojo and 2.6% on WorkBench, most of them by a few models (Qwen3.5 Flash 17.5%, Llama 3.3 70B 6.2%), whose replayed savings are upper bounds, as gpt-oss-120b's are on DTap-Bench.

  • scripts/ceiling.py counts a fifth ceiling, words by model: a value of the user's counts as bindable when the System-One model, shown what the user wrote so far and the call's tool and argument, picks it among the spans of the user's words. --questions writes the questions, scripts/model_questions.py asks them through stretto ask and its replay cache, and --model-answers reads the picks back; every other count is unchanged. Of what the user's words add, the model takes 39% in WorkBench, 53% in BFCL, 48% in AgentDojo, 30% in DTap-Bench, 53% in τ-bench and 50% in τ²-bench, where a pattern takes 0–31%; MCPMark's composed values are rarely a span of the request (2%). The picks are published (results).

  • scripts/ceiling.py counts a fourth ceiling, words by shape: a value of the user's counts as bindable when it is the only one of its shape (an email, an id, a number or a date) in what they wrote so far, as a pattern could take it with no model. Every other count is unchanged. Of the 3–7 points the user's words would add in BFCL, AgentDojo, WorkBench and DTap-Bench, a pattern recovers 0–4% in the first three and MCPMark, and 17% in DTap-Bench.

  • Constants, opt-in (stretto learn --constants, flow format 2's bindings.constants). An argument the agent passed with one value in every call of a lookup that passed it, at least five calls and at least half of the lookup's, and that was never found in an earlier output, such as a page size, is learned as a constant, unless it is a string a user wrote with a digit or an @, such as an id or an email (of the published flows, that drops only the repository EasyR1, which Kimi K2's MCPMark GitHub users named, where the flows save nothing either way): the binding passes it as the agent did, so that the lookup is the agent's own call, and a required argument with a constant needs no source. flow-show lists them, and flow-diff flags a new or changed constant for review, since it could be one user's value that every training session shared. Without the flag, learn writes the same flow as before. scripts/bench/replay.sh gains mcpm-const and dtap-const: each model's own flows save 21 of MCPMark Notion's turns instead of 13, with 1.6 detours an episode instead of 2.6 (o3's fall from 173 to 23, since its lookups now pass the page size it always passes), 82 of DTap-Bench CRM's instead of 62, and 262 of its OS files' instead of 155, where the agents always ask for a message's body as text; elsewhere they save the same turns.

  • scripts/ceiling.py counts a third ceiling, with constants: an argument that took one value in every call of its tool the agent made, at least five, counts as bindable. Every published count is unchanged. Constants would add 5.2 points of MCPMark Notion's turns (GPT-5's and o3's page size), 2.2 of DTap-Bench's OS files', 2.0 of its CRM's, 0.8 of its customer service's, 0.1 of BFCL's, and nothing elsewhere.

  • MCPMark's Notion server (scripts/mcpmark_to_tau2.py --service notion). A tool name's hyphens become underscores, and dtap_to_tau2.py's verb rule reads Notion's POST searches and database queries as reads: an HTTP post is a write only when no read verb follows it (no label of the other benchmarks changes, and the other three servers convert byte-identically). scripts/bench/replay.sh replays Notion with the pooled flows and adds mcpm-own, each newer model with a flow from its own training runs on all four servers. The ceiling is 7.9% of Notion's turns, 22–34% for the Claude models and Gemini 2.5 Pro; the flows save 0.2% of MCPMark's turns at θ = 0.3 pooled and 0.4% with each model's own, since they learn the walk through a page's blocks but not which block comes next.

  • θ priced as billed* (scripts/costs.py --price OUT,READ,WRITE, --sweep). Prices a saved turn's output tokens and, with prompt caching, its prompt as the previous turn's prompt read from the cache plus what came after it, written; a detour's result is written once and read after. --sweep prices the reach round's threshold sweep at those costs. The default, input tokens alone, reproduces Appendix B. At Anthropic's ratios θ* is 0.29/0.07/0.11 (retail/airline/telecom), at OpenAI's 0.23/0.05/0.08; the thresholds counted in input tokens keep at least 94% of the best swept utility (97% with caching). costs.py also finds scripts/bench/replay.sh's replay folders, which name the decider.

  • The benchmarks round, rebuilt from the repository (scripts/bench/). replay.sh learns every flow and runs every replay behind docs/results/benchmarks-2026-09-27.md, set by set, from the converters' output in $WORK; a replay that finished is skipped. tables.py prints and writes that page's tables, and README.md gives the order of steps. Checked against the published rows: every table the page draws from these replays comes out the same, and priced.py's JSON is identical. Customer service's pooled flows keep 90% of its own flows' turns with 60% of their detours; the page had given the four domains' 94% and 79% as customer service's. priced.py now skips the solo replays, whose detours are not counted.

  • DTap-Bench, across harnesses (scripts/dtap_to_tau2.py). DTap-Bench's benign runs of one domain become τ²-bench results, one file per harness and model (--name renames a domain that τ²-bench also has). Each run holds its request, its turns (consecutive calls, with any message the agent sent with them) and its judged outcome. The harnesses' own calls that list or load tools (List MCP Tools, ToolSearch) are left out with their results. Results are paired with calls in the order the calls were made, since the Claude Agent SDK names some wrongly, and MCP text blocks that the OpenAI Agents SDK logs as Python reprs are unwrapped. OpenClaw's runs, which log only the reply, are skipped. A tool is a read when the first verb in its name reads (exists reads; request, a test ordered for a patient, writes); the Claude Agent SDK's own tools are neither. Six domains are replayed: customer service, CRM, telecom, travel, an operating system's files and medical, 1,417 tasks. A flow learned in other harnesses keeps 59% of the turns the agent's own saves, with 64% of its detours. Medical, half the test turns, is a control: its agents order tests and question a simulated patient, six of them make 0–8 reads in 642 runs, and both deciders make the same lookups, gpt-oss-120b's (results).

  • MCPMark, real MCP servers (scripts/mcpmark_to_tau2.py). MCPMark's logged runs (OpenAI Responses format) of one server become τ²-bench results: the assistant's message and the calls it makes before their results arrive are one turn, results pair with calls by id, and MCP text blocks are unwrapped as for DTap-Bench. --learn-from-all gives every run a reward of 1, keeping the verdict as verified, since learn fits the habit on successful runs and few of MCPMark's pass. On the filesystem, PostgreSQL and GitHub servers the read-only ceiling is 6.6% of turns and the flows save 0.3% (results).

  • Saved turns in seconds and dollars (scripts/priced.py). pilot/check_flow.py rows now name the turns a replay saved (saved_at), and priced.py prices them at each turn's recorded generation time and cost, net of detours at the agent's fitted input price. In τ²-bench retail the use-before-write speculator's saved turns are 24% of the episodes' generation time and 18% of their cost (results).

  • scripts/remade.py --results counts, in any recorded runs laid out as τ²-bench's, the reads an agent made again with no write between, as an agent that ignored a lookup's result would.

  • A threshold per decision, evaluated (scripts/per_decision.py). Prices each lookup's detour (the tool's mean result tokens times the turns left) and saved turn (the next turn's input tokens) where it is decided, and evaluates that threshold, and flat ones, off-policy over the logged decisions of a replay. At flat thresholds it reproduces the sweep's turns saved to within 5.2%. On τ²-bench it changes the counted utility by −15% in retail and +7–10% in airline and telecom; --recalibrate WEIGHT scores each lookup by its site's own rate of use, cross-fitted over halves of the tasks, which brings every domain within 5%. The served threshold stays the domain's (results). pilot/check_flow.py --explore now logs each decision's place in the episode (at) and, for each used option, the message of the call it answers (use_at).

  • A search the agent always narrows is not made bare. A lookup whose arguments the agent passes in under 90% of its calls has no required arguments, and a flow made it with none. On WorkBench, whose searches take one of several optional filters, that was every lookup the flows made: a search of all tasks or all customers, which no agent makes, 289 detours for one saved turn. learn and compile now count, for each lookup without a required argument that the agent sometimes called with one, how many of its calls passed none (bindings.bare). Such a lookup is as likely as those calls, Laplace-smoothed as a string binding's agreement is. A read the agent always calls with no argument keeps the chance 1 and is not counted. flow-show shows the chance. A flow with the count is format version 2. Flows without it replay as before.

  • Bindings pass whole lists. A lookup can take a list, such as the hotels a city's listing named, passed to a lookup of their prices. learn and compile now trace a list argument to the path of an earlier result whose values contain every value of the list (bindings.lists), and the binding passes every value at that path of the most recent such result, in order, unless the lookup already had that list. Its chance is scored as a string argument's is. flow-show lists such a source as the path's "every value". A flow with lists is format version 2, since a build that ignored them would make fewer lookups than the flow was learned to. Flows without lists replay exactly as before: GLM-5's retail replay with the paper's flow gave the same decision log, byte for byte. On AgentDojo's travel suite, flows had made none of the lookups its 47% ceiling holds (results).

  • Replay from the record (pilot/check_flow.py --trace, pilot/trace_env.py). Replays another benchmark's published trajectories with no environment. The agent's calls return their recorded results. A lookup returns the result of the agent's own later call with the same arguments, if the agent made it before its next write, which is exact. Any other lookup is a detour, gets the nearest result of the same tool as a stand-in, and never counts as a saving. On τ²-bench's paper batch, it counts 96.7% of the environment replay's saved turns, 91% of the reach decider's lead, and 55–72% of the detours (results). A domain τ²-bench does not know takes its writes from its checkout's tools.py.

  • Four more benchmarks, as τ²-bench results (and DTap-Bench, above).

    • scripts/taubench_v1_to_tau2.py: τ-bench's historical trajectories.
    • scripts/bfcl_to_tau2.py: BFCL v4's multi-turn tasks, run on BFCL's own backends as ground truth; with --runs, also the leaderboard's logged runs of any model from a BFCL-Result snapshot, scored by its checker.
    • scripts/agentdojo_to_tau2.py: AgentDojo's benign runs, with results rendered as JSON.
    • scripts/workbench_to_tau2.py: WorkBench's runs, the logged Final Answer as the reply.

    Each also writes a checkout-shaped folder: tools.py with every tool marked a read or a write (checked against the benchmark's source), and a train/test split by task. stretto learn --results, pilot/check_flow.py --trace, scripts/ceiling.py and scripts/calibration.py read them unchanged (results).

  • scripts/learning_curve.py --prior OTHER.json... learns from those other agents' training sessions together with the first n of the agent's own, as a deployment that starts from other agents' sessions and adds its own; --prior-n K takes K of them, drawn at random, so that they weigh as K sessions do (results).

  • pilot/check_flow.py splits each replay row's used lookups by when they were used: used_next before the flow's next decision (before any call of the agent's that no lookup answered), used_later after one, so that a later decision could have made the lookup in time. At 0.3, 94% of the reach decider's lookups that paid in retail, airline and telecom were used before its next decision (results).

  • scripts/remade.py checks the replay's assumption on live episodes: whether the agent made a flow's lookup again before the next write. On the reach arm's 28 live episodes, GLM-5.3 made none of the flow's 101 lookups again (results).

  • stretto flow-show says when a flow carries the reach counts, which --decider reach serves.

  • scripts/costs.py counts a detour's cost and a saved turn's value in recorded episodes, in each agent's own input tokens, and so the threshold a read-only flow should use, θ* = δ/(β+δ). On τ²-bench's leaderboard episodes it gives 0.30 in retail, what the live paired run measured, and 0.12–0.13 in airline and telecom, whose contexts are longer and results shorter (results).

  • scripts/learning_curve.py measures how a flow's savings grow with the sessions it learns from. For each seed it orders an agent's training sessions at random, learns a habit-only flow from the first n (or, with --pool, from other agents' sessions), and replays it on the agent's own test episodes at --decider and --threshold: a row per n and seed (results).

  • scripts/paper_figures.py draws the working paper's figures, the reliability diagrams and the learning curves, from the round's published rows (docs/results/reach-2026-09-26.json).

  • Compiled procedures in stretto (#38). A whole workflow compiled once from agents' traces, writes included, for where no user speaks, is now a file stretto runs: the procedure IR (stretto_procedure: 1, formats), per site a decision tree over the run's state, calls whose identifiers are held as where they came from and bound again at run time, a guard against repeats, and the checks of each outcome a ticket may state. scripts/telecom_workflow.py --export writes one; stretto-procedure --procedure FILE --ticket TEXT -- SERVER… runs it against an MCP server with no model and prints the run: its calls, its check of the ticket's outcome, and its verdict (resolved, transferred, or hand_back when the check failed and the ticket should go to a model). stretto_report::procedure is the runtime. pilot/run_procedure.py runs it on τ²-bench telecom's 40 held-out solo tasks over tau2_mcp.py and scores each with τ²-bench's evaluator: the published procedure (docs/results/telecom-workflow-2026-09-26.procedure.json) made the same calls as the script on all 40, passed the same 35, and handed back the same 4.

  • Live solo runs and the cascade. pilot/run_episode.py --solo runs τ²-bench's no-user mode live (telecom): no customer, τ²-bench's solo system prompt with the ticket, the phone's tools and τ²-bench's done tool served by tau2_mcp.py --solo, and the episode scored as τ²-bench scores solo runs; as in τ²-bench, a turn that ends with neither done nor a transfer is an agent error. --handback RUNS.json makes the agent take over from the compiled workflow (scripts/telecom_workflow.py --json): tau2_mcp.py --prefix makes the workflow's calls first, with their effects in place, and the agent is told what they returned and that the workflow's check of the ticket's outcome failed. --arm reach serves a flow with --decider reach. The harness now stops an episode whose MCP server failed to start, rather than letting the agent run without its tools.

  • The replay counts only what a lookup answered. pilot/check_flow.py skipped any recorded call already made, the agent's own repeated calls included, and so counted as saved the 0.2–0.9% of LLM turns (3.3% for one agent) in which an agent repeated calls it had made. It now skips a recorded call only when a flow lookup answers it: the same call, a lookup no recorded call has used yet, that returned what the recorded call returned (with no write since, the agent's or the customer's, it did). --legacy replays with the old rule, which reproduces the published numbers. --explore 0 labels each option used only when the agent makes the call before its next write, as the rule has it, and also next when it is the agent's very next call.

  • scripts/ceiling.py classes every LLM turn of τ²-bench's recorded test episodes as a reply, reads or writes, each after the customer or after a tool, and counts the turns any read-only speculator could save: every call a read, with its arguments in earlier results, after a tool response since the last write. scripts/replay_stats.py gives replays' turns saved, detours and share of that ceiling with 95% intervals from a bootstrap over tasks, and scripts/calibration.py how calibrated a flow's scores were against the used labels (binned, with the expected calibration error, the Brier score and the AUC); with --versus, against the same episodes replayed under another decider, with 95% intervals for the differences from a bootstrap over tasks.

  • Deciding on the chance of use before the next write (--decider reach). A lookup pays off if the agent makes the call at any point before its next write, not only next, since a read's result stays current until then. learn and compile now also count, over the habit's histories, how often each action came before the agent's next write (reach in the flow IR; each action its own Beta–Bernoulli, backed off as the habit is), and serve, flow-serve, audit, promote and stretto-proxy --flow-decider take reach: of the lookups that bind, the one whose chance times its binding's is highest, when that clears the threshold. A flow learned before has no counts and refuses the decider; a build that does not know the field serves the flow as before, so the format version stays. On GLM-5's retail test episodes, flows learned from the 2025 runs scored lookups with a calibration error of 0.10 against the habit's next-step probability's 0.19, and at the threshold of 0.3 saved 358 turns with 55 detours against 307 with 33.

  • Counterfactual evaluation (RFC-001 §3.7, #15). stretto-proxy --flow-explore EPSILON, and serve --explore and flow-serve --explore, explore at read-only sites: with that probability the flow takes a lookup other than its rule's choice, drawn by the decider's probabilities among the lookups that bind. Each decision then logs its policy: every option and the chance that the flow took what it took. stretto evaluate reads such decisions, with each option's outcome, and estimates what another rule (the arbiter's or the habit's probabilities at a threshold) would have done, per site and in total: directly from every option's outcome, and by IPS, self-normalized IPS and a doubly robust estimate from the taken option's alone, with each site's effective sample size. pilot/check_flow.py --explore writes the labelled decisions from replays.

  • Predicate refinement (RFC-001 §3.4, #16). phase0 --candidates FILE asks candidate predicates alone at every asked next-step decision, with the same state as the next-step question, so every other answer keeps its cache key. Their answers go to --oracle-log, and --weigh ID weighs one in the arbiter. stretto refine reads such a log, fits the arbiter by cross-validation over tasks with each candidate added, and keeps candidates greedily while one raises the held-out log-likelihood of the agents' steps by more than a penalty (by default what BIC charges a parameter). A transfer target's decisions, judged but never fitted on, are scored alongside as an out-of-sample check. --examples writes examples from the sites where the arbiter is weakest, for whoever proposes the candidates.

  • A predicate can bear on one named lookup: "favors": {"lookup": TOOL} (formats).

  • Flow search (RFC-001 §3.10, #25). stretto search runs NSGA-II, from fugue-evo, over each site's threshold (0.10 to 0.95, or off) and the decider (the arbiter, or the habit alone). It scores each setting by a replay command that prints pilot/check_flow.py's CHECK totals, and reports the settings that save the most turns for the fewest detours, starting from the hand-set ones. It counts the decisions a replay could not answer, and says so when a setting on the front has any. --rescore replays an earlier search's front on other episodes, such as held-out ones.

  • Per-site thresholds in the flow IR (thresholds): at a listed site a flow acts on that threshold in place of the served one, and above 1 it never acts there. A flow with them is format version 2, which 0.1.0 refuses rather than ignores; this build reads versions 1 and 2. flow-show and flow-diff show them. flow-diff lists a changed threshold without flagging it, as it would the served one, and flags a site switched back on.

  • stretto-proxy --flow-tools TOOLS names the only tools a flow may call on its own. A read-only tool can still be metered, rate-limited or recorded as an access; without the option, a flow may call every tool it reads as a lookup, as before.

  • Pinned inputs. A flow learned from recorded sessions pins each tool's input contract (contracts): its arguments, their JSON types and which are required, as the server listed them. stretto-proxy makes no lookup of a tool whose server now lists another, and says so once. flow-show lists them and flow-diff shows a change.

  • A record the customer did not ask about. learn and compile count, per lookup, the picks the binding would make where the customer had mentioned a value at its sources and the binding would pass another, and how many of them the agent went on to pass (bindings.named_other). The binding's chance there is that rate, shrunk towards its unmentioned chance. flow-show and flow-diff show it. A flow with the count is format version 2. Replayed on GLM-5's test episodes, airline detours fell from 56 to 4 at no cost in turns (results).

  • scripts/proposal_check.py checks each write of τ²-bench episodes against what the agent proposed, with no model: it flags a write when the confirmation chose another record of the list its value came from (results).

  • scripts/anatomy.py classes every LLM turn, every decision after a tool returns and every argument of τ²-bench episodes by where its information came from: copied from an earlier result (one value at its path, or one of several), the customer's words, or made up, after TraceCompiler's binding classes; and how well the previous tool, structured features of the results and the goal predict each decision (results).

  • scripts/anatomy.py reads telecom too: τ²-bench's read-only tools for the agent and the phone, features from results in text (each Key: value line's value, or its |-separated parts), and results without ids paired with their calls in order. It also predicts each decision with a decision tree over the whole state (every tool's last result and the tools called), its depth chosen by cross-validation, as decision mining does at a process's branch points (results).

  • Sources in the order the agent used them at the site. learn and compile count, for each lookup argument with more than one source, where the agent took its values at each site, the tool whose result came last (bindings.site_sources), each read of a batch at the site of the read before it, as a flow makes them. Where a site has enough of them, the binding tries its sources in that order before recency: after reading a phone line, the next line id, not the line's plan id. Replayed on the nine leaderboard agents' telecom test episodes, a flow learned from τ²-bench's 2025 runs saved 12.8% of turns with 443 detours, against 1.9% with 2,936; retail and airline replay exactly as before on all nine agents (results). pilot/check_flow.py replays telecom: the customer's calls on their own phone are the customer's turn. A flow with them is format version 2.

  • The record the customer described, already read. learn and compile count, per lookup, the picks where another value of the same list was already looked up and returned a value the customer gave, which another's result did not (the line with the ticket's phone number), and how many of them the agent went on to pass (bindings.described_read). A value the customer gave has a digit and at least four characters, and the customer wrote it or the agent passed it before any result held it. The binding's chance there is scored as with named_other. In τ²-bench telecom's solo mode the agents almost never read another line after the one with the ticket's number (7 of 297), and the flow's detours fell from 387 and 430 to 203 and 231 with more turns saved (799 and 647, from 776 and 633); all nine leaderboard agents replay retail and airline exactly as before (results). flow-show and flow-diff show it. A flow with the count is format version 2.

  • flow-show lists each lookup argument's sources by site, in the order the binding tries them, and flow-diff lists a change in the source it tries first, without flagging it.

  • learn --results takes τ²-bench's solo runs (no-user), where the agent calls the customer's tools itself: those tools join the flow's, read-only or not as τ²-bench marks them. Only calls that returned count, since an agent that calls the customer's tools when the customer holds the phone is told they do not exist (Claude 3.7 Sonnet did 130 times in the 2025 dual-control runs, which had made a flow learned from them offer the phone's tools). pilot/check_flow.py --solo and pilot/tau2_mcp.py --solo serve them. In telecom's solo mode a habit-only flow saved 26.9–32.5% of LLM turns in replay, and 27.5–33.5% with bindings.described_read (results).

  • stretto-proxy --confirm-judge also logs, with each judgment, the proposal check (proposal_check): the call's values whose record the confirmation chose another of, with no model. Only values with a digit are checked, as ids carry them; on τ²-bench's 2025 runs that flags exactly what scripts/proposal_check.py's closed-choice rule flagged. It refuses nothing.

  • scripts/telecom_workflow.py runs a workflow compiled once from τ²-bench telecom's solo runs (a decision tree per site over the whole state, its steps whole calls) with no model in τ²-bench's environment, and scores it with τ²-bench's evaluator; --sure hands back where the tree is not sure, --no-guard lets it repeat itself, and --train-share learns from fewer tasks (results). --self-train ROUNDS then learns from its own runs, as expert iteration does: each round it tries every training ticket once as it stands and --rollouts times drawing from its leaves, keeps the shortest run its own check of the ticket's criterion says resolved, and is refitted (results); --bag N averages N trees fitted on resamples. --symbolic learns each identifier a call passes as where it came from (a result's path, with the fields that set its record apart in a list, or the ticket) and binds it again from the run's results, preferring the record that holds the ticket's value, and keeps a tool's state on the record the ticket names; --rename runs the test tasks for a customer no trace saw, and --as-customer ID for another customer of the database, keeping the tasks its gold actions still solve and whose faults the account still shows (τ²-bench's flags for the line: active, roaming allowed, data used up). Renamed, the workflow passed the same 35 of 40 as for the original customer, against 16 with identifiers as constants; moved to Michael Lee (one line, not three), 35 against 19 (results). anatomy.tree(counts=True) returns a leaf's calls by count.

  • scripts/detours.py tools lists each lookup tool's detours and uses over tagged replays, by the kind of task, with how many detours the agent made with other arguments (results).

  • scripts/detours.py says where a replayed flow's detours come from: a session hand-back simulated on the replay, each lookup's place in its chain against its probabilities and use, and, in airline, reads after a reservation the customer had described (results).

  • pilot/check_flow.py --same-result also counts a recorded call as made when a flow lookup of the same tool, agreeing on every argument both pass, already returned its recorded result, as an optional argument that changes nothing does (a limit above the number of bills), and counts that lookup as used, not as a detour. Each such pair is written to the episode's same-result.json. Scoring by exact arguments stays the default (results).

  • pilot/check_flow.py names each recorded episode it replays after its folder, or, when two folders share a name (a task's trials, each in its own folder), after its path below their common parent. Such episodes shared one replay folder before, and with --jobs one trajectory, which the flow reads; the published replays were of τ²-bench results files, whose episodes are named by task and trial, and are unaffected.

Released under the MIT License.

Copyright © 2026 Alex Nodeland