Roadmap
Every piece of the first design is built: the flow compiler, the MCP proxy, the policy guards, the audit, the confirmation judge and the shipped arbiters (implementation status). The evidence so far:
- offline replays on τ²-bench's published trajectories;
- live pilots of ten paired tasks per domain, with GLM-5.3 in Claude Code as the agent;
- live runs with Claude models as the agent and as the customer, on ten retail tasks and fewer (results);
- a paired run on every retail and airline test task: 80 pairs, with 25.5% fewer LLM turns and the pass rate within −7.5 to +6.25 points (results).
- a flow deciding on the chance of use before the next write, with no model, on 28 of those tasks: 27.9% fewer LLM turns (19.1% to 35.9%) (results).
- the same flow and tasks with Claude Sonnet 5 and Claude Haiku 4.5, three trials, pre-registered: 20.5% and 22.4% fewer LLM turns, where Anthropic's prompt for parallel tool calls saved 3.4% and 5.9% (results).
- the same flow live in AgentDojo's and BFCL's own environments, with GLM-5.3 and Claude Haiku 4.5: 10.1% fewer LLM turns (5.8% to 14.0%) in the AgentDojo suites where the replay found reads to take, with passes unchanged, and no effect in BFCL, as its replay projected (results).
- a smaller agent,
glm-5.3-flash, served a flow learned from three frontier models' traces with the policy guards: 20.8% fewer LLM turns on the pilots' twenty tasks, with passes unchanged (results). - the cold start on three more agents, twenty draws each: one five-session start in eight is narrow for the habit alone, and a shipped arbiter lifts most of them (results).
The work is tracked in GitHub issues, all of them sub-issues of #1. This page groups them and says why each mattered. Every one is done except the first release (#30), which waits on the owner's tag. The changes RFC-001 §3.9 asks of fugue itself are tracked in fugue's #61.
Evidence from live runs
Each of these spends the Z.ai coding-plan key, or Claude tokens where a Claude model plays a part, so each needs an approved budget before it starts. A pilot episode has cost about 12 credits in retail and 17 in airline.
| Issue | What | Why | Cost |
|---|---|---|---|
| #2 | A paired run on every test task | Done: 80 pairs. The flow saved 25.5% of LLM turns (95% interval 20.5% to 30.4%), and the pass rate moved −1.25 points (−7.5 to +6.25) (results). A one-point bound would take about 4,300 pairs | 1,897 credits |
| #3 | The cold start live: the habit alone against the habit with a shipped arbiter | Done in both domains. In retail, the habit from five sessions took 79 LLM turns against 110 with no flow, and with the shipped airline arbiter 70, with fewer detours (results). In airline, both took 102 against 127, and the shipped retail arbiter cut the habit's detours from 8 to 1 (results) | 205 credits for retail, 332 for airline |
| #4 | A simulated customer that is not the agent's own model | Done: with Claude Sonnet 5 as the customer to GLM-5.3, on five retail tasks, the flow saved 23.1% of turns, against 25.8% with GLM-5.3 as the customer, and made the same lookups on four (results) | 110,016 Claude tokens and 112 credits |
| #5 | More agent models live | Done for two Claude models. On the retail pilot's ten tasks, D0 saved Claude Haiku 4.5 18.6% of turns, less than GLM-5.3's 23.6%, as its more parallel calling predicts; on three tasks it saved Claude Sonnet 5 24.1% (results). Qwen3.5, the strongest candidate offline, waits on a key | 2.15M Claude tokens and 66 credits |
| #6 | The confirmation judge enforced | Done: enforced on 20 episodes, it refused none of 40 writes; logged on 20 more, it would have stopped 2 real lapses for 2 false alarms, and passes did not move. The recommended setting enforces the first question and logs the second (results) | 782 credits |
| #7 | stretto_commit live | Done offline: it could save at most 0.2–3.5% of LLM turns, too little for a ten-task pilot to see, so no live run for now (results) | Free |
| #8 | Flows and guards from frontier traces, serving a smaller agent | Done: glm-5.3-flash, served a flow learned from Claude Opus 4.5's, Claude Sonnet 4.5's and Gemini 3 Pro's training episodes with the guards (arm E), took 20.8% fewer LLM turns (11.0% to 29.9%) on the pilots' ten tasks per domain, 33.3% in retail and 10.1% in airline. Passes held at 18 of 20, and the guards refused one write, which the tool refuses too: alone, this model passed more of these tasks than GLM-5.3 had. The first attempt found that guards beside a flow the server runs must read the lookups it appends; they now do (results) | 629 credits, and 137 for the first attempt and smoke episodes |
| #34 | The habit with the named-other count live, paired against D0 (in place of the searched flows) | Done: on the pilots' ten tasks per domain, the habit with the count, asking no model, took 80 retail turns to D0's 78 and 95 airline turns to D0's 118, against 110 and 127 without a flow, and passed 9 of 10 in each domain (D0 9 and 7). Both made almost no detours live (2 and 0; D0 1 and 1), so the detours the count removes in replay were not there to remove results | 554 credits |
Evidence from offline runs
These cost Jev dollars, CPU time or people's time, and no LLM runs.
| Issue | What | Why | Cost |
|---|---|---|---|
| #9 | The cold start on other agents, with more draws | Done: on Qwen3.5-397B, Claude Sonnet 4.5 and Claude Opus 4.5, twenty draws of 5 and of 10 sessions per agent and domain. From five sessions the habit alone saved under half of what every training task gives in 16 of 120 draws, and from ten in 1. A shipped arbiter lifted 13 of the 17 narrow draws above half, at no more than 3.3 points of the median; the draw's own arbiter was below half in 27 of 120 (results) | $4.36 of Jev's answers; 726 replays |
| #10 | Telecom, and an arbiter fitted on retail and airline together | Done: telecom compresses like the others, but neither shipped arbiter nor one fitted on both carries to it; the habit alone does better there (results). Replayed, the habit-only flow saves 12.8% of the nine leaderboard agents' telecom turns once bindings order their sources by site (results) | $0.48 What is left there: the bills, which four agents read too but with a limit that returns the same bills (counted as used, 298 detours are left, from 443), and data usage in MMS tickets, an agent's habit (what is left). The habit already sees the line's status; a back-off to it alone changes nothing. Next: whether an agent that passes a limit repeats such a read live, which needs a live run with one of those agents |
| #11 | Independent labels for the confirmation judge | Done, with two model annotators, not people: the first question's tallies hold (Jev right on 30 of 40 by the majority of three); the second question's lapses are 11 of 30 by the majority, not 16, the difference being addresses the agent offered to update but never read back, and 19 under the stricter reading (confirm-second) | None |
| #12 | Score the pilots as τ²-bench does | Done: with every check in τ²-bench's reward basis, the natural-language assertions judged three times by Claude Haiku 4.5, 18 of 1,101 published episodes lose their pass, 17 on retail task 68, as often with a flow as without; no comparison changes direction, and the paired run's 80 pairs pass 70 in each arm. run_episode.py records both rewards (results) | None: the judge ran on the Claude subscription |
| #13 | Prompt injection against flows and the confirmation judge | Done: a flow stays inside its compiled lookups, but injected text steers the choice among them; one sentence in the questions and a higher bar absorb most of it (results) | $0.22 |
Features from the design that are not built
| Issue | What | Where the design asks for it |
|---|---|---|
| #14 | The confirmation judge in the proxy's guards, logged or enforced | Done: stretto-proxy --confirm-judge log|enforce |
| #15 | Counterfactual evaluation from logged decisions | Done: --flow-explore and stretto evaluate. On replays, the estimates match a rule's own replay for changes that keep a flow's chains, and miss a changed decider's detours (results) |
| #16 | Predicate refinement | Done: phase0 --candidates and stretto refine. One round in airline kept two of eight candidates, which fitted the agents the proposer read but not GLM-5, and did not move the savings; the three hand-written predicates stay (results) |
| #17 | Hand back when a session surprises the flow | Done: learn --surprise Q gives a flow a gate that hands back for the rest of a session once the agent's steps in a row surprise its habit more than the Q-quantile of training sessions do, and serve, flow-serve and the proxy serve it, or not (--surprise, --flow-surprise). Replayed on two agents in two domains, it rarely had anything left to stop, since the flow makes its lookups before a session surprises it, and no setting paid in more than one of the four, so the default is off (results) |
| #18 | Drift alarms | Done: stretto drift and learn --half-life. On τ²-bench, the alarm found agents that fit a flow clearly worse, or called tools it never saw, 7.5 to 10 sessions in, and sounded on 5 of 160 orders of one agent's own sessions (results). The proxy says why it leaves a tool whose listing changed to the agent |
| #19 | Shadow mode and per-site promotion | Done: stretto-proxy --flow-shadow and stretto promote; replayed, promotion halved D0's detours for 6% of its retail savings (results) |
| #20 | stretto flow show and flow diff | Done: stretto flow-show and stretto flow-diff (reviewing flows) |
| #21 | Streamable HTTP for the proxy | Done: stretto-proxy --upstream URL |
| #22 | Learn from several servers' logs of one session | Done: the proxies of one host share a host session (STRETTO_SESSION, else the host's process) and name their server (--server-name); learn, promote, audit and drift merge a host session's logs into one session, each tool named server::tool, and each proxy serves the flow's lookups on its own server (several servers). Lookups across servers would need the proxies to share a connection |
| #23 | Learn from OpenTelemetry GenAI spans | Done: stretto_trace::otel reads OTLP JSON exports, each trace a session, under the conventions of semantic-conventions 1.38.0, which a flow records; learn, audit, promote and drift take --otel, with a manifest for the tool kinds. Pydantic AI's export is the example (OpenTelemetry spans). Without the opt-in tool content, a flow learns the lookups but binds none |
| #24 | Privacy for recorded sessions | Done: an inventory, stretto redact and stretto-proxy --retain-days. Leaving fields out of the questions' state is still open |
| #25 | Flow search with fugue-evo | Done: stretto search and per-site thresholds in the flow IR. Replayed on held-out tasks, its front held flows with far fewer detours than D0 in both domains, for 7 more turns saved in airline and one fewer in retail, each by changing one or two sites' thresholds (results). They have not run live . The named-other count reaches the same gains from the traces, with no search (results) |
| #35 | Staged learning: update flows as sessions arrive, evaluate in the background, commit explicitly | Done: stretto stage learns the staged flow beside the committed one, scoring each new session with both flows before it learns from it; flow-commit, flow-rollback and flow-log keep every committed version with the evidence it rested on (staged flows), and the console's Staged tab shows the comparison, commits and rolls back (#36). Replayed along GLM-5's retail stream, committing the staged flow every 20 sessions found 21% more of the agent's lookups for 2.2 times the detours, and committing only when it netted more took most of the gain; in airline, and after a new agent took over, relearning gained nothing (results) |
| #36 | A web app: a sanctioned container and a UI for flows, servers and commits | Done: the console, stretto-console, with its container image and an API under the UI. It covers the servers behind the proxy with their host configuration and a connection test, each session call by call, each flow's review, graph and diff, its staged flow with commit and rollback (#35), the CLI's jobs, each server's guards, confirmation judge, stretto_commit, retention and surprise gate, the sessions that tripped a flow's gate (#17), the drift alarm (#18), and the tools each server's flow may call (--flow-tools). Hosts that connect over Streamable HTTP moved to #80 |
| #80 | stretto-proxy serves hosts over Streamable HTTP | Done: stretto-proxy --listen serves hosts that connect by URL, with a server, a log and a flow for each MCP session, behind a token or on loopback alone (the proxy); stretto init --listen and the console's servers give each host its URL |
| #37 | Macros that write: flows that prepare a write sequence for one confirmation | Done, and not built. On WorkBench, a write-heavy benchmark, bundling writes could save at most 2.0% of LLM turns, since most models already make a task's writes in one turn. The agents that write one call per turn (o3: 12.0%) get stretto_commit and a note on it in the server's instructions, with nothing to learn (results) |
| #38 | Compiled procedures for agents that work alone, handing back on the outcome | In telecom's solo mode a workflow compiled once from traces passed as many held-out tasks as the agents, with no model, and its outcome check caught 8 of its 9 failures (results). With that check as the verifier, its own tries in τ²-bench's environment took a workflow fitted on half the demonstrations from 18 to 22–23 of 40 (learning from its own runs). Learned as where they came from, its identifiers bound for a customer no trace saw: renamed throughout τ²-bench's database and tasks, it passed the same 35 of 40 as for the original, against 16 with constants, and 35 against 19 moved to another customer's account, with one line instead of three; on a five-line account it matched the original customer task for task, 14 of the 16 tasks whose faults survive the move (a customer no trace saw). The procedure is now a file stretto runs (stretto-procedure, the procedure IR), matching the script on all 40 held-out tasks, and live, GLM-5.3 on its 4 hand-backs took the cascade to 39 of 40 at 0.48 LLM turns per ticket (results). Done: on τ²-bench's 2,171 telecom tickets whose faults no base task combines, a second solo workload since τ²-bench has solo mode only in telecom, the procedure passed 84.3% with no model, the same tickets for a customer renamed throughout, and handed back 334, all failures; Claude Haiku 4.5 resolved 19 of 20 hand-backs, so the cascade projects to 99.0% at 0.80 LLM turns per ticket, where Haiku alone passed 8 of 20 (results) |
| #39 | Stop reading once the described record is found | What is left of the detours on an agent a flow never trained on is a semantic stop (results): a question at the sites that iterate, scored per agent. Where the description is an exact value (a ticket's phone number), the binding now scores it with no model (bindings.described_read), which halved the solo telecom flow's detours (results). Done, not shipped for airline: its cities can be read with no model, as aliases from the server's airport table, but no agent's own flow learns the stop from its training episodes, and a pooled one costs GLM-5 turns (results) |
Fixes and tooling
| Issue | What |
|---|---|
| #26 | Done: learn and phase0 refuse options they would ignore, such as --manifest-options with --habit-only or --arbiter-from |
| #27 | Done: run_episode.py --read-only-hints marks τ²-bench's read-only tools in tools/list, so pilot recordings learn without --manifest |
| #28 | Done: check_flow.py --in-process --jobs N makes the tools' calls in one process and replays in parallel; replay_study.py runs a resumable list of replays |
| #29 | Done: the pilots' recorded episodes are published |
| #30 | A first release, 0.1.0. Ready: the release workflow builds five targets with checksums, install.sh, install.ps1, a Homebrew formula and the image ghcr.io/alexnodeland/stretto, and a manual dry run checks them (releasing). It waits on the tag |
| #40 | Done: a call within half a second of its turn's last response joins the turn unless it needs what the turn returned, which split 20 of 210 multi-call LLM turns in Claude Code's own record, against 122 before. The proxy and promote leave the calls still on their way to the agent, and promotion now keeps none of the Slack sites whose lookups the agents had already asked for (results) |
Documentation
| Issue | What |
|---|---|
| #31 | Done: the flow IR and arbiter file formats (formats), with a test that every field is named |
| #32 | Done: the CLI reference, generated from the code and kept current by a test (cli) |
| #33 | Done: the walkthrough on the official MCP filesystem server, run in CI (walkthrough), and recorded as a video. The documentation site, with a guide, integrations, the reference and the research, is at stretto.alexnodeland.com |
| #74 | Done: the paper's PDF, built from the Markdown in paper/latex/, is served with the site at /paper/stretto.pdf and linked beside the paper wherever the site and the README link it |
In fugue
RFC-001 §3.9 asks six additive changes of fugue-ppl. They are tracked in fugue#61:
- async interpretation (#62);
- site metadata for handlers (#63);
- a delegating handler adapter (#64);
- a serializable program format (#65);
- vector-valued sites (#66);
- a how-to for sample-or-observe sites (#67).
All six are built. A live flow now runs as a fugue program: the flow IR holds its run in fugue's program format, and the proxy runs it with run_async. stretto depends on fugue by git revision until fugue's next release.
A suggested order
Every step below is done. What is left is the release (#30).
- What cannot wait. Publish the pilots' episodes (#29) while the recordings exist, done.
- What costs nothing and unblocks the rest:
- the bug (#26), done;
- the pilot's read-only hints (#27), done;
- faster replays (#28), done;
- the judge in the proxy (#14), done.
- Offline evidence, cheapest first:
- airline's communication check (#12), done;
- prompt injection (#13), done;
- telecom (#10), done;
- the cold start's other agents and draws (#9), done;
- independent labels (#11), done.
- Live runs, as budgets are approved. The paired run (#2), the cold start live in both domains (#3), Claude models as the agent and the customer (#4, #5) and the confirmation judge enforced (#6) are done. The paired run puts the pass-rate change between −7.5 and +6.25 points, not within one (−7.5 to +7.5 with every check in τ²-bench's reward basis, #12). The habit with the named-other count live against D0 (#34) and flows and guards serving a smaller agent (#8) are done too.
- Phase 3 features as the evidence calls for them:
- counterfactual evaluation (#15), done;
- predicate refinement (#16), done;
- flow search (#25), done.