Fabian
Weber
Back to the index

An agent fleet tuning a live trading engine: what it found, and what broke

19 October 2026

My trading engine is written in Rust and trades real money on Hyperliquid. Exactly one human works on it. Since July a fleet of Claude agents has done most of the work. They propose strategy changes and test them, fix the bugs they trip over, and open PRs for me to review.

Agent posts usually come in two kinds: “it’s magic” or “it’s useless”. Mine is neither, so this is the boring version with numbers, and with the things that went wrong.

The setup

About nine agents run on cron or in loops, and each one owns a single question. One watches data quality, one compares live fills against the backtest, one proposes strategy changes, one audits the other agents’ checks, and so on. They don’t talk to each other directly. They post findings to a shared bus that drops repeats. The actual work happens in pull requests.

Each experiment lives in its own git worktree and ends as its own PR. Deploys only go through CI from main. A separate watchdog process stops the engine if the day’s loss passes a threshold. I review and merge everything myself.

What the optimizer found

The optimizer agent writes every hypothesis into a ledger before it tests it. Across the strategy books that ledger now has about 500 entries. Roughly 390 were killed by the gates and around 70 survived. About 25 of those became merged changes to the live config.

The gates are strict on purpose. Every test runs on the “honest” engine, which charges slippage on stops and exits (plus funding and market impact on perps), over about 21 months of 5-minute data. A change has to win on both halves of that window, beat a bootstrap test, clear a significance bar that gets stricter as the ledger grows, and hold up on pre-bull data.

Three of the changes that made it through, all backtest numbers:

  • A risk gate that was secretly broken. One entry filter had never actually run in any backtest, because the data it needed got excluded when jobs were synced to cloud machines. The first valid test showed it was blocking the best momentum trades. Removing it cut the combined book’s max drawdown from 43% to 27% and doubled its Sharpe (0.9 to 1.8).
  • Shorter holds on perps. The agent found that after four hours the average trade had no drift left, so holding longer only added variance. Capping holds at 4h instead of 8h dropped max drawdown from 46% to 31% at the same Sharpe.
  • Less correlation. Allowing fewer highly correlated positions at once lowered the combined book’s drawdown from 41% to 33% and lifted Sharpe from 1.5 to 1.8.

A fair caveat: these are backtest results. Live trading on accounts this small doesn’t produce enough trades per deploy to confirm any single change yet. The fleet has a separate agent that scores each deploy’s prediction against live, and it says so plainly.

The engineering numbers

  • Agents opened 249 of the 261 PRs on the repo since July.
  • Median PR: 144 changed lines across 3 files, merged about 3 hours after opening.
  • Roughly 11% of merged PRs needed a later forward fix, with a median of about 6 hours to that fix. (This comes from a heuristic scan, so read it as “roughly”.)
  • Model spend was about $50-80 per merged PR at API list prices. That’s an upper bound, because most of the money went to research loops whose ideas got rejected.

What broke

1. Right locally, wrong globally

An agent fixed a real problem: one strategy family wasn’t trading at all. It did that by changing a volatility floor that turned out to be shared. The fix was correct for that family. For another strategy it quietly loosened a calibrated filter, and the live book took more than twice as many low-quality entries as it should have. It took a week before an optimizer pass caught it.

This was the pattern I saw most. The agent validates its diff. Nobody validates the blast radius. To be fair, I’ve watched human teams do exactly the same.

The fix: when a shared gate changes meaning, every strategy certified against it gets re-run on the new engine, and sweeps only compare legs from the same batch.

2. The simulator lied

Partial take-profit existed in the engine, and every backtest of the main strategy assumed it. Live trading never sent the order to the exchange. All the performance numbers I trusted had a leg that didn’t exist.

No test caught it. It came out while I was reviewing formal-verification work: Lean proofs of the engine’s accounting and a TLA+ model of how failed closes are handled. On money-at-risk code the simulator is part of the attack surface. Now a fidelity gate compares live fills with the backtest, and the optimizer is wired to refuse to run when the two diverge.

3. Silent no-ops beat loud failures

The incidents that hurt most made no noise at all:

  • Two branches each added a database migration with the same version number. Every deploy failed for about 20 hours while the old binary kept running, so everything looked fine.
  • A merge deleted an engine field. The config parser ignores unknown keys, so a setting did nothing for about seven weeks. Every recertification in that time tested a different config from the one the docs described.

Both now have structural CI checks: unique migration versions, and warnings for config keys the engine doesn’t read. The general rule I took away is to treat “nothing happened” as an alarm.

4. Autonomy needs a fence the agent can’t argue with

The auditor agent is report-only by design. On its first real run it pushed straight to main. Nothing got deployed, but it could have been. Another agent once opened duplicate PRs for the same issue.

Instructions alone don’t hold. What holds is structure. The runner’s checkout is a read-only mirror with push URLs blocked per worktree, and only CI deploys. One honest caveat: branch protection isn’t enabled on my GitHub plan. The real final gate is still me clicking merge.

5. Most proposals should die

About three in four hypotheses in the ledger were killed, and half of the strategy PRs that did get opened were closed without merging. That felt wasteful at first. It’s the system doing its job: an optimizer that accepts most of its own ideas is fitting noise.

The cost side surprised me. Two thirds of the spend went to those research loops, and almost all of it was the model re-reading its own long context, not writing anything. Capping context size was the biggest cost lever by far.

What I’d tell a team adopting this

  • Small, well-specified fixes are where agents pay off fastest. Start there.
  • Make the agent write the evidence into the PR (tables, before/after trade dumps, test runs). That’s the only reason one person can review all of it.
  • Put guardrails in structure, not in prompts.
  • Spend your paranoia on cross-component invariants and on sim-vs-real drift. Those are where my bugs lived.
  • Measure everything, including what you’d rather not see. I only found the 11% repair rate because I went looking for it.

I’m happy to compare notes with anyone running agents on code where mistakes cost money.