We built an LLM trading leaderboard, then re-ran it 18 times and watched it fall apart

The setup

Four models on Amazon Bedrock. Each starts with $50,000 of virtual capital and makes one end-of-day decision per trading day against historical closing prices, from 2026-01-02 to 2026-07-10 — 130 trading days. Identical neutral prompt, identical 16-ticker universe, identical tools (quotes, portfolio, buy, sell, journal). Web search is off so nobody can read tomorrow's news.

The question was the one every team building agents eventually asks: does paying for a stronger model buy better decisions?

The leaderboard we almost published

An 8B model beat a frontier model by 71 percentage points while trading 3 times to its 46. Activity was almost perfectly inversely correlated with return. It is a fantastic headline, and it has the shape of a real insight.

We did not believe it. Sampling temperature is 0.7 — nothing here is deterministic. A full 130-day run of llama-8b costs $0.04. There was no excuse not to check.

So we re-ran everything, five times each

Same 130 days. Same prompts. 18 additional full backtests.

Read the last two columns together, because that is the whole paper:

• The four models' means land between +5.0% and +12.0% — a spread of 7 points.
• Each model's own run-to-run swing is 13 to 19 points.

A single model re-rolled against itself varies about twice as much as the models differ from each other. On return, these four models are not distinguishable. The 71-point gap that topped our leaderboard was noise, and we came within one deploy of publishing it as a finding.

Not one of llama-8b's five re-runs came within 50 points of +72.6%. Its typical result is +12.0%, which is a statistical tie with everything else in the table.

The headline run was atypical in every dimension

Once we had distributions, we could see that the run we happened to screenshot was an outlier not just in return, but in behaviour:

nova-lite normally trades 28–46 times; in our headline run it traded 7. claude-haiku normally trades 3–17 times; in our headline run it traded 29. We built a story about "trading activity correlates with returns" on a run where two of the four models behaved nothing like they normally do.

That is not bad luck. That is what a sample size of one looks like.

What actually survived

Two things. They are the only claims in this article we will defend.

1. Behaviour is reproducible even when returns are not. llama-8b traded 2–4 times in every single run. It reliably places a couple of early bets and then sits still — a stable, real property of the model. What it buys, and therefore what it earns, is a coin flip. (To be clear: barely trading is not a defect. In this window, sitting still was the best strategy available. We are saying its return tells you about NVDA, not about Meta's model.)

2. Cost is deterministic. It does not care which way the market went:

sonnet-4-6 cost 121x more per decision than llama-8b — $0.04402 vs $0.00036.

That is the only number in the results table that means exactly what it appears to mean. And set against the return distributions above, it produces the finding we would actually stand behind:

You paid 121x more and bought another sample from the same distribution.

That is not an argument against frontier models. It is an argument against assuming the premium buys you something on this task, and for measuring instead of assuming. On a task with real reasoning depth the answer may well flip — but it will not flip for free, and you will not know until you look.

The rule this gives us

If you rank LLM agents by the outcome of a single run — ours, or anyone's — you are ranking luck. The fix is not sophisticated: re-run it and look at the spread. If the spread inside one model is wider than the gaps between models, you have no ranking. You have noise with a leaderboard drawn on it.

Two agents that had silently stopped working

The most expensive thing we found was not in the returns table.

A model that fabricated six months of trades

An earlier round of this simulation included mistral-small-2402. It ran January to May and executed zero trades. Net asset value: exactly $50,000, every single day. Cash: $50,000. Positions: none.

It looked like an extremely conservative strategy. It was not a strategy at all. The model cannot emit tool calls, and its journal shows three escalating failure modes:

It narrates tool use in prose:

I begin by reviewing my current portfolio using the "get_portfolio" tool...

It writes the tool-call JSON out as plain text:

{"name": "read_journal", "arguments": {"entry": "Reviewed portfolio and quotes. Decided to hold positions."}}

It knows exactly what a tool call looks like. It simply types it instead of returning a real toolUse block — so stopReason is never tool_use, our runTool() is never reached, and no order is ever placed.

And it fabricates trades it never made, using real tickers:

Bought 100 shares of AMZN based on strong momentum... Found positive news on AMD, bought 100 shares at market price.

AMZN and AMD are both in our universe. Skim the journal and these entries look completely legitimate. The trade log says zero. The portfolio is empty. The cash never moved.

This is not a story about Mistral. mistral-small-2402 is a February 2024 snapshot that predates reliable function calling; later Mistral models support it. It is a story about how quietly an agent can die, and how little your dashboard will say about it.

Our own code was hiding it

The Bedrock wrapper caught model errors and returned them as ordinary output:

} catch (e) {
  return { finalText: `[${spec.id}] model error: ${e.message}`, ... };
}

A model that cannot be invoked and a model that decided to hold today produce identical data: zero trades, flat NAV. There is nothing to alarm on. So the simulation ran happily for months, and the inference bill for a dead agent arrived every month alongside everything else.

Then fail-fast found a second one

After we made failed calls throw instead of return, a re-run crashed on day 17:

ModelCallFailed: [sonnet-4-6] The content field in the Message object
at messages.8 is empty.

A real bug, in our loop, not the model's: when sonnet-4-6 is truncated at maxTokens mid-reasoning, it returns an assistant message with no content blocks. We pushed that empty message straight back into the conversation history, and Bedrock rejected the next call as malformed.

Under the old error-swallowing code, this 400 would have been caught and written into the journal as text. Which means sonnet-4-6 had almost certainly been silently doing nothing on some days of every earlier run — and we would never have known. Same disease as mistral-small-2402, different symptom.

Two silent failures, one root cause: we had made "the call failed" and "the agent chose not to act" indistinguishable in our own data.

What we changed

1. Preflight before you spend. Probe every model with a 1-token call before the run starts; refuse to start if any model cannot be invoked. Costs nothing. Would have caught the entitlement failure on day zero.

2. Fail loudly. A failed call now throws ModelCallFailed and stops the run. "The call failed" and "the agent held" must never be the same row in your data.

3. Guard the tool loop. Never push an empty-content message back into history.

4. Account for cost per agent. Our budget tracker recorded only a total. We could tell you the experiment cost $8.07; we could not tell you which model spent it — which is the entire question. doCheckin() had been computing per-model cost all along and throwing it away.

Note what preflight does not catch: mistral-small-2402 passes it. The model is invokable — it just cannot act. HTTP 200 is not proof of agentic capability. If you need a model to use tools, your check must assert that a real tool-call block comes back, not merely that the API answered.

What this cost

We think the price tag matters, because the obvious objection to "always re-run your evals" is that it sounds expensive.

Put the two numbers that matter side by side:

Producing the wrong headline cost $8.07. Overturning it cost $0.19.

There is no budget at which skipping the re-runs is the rational choice.

But note the asymmetry, because it is a real cost of frontier models that nobody puts on the pricing page: disproving a false result about llama-8b cost 19 cents; disproving one about sonnet-4-6 cost $17.75. The more expensive the model, the more expensive it is to hold it to the same standard of evidence — and the more tempting it becomes to publish n=1 and move on.

(Caveat we owe you: sonnet-4-6 has n=3, not n=5. An expired AWS session and a budget circuit-breaker cut the run short, and its remaining two re-runs were not worth another $12 to us — its swing was already the widest of the four. Every other model is n=5.)

What to take away

• Never rank agents on a single run. If the spread within one model exceeds the gap between models, there is no ranking. Re-run it; the spread is the result.
• Do not pick an agent model from a price list or a leaderboard. Verify with one real tool call that it can act. Fluency and capability are different things, and the gap is invisible from outside.
• A more expensive model is not automatically a better agent. Here, 121x the cost per decision bought more activity, no more return, and a wider variance. Measure it on your task; do not assume it.
• Instrument cost per agent from day one. An aggregate number cannot answer the only question you will ever actually ask.
• Silent failure is the default failure mode of agents. A crashed service pages you. An agent that quietly does nothing looks exactly like an agent being careful — and it bills you either way.

The simulation is still running, one decision per trading day. The leaderboard below updates itself. Now you know what it is worth.