Skip to content

Backtest metrics reference

Every metric a backtest reports, the results-page diagnostics, and the three drill-down fields (regimeBreakdown, outOfSample, roundTrips). Computed in packages/strategy/backtest/src/metrics.ts. For how the engine produces these — the fill model, look-ahead safety, and why a backtest is optimistic relative to live trading — see Backtesting.

Metrics

Reading the ratios. CAGR — compound annual growth rate, the yearly rate the run would grow at. Max drawdown — the worst peak-to-trough drop, i.e. how deep the balance fell before recovering. Calmar — return per unit of that worst drop (higher = a steadier ride). Sharpe — return per unit of overall volatility; Sortino — the same but penalising only downside moves. SQN — System Quality Number, how consistent the trade-by-trade edge is. Win rate — the share of trades that closed positive. Profit factor — gross profit ÷ gross loss (above 1 is profitable). Expectancy — average profit per trade. As a rule, prefer higher Sharpe / Sortino / Calmar / profit factor and a shallower max drawdown.

  • Total return / absolute profit / final balance — from the equity curve endpoints.
  • CAGR — annualised from the equity-curve span; a run shorter than about 1 day is not annualised (the 1/years exponent explodes) and reports 0.
  • Max drawdown — the deepest peak-to-trough on the equity curve, with its start/end timestamps.
  • Calmar = CAGR / |max drawdown|.
  • Sharpe, Sortino, SQN, win rate, profit factor, expectancy, best/worst trade — computed over closed round-trip trades (a buy paired with its exit), not per-candle equity returns, so they stay meaningful on short runs. Sharpe/Sortino are therefore per-trade and not annualised (the UI labels them "Sharpe (per-trade)" / "Sortino (per-trade)" to avoid implying the annualised standard). A run with no closed trade reports the null/zero of each.
  • Buy & hold (marketChangePct) — the equal-weighted buy-and-hold benchmark across the symbols, anchored on the first traded (post-warm-up) close through the last close, so it spans the same window as the strategy's equity curve. Warm-up candles are excluded; including them measured the benchmark over candles the operator never asked to evaluate and distorted alpha.
  • Dollar-cost average (dcaChangePct) — the equal-weighted benchmark of investing a fixed cash slice at every post-warm-up close. Per symbol the return is lastClose * mean(1/close) - 1, the honest comparator for a dip-buyer.
  • Alpha (alphaVsHoldPct, alphaVsDcaPct) — total return minus each passive benchmark. This is the scorecard that matters: a positive total return with negative alpha means the strategy lost to simply holding (or averaging into) the basket.

When alphaVsHoldPct < 0 the result view shows an explicit prefer-hold banner (recommendTradeOrHold, @app/contracts): holding the basket would have beaten the strategy net of fees, so the honest recommendation is to hold, not trade. The boundary matches the live-enablement gate's alpha floor (alphaVsHoldPct >= minAlphaVsHoldPct, default 0) — a config that exactly ties holding clears the gate and shows no banner — so the advisory and the gate agree. This makes the "just hold" answer explicit rather than leaving the operator to infer it from a negative alpha cell.

A run with zero closed trades is handled first and independently of alpha sign: the result view shows a dedicated zero-trade banner (naming the dominant entry-block reason from the decision breakdown in plain language), the "your result" cards drop their green/red tint, and recommendTradeOrHold returns hold regardless of alpha — a positive alpha with no trades is just cash sitting out a falling market, not a repeatable edge. The banner copy distinguishes a run that never entered (zero fills) from one still holding an open position at the window end (fills but no closed round-trip).

Why it traded (or did not): legible funnel + guarded suggestions

The raw decision breakdown is per-tick counters with strategy-internal names (tt-technicals-gate-veto, tt-indicator-gate-veto, tt_first_buy_skipped). The gates evaluate in order — technical-rating → indicator → order sizing, each short-circuiting — so the per-reason counts partition the entries that reached each gate. summarizeDecisionBreakdown (apps/web/src/features/backtest/lib/decision-breakdown.ts) rebuilds that funnel into a plain-language view: how many entries each gate blocked versus let through, a ranked bar per blocking reason (glossed, tinted by whether it is the operator's setting, the market reading correctly, a size floor, or warm-up), and the single binding-constraint line ("of the N entries that passed the rating gate, M (X%) were then stopped by …"). The raw counters stay, shown in full beneath the plain-language summary, for verification. tt_tick_pure_path is not a blocker — it is the bucket the gate-veto logs subdivide, so it is never counted as one (double-count guard).

Deterministic diagnosis spine

Above the metrics and the evidence, the result view leads with a deterministic diagnosis spine (apps/web/src/features/backtest/lib/diagnosis-spine.ts, buildDiagnosisSpine): a ranked list of only the provable causes of the run's outcome. Each item is one of four kinds, emitted in a fixed order: funnel blockers (from summarizeDecisionBreakdown, ranked by count descending, each naming its config lever) → live-gate misses (the threshold checks the metrics fail, grouped into one item via the shared failedGateChecks helper) → segment reads (facts the metrics state outright: zero closed trades, negative alpha vs hold, a regime row with negative alpha, an out-of-sample holdout that was too short or underperformed) → a single "no deterministic cause" fallback. The fallback fires only when the run is a loss and nothing provable was found; it routes the operator to the advisor in "What next?" rather than inventing a reason. The spine never fabricates a cause from a PnL or drawdown heuristic — a deep drawdown or a red number is an outcome, not a cause — so a losing run is never left silent and a clean winning run yields an empty spine. Every per-code blocker detail is resolved off the active strategy's descriptor (reasonAttribution, served from GET /strategies), not a table hardcoded in the web: the gloss label, the kind tint, and the "set by config.path = value" lever line all come from the one entry — invariant 1, so adding a strategy needs no web edit. A code the descriptor does not attribute falls back to its raw name and the neutral data tint.

recommendConfigChanges turns the dominant blockers into guarded suggestions. Each suggestion only removes an entry constraint the operator actually armed (the RSI/SMA/EMA/mean-reversion gates, or arming a bullish rating level that was off), and is rendered with a "Load into form & re-test" button that seeds the Setup form (via the same runConfigSeed remount the past-run loader uses) and switches tabs — it never writes the live config. The honest loop is preserved: load → re-run → the new run must clear the out-of-sample gate → only then does "Apply to live config" appear. Two things are deliberately never suggested: bypassing the bearish technicals-sell veto (that is the gate keeping the bot out of a downtrend), and any ranking of configs by in-sample return (the out-of-sample gate exists to distrust exactly that). Suggestions are framed as hypotheses to measure, not predictions of profit.

A second, on-demand suggestion source feeds the same review loop: the LLM advisor, now a durable, background artifact rather than a synchronous call. Each generation is a per-(profile, run, variant) row in backtest_advisor_result that survives page reload and tab-close. GET /profiles/:id/backtests/:runId/advisor lists the saved variant results, so opening the run rehydrates prior advice with no new model call and no re-bill. POST …/backtests/:runId/advisor/{variant} enqueues a background job on the study-role worker and returns 202; it returns 503 ("AI advisor unavailable — study worker offline or no AI provider configured") when the advisor:ready readiness flag is absent. The web UI polls GET …/advisor (a TanStack Query refetchInterval while any variant is running) instead of holding a stream open. Single-flight is a conditional DB upsert, not a BullMQ jobId: the start route transitions the row to running only from a non-running state, so a duplicate click enqueues nothing and the poll picks up the in-flight job. Regenerate re-enqueues from a done/error row; a row stranded running by a lost or hard-killed job is reclaimed (→ error) by a periodic study-role stale sweep, so the polling UI never watches a dead row forever.

The job sends the model the run's full context — the config tested, parameters, metrics, the live-gate checklist (each go-live bar with its current pass/fail under the profile's policy, from the same gateThresholdChecks the gate-status card uses, so the advisor targets the failing checks instead of an abstract "improve performance"), the data-coverage warnings (patchy-candle flags, so it never tunes on a run it should distrust), the trade-or-hold baseline (recommendTradeOrHold — the honest "just hold" verdict when the run lost to a fee-free buy-and-hold or closed no trades), the fill-model realism (detail vs strategy interval and whether spread/volume-cap were modeled, so it discounts optimistic intra-candle fills), the why-it-did-not-trade breakdown, the regime split, the out-of-sample holdout, the prior same-market runs (each earlier config→metrics on the same symbols+window, so the model reasons over the response surface instead of a single point), the downsampled equity/drawdown curves (the per-bar series capped to about 80 points so the model sees where capital sat idle and where drawdowns landed), and the exit-reason mix across every sell (which mechanism closed positions) — under a system prompt that asks for changes likely to improve forward (out-of-sample) performance, never the in-sample number, never relaxing the bearish-rating veto, and never tuning the risk-adjusted metrics (cagr/calmar/sharpe/sortino/sqn) the live gate deliberately ignores. The model returns path/value config patches; the worker re-applies each onto the run's config and re-validates against the strategy schema, dropping any that do not parse (surfaced as dropped), so only schema-valid suggestions reach the operator. They render as the same multi-select cards (with an overfit-risk flag) and load into Setup through the identical applyRecommendations → re-run → out-of-sample gate loop — the LLM stays strictly on the backtest side of the gate. The run context is serialised to the prompt in a compact, token-oriented encoding (the bulk is uniform arrays — equity/drawdown curves, round-trips, regime split — that cost fewer tokens that way), while the config schema stays JSON and the model still returns JSON. The advisor drives a provider-agnostic client seam, so any configured provider works — Anthropic, or an OpenAI-compatible endpoint like Ollama, vLLM, or OpenAI. The encoding choice and the per-provider adapter mechanics are contributor detail — see Backtesting internals. It advises only and never writes config or runs anything. The provider and its credentials are DB-configured (the ai_provider_config singleton, edited at Account → AI assistant), resolved fresh per job by the study-role worker so switching provider takes effect without a restart. Because a fast suggest → apply → re-run loop makes overfitting frictionless, the out-of-sample holdout in the live-enablement gate remains the backstop.

The advisor runs in one of five variants, one per button on the card and selected by the {variant} path segment on the start route (safe default). safe proposes only changes likely to improve forward performance and says HOLD when nothing beats holding cash. The other four are opt-in EXPLORE lenses that steer the same bold advisor at a different lever: ride-trend loosens exits so winners run, trade-more loosens entry throttles to raise the trade count, aggressive leans on larger sizing / exposure, defensive cuts drawdown. Each EXPLORE variant proposes higher-variance hypotheses flagged medium/high overfit-risk. To widen the set, an EXPLORE variant samples the model twice and merges the deduped union (mergeImproveResponses, dropping same-edit repeats); safe is a single call. The hard guardrails are identical across variants — never disable the bearish veto, respect documented bounds, lead with "re-run on clean data" when the metrics are unreliable, and be honest that position sizing amplifies edge rather than creating it — and the out-of-sample gate still decides go-live, so an EXPLORE suggestion is only ever a hypothesis until a re-run clears the gate.

Manual loop (no server provider). An operator who configures no server-side provider can still use the advisor: a "Run it myself" button hands you the exact prompt the server would have sent, you copy it into claude.ai, and paste the reply back. It runs the same strategy-schema re-validation as the server path and persists to the run's own manual slot, and it needs no server-side credential. The route mechanics and the Anthropic Console-key vs subscription-OAuth detail are contributor detail — see Backtesting internals.

Live-gate quality scorecard

The result view also shows a gate scorecard: the three quality thresholds the live-enablement gate enforces (net profit factor, closed-trade count, alpha vs hold), each marked pass/fail against the profile's EnablementPolicy. It exists so the operator learns whether a config clears the bar at backtest time, not as a 409 when they later try to enable live. The scorecard and the gate call the same pure gateThresholdChecks (@app/contracts), so the criteria can never drift.

Scope: the scorecard judges only the quality thresholds of these results. Actually going live additionally requires the profile's saved config to match a recent backtest (the fingerprint + freshness checks) — that is the gate-status card's job, not the scorecard's. So a "clears the bar" verdict means the metrics are good enough, not that the profile can be enabled right now.

Performance by market regime (regimeBreakdown)

metrics.ts also splits the run by market regime and the web view renders it as a "Performance by market regime" table. The lens is the benchmark symbol's (the first requested symbol) daily close vs its 50-day SMA, confirmed over 2 closes. It shares the bull / neutral / bear vocabulary with the live Market Trend card but uses a simpler, self-contained rule (close vs the 50-day SMA only, no EMA-cross), computed generically inside regime.ts so the strategy-agnostic backtest package never reads a plugin's regime config. For each regime the record carries the strategy's compounded equity return, the benchmark buy-and-hold over the same steps (the strategy equity is sampled on the benchmark symbol's candle cadence so a basket run stays apples-to-apples), their difference (alphaVsHoldPct), and per-regime trade win rate, profit factor, and expectancy — the table renders all but expectancy. The classification is no-lookahead (a day is labelled from daily closes that closed before it). A window too short to fill the 50-day MA yields an empty breakdown.

This is the honest test of the long-only ceiling: a strategy whose only positive alphaVsHoldPct is in the bull row is just holding in a rising market, not finding an edge. regimeBreakdown is an additive, defaulted field — runs persisted before it shipped parse as [].

Out-of-sample check (outOfSample)

metrics.ts also recomputes the headline figures over only the most-recent 30% of the run's time span (HOLDOUT_FRACTION in regime.ts, a fixed analysis constant — the engine computes one canonical holdout and never reads a policy, the same separation the regime MA period keeps). An operator tunes a config against the full window, so the full-run metrics are in-sample and can be curve-fit; the recent slice the tuning never targeted is the honest test set. The slice reuses the regime segment's compounding (strategy return, buy-and-hold over the same steps, their difference alphaVsHoldPct, plus trades opened in the holdout, win rate, profit factor, and expectancy) and the web view renders it as an "Out-of-sample check" panel. A strategy whose strong full-run profit factor collapses out-of-sample was fit to the window, not to an edge. It is surfaced for the operator on the results page, and the live-enablement gate (below) requires it to clear the same bars by default (requireOutOfSample). outOfSample is additive, nullable, and defaulted: runs persisted before it shipped, and runs too short to carve a holdout (< 2 equity points), parse as null — which fails the gate's out-of-sample check, so the operator re-runs the backtest.

Round-trip trades (roundTrips)

The headline trade quality metrics (win rate, profit factor, expectancy) are a reduction of the run's closed round-trips, which metrics.ts otherwise discards. The full per-trade list is surfaced as roundTrips so the web view can render a Trades drill-down: a per-exit-reason rollup (which exits make or lose money) above a per-trade table. A round-trip pairs each reducing sell against the position's average cost, so a grid that stacks several buys before one sell is one round-trip per reducing sell, not one per fill — unlike the raw trades (fills) list. Each record carries the fee-free average entry price, the sell exit price, quantity closed, realised P&L net of all fees, return over the closed portion's cost basis, the fees attributed to it (allocated buy fees plus the sell fee), open/close timestamps, and the closing sell's reason. The display figures are derived from a parallel fee-free cost accumulator that never touches the P&L math, so adding them left the golden replay byte-identical. roundTrips is an additive, defaulted field — runs persisted before it shipped parse as [], and the view falls back to the raw trades list.