> For the complete documentation index, see [llms.txt](https://docs.algenta.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.algenta.ai/machine-learning/ml-model-comparison.md).

# Statistical model comparison

Paired bootstrap CIs, exact McNemar and Diebold-Mariano tests that decide whether a candidate model is genuinely better than the champion.

`model_compare` decides whether model B is genuinely better than model A — with uncertainty, not a single point estimate. Comparing two accuracy scalars answers nothing; these tests answer "is the difference real?"

## The modules

| Module          | What it does                                                                                                                               |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `model_compare` | Paired bootstrap CIs, exact McNemar and Diebold-Mariano tests that decide whether a candidate model is genuinely better than the champion. |

## Functions

| Function                                                              | What it does                                                                                                                                                              |
| --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `paired_bootstrap_diff_ci(metric_a, metric_b, n_boot, alpha, seed)`   | Confidence interval on `mean(a - b)` by resampling **pairs** (per-prediction correctness, per-trade PnL). If the CI excludes 0, the difference is real at level `alpha`.  |
| `mcnemar_exact_p(b01, b10)`                                           | Exact two-sided McNemar test on disagreement counts (A-wrong/B-right vs A-right/B-wrong). Log-space binomial; exact for any n.                                            |
| `diebold_mariano(err_a, err_b, h)`                                    | Forecast-accuracy comparison on squared-error loss differentials with Newey–West variance (lag `h-1`) — the standard test for comparing two forecasting models over time. |
| `sharpe_diff_bootstrap_ci(returns_a, returns_b, n_boot, alpha, seed)` | Paired bootstrap CI on the Sharpe-ratio difference between two return streams — compares models by **trading outcome**, not accuracy.                                     |

## Recommended promotion gate

For a candidate model to replace a champion:

1. `paired_bootstrap_diff_ci` on per-sample correctness → CI lower bound > 0
2. `mcnemar_exact_p` < 0.05 on the disagreement table
3. `sharpe_diff_bootstrap_ci` on realized signal PnL → CI lower bound > 0

Any single point estimate can flatter a candidate; requiring the interval to clear zero is what "only promote if better" means statistically.

## Serial dependence caveat

`paired_bootstrap_diff_ci`, `mcnemar_exact_p` and `sharpe_diff_bootstrap_ci` assume exchangeable pairs. Overlapping forecast targets (stride-1 windows predicting h-step returns) violate that badly — measured \~3× type-I inflation at h = 5 measured in production use. For such data either subsample every h-th observation before calling them, or use `diebold_mariano(err_a, err_b, h)`, whose Newey–West variance (lags `1..h-1`) is designed exactly for overlapping h-step forecast errors and needs no subsampling.

## Verification

Verified against independent references: McNemar to 2e-9 relative error; Diebold–Mariano re-derived from the Newey–West definition.

**Next:** validate on honest splits with [time-series model selection](/machine-learning/ml-model-selection.md), or browse the [engine catalog](/engine-runtime/catalog.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.algenta.ai/machine-learning/ml-model-comparison.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.