pybench

While Pytest does deterministic tests, Pybench does statistical tests.

pybench discovers benchmark functions, runs them across many seeds, and statistically detects regressions against a saved baseline.

It reruns each benchmark on the same stored seeds as its baseline, so the comparison is paired (far more sensitive than a two-sample test), and judges the whole benchmark with a within-seed sign-flip permutation test that respects correlation across metrics and steps.

• Getting started
  • What is pybench?
  • Quickstart
• User guide
  • Synthetic example
  • MNIST example
• How it works
  • The problem: the metric is noisy
  • Idea 1 — pairing cancels the noise
  • Idea 2 — a per-slot paired t-test
  • Idea 3 — combine slots without assuming independence
  • Putting it together
• CLI reference
  • Default — discover & run benchmarks
  • Subcommands