Permutation tests

LLM metrics are rarely normal: latency is long-tailed, cost is spiky, p95s are order statistics. Parametric tests can mislead there. A permutation test builds the null distribution empirically — shuffle the variant labels, recompute the statistic, repeat thousands of times — and asks how extreme the observed difference is against that.

How LLMJury uses them

• Percentile metrics (e.g. p95 latency) and count metrics route to a permutation test by default (see method selection).
• The permutation count is configurable per experiment (default 1,000, clamped to your plan's ceiling); p-values quantize in steps of 1/permutations, and the effective permutations_used is recorded on every result for reproducibility.
• Because a permutation test yields no confidence interval, LLMJury pairs it with a bootstrap CI: resample with replacement, recompute the effect, take the central interval.

When to prefer them

Skewed distributions, small samples, tail-focused questions ("did p95 regress?"), or any metric where you'd rather not defend a normality assumption in a launch review.