Method selection
Every metric category auto-routes to the statistically correct default test — the safe path is one click and protects correctness. LLMJury deliberately keeps the default method set minimal: one distribution-free test for everything non-binary, plus one closed-form fast path for binary rates. Fewer default methods means fewer statistical code paths to validate, and the permutation test is correct for any distribution shape — including the heavy-tailed latency and cost data where parametric tests give wrong p-values.
| Category | Recommended test | Why |
|---|---|---|
| Continuous | Permutation + bootstrap CI | Correct for heavy-tailed latency/cost; no normality assumption |
| Binary | Two-proportion z-test | Closed-form and fast for rates (conversion, pass/fail) |
| Ordinal | Permutation + bootstrap CI | Valid for ranked scores (1–5 judge ratings) with one machinery |
| Percentile | Permutation + bootstrap CI | Tail statistics (p95 latency) are non-normal |
| Count | Permutation + bootstrap CI | Events-per-user distributions are skewed |
Every non-binary result pairs a permutation p-value ("is the effect real?") with a bootstrap confidence interval on the effect size ("is it big enough to act on?") — a p-value is never shown without its effect-size CI.
Small-sample binary fallback: the z-test's normal approximation needs an expected count of at least 10 in every cell of the 2×2 table. When a binary metric is below that (rare events like toxicity, or very high pass rates with scarce failures), the engine automatically falls back to a permutation test on the 0/1 indicator — the small-sample exact equivalent. The result records which test actually ran.
Welch's t-test, Mann–Whitney U, and Fisher's exact remain in the catalogue as advanced overrides only — they are never auto-selected. Fisher's exact is exempt from per-arm sample caps so rare binary events keep every observation.
Overriding
Advanced users can override the test per metric — in the metric builder or the experiment
wizard. When your choice diverges from the recommendation, LLMJury shows an inline warning
at decision time, records the override (and the warning) on every result as
method_override_warning, and writes it to the audit log. The default path stays one click.
What's recorded
Each result carries test, permutations_used, sample_cap_used, window_days, and both raw
and FDR-corrected p-values — enough to reproduce or defend the analysis later. See
FDR correction and permutation tests.