LLMJury documentation
LLMJury is statistically defensible A/B testing for LLM products: deterministic variant assignment, LLM-as-judge quality scoring, and SRM-gated, FDR-corrected results. Start with Getting started or jump to your SDK.
Start here
SDKs
- Python SDKInstall the llmjury Python package: get_prompt with in-code fallbacks, setup-once wrap() interception, and deterministic local bucketing.
- TypeScript SDKInstall @llmjury/sdk for Node 18+ or the browser: getPrompt with in-code fallbacks, setup-once wrap() interception, and deterministic bucketing.
- Java SDKAdd com.llmjury:sdk (Java 11+): ExperimentSession resolves the prompt, intercepts the model call, and tracks the outcome.
Concepts
- SRM: sample ratio mismatchWhat a sample ratio mismatch is, why it invalidates results, and how LLMJury's hard gate halts analysis at p < 0.001.
- FDR correctionWhy multi-metric experiments need Benjamini–Hochberg false-discovery-rate correction and how LLMJury applies it.
- Permutation testsHow permutation tests and bootstrap confidence intervals handle skewed LLM metrics like p95 latency.
- Bucketing & sticky assignmentThe frozen MurmurHash3 bucketing hash, integer allocation boundaries, and why assignments never reshuffle.
- LLM-as-judgeHow LLMJury scores outputs with a judge model: sampling, caching, versioned rubrics, and hard budgets.
- Analysis windows & retentionThe analysis window is the experiment's own duration; plan tiers bound raw-event retention, never the analysis.
Guides
- Bring your own data (CSV/JSON import)Bulk-import historical events through POST /v1/events/import with a secret key.
- Custom metricsDefine a judge metric as a natural-language rubric plus a structured output schema, versioned per experiment.
- Method selectionHow each metric category routes to its recommended statistical test, and how (and when) to override it.