Getting started

Ship your first statistically defensible LLM experiment in about five minutes. You need an LLMJury account (every plan starts free, no credit card).

1. Install an SDK

Pick your language — each has a dedicated quickstart with the full API surface: Python · TypeScript · Java.

pip install llmjury          # Python 3.8+
npm install @llmjury/sdk     # Node 18+ / browser
# Java 11+: com.llmjury:sdk (Gradle/Maven)

2. Add your API key

Grab your org's publishable key from the dashboard under Settings → API keys — the raw key is shown once when it's issued, so copy it right away. Publishable keys are prefixed llmj_pk_ and are safe in client-side code: they can only assign, track, and read config. Secret keys (llmj_sk_) are server-side only and unlock privileged operations like bulk import — never ship one to a browser.

Set the key in the LLMJURY_API_KEY environment variable — every SDK reads it automatically, so it never lives in your code:

export LLMJURY_API_KEY=llmj_pk_...   # shell / CI secret / deployment environment / .env file

If your platform injects secrets another way, pass the key explicitly instead: Client(api_key=...) in Python, new Client({ apiKey }) in TypeScript, LlmjuryClient.builder(apiKey) in Java. The SDK defaults to the production API — no localhost, no base URL to configure.

3. Create an experiment

In the dashboard, create an experiment with two variants (control, treatment) and pick your metrics — built-ins like quality and latency, or a custom judge metric. Give the experiment a unique name (like checkout-prompt): that name is how your code addresses it — no UUIDs. You can also configure each variant's prompt and custom variables (model, temperature, …) right in the dashboard, so iterating never needs a deploy. The recommended statistical test per metric is pre-selected; see method selection.

4. Integrate

The recommended integration is two calls on the request path: get_prompt resolves the user's variant prompt from client memory (your in-code default keeps the app working through a full LLMJury outage), and the setup-once wrap() intercepts your provider client so every model call records latency, tokens, and errors with no call-site code. The only event you track explicitly is your business outcome:

from llmjury import Client

# ---- setup, once at startup -------------------------------------------------
# Reads LLMJURY_API_KEY from the environment; defaults to https://api.llmjury.com.
client = Client(experiments=["checkout-prompt"])          # prefetch by experiment NAME
llm = client.wrap(anthropic_client, "checkout-prompt")    # every model call is now traced

# ---- per request ------------------------------------------------------------
with client.as_user(user_id):
    # Variant prompt from client memory; your in-code default survives an outage.
    p = client.get_prompt("checkout-prompt", user_id, default="You are a helpful assistant.")
    # Call your provider client DIRECTLY — latency/tokens/errors are intercepted.
    response = llm.messages.create(model="claude-haiku-4-5", max_tokens=1024,
                                   system=p.prompt,
                                   messages=[{"role": "user", "content": user_input}])

# The ONLY explicit metric: your business outcome.
client.track("business_event", {"experiment_id": "checkout-prompt", "user_id": user_id,
                                "variant": p.variant, "business_metric": "conversion", "value": 1})

Every call is deterministic and sticky — the same user gets the same variant in every language — and nothing here ever blocks or throws into your app. The same surface exists in TypeScript and Java, and the lower-level assign + track calls remain available if you want full manual control.

5. Read the result

Results appear after the daily pipeline runs with enough judged events. Every result shows the effect with a confidence interval, raw and FDR-corrected p-values, the test used, and the SRM check status. Significance is judged on the corrected p-value.