> This is part 1 of 5 of the full documentation (pages 1–100 of 470).
> The content is paginated: fetch every part to see all of it.
> Next part: https://docs.layerlens.ai/llms-full.txt/1
> Page index: https://docs.layerlens.ai/llms.txt

# LayerLens Stratix Documentation

Welcome to LayerLens Stratix documentation. Three customer-facing experiences, one platform — pick a path and ship faster.

LayerLens Stratix is the AI evaluation platform for teams that need to know — before customers do — whether a model, prompt, or agent is good enough to ship. Browse the world's largest public catalog of model evaluations, run private evaluations on your own data, score traces with deterministic rules and LLM judges, and ship AI you can trust.

## Three customer-facing experiences

{% tabs %}
{% tab title="Stratix Public" %}
**`stratix.layerlens.ai`** — anonymous browsing of 175+ models, 52+ benchmarks, and 2,000+ public evaluations. Compare any two models head-to-head. No sign-up required.

[Start with Stratix Public →](/2.-get-started/public)
{% endtab %}

{% tab title="Stratix Premium" %}
**`stratix.layerlens.ai` — after you sign in.** The logged-in workspace where teams run private evaluations, build judges, score traces, manage scorers, run agentic evaluations, and govern AI quality across an organization.

[Start with Stratix Premium →](/2.-get-started/premium)
{% endtab %}

{% tab title="Stratix SDK" %}
**`pip install layerlens --extra-index-url https://sdk.layerlens.ai/package`** — the Python client SDK (v1.3.0) for programmatic evaluation, judge orchestration, and trace ingestion. 138 sample programs.

[Start with the SDK →](/2.-get-started/sdk)
{% endtab %}
{% endtabs %}

## Pick a path

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>I'm a builder</strong></td><td>Ship faster with evaluations wired into your dev loop.</td><td><a href="/1.-introduction/personas-paths#builder">Builder path</a></td></tr><tr><td><strong>I'm an operator</strong></td><td>Run live AI in production with quality gates and continuous evaluation.</td><td><a href="/1.-introduction/personas-paths#operator">Operator path</a></td></tr><tr><td><strong>I'm a researcher</strong></td><td>Explore 175+ models against 52+ benchmarks, browse 2,000+ public evaluations.</td><td><a href="/1.-introduction/personas-paths#researcher">Researcher path</a></td></tr><tr><td><strong>I'm an admin or buyer</strong></td><td>Set up your org, manage seats and credits, evaluate enterprise readiness.</td><td><a href="/1.-introduction/personas-paths#admin-buyer">Admin/Buyer path</a></td></tr></tbody></table>

## The Stratix Workflow

Five stages from raw model to governed production. Every Premium capability maps to one of them.

{% hint style="info" %}
**Select → Build → Observe → Evaluate → Improve.** This is the spine. Every concept, how-to, tutorial, and recipe in this documentation pins to one of these stages. [Learn the workflow →](/1.-introduction/the-stratix-workflow)
{% endhint %}

## What's new

* **Q1 2026 model leaderboard** — refreshed quarterly with GPT-5.3, Claude Opus 4.6, Gemini 3.1 Pro/Flash, and 200+ more.
* **Agentic evaluations** — pre- and post-deployment quality gates for multi-step agents. [Read the announcement](/4.1-general-use-cases/agentic-evals-overview).
* **GEPA judge optimization** — automatically tune your LLM judges against ground-truth labels. [Read the concept](/8.-evaluate-score-the-outputs/judges-1#judge-optimization-gepa).

## Need help?

* **In-app:** click the Assistant icon (Premium only) for context-aware help.
* **Email:** <support@layerlens.ai>
* **Docs feedback:** open an issue or reach out via the in-app feedback form.

## Where to next

* New here? → [What is LayerLens Stratix?](/1.-introduction/what-is-layerlens-stratix)
* Want to ship today? → [Getting Started](/2.-get-started/02-get-started)
* Researching for purchase? → [How Stratix compares](/1.-introduction/how-stratix-compares-deep)
* Building a business case? → [Pricing](/)


# Quickstart

5-minute quickstart — pick a path and ship.

You're in a hurry. Pick the line that matches you.

## "I want to know which model is best at X"

1. [`stratix.layerlens.ai/models`](https://stratix.layerlens.ai/models) — browse the catalog
2. [Compare models](/5.-select-pick-the-model/compare-models) head-to-head
3. Read the relevant [Quarterly report](/5.-select-pick-the-model/quarterly-reports)

No signup. Five minutes.

## "I want to evaluate my AI on my data"

```bash
pip install layerlens --extra-index-url https://sdk.layerlens.ai/package
export LAYERLENS_STRATIX_API_KEY="..."
```

```python
from layerlens import Stratix
client = Stratix()
model = client.models.get_by_key("openai/gpt-4o")
benchmark = client.benchmarks.get_by_key("mmlu")
e = client.evaluations.create(model=model, benchmark=benchmark)
print(client.evaluations.wait_for_completion(e).accuracy)
```

[Sign up](/2.-get-started/sign-up) → [First evaluation](/2.-get-started/first-evaluation).

## "I just want to verify my key works (curl, language-agnostic)"

```bash
curl -H "X-API-Key: $LAYERLENS_STRATIX_API_KEY" \
 "https://stratix.layerlens.ai/api/v1/models?limit=3"
```

JSON back → you're good. 401 → see [Authentication](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/api/auth.md).

## "I want to gate AI changes in CI"

[Tutorial 3: CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates). 30 minutes from "no gate" to "regression blocks merge."

## "I want to score live production traces"

[Tutorial 4: Score live traces](/8.-evaluate-score-the-outputs/04-score-traces). Continuous evaluation on real traffic.

## "I'm evaluating Stratix for my company"

* [What is LayerLens Stratix?](/1.-introduction/what-is-layerlens-stratix)
* [How Stratix compares](/1.-introduction/how-stratix-compares-deep)
* [Pricing](https://github.com/LayerLens/gitbook-full/blob/main/assets/screenshots/README.md)
* [Industry patterns](/4.2-industry-use-cases/travel-hospitality)

## Got 30 seconds?

Stratix is the AI evaluation platform. Public catalog of 175+ models on `stratix.layerlens.ai`. Private workspace at `stratix.layerlens.ai`. Python SDK on PyPI. Read the [README](/).


# Glossary

Definitions for every Stratix term — models, evaluations, judges, scorers, traces, and more.

## Agentic evaluation

Pre- and post-deployment evaluation of multi-step AI agents. Combines natural-language assertions, deterministic rules, and LLM judges to verify that an agent reaches a correct state, takes legal actions along the way, and satisfies regression criteria. Surfaced in Stratix Premium under **Agent Evaluation** and exercised at `stratix.layerlens.ai/dashboard/agent-evaluation`. See [Agentic evaluations](/4.1-general-use-cases/agentic-evals-overview).

## Benchmark

A standardized dataset and grading procedure used to measure model performance. Stratix hosts 52+ public benchmarks (MMLU, HumanEval, GSM8K, etc.) in the public catalog. See [Benchmarks catalog](/5.-select-pick-the-model/benchmarks-catalog).

## BYOK (Bring Your Own Key)

The ability to use your own LLM provider API keys with Stratix. Two scopes:

* **BYOK custom models** — register an OpenAI-compatible endpoint as a custom model the Premium UI can target.
* **BYOK provider keys** — supply your own OpenAI/Anthropic/Google keys for evaluations that consume tokens.

See [BYOK custom models](/5.-select-pick-the-model/byok-custom-models) and [BYOK models concept](/5.-select-pick-the-model/byok-models).

## Compare models

A Stratix Public and Premium feature that puts two models head-to-head across a benchmark or your own dataset. See [Compare models](/5.-select-pick-the-model/compare-models).

## Continuous evaluation

The practice of running evaluations on a recurring schedule against live traffic — not only at release time. Stratix supports continuous evaluation via trace-evaluations on real production traces. See [Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation).

## ECU (Evaluation Compute Unit)

The Stratix billing unit for compute-intensive operations: model inference, judge runs, GEPA optimization, etc. Free tier includes a starter ECU balance; Premium tier is pay-as-you-go. See [ECU credits](/11.-admin/concepts-ecu-credits).

## Evaluation

A run that scores model output against a benchmark or dataset. Results are stored, browsable, and shareable. Two flavors:

* **Public evaluations** — runs visible to everyone on `stratix.layerlens.ai`.
* **Private evaluations** — runs scoped to your organization in `stratix.layerlens.ai`.

See [Evaluations](/8.-evaluate-score-the-outputs/evaluations-1).

## Evaluation space

A workspace bundling a model selection, dataset/benchmark selection, and scoring config. Spaces let you re-run the same evaluation as conditions change. Public spaces are shareable; Premium spaces are org-scoped. See [Evaluation spaces](/8.-evaluate-score-the-outputs/evaluation-spaces).

## GEPA judge optimization

Automatic tuning of an LLM judge's prompt against a labeled ground-truth set. GEPA finds the prompt that best matches your labels, raising judge agreement with humans. See [Judge optimization (GEPA)](/8.-evaluate-score-the-outputs/judges-1#judge-optimization-gepa).

## Judge

An LLM-based grader that scores model output along subjective dimensions (helpfulness, accuracy, faithfulness, etc.). Stratix Premium supports building custom judges, optimizing them with GEPA, and applying them to evaluations and traces. See [Judges](/8.-evaluate-score-the-outputs/judges-1).

## Model

An LLM (e.g., GPT-5.3, Claude Opus 4.6, Gemini 3.1 Pro). Stratix catalogs 175+ models with metadata, benchmark scores, and per-context information. See [Models and benchmarks](/5.-select-pick-the-model/models-and-benchmarks).

## Multi-org / Organization

Stratix supports users belonging to multiple organizations. After sign-in, the user picks an active organization. Org-switch updates the session and reloads org-scoped data. See [Organizations](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/organizations.md).

## Public catalog

The set of models, benchmarks, evaluations, and spaces visible without authentication on `stratix.layerlens.ai`. See [Stratix Public](https://github.com/LayerLens/gitbook-full/blob/main/assets/screenshots/README.md).

## Quarterly Reports

Long-form research reports (Q1-Q4) summarizing model performance, benchmark winners, and AI evaluation trends. See [Quarterly reports](/5.-select-pick-the-model/quarterly-reports).

## Scorer

A code grader (regex match, JSON-schema validation, exact match, semantic similarity, etc.). Scorers are fast and cheap and suit objective dimensions. See [Scorers](/8.-evaluate-score-the-outputs/scorers-1).

## Span

A child unit of work within a trace. A span represents one tool call, one LLM call, or one logical step. See [Traces and spans](/6.-build-wire-your-code/traces-and-spans).

## Stratix Premium

The logged-in workspace at `stratix.layerlens.ai`. Run private evaluations, build judges, score traces, manage scorers, run agentic evals, govern AI quality.

## Stratix Public

The anonymous browsing experience at `stratix.layerlens.ai`. Browse models, benchmarks, public evaluations, public spaces, and compare models.

## Trace

A record of an AI call (or a chain of calls) — inputs, outputs, latencies, costs, errors, and any spans. Traces are uploaded to Stratix Premium for inspection and trace-evaluation. See [Traces and spans](/6.-build-wire-your-code/traces-and-spans).

## Trace evaluation

A scoring run applied to ingested traces using one or more scorers and judges. Lets you grade real production traffic. See [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations).

***

Seeded from the [LayerLens "25 AI evaluation terms" blog post](https://layerlens.ai/blog) and the live product surface. If a term is missing, open an issue or PR.


# Search & Persona Index

Persona and goal index — find the right page in seconds.

## By persona

### Builder

*"I'm building an AI feature and want it to be good."*

* [Personalized onboarding](/2.-get-started/personalized-onboarding)
* [Your first evaluation](/2.-get-started/first-evaluation)
* [Your first judge](/2.-get-started/first-judge)
* [Tutorial: First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation)
* [Workflow: Instrument](/9.-improve-tune-the-system/workflow)
* [Cookbook recipes](/2.-get-started/all-cookbook-recipes)
* [Python SDK reference](/6.-build-wire-your-code/sdk-python)
* [Integrations](https://github.com/LayerLens/gitbook-full/blob/main/assets/screenshots/README.md)

### Operator

*"I run live AI in production and need to keep it healthy."*

* [Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation)
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations)
* [Workflow: Observe](/9.-improve-tune-the-system/workflow)
* [Workflow: Evaluate](/9.-improve-tune-the-system/workflow)
* [AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd)
* [Tutorial: Wire CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates)
* [Status and reliability](/13.5-operational/status)

### Researcher

*"I want to know which model is best for my use case."*

* [Stratix Public](https://github.com/LayerLens/gitbook-full/blob/main/assets/screenshots/README.md)
* [Models catalog](/5.-select-pick-the-model/models-catalog)
* [Compare models](/5.-select-pick-the-model/compare-models)
* [Public evaluations](/5.-select-pick-the-model/public-evaluations)
* [Quarterly reports](/5.-select-pick-the-model/quarterly-reports)
* [Browse the leaderboard](/5.-select-pick-the-model/leaderboard)
* [Benchmarks catalog](/5.-select-pick-the-model/benchmarks-catalog)

### Admin

*"I'm setting up Stratix for my team."*

* [Sign up](/2.-get-started/sign-up)
* [Organizations and multi-org](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/organizations.md)
* [Team management](/11.-admin/team-management)
* [ECU credits and billing](/11.-admin/ecu-credits-billing)
* [Settings](/11.-admin/settings)
* [Enterprise](https://github.com/LayerLens/gitbook-full/blob/main/assets/screenshots/README.md)
* [Security and compliance](https://github.com/LayerLens/gitbook-full/blob/main/assets/screenshots/README.md)

### Buyer

*"I'm evaluating Stratix for purchase."*

* [What is LayerLens Stratix?](/1.-introduction/what-is-layerlens-stratix)
* [Three experiences](/1.-introduction/three-experiences)
* [How Stratix compares](/1.-introduction/how-stratix-compares-deep)
* [Pricing](https://github.com/LayerLens/gitbook-full/blob/main/assets/screenshots/README.md)
* [Industry patterns](/4.2-industry-use-cases/travel-hospitality)
* [Use cases](/4.1-general-use-cases/general)
* [Enterprise](https://github.com/LayerLens/gitbook-full/blob/main/assets/screenshots/README.md)
* [Security and compliance](https://github.com/LayerLens/gitbook-full/blob/main/assets/screenshots/README.md)

## By goal

| I want to...                    | Go to                                                                                       |
| ------------------------------- | ------------------------------------------------------------------------------------------- |
| Browse 175+ models              | [Models catalog](/5.-select-pick-the-model/models-catalog)                                  |
| Compare two models head-to-head | [Compare models](/5.-select-pick-the-model/compare-models)                                  |
| Run my first evaluation         | [Tutorial 1](/8.-evaluate-score-the-outputs/01-first-evaluation)                            |
| Build a custom LLM judge        | [Tutorial 2](/8.-evaluate-score-the-outputs/02-first-judge)                                 |
| Optimize a judge with GEPA      | [Tutorial 5](/9.-improve-tune-the-system/05-gepa-optimize)                                  |
| Wire evals into CI/CD           | [Tutorial 3](/6.-build-wire-your-code/03-cicd-gates)                                        |
| Evaluate live traces            | [Tutorial 4](/8.-evaluate-score-the-outputs/04-score-traces)                                |
| Use my own LLM key              | [BYOK custom models](/5.-select-pick-the-model/byok-custom-models)                          |
| Migrate from LangSmith          | [From LangSmith](/6.-build-wire-your-code/from-langsmith)                                   |
| Migrate from Braintrust         | [From Braintrust](/6.-build-wire-your-code/from-braintrust)                                 |
| Set up SSO                      | **Enterprise SSO** (Enterprise tier — contact sales)                                        |
| Understand pricing              | [Pricing](https://github.com/LayerLens/gitbook-full/blob/main/assets/screenshots/README.md) |
| File a support ticket           | [Support](https://github.com/LayerLens/gitbook-full/blob/main/assets/screenshots/README.md) |

## By source surface

| Surface                            | Section                                                                                                                |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `stratix.layerlens.ai` (anonymous) | [Stratix Public](https://github.com/LayerLens/gitbook-full/blob/main/assets/screenshots/README.md)                     |
| `stratix.layerlens.ai` (logged-in) | [Stratix Premium](https://github.com/LayerLens/gitbook-full/blob/main/05-select/catalog/premium-workspace-overview.md) |
| Python SDK                         | [Python SDK](/6.-build-wire-your-code/sdk-python)                                                                      |
| REST API                           | [API reference](/13.1-sdk-and-apis/api)                                                                                |
| CLI                                | [CLI reference](/13.1-sdk-and-apis/cli)                                                                                |
| VS Code extension                  | [VS Code](/6.-build-wire-your-code/vscode-extension)                                                                   |
| Slack                              | [Slack](/6.-build-wire-your-code/slack)                                                                                |


# What is LayerLens Stratix?

LayerLens Stratix is the AI evaluation platform — public catalog, private workspace, and SDK.

LayerLens Stratix is a complete platform for evaluating AI — from the moment you pick a model, through the build, into production, and across an organization. The platform is structured as **two product lines** (Model Evaluations and Agentic Evaluations) delivered through **three customer-facing experiences** (Stratix Public, Stratix Premium, and the Python SDK) that share one catalog, one judge engine, and one trace pipeline.

If you came in via the marketing site, you've seen the two product lines. If you're starting from this documentation, you'll mostly hear about the three experiences. They reconcile: the products are *what* you can do; the experiences are *where* you do it.

## High Level Overview

Most AI teams ship by intuition. They pick a model because it's popular. They iterate prompts because the last few examples felt better. They watch CSAT in production because that's the only signal that survives. By the time they know something's wrong, customers have already noticed.

Stratix gives you a real signal. Three of them, actually:

1. **Public catalog** — 175+ models scored against 52+ benchmarks, refreshed quarterly. See what's actually best at the task you care about, not what's loudest.
2. **Private workspace** — run evaluations on your own data, build LLM judges that match your team's quality bar, and score live traces. Stop guessing whether the new model is better; measure it.
3. **CI/CD gates** — wire evaluations into your pipeline. Block regressions. Promote winners. Ship with the same confidence you ship traditional code.

## The two product lines (what you can do)

* **Model Evaluations** — what the public catalog showcases: model + benchmark + score. Use it to pick the right model and to track how the frontier is moving.
* **Agentic Evaluations** — pre- and post-deployment quality gates for multi-step AI agents: assertions, deterministic rules, and LLM judges combined into a verdict + root-cause report.

Both share the same trace pipeline, the same judge engine, the same scorer library. You don't pick "one product" — you pick the workflow.

## What ships today

* **Models:** 200+ in the catalog, including GPT-5.3, Claude Opus 4.6, Gemini 3.1 Pro/Flash.
* **Benchmarks:** 52+ public benchmarks (MMLU, HumanEval, GSM8K, AGIEval, MATH, etc.).
* **Public evaluations:** 2,000+ runs visible at `stratix.layerlens.ai`.
* **Compare models:** head-to-head model comparisons across any benchmark.
* **Evaluation spaces:** public and private workspaces bundling models + datasets + scoring.
* **Scorers:** code graders for objective dimensions.
* **Judges:** LLM-as-a-judge for subjective dimensions, with the GEPA optimizer.
* **Traces:** upload trace JSON; run trace-evaluations against scorers and judges.
* **Learning library + paths:** structured curriculum on AI evaluation.
* **ECU credits:** pay-as-you-go billing for compute-intensive runs.
* **BYOK custom models:** point at your own OpenAI-compatible endpoint.
* **Multi-org:** users belong to multiple orgs; switch from the picker.
* **Auth:** GitHub OAuth, Google OAuth, email/password.
* **Search, notifications, onboarding wizard.**

## Where to next

* [Three experiences](/1.-introduction/three-experiences) — Public, Premium, SDK side-by-side
* [Agentic evaluations](/4.1-general-use-cases/agentic-evals-overview) — the pre- and post-deployment workflow
* [Pricing](/1.-introduction/01-introduction) — what each tier includes
* [Getting started](/2.-get-started/02-get-started) — pick a path and ship


# Overview

Overview of LayerLens Stratix — three experiences, the workflow spine, what's in the platform.

LayerLens Stratix is the AI evaluation platform for teams who need to know — before customers do — whether a model, a prompt, or an agent is good enough to ship. Three customer-facing experiences cover the full evaluation lifecycle: **Stratix Public** for anonymous research, **Stratix Premium** for team workflows, and the **Python SDK** for programmatic access.

{% hint style="info" %}
**Want to see Stratix in action?** [Schedule a 30-minute demo →](https://github.com/LayerLens/gitbook-full/blob/main/01-introduction/schedule-a-demo.md) — or jump straight into [Stratix Public](/2.-get-started/public) (no signup) and the [Free tier of Premium](/2.-get-started/sign-up).
{% endhint %}

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>What is it?</strong></td><td>High Level Overview and the "two products" framing.</td><td><a href="/1.-introduction/what-is-layerlens-stratix">Read</a></td></tr><tr><td><strong>Three experiences</strong></td><td>How Public, Premium, and the SDK fit together.</td><td><a href="/1.-introduction/three-experiences">Read</a></td></tr><tr><td><strong>Agentic evaluations</strong></td><td>Pre- and post-deployment quality gates for multi-step agents.</td><td><a href="/4.1-general-use-cases/agentic-evals-overview">Read</a></td></tr><tr><td><strong>Platform architecture</strong></td><td>The components: catalog, evaluation engine, judges, traces, learning.</td><td><a href="/13.3-concepts-library-+-architecture/architecture">Read</a></td></tr><tr><td><strong>Personas and paths</strong></td><td>Builder, Operator, Researcher, Admin, Buyer — pick yours.</td><td><a href="/1.-introduction/personas-paths">Read</a></td></tr><tr><td><strong>Quarterly reports</strong></td><td>Q1-Q4 research reports on model performance and benchmark trends.</td><td><a href="/5.-select-pick-the-model/quarterly-reports">Read</a></td></tr><tr><td><strong>Pricing tiers</strong></td><td>Free, Premium, Enterprise — what's included and what's not.</td><td><a href="/3.-pricing/pricing-tiers-overview">Read</a></td></tr><tr><td><strong>How Stratix compares</strong></td><td>Stratix vs Braintrust, LangSmith, Galileo, Promptfoo, Arize.</td><td><a href="/1.-introduction/how-stratix-compares">Read</a></td></tr><tr><td><strong>Trust and security</strong></td><td>Encryption, compliance, tenant isolation, audit, BYOK.</td><td><a href="https://github.com/LayerLens/gitbook-full/blob/main/01-introduction/trust.md">Read</a></td></tr><tr><td><strong>Schedule a demo</strong></td><td>Book a 30-minute walk-through — or start hands-on today.</td><td><a href="https://github.com/LayerLens/gitbook-full/blob/main/01-introduction/schedule-a-demo.md">Read</a></td></tr><tr><td><strong>Brand assets</strong></td><td>Product names, logos, voice, and boilerplate for partners.</td><td><a href="https://github.com/LayerLens/gitbook-full/blob/main/01-introduction/brand-assets.md">Read</a></td></tr></tbody></table>

## The Stratix Workflow

Six stages from picking a model to governing it in production:

1. **Select** — review leaderboards and benchmarks; optionally compare candidates side-by-side.
2. **Build** — wire your code or agent to emit traces.
3. **Observe** — see real production behavior across surfaces.
4. **Evaluate** — score outputs with LLM judges, LLM-backed scorers, and code graders.
5. **Improve** — tune prompts and judges (GEPA); re-run.
6. **Govern** — enforce gates in CI/CD and across your org.

Every concept, how-to, tutorial, and recipe in these docs maps to one of these stages. [Learn the workflow →](/1.-introduction/the-stratix-workflow)


# Why evaluate?

Why AI teams need evaluation discipline — the cost of shipping unmeasured AI.

Most teams ship AI by intuition. They pick a model because it's popular. They iterate prompts because the last few examples felt better. They watch CSAT in production because that's the only signal that survives. By the time they know something's wrong, customers have noticed.

## The cost of unmeasured AI

* **Wrong model from the start.** A 4-week feature build gets thrown out when the foundation model can't do the task.
* **Silent regressions.** A prompt change five PRs ago broke an edge case nobody noticed; customers find it first.
* **Vendor migrations driven by hunches.** Switching from one provider to another without quantitative justification.
* **Eval-by-eyeballing.** A handful of cherry-picked examples drive ship/no-ship decisions.

Evaluation discipline gets you out of all four traps.

## What evaluation discipline looks like

1. **Quantitative model selection.** You pick a model because it scored higher on the benchmarks closest to your task. Not because it was on the front page of HN.
2. **Versioned regression suites.** Every prompt change runs against a known set with a known baseline.
3. **Continuous scoring on production traffic.** When live behavior drifts, you know before customers do.
4. **Auditable evidence.** Releases cite the evaluation that gated them.

## What it takes

* A shared **mental model** for evaluation (this documentation)
* A **catalog** to anchor your model choices ([Stratix Public](/1.-introduction/01-introduction))
* A **workspace** to run private evaluations ([Stratix Premium](https://github.com/LayerLens/gitbook-full/blob/main/05-select/catalog/premium-workspace-overview.md))
* **Automation** to wire it into CI/CD ([SDK](/6.-build-wire-your-code/sdk-python), [CLI](/13.1-sdk-and-apis/cli))

## Three failure modes Stratix is shaped to prevent

1. **The "hopeful" launch.** Shipping without quantitative evidence the AI works.
2. **The slow regression.** Quality erosion over a quarter that nobody catches because nobody's watching the trend.
3. **The vendor lock-in mistake.** Picking a model based on superficial features and discovering, six months later, that switching is now expensive.

## Where to next

* [What is LayerLens Stratix?](/1.-introduction/what-is-layerlens-stratix)
* [Workflow](/1.-introduction/the-stratix-workflow)
* [Use cases](/4.1-general-use-cases/general)
* [Industry patterns](/4.2-industry-use-cases/travel-hospitality)


# Personas and paths

Five personas, five paths — Builder, Operator, Researcher, Admin, Buyer.

Stratix is read by five kinds of people. Each one has a different first question and a different "ah-ha" moment. Pick your path; we'll get you there fast.

## Builder

> *"I'm shipping an AI feature and I want it to be good."*

You write code. Your name is on the PR that adds the new prompt or the new agent. You want a tight feedback loop: write a change, run an eval, see the score change.

**First questions:** "How do I write my first eval? How do I score outputs my eyes can't grade?"

**Path:**

1. [Sign up for Premium](/2.-get-started/sign-up)
2. [Tutorial 1: First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation)
3. [Tutorial 2: Build your first judge](/8.-evaluate-score-the-outputs/02-first-judge)
4. [Workflow: Instrument](/9.-improve-tune-the-system/workflow) → [Evaluate](/9.-improve-tune-the-system/workflow)
5. [SDK quickstart](/2.-get-started/sdk)
6. [Cookbook](/2.-get-started/all-cookbook-recipes) — recipes by problem, framework, and integration

**You'll know you're winning when:** every prompt change has an eval result attached and you've stopped trusting your gut.

## Operator

> *"I run live AI in production and I need to keep it healthy."*

You own the on-call. You watch dashboards. You ship rollback scripts. AI quality is your latest reliability surface.

**First questions:** "How do I score real production traffic? How do I get paged when quality drifts?"

**Path:**

1. [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation)
2. [Concept: Traces and spans](/6.-build-wire-your-code/traces-and-spans)
3. [Tutorial 4: Score live traces](/8.-evaluate-score-the-outputs/04-score-traces)
4. [Workflow: Observe](/9.-improve-tune-the-system/workflow) → [Govern](/9.-improve-tune-the-system/workflow)
5. [Use case: AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd)
6. [Status and reliability](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/status/README.md)

**You'll know you're winning when:** quality regressions surface to you before they surface to customers.

## Researcher

> *"I want to know which model is best for my use case."*

You don't (yet) need to wire anything up. You need ground truth — what's the frontier, what's commoditized, what's punching above its weight.

**First questions:** "Which model is best at structured-output tasks? How is open-weight catching up?"

**Path:**

1. [Stratix Public](/1.-introduction/01-introduction) — start here, no signup
2. [Models catalog](/5.-select-pick-the-model/models-catalog)
3. [Benchmarks catalog](/5.-select-pick-the-model/benchmarks-catalog)
4. [Compare models](/5.-select-pick-the-model/compare-models) — head-to-head
5. [Quarterly reports](/5.-select-pick-the-model/quarterly-reports)
6. [Public evaluations](/5.-select-pick-the-model/public-evaluations)
7. (Optional) [Sign up for Premium](/2.-get-started/sign-up) to run a private evaluation on your own data

**You'll know you're winning when:** you can answer "is X actually better than Y for our task" in under 5 minutes.

## Admin

> *"I'm setting up Stratix for my team."*

You provision seats, manage SSO, watch the bill, write the IT-approval memo.

**First questions:** "How does auth work? How do I manage multiple orgs and seats? What's the data residency story?"

**Path:**

1. [Sign up](/2.-get-started/sign-up)
2. [Organizations and multi-org](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/organizations.md)
3. [Team management](/11.-admin/team-management)
4. [ECU credits and billing](/11.-admin/ecu-credits-billing)
5. [Settings](/11.-admin/settings)
6. [Enterprise](/1.-introduction/01-introduction) — SSO, data residency,
7. [Security and compliance](/1.-introduction/01-introduction)

**You'll know you're winning when:** auth, billing, and access control are boring.

## Buyer

> *"I'm evaluating Stratix for purchase."*

You may not be the user. You're the budget owner, the AI-platform PM, the head of engineering. You need to know whether Stratix is the right bet — and whether your team will adopt it.

**First questions:** "How does Stratix compare to Braintrust/LangSmith/etc? What's the ROI? Who else uses it?"

**Path:**

1. [What is LayerLens Stratix?](/1.-introduction/what-is-layerlens-stratix)
2. [Three experiences](/1.-introduction/three-experiences)
3. [How Stratix compares](/1.-introduction/how-stratix-compares-deep)
4. [Pricing](/1.-introduction/01-introduction)
5. [Industry patterns](/4.2-industry-use-cases/travel-hospitality)
6. [Use cases](/4.1-general-use-cases/general)
7. [Enterprise](/1.-introduction/01-introduction)
8. [Security and compliance](/1.-introduction/01-introduction)

**You'll know you're winning when:** you have a one-page rationale and your team has run a real evaluation in the trial.

## Where to next

* [Getting started](/2.-get-started/02-get-started) — pick a path and ship
* [Workflow](/1.-introduction/the-stratix-workflow) — the spine that ties all five personas together


# Three experiences (Public · Premium · SDK)

Stratix Public, Stratix Premium, and the Python SDK — what each one is for and how they work together.

LayerLens Stratix is delivered as three distinct customer-facing experiences. They share one catalog, one judge engine, and one trace pipeline — but each one is shaped for a specific kind of user and a specific kind of question.

## Side by side

|                             | Stratix Public                      | Stratix Premium                            | Python SDK                                                                 |
| --------------------------- | ----------------------------------- | ------------------------------------------ | -------------------------------------------------------------------------- |
| **Where**                   | `stratix.layerlens.ai` — signed out | `stratix.layerlens.ai` — after you sign in | `pip install layerlens --extra-index-url https://sdk.layerlens.ai/package` |
| **Auth**                    | none                                | OAuth (Google, GitHub) + email             | API key (`X-API-Key`)                                                      |
| **Audience**                | researchers, buyers, the curious    | builders, operators, admins                | builders, automation, CI/CD                                                |
| **What you do**             | browse, compare, read               | build, evaluate, govern                    | automate, integrate, batch                                                 |
| **Catalog visibility**      | full public catalog                 | full public + private                      | full public + private                                                      |
| **Private evaluations**     | no                                  | yes                                        | yes                                                                        |
| **Build judges**            | no                                  | yes                                        | yes                                                                        |
| **Trace upload**            | no                                  | yes                                        | yes                                                                        |
| **GEPA judge optimization** | no                                  | yes                                        | yes                                                                        |
| **Org and team management** | no                                  | yes                                        | n/a                                                                        |
| **Cost**                    | free                                | free + ECU PAYG                            | free + ECU PAYG                                                            |

## Stratix Public — the showcase

Anonymous browsing. No account required. Built for researchers, AI buyers, and anyone who wants to know "which model is actually best at X right now."

You can:

* Browse 175+ models with metadata, benchmark scores, and the latest leaderboard
* Browse 52+ benchmarks with descriptions and per-model scores
* Browse 2,000+ public evaluations
* Compare any two models head-to-head across any shared benchmark
* Browse public evaluation spaces — curated bundles of model + dataset + scoring
* Search the entire catalog
* Read the quarterly research reports

You cannot:

* Run a private evaluation on your data
* Save or share work to a personal workspace
* Build judges or scorers
* See trace data

Stratix Public is the doorway. Most users start here, find a model or a benchmark they care about, and continue into Premium when they're ready to evaluate against their own data.

## Stratix Premium — the workspace

Logged-in. Multi-org. Pay-as-you-go on compute. Built for teams shipping AI features.

You can:

* Run **private evaluations** on your own data
* Build **LLM judges** for subjective dimensions and optimize them with **GEPA**
* Upload **traces** and run **trace evaluations** against scorers and judges
* Manage **scorers** (code graders) at the org level
* Run **agentic evaluations** — pre- and post-deployment quality gates for multi-step agents
* Use **BYOK custom models** — register your own OpenAI-compatible endpoint
* Manage **organizations**, multi-org membership, and team seats
* Buy and consume **ECU credits**
* Browse the **Learning Library** and follow **Paths**

The left rail is: Home, Evaluations, Models, Benchmarks, Scorers, Spaces, Agent Evaluation (Traces, Judges), Learning (Library, Paths), Settings, Notifications. ECU balance and Buy Now appear in the header. The in-app **Assistant** offers context-aware help. The **Switch toggle** lets you flip between the Public and Premium catalogs without losing context.

## Python SDK — the automation surface

`pip install layerlens --extra-index-url https://sdk.layerlens.ai/package` (v1.3.0). 11 resources, 138 sample programs.

You can:

* Programmatically create evaluations, judges, scorers, traces
* Run trace evaluations as part of CI/CD
* Sync results into your own data warehouse or BI
* Automate judge GEPA optimization
* Integrate with frameworks (LangChain, LlamaIndex, Haystack, etc.) via instrumentation samples

The SDK is sync-and-async. Both `client.evaluations.create()` and `await client.evaluations.acreate()` work.

## How they work together

A typical team uses all three:

1. **A researcher** browses the public catalog and picks the three candidate models that look strongest on a relevant benchmark.
2. **A builder** logs into Premium, runs a private evaluation on the team's own dataset, and chooses a winner. They build a judge that captures the team's quality bar.
3. **A platform engineer** wires the SDK into CI/CD: every PR that changes a prompt re-runs the evaluation. Judges and scorers gate the merge.
4. **An operator** ingests live traces and runs trace evaluations on a daily cadence. Quality regressions surface in a dashboard before they surface to customers.

## Where to next

* [Stratix Public](/1.-introduction/01-introduction)
* [Stratix Premium](https://github.com/LayerLens/gitbook-full/blob/main/05-select/catalog/premium-workspace-overview.md)
* [Python SDK](/6.-build-wire-your-code/sdk-python)
* [Getting started](/2.-get-started/02-get-started) — choose a path and ship


# Public vs Premium — by user intent

Public vs Premium organized by user intent — Select, Build / Instrument, Observe, Evaluate, Improve, Govern.

This page answers "what do I do with each tier" by walking through the six user intents that map to the [Stratix Workflow](/1.-introduction/the-stratix-workflow). Each intent gets a same-axis comparison between **Stratix Public** (free, anonymous) and **Stratix Premium** (logged-in workspace).

## Select — pick the model

> "Which model should I use to build this feature?"

| Surface     | What you can do                                                                                                                                                                                                                                                                                | Limitation                                                                                                                                                           |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Public**  | Browse the [leaderboard](/5.-select-pick-the-model/leaderboard). Compare candidates head-to-head on the [public benchmarks catalog](/5.-select-pick-the-model/benchmarks-catalog). Test a prompt against any catalog model (rate-limited, anonymous).                                          | Public benchmarks are general-capability. They won't decide for your specific domain (pricing in your tariff, citing your jurisdiction, summarizing your documents). |
| **Premium** | Everything Public does — and your dashboard's catalog augments with **BYOK custom models** you've registered. Author **custom benchmarks** from your own data and run any candidate model against them. The selection narrowed by Public gets confirmed by a domain-aligned private benchmark. | None for the intent.                                                                                                                                                 |

## Build / Instrument — wire your code

> "How do I get observability into what my AI is actually doing?"

| Surface     | What you can do                                                                                                                                                                                                                                                                           | Limitation                                                                                         |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| **Public**  | Read the docs. Use the SDK's free tier (anonymous test calls).                                                                                                                                                                                                                            | No persistent traces, no organization-scoped workspace.                                            |
| **Premium** | Install the Python SDK (`pip install layerlens --extra-index-url https://sdk.layerlens.ai/package`) with your API key. Use `@trace`, `span()`, `instrument_openai()`, `instrument_anthropic()`, framework-specific callbacks (LangChain handler, etc.). Traces persist in your workspace. | Some framework adapters in private preview — see [Integrations](/1.-introduction/01-introduction). |

## Observe — see real production behavior

> "What's my AI actually doing in production?"

| Surface     | What you can do                                                                                                                                                                                  | Limitation                            |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------- |
| **Public**  | n/a — Public is for catalog browsing, not production observability.                                                                                                                              | No persistent trace ingestion at all. |
| **Premium** | Ingest traces from any application via SDK, CLI, or REST. Browse, search, and filter traces in the dashboard. Drill into individual traces with full span trees. Real-time and historical views. | None.                                 |

## Evaluate — score the outputs

> "Is this output good? Against my benchmark, my rubric, my labels?"

| Surface     | What you can do                                                                                                                                                                                                                    | Limitation                                                                    |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| **Public**  | View public evaluations (2,000+ pre-run scoring runs across the catalog). Read methodology. Reproduce scores in your own workspace if you upgrade.                                                                                 | Cannot run an evaluation on your data. Cannot author or run scorers / judges. |
| **Premium** | Run private evaluations on your benchmarks. Author **LLM-backed scorers** (model + prompt) reusable across evaluations. Author **LLM judges** (versioned, GEPA-tunable). Apply judges to traces directly via **trace evaluation**. | None.                                                                         |

## Improve — tune prompts, judges, models

> "My evaluation isn't where I need it. What do I tune?"

| Surface     | What you can do                                                                                                                                                                                                                                          | Limitation                             |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- |
| **Public**  | n/a — improvement is workflow-internal.                                                                                                                                                                                                                  | No iteration loop available on Public. |
| **Premium** | Iterate on prompts and re-evaluate against the same benchmark. Run **GEPA optimization** to tune judges against labeled examples — typically lifts judge agreement-with-humans from 60–70% to 85–95%. Maintain prompt versions as first-class artifacts. | None.                                  |

## Govern — enforce gates in CI/CD and across the org

> "How do I make sure regressions never reach production?"

| Surface     | What you can do                                                                                                                                                                                                                                                                                                                | Limitation      |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------- |
| **Public**  | n/a — governance requires your team and your CI/CD.                                                                                                                                                                                                                                                                            | Not applicable. |
| **Premium** | Wire evaluations into CI/CD as quality gates (drop-in [GitHub Actions](/6.-build-wire-your-code/cicd-github-actions), GitLab, Buildkite recipes). Run **continuous trace evaluation** on production samples with threshold alerts. SSO + role-based access (Enterprise tier). Audit log of every evaluation (Enterprise tier). | None.           |

## Summary table

| Intent     |    Public   | Premium |
| ---------- | :---------: | :-----: |
| Select     |      ●      |    ●    |
| Instrument | (read-only) |    ●    |
| Observe    |      —      |    ●    |
| Evaluate   |   (browse)  |    ●    |
| Improve    |      —      |    ●    |
| Govern     |      —      |    ●    |

## Where to next

* [The Stratix Workflow](/1.-introduction/the-stratix-workflow) — the six-stage spine these intents map to
* [Stratix Public](/1.-introduction/01-introduction) — full Public surface
* [Stratix Premium](https://github.com/LayerLens/gitbook-full/blob/main/05-select/catalog/premium-workspace-overview.md) — full Premium surface
* [Public vs Premium feature matrix](/5.-select-pick-the-model/public-vs-premium-1) — feature-by-feature table
* [Pricing](/1.-introduction/01-introduction)


# The Stratix Workflow — the six intents

The Stratix Workflow — six stages from picking a model to governing it in production.

Six stages from "I need to pick a model" to "this AI is governed in production." Every concept, how-to, tutorial, and recipe in these docs maps to one of them.

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>1. Select</strong></td><td>Review leaderboards and benchmarks; optionally compare candidates side-by-side.</td><td><a href="/5.-select-pick-the-model/workflow">Read</a></td></tr><tr><td><strong>2. Instrument</strong></td><td>Wire your code to emit traces.</td><td><a href="/6.-build-wire-your-code/workflow">Read</a></td></tr><tr><td><strong>3. Observe</strong></td><td>See real production behavior.</td><td><a href="/7.-observe-see-whats-happening/workflow">Read</a></td></tr><tr><td><strong>4. Evaluate</strong></td><td>Score with scorers, judges, and code graders.</td><td><a href="/8.-evaluate-score-the-outputs/workflow">Read</a></td></tr><tr><td><strong>5. Improve</strong></td><td>Tune prompts, judges, models. Re-run.</td><td><a href="/9.-improve-tune-the-system/workflow">Read</a></td></tr><tr><td><strong>6. Govern</strong></td><td>Enforce gates in CI/CD and across the org.</td><td><a href="/11.-admin/11-admin">Read</a></td></tr></tbody></table>

## Why a workflow spine

Without a spine, evaluation is a grab-bag of features. With a spine:

* Each stage has a clear question it answers
* The path from "picking a model" to "shipping confidently" is named
* Documentation links naturally between stages

## How the stages relate

The stages aren't strictly sequential; they're a cycle. Your AI is always at some stage, and the cycle goes around as it matures:

```
Select → Build → Observe → Evaluate → Improve → Govern
 ↑
 └─ back to Observe
```

The exception: **Select** can also reappear mid-cycle when a new model generation drops or your evaluation surfaces a model that's outgrown its envelope. Returning to Select means re-running the side-by-side and confirming your incumbent is still the right choice.

## Pick a starting stage

| If you're...             | Start at                                                                    |
| ------------------------ | --------------------------------------------------------------------------- |
| Thinking about adding AI | [Select](/9.-improve-tune-the-system/workflow) — pick a model               |
| Already shipping         | [Observe](/9.-improve-tune-the-system/workflow) — what's it actually doing? |
| Mid-build                | [Evaluate](/9.-improve-tune-the-system/workflow) — score it                 |
| In production            | [Govern](/9.-improve-tune-the-system/workflow) — gate it                    |

## Where to next

* [Select](/9.-improve-tune-the-system/workflow)
* [Concepts](/13.3-concepts-library-+-architecture/concepts-library)
* [Tutorials](/2.-get-started/all-tutorials)


# How Stratix compares

Stratix vs Braintrust, LangSmith, Galileo, Promptfoo, Arize — the short version.

The AI evaluation space is crowded. This page is the short, honest comparison. The deep version with feature-by-feature tables lives in [How Stratix compares (deep)](/1.-introduction/how-stratix-compares-deep).

## What's distinctly Stratix

These are the things you can do on Stratix today that are **not** equivalent on the alternatives:

* **Public model leaderboard.** 175+ models, 52+ benchmarks, refreshed quarterly. None of the alternatives operate a public catalog at this scale.
* **Public benchmark catalog.** Open browsing, no signup. Most alternatives keep benchmarks behind login.
* **Quarterly research reports.** Long-form, dated, methodologically transparent.
* **Industry pages.** 13 industries, scenario-mapped to platform capabilities.
* **Migration guides** from LangSmith, Braintrust, Galileo, Promptfoo, MLflow.
* **Pricing transparency.** Public Free tier, ECU PAYG visible without a sales call.

## What the alternatives do that we don't (yet)

Honesty up front:

* **Some have larger out-of-the-box framework integrations.** We ship a focused set; see [Integrations](/1.-introduction/01-introduction). When we don't have a turnkey integration, we have a sample.

## Honest competitive table

| Capability               | Stratix    | Braintrust | LangSmith | Galileo | Promptfoo | Arize   |
| ------------------------ | ---------- | ---------- | --------- | ------- | --------- | ------- |
| Public model leaderboard | yes        | no         | no        | no      | no        | no      |
| Public benchmark catalog | yes        | no         | no        | no      | no        | no      |
| Quarterly research       | yes        | no         | no        | no      | no        | no      |
| Migration guides         | 6          | minimal    | minimal   | minimal | minimal   | minimal |
| LLM-as-a-judge           | yes        | yes        | yes       | yes     | yes       | yes     |
| Judge optimization       | yes (GEPA) | partial    | no        | partial | no        | no      |
| Trace ingestion          | yes        | yes        | yes       | yes     | partial   | yes     |
| Trace evaluation         | yes        | yes        | yes       | yes     | partial   | yes     |
| Compare models           | yes        | partial    | no        | partial | yes       | partial |
| Public evaluation spaces | yes        | no         | no        | no      | no        | no      |
| Industry pages           | 13         | no         | no        | no      | no        | partial |
| Pricing transparency     | yes        | partial    | partial   | no      | yes       | partial |
| SDKs                     | yes        | yes        | yes       | yes     | yes       | yes     |

## Where to next

* [How Stratix compares (deep)](/1.-introduction/how-stratix-compares-deep) — feature-by-feature, with citations
* [Migration guides](/6.-build-wire-your-code/migration) — from LangSmith, Braintrust, Galileo, Promptfoo, MLflow, Arize
* [Pricing](/1.-introduction/01-introduction)
* [Industry patterns](/4.2-industry-use-cases/travel-hospitality)


# Deep comparison

Stratix vs Braintrust, LangSmith, Galileo, Promptfoo, Arize — feature-by-feature comparison.

The short version is on [Overview → How Stratix compares](/1.-introduction/how-stratix-compares). This page is the long version: feature-by-feature tables and the qualitative trade-offs.

## What's distinctly Stratix

These are capabilities Stratix offers today that the alternatives don't match:

1. **Public model leaderboard.** 175+ models scored against 52+ benchmarks at `stratix.layerlens.ai`. Anonymous browsing, refreshed quarterly. No competitor operates a public catalog of this scale.
2. **Public benchmark catalog.** Open browsing of 52+ benchmarks with descriptions and per-model scores.
3. **Quarterly research reports.** Long-form, dated, methodologically transparent reports on the model frontier.
4. **Industry pages.** 13 industries with scenario mappings to platform capabilities.
5. **Migration guides** from LangSmith, Braintrust, Galileo, Promptfoo, MLflow, Arize.
6. **Pricing transparency.** Free tier, ECU PAYG visible without a sales call.

## Feature-by-feature

### Public catalog and research

| Feature                  | Stratix      | Braintrust | LangSmith | Galileo | Promptfoo | Arize |
| ------------------------ | ------------ | ---------- | --------- | ------- | --------- | ----- |
| Public model leaderboard | yes          | no         | no        | no      | no        | no    |
| Public benchmark catalog | yes          | no         | no        | no      | no        | no    |
| Public evaluations       | yes (2,000+) | no         | no        | no      | no        | no    |
| Public evaluation spaces | yes          | no         | no        | no      | no        | no    |
| Quarterly research       | yes          | no         | no        | no      | no        | no    |
| Anonymous browsing       | yes          | no         | partial   | no      | no        | no    |

### Evaluation and judging

| Feature                               | Stratix    | Braintrust | LangSmith | Galileo | Promptfoo | Arize   |
| ------------------------------------- | ---------- | ---------- | --------- | ------- | --------- | ------- |
| LLM-as-a-judge                        | yes        | yes        | yes       | yes     | yes       | yes     |
| Built-in judge optimization           | yes (GEPA) | partial    | no        | partial | no        | no      |
| LLM-backed scorers                    | yes        | yes        | yes       | yes     | yes       | yes     |
| Code graders for deterministic checks | yes        | yes        | yes       | yes     | yes       | yes     |
| Compare-models head-to-head           | yes        | partial    | no        | partial | yes       | partial |
| Trace ingestion                       | yes        | yes        | yes       | yes     | partial   | yes     |
| Trace evaluation                      | yes        | yes        | yes       | yes     | partial   | yes     |
| Pre-deploy agentic evaluation         | yes        | partial    | partial   | partial | partial   | partial |

### Pricing and onboarding

| Feature                       | Stratix | Braintrust | LangSmith | Galileo | Promptfoo | Arize   |
| ----------------------------- | ------- | ---------- | --------- | ------- | --------- | ------- |
| Free tier with public catalog | yes     | partial    | partial   | no      | yes (OSS) | partial |
| Pricing visible without sales | yes     | partial    | partial   | no      | yes       | partial |
| Self-serve sign-up            | yes     | yes        | yes       | partial | yes       | partial |

## Migration paths

Coming from another tool? See:

* [From LangSmith](/6.-build-wire-your-code/from-langsmith)
* [From Braintrust](/6.-build-wire-your-code/from-braintrust)
* [From Galileo](/6.-build-wire-your-code/from-galileo)
* [From Promptfoo](/6.-build-wire-your-code/from-promptfoo)
* [From MLflow](/6.-build-wire-your-code/from-mlflow)
* [From Arize](/6.-build-wire-your-code/from-arize)

## Where to next

* [Pricing](/3.-pricing/03-pricing)
* [Industry patterns](/4.2-industry-use-cases/industry)


# Brand assets

Brand assets, product names, voice, and boilerplate for partners and press covering LayerLens Stratix.

Use this page to reference LayerLens Stratix correctly in partner materials, press, conference programs, and integrations. For anything not on this page — extended logo lockups, motion assets, talking points for a specific announcement — email <marketing@layerlens.ai>.

## Product naming

Use the name that matches the level of context the reader already has.

| Form                                                     | When to use                                                                                                                                                       |
| -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **LayerLens Stratix**                                    | First mention in any document, headline, press release, or partner page.                                                                                          |
| **Stratix**                                              | Second and subsequent mentions inside the same document. Also fine in product UI and conversational copy where the LayerLens parent brand is already established. |
| **Stratix Public**, **Stratix Premium**, **Stratix SDK** | When the specific experience matters — public catalog, logged-in workspace, or Python client.                                                                     |

Do **not** refer to the product as "LayerLens" on its own when the context is the evaluation platform. LayerLens is the parent brand; Stratix is the product. Calling the product "LayerLens" creates ambiguity for partners and analysts.

Also avoid the legacy internal codename ATEAM in customer-facing materials.

## Logos and isotype

The canonical logo files live in the `docs/logos/` directory of this repository.

| Asset                   | File                             |
| ----------------------- | -------------------------------- |
| Wordmark, color         | `docs/logos/LL-logo.svg`         |
| Wordmark, light-on-dark | `docs/logos/LL-logo-white.svg`   |
| Isotype (symbol only)   | `docs/logos/isotype.png`         |
| Isotype, 512x512 PNG    | `docs/logos/isotype-512x512.png` |

For social avatars, favicons, and small-format placements, use the isotype. For everywhere else, use the wordmark. Keep at least the height of the "L" as clear space on every side.

For high-resolution vector assets, EPS, or extended lockups, contact <marketing@layerlens.ai>.

## Color palette

These are placeholder reference values for documentation. The authoritative palette — including AA / AAA contrast pairings, dark-mode tokens, and gradient stops — is owned by the design team and ships with the design system. Confirm any production use with <marketing@layerlens.ai> before publishing.

| Role            | Token                   | Hex       |
| --------------- | ----------------------- | --------- |
| Primary         | `stratix.primary`       | `#5046E5` |
| Secondary       | `stratix.secondary`     | `#1A1A2E` |
| Accent          | `stratix.accent`        | `#22D3EE` |
| Surface — light | `stratix.surface.light` | `#FFFFFF` |
| Surface — dark  | `stratix.surface.dark`  | `#0B0B1A` |
| Text — primary  | `stratix.text.primary`  | `#0F172A` |
| Text — muted    | `stratix.text.muted`    | `#475569` |

## Voice

The Stratix voice is declarative and technical. We respect the reader's time and assume they are smart, busy, and skeptical of marketing claims.

* **Declarative, not hyped.** State what the product does. Skip the adjectives.
* **No exclamation marks.** None.
* **No superlatives** ("the best", "world-class", "revolutionary", "game-changing"). Replace with what the product actually does.
* **Specific over vague.** "175+ models, 52+ benchmarks" beats "comprehensive coverage."
* **Honest about scope.** If a capability is on the roadmap, say "on the roadmap." Do not imply it ships today.
* **Active voice.** "Stratix scores your traces" not "Your traces are scored by Stratix."
* **Sentence case** for headings.

When in doubt, lean toward the tone of a senior engineer explaining the system to a peer.

## Boilerplate — one paragraph

> LayerLens Stratix is an AI evaluation platform built for teams shipping AI to customers. The public catalog scores 175+ models against 52+ benchmarks and is refreshed quarterly, so anyone evaluating a model can answer "which is actually best at my task" in minutes — no signup required. The workspace lets teams run private evaluations on their own data, build LLM judges that capture their team's quality bar, ingest live traces, and wire all of it into CI/CD. Stratix is delivered as three experiences — a free public catalog, a logged-in workspace with pay-as-you-go compute, and a Python SDK — sharing one catalog, one judge engine, and one trace pipeline.

## Boilerplate — one sentence

> LayerLens Stratix is the AI evaluation platform that lets teams pick the right model, score it on their own data, and govern its quality in production.

## Press kit

For the full press kit — extended copy, headshots, screenshots, customer quotes (where permitted), and the latest fact sheet — contact <marketing@layerlens.ai>.

## Where to next

* [What is LayerLens Stratix?](/1.-introduction/what-is-layerlens-stratix)
* [Trust and security](/more-in-this-section/trust)
* [Schedule a demo](/more-in-this-section/schedule-a-demo)
* [How Stratix compares](/1.-introduction/how-stratix-compares)


# Schedule a demo

Book a 30-minute LayerLens Stratix walk-through, or start in the product today — for free.

See LayerLens Stratix on real data, with real models, in 30 minutes. A solutions engineer walks you through picking a model from the public catalog, instrumenting a sample agent, running an evaluation with an LLM judge, and tightening that judge with GEPA — end-to-end. You leave with a working playbook for your team's first evaluation, not a sales pitch.

If you'd rather start hands-on right now, both Stratix Public and the Free tier of Stratix Premium are open — no calendar invite required.

## Three ways to start

### 1. Schedule a 30-minute walk-through

Email <sales@layerlens.ai> with a couple of times that work for you and a one-line description of what you're trying to evaluate. We'll send a calendar invite, a short pre-call questionnaire, and a copy of the demo dataset so you can follow along live.

Bring a teammate. The demo is more useful when the builder, the operator, and the buyer are all in the room.

### 2. Try Stratix Public — no signup

Browse 175+ models and 52+ benchmarks anonymously at the public catalog. Run head-to-head model comparisons. Read the quarterly reports. Decide if the catalog answers your "which model is best for X" question before you spend any time on procurement.

* [Stratix Public — anonymous browsing](/2.-get-started/public)
* [Browse models](/2.-get-started/browse-models)
* [Compare two models](/2.-get-started/compare-two-models)

### 3. Sign up for Stratix Premium — free tier

Create a workspace, invite a teammate, run your first private evaluation on your own data. The free tier is permanent; you only pay for compute-heavy runs in ECU credits once you're ready.

* [Sign up for Premium](/2.-get-started/sign-up)
* [Your first evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation)

## What we cover in a 30-minute demo

The demo follows the Stratix workflow. We pick whichever of your real use cases is closest to "we need to know if this is good enough to ship" and walk it through four of the six stages live.

1. **Select** — open the public catalog, filter to the model family relevant to your problem, compare two finalists head-to-head, and pull up the latest quarterly report on that family.

* Reference: [Select — workflow stage](/5.-select-pick-the-model/workflow)

2. **Build** — instrument a sample agent (LangChain, LlamaIndex, or your stack of choice) with the Python SDK so it emits traces into the workspace.

* Reference: [Build — workflow stage](/6.-build-wire-your-code/workflow)

3. **Evaluate** — run an evaluation on a small dataset, score the outputs with a code grader plus an LLM judge, and read the verdict + root-cause panel.

* Reference: [Evaluate — workflow stage](/8.-evaluate-score-the-outputs/workflow)

4. **Improve** — tighten the judge with GEPA against a labelled fixture set, re-run the evaluation, and watch the regression / improvement land in the dashboard.

* Reference: [Improve — workflow stage](/9.-improve-tune-the-system/workflow)

If your audience includes a buyer or an IT reviewer, we also cover the [trust and security posture](/more-in-this-section/trust), pricing, and what a typical Enterprise rollout looks like.

## FAQ

### Can I bring my own data?

Yes. Most teams do. You can upload a dataset in CSV / JSONL, register a private benchmark on top of it, and run any model in the catalog — or a BYOK custom model — against it. See [BYOK custom models](/5.-select-pick-the-model/byok-custom-models).

### How long is the trial?

There is no trial countdown.

* **Stratix Public** is permanently free and anonymous.
* **Stratix Premium Free tier** is permanent. You can build evaluations, run judges, and ingest traces up to the free-tier limits indefinitely.
* **Stratix Premium PAYG** charges only for compute-heavy runs in ECU credits — you pay for what you use, when you use it.

Details: [Pricing — overview](/3.-pricing/03-pricing), [Free tier details](/3.-pricing/free-tier-details).

### Do I need to install anything before the demo?

No. The 30-minute demo runs entirely in the browser plus a shared notebook. If you'd like to follow along in your own environment, `pip install layerlens --extra-index-url https://sdk.layerlens.ai/package` ahead of time and we'll wire your local SDK into the demo workspace.

### Can my whole team join?

Yes. We recommend pulling in at least one builder, one operator (or on-call lead), and one decision-maker so the conversation covers the full lifecycle in one pass.

### What if I'm evaluating Stratix for procurement?

Send a note to <sales@layerlens.ai> and we'll route you to a longer technical session plus a security review track with <security@layerlens.ai>. See [Trust and security](/more-in-this-section/trust).

## Where to next

* [Trust and security](/more-in-this-section/trust)
* [How Stratix compares](/1.-introduction/how-stratix-compares)
* [Pricing — overview](/3.-pricing/03-pricing)
* [Personas and paths](/1.-introduction/personas-paths)


# Trust and security

Trust, security, and compliance posture for LayerLens Stratix — encryption, tenancy, audit, and how buyers run a security review.

LayerLens Stratix is built for teams whose AI ships in front of paying customers. That means evaluation data, traces, judge outputs, and audit history are treated with the same care as the production systems they govern. This page is the consolidated security and compliance summary for buyers, IT, and security reviewers.

For an in-depth review, contact <security@layerlens.ai>. For procurement and contract questions, contact <sales@layerlens.ai>.

## Encryption

* **In transit:** All traffic between clients, the SDK, the public catalog, and the workspace is TLS 1.2 or higher. Internal service-to-service traffic is mutually authenticated.
* **At rest:** All persistent data — evaluation results, traces, judge configurations, datasets, customer-supplied content — is encrypted with AES-256.
* **Customer-managed keys:** Available on the Enterprise tier.

## Compliance commitments

* **GDPR-aligned data handling** — DPA available; sub-processor list and data-flow diagram available on request.
* **CCPA** — consumer-data subject requests are supported through the same flow as GDPR data-subject requests.
* **HIPAA** — available on the Enterprise tier with an executable Business Associate Agreement (BAA). PHI workloads must be provisioned under a BAA-enabled tenant before any PHI is sent to the platform.

For the current compliance posture and timeline, request the latest summary from <security@layerlens.ai>.

## Tenant isolation

Every record — evaluations, judges, scorers, traces, datasets, API keys, audit events — is scoped to the organization that created it.

* **Per-org data scoping** at the database layer with row-level access controls. No application code path returns another organization's data.
* **No cross-tenant flow:** A judge built in Org A cannot read Org B's traces. A scorer registered in Org A is not visible to Org B unless explicitly published to the public catalog.
* **Workspace isolation:** Each organization is provisioned with its own logical workspace, including its own catalog overlay (private models, private benchmarks, private evaluations).
* **Cache and queue isolation:** All caches and processing queues are keyed by organization so that one tenant's burst cannot affect another tenant's latency or quality of service.

## Authentication

Three customer-facing authentication paths, all always active:

* **Workspace login** (Stratix Premium) — managed identity with email + password, Google OAuth, and GitHub OAuth out of the box. Sessions issued as short-lived JWTs.
* **API keys** (SDK, CLI, automation) — created from the workspace, rotated by an admin, scoped to one organization. Old keys remain valid for 24 hours after rotation so deployments can roll forward without downtime.
* **Single Sign-On (SSO)** — Enterprise tier. Okta, Azure AD / Entra ID, and any SAML 2.0 or OIDC identity provider. Group-to-role mapping is configurable per organization.

Multi-organization users switch between organizations from the workspace picker; every API call carries the active organization context.

## BYOK custom models

Customers can register their own OpenAI-compatible endpoints and bring their own provider keys (BYOK). When BYOK is configured:

* Keys are encrypted at rest within the customer's tenant boundary.
* Keys are used **only** to fulfill that customer's own evaluation, judge, or trace workload.
* Keys are never logged, never used outside the requesting organization, and never shared with the Stratix Public catalog.
* A customer can rotate or revoke a BYOK key from the workspace at any time; in-flight runs that depend on it will fail closed.

## Audit trail

Every meaningful action on the platform is recorded to an append-only audit log:

* Evaluation created, started, completed, cancelled
* Judge created, updated, run, optimized
* Trace ingested
* Dataset uploaded or deleted
* API key created, rotated, revoked
* Organization, team, or role change
* BYOK key registered or rotated

Workspace admins can view audit history from the Admin section. **Enterprise tier** customers can export the audit log on a schedule (CSV / JSON / Parquet) into their own SIEM or data warehouse.

## Data residency

* **Default:** US region.
* **Enterprise:** Additional regional residency options on request, including dedicated-tenant deployments and hybrid configurations where evaluation execution runs inside the customer's own VPC.

The data-residency story is part of every Enterprise procurement conversation. Reach out to <sales@layerlens.ai> for the current set of supported regions.

## Sub-processors

LayerLens Stratix relies on a short list of sub-processors for hosting, identity, and operational telemetry. The current sub-processor list is maintained as part of the DPA and is available to current and prospective Enterprise customers from <security@layerlens.ai>. The list is versioned; customers under contract are notified before any material change.

## Vulnerability disclosure

We welcome responsible disclosure of security issues. Report findings to <security@layerlens.ai>. Please include:

* A description of the issue and the affected surface (catalog, workspace, SDK, API).
* Steps to reproduce.
* Impact, if known.

We will acknowledge receipt within two business days, work with you on a remediation timeline, and credit you once a fix is shipped if you would like to be named.

## Run a security review

For a guided security review — questionnaire response, architecture walk-through, sub-processor list, DPA, BAA, contact <security@layerlens.ai>.

For a product demo that includes the security and compliance surface, [schedule a demo](/more-in-this-section/schedule-a-demo).

## Where to next

* [Schedule a demo](/more-in-this-section/schedule-a-demo)
* [Pricing — Enterprise tier details](/3.-pricing/enterprise-tier-details)
* [Three experiences](/1.-introduction/three-experiences)
* [Brand assets](/more-in-this-section/brand-assets)


# Getting started

Getting started with LayerLens Stratix — go from zero to your first evaluation, or pick the entry point that fits your role.

Welcome. This section gets you from "I just heard of LayerLens Stratix" to "I have a real evaluation running against my data" — fast. The page you're on is a launchpad; the links below are the actual ride.

{% hint style="warning" %}
**Hit an error already?** Skip ahead to [Common first-day errors](/2.-get-started/first-day-errors) — install, auth, env-var, and network problems most people see in their first hour, with fixes.
{% endhint %}

## Fastest path: 5 minutes to a real score

If you have a Python environment and 5 minutes, do this first. One file, copy-paste-runnable, ends with a real accuracy number and a UI link.

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>Run your first evaluation in 5 minutes</strong></td><td>One Python script. Install, auth, fetch a model, fetch a benchmark, run, inspect results.</td><td><a href="/more-in-this-section-1/first-eval-5-minutes">Start</a></td></tr></tbody></table>

Did the 5-minute run. Now what?

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>Top 5 starter recipes</strong></td><td>The five highest-ROI cookbook recipes for week one — one per workflow stage.</td><td><a href="/more-in-this-section-1/top-5-starter-recipes">Open</a></td></tr><tr><td><strong>Choose your path</strong></td><td>Match your role and goal to the right entry point: Public, Premium, or SDK.</td><td><a href="/2.-get-started/choose-your-path">Open</a></td></tr></tbody></table>

## Prefer to start in the UI?

Not everyone wants to start in Python. Three browser-first entry points — pick whichever matches what you're trying to learn.

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>Stratix Public</strong></td><td>Browse models, compare them, read benchmark scores. No account needed.</td><td><a href="/2.-get-started/public">Browse</a></td></tr><tr><td><strong>Stratix Premium</strong></td><td>Sign up, run your first private evaluation, build a judge in the UI.</td><td><a href="/2.-get-started/premium">Sign up</a></td></tr><tr><td><strong>Python SDK</strong></td><td><code>pip install layerlens --extra-index-url https://sdk.layerlens.ai/package</code> and run an eval from code.</td><td><a href="/2.-get-started/sdk">Install</a></td></tr></tbody></table>

Most teams use all three together. A typical first day: **Public** to shortlist candidate models, **Premium** to run a private eval on your data, **SDK** to wire the winner into CI.

## Browse by depth

Once you've run your first eval, dig into whichever shape of content matches how you learn.

| Format        | When to use it                                                             | Index                                                        |
| ------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------ |
| **Tutorials** | Guided, end-to-end, sequenced. Take an evening, finish a working pipeline. | [All tutorials](/2.-get-started/all-tutorials)               |
| **Cookbook**  | Problem-shaped recipes. "I have task X, give me code."                     | [All cookbook recipes](/2.-get-started/all-cookbook-recipes) |
| **Guides**    | Task-oriented walkthroughs. Bigger than a recipe, smaller than a tutorial. | [All guides](/2.-get-started/all-guides)                     |

## The workflow under the hood

Everything in Stratix maps to five stages. Knowing the stage helps you find the right doc fast.

| Stage        | What you're doing                                     | Where it lives                                         |
| ------------ | ----------------------------------------------------- | ------------------------------------------------------ |
| **Select**   | Pick models, benchmarks, datasets to evaluate against | [§5 Select](/5.-select-pick-the-model/workflow)        |
| **Build**    | Instrument your code so Stratix can see what it does  | [§6 Build](/6.-build-wire-your-code/workflow)          |
| **Observe**  | Capture traces, route them to where they're scored    | [§7 Observe](/7.-observe-see-whats-happening/workflow) |
| **Evaluate** | Score runs with judges and code graders               | [§8 Evaluate](/8.-evaluate-score-the-outputs/workflow) |
| **Improve**  | Optimize judges, cut cost, regression-protect         | [§9 Improve](/9.-improve-tune-the-system/workflow)     |

See [The Stratix workflow](/1.-introduction/the-stratix-workflow) for the full picture.

## When something breaks

* [Common first-day errors](/2.-get-started/first-day-errors) — auth, install, network, env-var problems.
* [SDK troubleshooting](/more-in-this-section-9/troubleshooting) — deeper SDK-specific issues.
* [Support](/12.-support/contact) — how to reach a human.
* [Status and reliability](/13.5-operational/status) — is it us or you?

## Where to next

The honest answer depends on what you're trying to do. If you're not sure:

1. Run [Your first evaluation in 5 minutes](/more-in-this-section-1/first-eval-5-minutes).
2. Pick one of the [Top 5 starter recipes](/more-in-this-section-1/top-5-starter-recipes) that matches a real problem you have.
3. When you hit something specific you need to learn, return to [All tutorials](/2.-get-started/all-tutorials) / [All cookbook recipes](/2.-get-started/all-cookbook-recipes) / [All guides](/2.-get-started/all-guides).


# Choose your path

Pick the Stratix entry point that fits what you're trying to do.

| If you...                                            | Start here                                                                                                                  |
| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| ...want to know which model is best at X             | [Stratix Public](/2.-get-started/public)                                                                                    |
| ...want to compare two models head-to-head           | [Stratix Public — Compare two models](/2.-get-started/compare-two-models)                                                   |
| ...want to run a private evaluation on your own data | [Stratix Premium — Sign up](/2.-get-started/sign-up) → [First evaluation](/2.-get-started/first-evaluation)                 |
| ...want to build an LLM judge                        | [Stratix Premium — First judge](/2.-get-started/first-judge)                                                                |
| ...want to upload a trace and score it               | [Stratix Premium — First trace](/2.-get-started/first-trace)                                                                |
| ...want to integrate evaluations into code or CI/CD  | [SDK quickstart](/2.-get-started/sdk)                                                                                       |
| ...are setting up for a team                         | [Stratix Premium — Sign up](/2.-get-started/sign-up) and [Personalized onboarding](/2.-get-started/personalized-onboarding) |
| ...just want to test your AI quickly                 | [Stratix Public — Test your AI](/2.-get-started/test-your-ai)                                                               |

## All three at once

Many teams use all three surfaces. A typical first day looks like:

1. **Public** — research the catalog, shortlist candidate models.
2. **Premium** — sign up, run a private evaluation against your data on the shortlist.
3. **SDK** — wire the winning evaluation into your code or CI gate.

You can switch between Public and Premium with the **Switch toggle** in the Premium top bar.

## Common pitfalls

* **Skipping the Public catalog and going straight to Premium.** You'll waste ECU running candidates that public scores would have ruled out.
* **Building a judge before you have a labeled set.** Judges only work as well as the labels you tune them against. Even 30 labels makes a meaningful difference.
* **Wiring CI gates before you've run the eval interactively.** Start in the Premium UI; promote to CI once you trust the result.

## Where to next

* [Stratix Public quickstart](/2.-get-started/public)
* [Stratix Premium quickstart](/2.-get-started/premium)
* [SDK quickstart](/2.-get-started/sdk)


# First-day errors

First-day errors — the auth, install, and network problems most people hit on day 1, with fixes.

Most new users hit one of these in their first hour. Find your symptom, apply the fix.

## "command not found: layerlens" after pip install

**Cause:** the `layerlens` CLI installed but isn't on your PATH.

**Fix:**

* On macOS/Linux: confirm `~/.local/bin` (or your venv `bin/`) is on PATH
* In a venv: activate it (`source.venv/bin/activate`)
* Reinstall in the active venv: `pip install layerlens --extra-index-url https://sdk.layerlens.ai/package`

## `pip install` returns "Could not find a version"

**Cause:** missing the `--extra-index-url` flag.

**Fix:** always use the LayerLens package index:

```bash
pip install layerlens --extra-index-url https://sdk.layerlens.ai/package
```

## `AuthenticationError: invalid API key`

**Cause:** wrong key, wrong env-var name, or expired key.

**Fix:**

1. Verify env var name is `LAYERLENS_STRATIX_API_KEY` (not `LAYERLENS_API_KEY`)
2. Echo it: `echo $LAYERLENS_STRATIX_API_KEY` — should show your key (or its prefix)
3. Generate a fresh key in **Settings → API keys**
4. If using a CI runner, check the secret name matches

## `ConnectionError` / `httpx.ConnectError` to `stratix.layerlens.ai`

**Cause:** firewall, proxy, or DNS.

**Fix:**

1. Confirm DNS: `nslookup stratix.layerlens.ai` should resolve
2. Confirm reachability: `curl -I https://stratix.layerlens.ai/health`
3. If behind a corporate proxy: set `HTTPS_PROXY` env var; ensure CA bundle is configured
4. If using `--extra-index-url`, confirm `sdk.layerlens.ai` is also reachable

## `ImportError: cannot import name 'Stratix' from 'layerlens'`

**Cause:** old SDK version, or `layerlens` namespace collision.

**Fix:**

1. `pip show layerlens` — verify version ≥ 1.3.0
2. Upgrade: `pip install --upgrade layerlens --extra-index-url https://sdk.layerlens.ai/package`
3. If still failing, check for a local file or directory named `layerlens.py` shadowing the install

## `RateLimitError` on first 5 calls

**Cause:** rare on day 1; usually concurrent retries from a flaky network.

**Fix:** the SDK retries with backoff automatically. If it persists, drop concurrency and confirm your API key isn't shared across multiple processes.

## "I don't see my evaluation in the UI"

**Cause:** wrong active org.

**Fix:** the API key is scoped to one org; the dashboard's active org may differ. Switch active org in the top bar to match the key's org. See [Organizations](/more-in-this-section-9/organizations-1).

## CI: `LAYERLENS_STRATIX_API_KEY` not set

**Cause:** secret not configured on the runner.

**Fix (GitHub Actions):**

```yaml
env:
 LAYERLENS_STRATIX_API_KEY: ${{ secrets.LAYERLENS_STRATIX_API_KEY }}
```

For other CIs, see [CI/CD integrations](/more-in-this-section-4/cicd).

## Still stuck

* [SDK troubleshooting](/more-in-this-section-9/troubleshooting)
* [Support — contact](/12.-support/contact)
* [Status and reliability](/13.5-operational/status)


# Stratix Public — anonymous browsing

Stratix Public quickstart — browse, compare, test your AI without signup.

`stratix.layerlens.ai` is anonymous and free. No signup. Four things to try first.

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>Browse models</strong></td><td>Filter and search the catalog of 175+ models.</td><td><a href="/2.-get-started/browse-models">Start</a></td></tr><tr><td><strong>Compare two models</strong></td><td>Put any two models head-to-head across a benchmark.</td><td><a href="/2.-get-started/compare-two-models">Start</a></td></tr><tr><td><strong>Browse benchmarks</strong></td><td>52+ benchmarks with descriptions and per-model scores.</td><td><a href="/2.-get-started/browse-benchmarks">Start</a></td></tr><tr><td><strong>Test your AI</strong></td><td>Quick test of a model on a prompt — no signup.</td><td><a href="/2.-get-started/test-your-ai">Start</a></td></tr></tbody></table>

## What you can do anonymously

* Browse and search the **models catalog**, with full metadata (provider, context window, modalities, licensing)
* Browse and search the **benchmarks catalog**
* View **public evaluations** — 2,000+ runs by the LayerLens team and partners
* **Compare any two models** head-to-head across any shared benchmark
* Browse **public evaluation spaces** — curated bundles
* Browse the **leaderboard** sorted by capability
* Read the **quarterly research reports**

## What requires Premium

* Running a private evaluation on your own dataset
* Saving spaces to a personal workspace
* Building judges or scorers
* Uploading traces

When you're ready, head to [Premium quickstart](/2.-get-started/premium).

## Where to next

* [Browse models](/2.-get-started/browse-models)
* [Compare two models](/2.-get-started/compare-two-models)
* [Browse benchmarks](/2.-get-started/browse-benchmarks)
* [Test your AI](/2.-get-started/test-your-ai)
* [Stratix Public reference](/2.-get-started/public)


# Browse models

Browse the public Stratix catalog of 175+ models, filter by provider, capability, and licensing.

The Stratix Public **Models** page lists 175+ models with rich metadata and benchmark scores. No account needed.

## Steps

### 1. Open the catalog

Go to [`stratix.layerlens.ai/models`](https://stratix.layerlens.ai/models). The full catalog is on screen.

### 2. Filter

Use the filters in the sidebar to narrow:

* **Provider** — OpenAI, Anthropic, Google, Meta, Mistral, etc.
* **Capability** — chat, code, reasoning, vision, embeddings
* **Context window** — pick a minimum
* **Licensing** — proprietary, open-weight
* **Modalities** — text, vision, audio

### 3. Sort

Sort by:

* Score on a chosen benchmark (top of list = best at that benchmark)
* Most recent (newest models first)
* Most popular (by view count)

### 4. Open a model

Click any model card. The model page shows:

* Provider and licensing
* Context window and modalities
* Per-benchmark scores
* Score history over time
* Public evaluations that include this model

### 5. Compare

From any model page, click **Compare** to put it side-by-side with another model. See [Compare two models](/2.-get-started/compare-two-models).

## Verify

You should be able to:

* Find your favorite frontier model on the first page
* Filter to "open-weight" and see open-weight models only
* Sort by score on MMLU and see the leaderboard

## Where to next

* [Compare two models](/2.-get-started/compare-two-models)
* [Browse benchmarks](/2.-get-started/browse-benchmarks)
* [Stratix Public — Models catalog reference](/5.-select-pick-the-model/models-catalog)
* [Concept: Models and benchmarks](/5.-select-pick-the-model/models-and-benchmarks)


# Compare two models

Compare any two models head-to-head across a shared benchmark — no signup needed.

Putting two models side-by-side is one of the highest-leverage things Stratix does. Use it during model selection, when a new model drops, or when leadership asks "is X better than Y."

## Steps

### 1. Open Compare models

Go to [`stratix.layerlens.ai/compare-models`](https://stratix.layerlens.ai/compare-models).

### 2. Pick the first model

Search and select a model in the left column. The page populates with that model's benchmark scores.

### 3. Pick the second model

Same in the right column. The page now shows both models' scores on every benchmark either has been evaluated against.

### 4. Read the comparison

The score table highlights:

* Where each model wins (green)
* Where they tie (gray)
* Where the gap is large vs small

### 5. Pick a benchmark to focus on

Click any benchmark row to drill into the comparison for that benchmark — including raw scores, sample inputs, and (where available) sample outputs from each model.

### 6. Save the comparison URL

The URL contains both model IDs. Bookmark it or share with your team.

## Verify

You should be able to compare GPT-5.3 vs Claude Opus 4.6 on MMLU and see both models' scores on the same row.

## Common patterns

* **Frontier vs cost-optimized.** Compare a frontier model against a smaller, cheaper one for tasks where you suspect the smaller model is good enough.
* **Provider against provider.** Compare an OpenAI model against an Anthropic model on the benchmark closest to your use case.
* **Old vs new generation.** When a new generation drops, compare it against the predecessor on benchmarks you care about.

## Where to next

* [Browse benchmarks](/2.-get-started/browse-benchmarks)
* [Premium — Compare models with your dataset](/2.-get-started/first-evaluation)
* [Stratix Public — Compare models reference](/5.-select-pick-the-model/compare-models)
* [Use case: Model evaluation](/4.1-general-use-cases/model-evaluation)


# Browse benchmarks

Browse 52+ public benchmarks with descriptions, methodology, and per-model scores.

The Stratix Public **Benchmarks** catalog is the broadest open-browsable collection of LLM benchmarks today. Each benchmark has metadata, methodology notes, and per-model scores.

## Steps

### 1. Open the catalog

Go to [`stratix.layerlens.ai/benchmarks`](https://stratix.layerlens.ai/benchmarks).

### 2. Filter

* **Capability** — reasoning, code, math, multilingual, vision, multi-turn
* **Difficulty** — easy / medium / hard
* **Sample size** — small / medium / large

### 3. Open a benchmark

Click any benchmark card. The page shows:

* Description and methodology
* Sample tasks
* Per-model scores
* Top performers
* Score-history-over-time chart
* Public evaluations that ran this benchmark

### 4. Pick a model and see its score

From the benchmark page, click any model in the score table to jump to that model on this benchmark.

## Verify

You should be able to find MMLU, HumanEval, GSM8K on the first page.

## How to pick a benchmark for your use case

Anti-pattern: "everyone uses MMLU, so I'll use MMLU."

Instead:

* **Reasoning-heavy task?** Look at GSM8K, MATH, ARC.
* **Code?** HumanEval, MBPP, SWE-Bench.
* **Multi-turn dialog?** MT-Bench.
* **Multilingual?** MMLU translated, FLORES.
* **Tool use / agents?** ToolBench, AgentBench.
* **Reading comprehension?** SQuAD, DROP.

## Where to next

* [Browse models](/2.-get-started/browse-models)
* [Compare two models](/2.-get-started/compare-two-models)
* [Stratix Public — Benchmarks catalog reference](/5.-select-pick-the-model/benchmarks-catalog)
* [Concept: Models and benchmarks](/5.-select-pick-the-model/models-and-benchmarks)


# Test your AI

Test a model on a prompt anonymously — no signup required.

Stratix Public includes a quick sandbox: pick a model, paste a prompt, see the response. Use it for ad-hoc model intuition; use Premium for anything you want to track.

## Steps

### 1. Open the sandbox

Go to [`stratix.layerlens.ai`](https://stratix.layerlens.ai) and open the **Test your AI** entry from the top nav (or model page).

### 2. Pick a model

Search and select.

### 3. Paste your prompt

Free-form text input. System message and user message both supported.

### 4. Run

Click **Run**. Response appears below.

### 5. Try another model

Switch models and rerun the same prompt; compare responses.

## Limits (anonymous)

* Rate-limited per IP — fine for ad-hoc, not for batch
* No persistence — close the tab and the run is gone
* No metrics, no scoring — just the response

## When to upgrade to Premium

When you want to:

* Persist results to a workspace
* Score responses (deterministic + LLM judges)
* Run against a dataset, not one prompt at a time
* Compare across many models in one job
* Save and re-run

[Sign up for Premium →](/2.-get-started/sign-up)

## Where to next

* [Stratix Premium — First evaluation](/2.-get-started/first-evaluation)
* [Compare two models](/2.-get-started/compare-two-models)
* [Concept: Evaluations](/8.-evaluate-score-the-outputs/evaluations-1)


# Stratix Premium — logged-in workspace

Stratix Premium quickstart — sign up, run your first evaluation, build your first judge.

Five steps to a working Premium account with a real evaluation on your data.

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>1. Sign up</strong></td><td>Create an account with GitHub, Google, or email.</td><td><a href="/2.-get-started/sign-up">Start</a></td></tr><tr><td><strong>2. Personalized onboarding</strong></td><td>Let the wizard tailor your workspace.</td><td><a href="/2.-get-started/personalized-onboarding">Start</a></td></tr><tr><td><strong>3. First evaluation</strong></td><td>Run a private evaluation on your data.</td><td><a href="/2.-get-started/first-evaluation">Start</a></td></tr><tr><td><strong>4. First trace</strong></td><td>Upload a trace and inspect it.</td><td><a href="/2.-get-started/first-trace">Start</a></td></tr><tr><td><strong>5. First judge</strong></td><td>Build an LLM judge for a subjective dimension.</td><td><a href="/2.-get-started/first-judge">Start</a></td></tr></tbody></table>

## What you'll have at the end

* A Stratix Premium account at `stratix.layerlens.ai`
* One organization with starter ECU credits
* One private evaluation run with results
* One uploaded trace, inspected
* One custom judge ready to use

Total time: \~30 minutes if you already have data; \~60 minutes from scratch.

## Where to next

* [Sign up](/2.-get-started/sign-up)
* [Stratix Premium reference](https://github.com/LayerLens/gitbook-full/blob/main/05-select/catalog/premium-workspace-overview.md)


# Sign up

Create a Stratix Premium account with GitHub, Google, or email — no card required for Free.

Three auth methods. Pick whichever your team uses today.

## Steps

### 1. Open the sign-up page

Go to [`stratix.layerlens.ai/sign-up`](https://stratix.layerlens.ai/sign-up).

### 2. Pick an auth method

* **Continue with GitHub** — OAuth, fastest if you're a developer
* **Continue with Google** — OAuth via your Google account
* **Email** — email + password; verification email sent

### 3. Verify (email only)

If you signed up with email, check your inbox for the verification link. Click it. You're back in the app.

### 4. Pick or create your organization

If your email domain matches an existing org, you can request to join. Otherwise, create a new org. You can be invited to additional orgs later and switch between them.

### 5. Optional: complete the onboarding wizard

The first time you land in Premium, the wizard offers to personalize your workspace. See [Personalized onboarding](/2.-get-started/personalized-onboarding).

## Verify

You should see the Premium home dashboard at `stratix.layerlens.ai/dashboard`. The left rail shows: Home, Evaluations, Models, Benchmarks, Scorers, Spaces, Agent Evaluation, Learning, Settings, Notifications.

## Multi-org and switching

If you're invited to a second org later:

1. Accept the invitation from the email
2. Use the org switcher in the top bar to flip between orgs
3. The dashboard data refreshes for the active org

## Common pitfalls

* **Email signup but no verification email.** Check spam. If still missing, see [Troubleshooting auth](/2.-get-started/premium).
* **GitHub/Google OAuth blocked by org.** Some enterprise GitHub orgs require admin approval before third-party OAuth apps work. Ask your admin to allow the LayerLens app.

## Where to next

* [Personalized onboarding](/2.-get-started/personalized-onboarding)
* [First evaluation](/2.-get-started/first-evaluation)
* [Organizations and multi-org](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/organizations.md)


# Personalized onboarding

Use the Stratix Premium onboarding wizard to tailor your workspace to your role and goals.

The first time you land in Premium, the onboarding wizard offers to tailor the workspace. It's optional — skip it if you'd rather explore — but it saves 10 minutes if you let it.

## Steps

### 1. Open the wizard

It opens automatically on first sign-in. If you skipped, you can re-run it from **Settings → Show tutorial**.

### 2. Tell us your role

* **Builder** — shipping AI features
* **Operator** — running AI in production
* **Researcher** — evaluating models for purchase or study
* **Admin** — setting up for a team
* **Buyer** — evaluating Stratix for purchase

The wizard tailors home dashboard widgets and the in-app Assistant prompts to your role.

### 3. Tell us your goals

Pick one or more:

* Choose a model
* Build an evaluation
* Score live traces
* Build a judge
* Gate CI/CD

The wizard surfaces the relevant first-time tutorials in the home dashboard.

### 4. Pick a starter dataset (optional)

If you don't have a dataset ready, you can run your first evaluation on a starter dataset. This lets you exercise the workflow without waiting on data prep.

### 5. Finish

The wizard drops you into a tailored home dashboard. Quick links to the next step in your goal.

## Verify

The home dashboard shows widgets relevant to your role. If you picked "build an evaluation" as a goal, you should see a "Start your first evaluation" card front-and-center.

## Where to next

* [First evaluation](/2.-get-started/first-evaluation)
* [First trace](/2.-get-started/first-trace)
* [First judge](/2.-get-started/first-judge)
* [Stratix Premium — Onboarding wizard](/11.-admin/onboarding-wizard)


# Your first evaluation

Run your first private evaluation in Stratix Premium — pick a model, pick a dataset, see scores in minutes.

A private evaluation runs a model against a dataset and scores the outputs. Here's the fastest way to do one.

## Prerequisites

* [ ] A Stratix Premium account ([sign up](/2.-get-started/sign-up))
* [ ] Either: a small dataset (CSV or JSONL with input → expected output columns), or willingness to use a starter dataset

## Steps

### 1. Open Evaluations

In the left rail, click **Evaluations**. Click **New evaluation**.

### 2. Pick a model

Search and select. The full public catalog is searchable, plus any BYOK custom models you've registered.

### 3. Pick a dataset

Two options:

* **Upload a dataset** — drag a CSV/JSONL onto the upload zone. Stratix infers the schema; you confirm input and expected-output columns.
* **Use a starter dataset** — pick a benchmark from the public catalog (e.g., MMLU subset).

### 4. Pick scoring

For your first eval, pick a code grader that fits your data:

* **Exact match** — outputs equal to expected
* **Semantic similarity** — fuzzy match using an embedding model
* **Regex match** — outputs match a pattern
* **JSON schema** — outputs validate against a schema

Skip judges for now — you'll build one later.

### 5. Preview cost

Stratix shows the worst-case ECU consumption before you run. Approve.

### 6. Run

Click **Run**. The evaluation queues; results stream in row-by-row.

### 7. Read the results

The results page shows:

* Overall score
* Per-row scores (input → output → score)
* Score distribution
* Latency and cost per row

## Verify

You should have a completed evaluation with a top-line score. Click any failing row to see the input, the model output, and the expected output side-by-side.

## What just happened

Stratix called the model on each input row, ran the scorer against each output, and aggregated the result. Every row is stored — you can re-open this evaluation later or compare it against a future run.

## What to try next

* **Run the same evaluation against a different model.** Stratix's compare-models view shows both runs side-by-side.
* **Iterate on the prompt.** If your dataset includes a prompt template column, change the template and rerun.
* **Add a judge.** See [First judge](/2.-get-started/first-judge) for subjective dimensions.

## Where to next

* [First judge](/2.-get-started/first-judge)
* [First trace](/2.-get-started/first-trace)
* [Tutorial: First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation)
* [Stratix Premium — Evaluations](/8.-evaluate-score-the-outputs/evaluations)


# Your first trace

Upload a trace to Stratix Premium and inspect it — see every span, every cost, every error.

A trace is a record of an AI call (or a chain of calls). Uploading a trace into Premium lets you inspect it in the UI and run trace evaluations against it.

## Prerequisites

* [ ] A Stratix Premium account ([sign up](/2.-get-started/sign-up))
* [ ] Either: a JSON file with one trace, or willingness to use the SDK to ingest a sample trace

## Option A — Upload via UI

### 1. Open Traces

Left rail: **Agent Evaluation → Traces**. Click **Upload trace**.

### 2. Drop the JSON file

A valid trace has at least:

```json
{
 "id": "trace-1",
 "name": "answer-question",
 "inputs": {"prompt": "What is the capital of France?"},
 "outputs": {"response": "Paris."},
 "spans": [
 {"name": "llm-call", "kind": "llm", "model": "claude-opus-4-7",...}
 ]
}
```

### 3. Confirm

Stratix parses the JSON, infers the schema, and shows a preview. Click **Save**.

### 4. Inspect

The trace page shows:

* The full span tree
* Per-span model, latency, cost, errors
* Inputs and outputs at every level

## Option B — Ingest via SDK

```python
import os
from layerlens import Stratix

client = Stratix(api_key=os.environ["LAYERLENS_STRATIX_API_KEY"])
client.traces.create(
 id="trace-1",
 name="answer-question",
 inputs={"prompt": "What is the capital of France?"},
 outputs={"response": "Paris."},
 spans=[
 {"name": "llm-call", "kind": "llm", "model": "claude-opus-4-7"}
 ],
)
```

The trace appears in **Agent Evaluation → Traces** within seconds.

## Verify

Open the trace page; you should see your trace's span tree, inputs, and outputs.

## What to try next

* **Score this trace.** See [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations).
* **Ingest more traces.** A representative trace set is the starting point for agentic evaluations.

## Where to next

* [First judge](/2.-get-started/first-judge)
* [Stratix Premium — Traces](/7.-observe-see-whats-happening/traces)
* [Concept: Traces and spans](/6.-build-wire-your-code/traces-and-spans)
* [Tutorial: Score live traces](/8.-evaluate-score-the-outputs/04-score-traces)


# Your first judge

Build your first LLM judge in Stratix Premium — define a rubric, test it, optionally GEPA-optimize.

A judge is an LLM that scores subjective dimensions — helpfulness, faithfulness, tone, safety — that a code grader can't grade. Building one takes about 15 minutes.

## Prerequisites

* [ ] A Stratix Premium account ([sign up](/2.-get-started/sign-up))
* [ ] A clear definition of what you want to grade (write it in plain English first)
* [ ] At least 10 example outputs labeled "good" or "bad" — more is better

## Steps

### 1. Open Judges

Left rail: **Agent Evaluation → Judges**. Click **New judge**.

### 2. Name and describe

* **Name:** e.g., "helpfulness-customer-support"
* **Description:** what this judge grades, in one sentence
* **Output type:** binary (pass/fail), score (e.g., 1-5), or labeled (e.g., "helpful" / "neutral" / "unhelpful")

### 3. Pick a judging model

The model that runs the rubric. Default is a balanced choice; pick a stronger model for harder rubrics.

### 4. Write the rubric

The rubric is the prompt the judging model uses. Best practices:

* State the dimension you're grading
* Describe what "good" looks like with examples
* Describe what "bad" looks like with examples
* Show the output format you want back

The Premium UI provides a starter template you can adapt.

### 5. Test on a few examples

Paste 3-5 sample outputs and run the judge. Read the verdicts. If they don't match your intuition, iterate the rubric.

### 6. Optional: GEPA-optimize

If you have ≥30 labeled examples, run **GEPA optimization**. Stratix tunes the rubric to better match your labels. Agreement with humans typically rises 10-20 percentage points.

[More about GEPA optimization](/8.-evaluate-score-the-outputs/judges-1#judge-optimization-gepa)

### 7. Save

The judge is now reusable in any evaluation, trace evaluation, or agentic evaluation in your org.

## Verify

You should be able to apply your judge to a single sample input/output and get back a verdict.

## What to try next

* **Apply the judge to your first evaluation.** Open the eval, add the judge, rerun.
* **Apply the judge to a trace evaluation.** Real production data, judged.
* **Build a second judge** for a different dimension. Stratix evaluations let you stack many judges.

## Where to next

* [Tutorial: Build your first judge](/8.-evaluate-score-the-outputs/02-first-judge)
* [Tutorial: Optimize a judge with GEPA](/9.-improve-tune-the-system/05-gepa-optimize)
* [Concept: Judges](/8.-evaluate-score-the-outputs/judges-1)
* [Stratix Premium — Judges](/8.-evaluate-score-the-outputs/judges)


# SDK

Stratix Python SDK quickstart — install, authenticate, run an evaluation in code.

The `layerlens` PyPI package (v1.3.0) is the Python client SDK for Stratix. Install, set an API key, and you can do everything the Premium UI does — programmatically.

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>1. Install</strong></td><td>One pip command, three minutes.</td><td><a href="/2.-get-started/install">Start</a></td></tr><tr><td><strong>2. First evaluation via SDK</strong></td><td>Run a benchmark evaluation from Python.</td><td><a href="/2.-get-started/first-eval-via-sdk">Start</a></td></tr></tbody></table>

## Why use the SDK

* **Automate.** CI/CD gates, scheduled trace evaluations, batch jobs.
* **Integrate.** Wire Stratix into your existing data pipeline.
* **Share recipes.** Sample programs across every major AI agent surface — copy, paste, ship.

## Samples for the AI agent ecosystem

Stratix ships first-class samples for the platforms your team is most likely already on:

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>MCP Server</strong></td><td>Evaluate Model Context Protocol servers and tool-use trajectories.</td><td><a href="/2.-get-started/samples-overview">Browse</a></td></tr><tr><td><strong>Claude Code Skills</strong></td><td>Score Claude Code skill invocations, deltas, and policy gates.</td><td><a href="/2.-get-started/samples-overview">Browse</a></td></tr><tr><td><strong>Claude CoWork</strong></td><td>Trace multi-agent CoWork sessions and compare runs.</td><td><a href="/2.-get-started/samples-overview">Browse</a></td></tr><tr><td><strong>CopilotKit</strong></td><td>Wire CopilotKit-powered agents into Stratix traces, judges, and CI gates.</td><td><a href="/2.-get-started/samples-overview">Browse</a></td></tr><tr><td><strong>OpenAI Agents SDK</strong></td><td>End-to-end coverage for the OpenAI Agents pattern.</td><td><a href="/2.-get-started/samples-overview">Browse</a></td></tr><tr><td><strong>LangChain / LlamaIndex / Haystack / DSPy</strong></td><td>Framework-native instrumentation samples.</td><td><a href="/2.-get-started/samples-overview">Browse</a></td></tr></tbody></table>

## What's the SDK shape

```python
from layerlens import Stratix

client = Stratix() # picks up LAYERLENS_STRATIX_API_KEY from env

# 11 resource handlers
client.models # public model catalog
client.benchmarks # public benchmark catalog
client.evaluations # private evaluations
client.scorers # code graders
client.judges # LLM judges
client.judge_optimizations # GEPA runs
client.traces # trace ingest + browse
client.trace_evaluations # score traces
client.results # result browsing
client.public # public-only convenience namespace
client.errors # exception types
```

Every method has a sync and an async form. `client.evaluations.create(...)` and `client.evaluations.acreate(...)`.

## Where to next

* [Install](/2.-get-started/install)
* [First evaluation via SDK](/2.-get-started/first-eval-via-sdk)
* [Python SDK reference](/6.-build-wire-your-code/sdk-python)
* [Samples overview](/2.-get-started/samples-overview) — MCP, Claude Code Skills, CoWork, CopilotKit, OpenAI Agents, and framework-native examples
* [Cookbook](/2.-get-started/all-cookbook-recipes) — recipes ported from `stratix-python/samples/`


# Install

Install the Stratix Python SDK from the LayerLens package registry.

## Requirements

* Python 3.8 or higher
* pip

## Install

```bash
pip install layerlens --extra-index-url https://sdk.layerlens.ai/package
```

## Verify

```python
import layerlens
print(layerlens.__version__)
```

You should see `1.3.0` (or later).

## Set your API key

Get an API key from `stratix.layerlens.ai/settings/api-keys`. Then export:

```bash
export LAYERLENS_STRATIX_API_KEY="your-api-key"
```

Or pass it explicitly when creating the client:

```python
from layerlens import Stratix
client = Stratix(api_key="your-api-key")
```

## Virtual environment (recommended)

```bash
python -m venv.venv
# Linux/macOS:
source.venv/bin/activate
# Windows PowerShell:
#.venv\Scripts\Activate.ps1

pip install layerlens --extra-index-url https://sdk.layerlens.ai/package
```

## Upgrade

```bash
pip install --upgrade layerlens --extra-index-url https://sdk.layerlens.ai/package
```

## Dependencies installed

* `httpx` — HTTP client
* `pydantic` — data validation
* `requests` — file upload (presigned S3 URLs)

## Where to next

* [First evaluation via SDK](/2.-get-started/first-eval-via-sdk)
* [SDK authentication reference](/2.-get-started/sdk)
* [SDK API reference](/2.-get-started/sdk)


# First evaluation via SDK

Run your first benchmark evaluation programmatically via the Stratix Python SDK.

Twenty lines of Python that run a real evaluation.

## Prerequisites

* [ ] [SDK installed](/2.-get-started/install)
* [ ] `LAYERLENS_STRATIX_API_KEY` exported
* [ ] A Stratix Premium account with starter ECU

## Sync version

```python
from layerlens import Stratix

client = Stratix()

# Pick a model and a benchmark from the public catalog
model = client.models.get_by_key("openai/gpt-4o")
benchmark = client.benchmarks.get_by_key("arc-agi-2")

# Create the evaluation
evaluation = client.evaluations.create(
 model=model,
 benchmark=benchmark,
)

# Wait for completion (polling)
result = client.evaluations.wait_for_completion(evaluation)
print(f"Accuracy: {result.accuracy}")
```

## Async version

Every method has an async counterpart on `AsyncStratix`:

```python
import asyncio
from layerlens import AsyncStratix

async def main():
 client = AsyncStratix()
 model = await client.models.get_by_key("openai/gpt-4o")
 benchmark = await client.benchmarks.get_by_key("arc-agi-2")
 evaluation = await client.evaluations.create(model=model, benchmark=benchmark)
 result = await client.evaluations.wait_for_completion(evaluation)
 print(f"Accuracy: {result.accuracy}")

asyncio.run(main())
```

## What just happened

1. The SDK authenticated with your API key.
2. It fetched the model and benchmark records from the public catalog.
3. It created an evaluation in your org.
4. The platform queued the work; the SDK polled for completion.
5. The result was returned; you got an accuracy score.

## Verify

The same evaluation should appear in **Premium → Evaluations**. Same record, same ID, same scores.

## Try variations

### Different model

```python
model = client.models.get_by_key("anthropic/claude-opus-4-7")
```

### Different benchmark

```python
benchmark = client.benchmarks.get_by_key("mmlu")
```

### Browse public results before running

```python
public_models = client.public.models.get()
for m in public_models.models[:5]:
 print(m.key, m.name)
```

## Error handling

```python
from layerlens import Stratix, NotFoundError, AuthenticationError, APIError

try:
 model = client.models.get_by_id("nonexistent-id")
except NotFoundError as e:
 print(f"Not found: {e.message}")
except AuthenticationError as e:
 print(f"Auth failed: {e.message}")
except APIError as e:
 print(f"API error ({e.status_code}): {e.message}")
```

## Where to next

* [Python SDK reference](/2.-get-started/sdk)
* [Tutorial: First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation)
* [Cookbook recipes](/2.-get-started/all-cookbook-recipes)


# All tutorials

Every Stratix tutorial — guided, end-to-end walkthroughs organized by workflow stage.

Tutorials are the guided learning path. Each one is end-to-end, sequenced, and produces something working. If you want the fastest possible first-touch instead, see [Run your first evaluation in 5 minutes](https://github.com/LayerLens/gitbook-full/blob/main/02-get-started/first-eval-5-minutes.md).

Tutorials are organized along the five stages of [the Stratix workflow](/1.-introduction/the-stratix-workflow): **Select**, **Build**, **Observe**, **Evaluate**, **Improve**.

## Select

Pick what you're evaluating — models, benchmarks, datasets. No standalone tutorial in this stage today; for guided model-selection flows, see the cookbook:

* [Pick a model for a task](https://github.com/LayerLens/gitbook-full/blob/main/05-select/cookbook/pick-a-model.md)
* [Compare prompts on a small test set](https://github.com/LayerLens/gitbook-full/blob/main/05-select/cookbook/compare-prompts.md)

## Build

Get Stratix wired into your code or CI.

| # | Tutorial                                                      | Time   | What you'll have at the end                                |
| - | ------------------------------------------------------------- | ------ | ---------------------------------------------------------- |
| 3 | [CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates) | 30 min | A GitHub Actions workflow that blocks merges on regression |

## Observe

Capture and route traces so they can be scored.

No standalone tutorial in this stage today. For trace ingestion, start with:

* [Set up trace ingestion](https://github.com/LayerLens/gitbook-full/blob/main/06-build/guides/setup-trace-ingestion.md) (guide)
* [Backfill from existing logs](/7.-observe-see-whats-happening/backfill-from-logs) (recipe)
* [Schedule continuous trace evaluation](https://github.com/LayerLens/gitbook-full/blob/main/07-observe/guides/schedule-continuous.md) (guide)

## Evaluate

Score model runs and traces with judges and code graders.

| # | Tutorial                                                                             | Time   | What you'll have at the end                           |
| - | ------------------------------------------------------------------------------------ | ------ | ----------------------------------------------------- |
| 1 | [First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation) | 10 min | A scored model vs. benchmark, in UI and SDK           |
| 2 | [First judge](/8.-evaluate-score-the-outputs/02-first-judge)                         | 20 min | An LLM judge scoring a subjective dimension           |
| 4 | [Score live traces](/8.-evaluate-score-the-outputs/04-score-traces)                  | 30 min | Continuous evaluation running over production traffic |

## Improve

Tune judges, cut costs, harden regression protection.

| # | Tutorial                                                              | Time   | What you'll have at the end                         |
| - | --------------------------------------------------------------------- | ------ | --------------------------------------------------- |
| 5 | [GEPA-optimize a judge](/9.-improve-tune-the-system/05-gepa-optimize) | 45 min | An LLM judge with measurably higher human agreement |

## Reading order

If you're new, take them in numeric order — each builds on what came before.

1. [First evaluation](/8.-evaluate-score-the-outputs/01-first-evaluation) — Evaluate
2. [First judge](/8.-evaluate-score-the-outputs/02-first-judge) — Evaluate
3. [CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates) — Build
4. [Score live traces](/8.-evaluate-score-the-outputs/04-score-traces) — Evaluate / Observe
5. [GEPA-optimize](/9.-improve-tune-the-system/05-gepa-optimize) — Improve

Total time investment for all five: \~2.5 hours.

## Where to next

* [Top 5 starter recipes](https://github.com/LayerLens/gitbook-full/blob/main/02-get-started/top-5-starter-recipes.md) — same week-one outcomes, recipe-shaped instead of tutorial-shaped
* [All cookbook recipes](/2.-get-started/all-cookbook-recipes) — 59 recipes by problem, framework, industry, integration
* [All guides](/2.-get-started/all-guides) — task-oriented walkthroughs
* [The Stratix workflow](/1.-introduction/the-stratix-workflow) — the spine these tutorials map to


# All cookbook recipes

Every Stratix cookbook recipe — runnable snippets organized by the workflow stage they belong to.

Recipes are the "give me code I can paste" shape. Each one is small, self-contained, and runnable. They're organized below by the [workflow stage](/1.-introduction/the-stratix-workflow) they map to — **Select**, **Build**, **Observe**, **Evaluate**, **Improve** — plus cross-cutting groupings (industry, integration, migration, SDK basics) at the bottom.

If you're new, start with [Top 5 starter recipes](https://github.com/LayerLens/gitbook-full/blob/main/02-get-started/top-5-starter-recipes.md). Otherwise, browse below.

## Select

Pick the model, benchmark, or prompt before you commit ECU.

* [Pick a model for a task](https://github.com/LayerLens/gitbook-full/blob/main/05-select/cookbook/pick-a-model.md)
* [Compare prompts on a small test set](https://github.com/LayerLens/gitbook-full/blob/main/05-select/cookbook/compare-prompts.md)

## Build

Instrument your code and ship the wiring.

### Framework adapters

* [Instrument a LangChain agent](/6.-build-wire-your-code/instrument-langchain)
* [Instrument a LlamaIndex pipeline](/6.-build-wire-your-code/instrument-llamaindex)
* [Instrument a Haystack pipeline](/6.-build-wire-your-code/instrument-haystack)
* [Instrument an OpenAI Assistants run](/6.-build-wire-your-code/instrument-openai-assistants)
* [Instrument a Vercel AI SDK call](/6.-build-wire-your-code/instrument-vercel-ai-sdk)
* [Instrument an Anthropic SDK call](/6.-build-wire-your-code/instrument-anthropic-sdk)
* [Instrument a custom Python pipeline](/6.-build-wire-your-code/instrument-custom-python)
* [OTel-style trace ingestion](/6.-build-wire-your-code/otel-trace-ingestion)

### CI / release

* [Containerized CI runner](/6.-build-wire-your-code/containerized-ci)
* [Pre-release readiness eval](/6.-build-wire-your-code/prerelease-readiness)

## Observe

Capture, route, and warehouse what your system is doing.

* [Backfill from existing logs](/7.-observe-see-whats-happening/backfill-from-logs)
* [Post-release continuous eval](/7.-observe-see-whats-happening/postrelease-continuous)

## Evaluate

Score what you've captured.

### Score by output shape

* [Score a structured-output prompt](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/score-structured-output.md)
* [Score a free-form chat prompt](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/score-chat.md)
* [Score code outputs (HumanEval-style)](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/score-code.md)
* [Score reasoning chains](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/score-reasoning.md)
* [Score retrieval quality (precision@k)](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/score-retrieval.md)
* [Score multilingual outputs](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/multilingual.md)
* [Validate JSON output with schema](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/json-schema-validate.md)

### Score by failure mode

* [Catch hallucinations](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/catch-hallucinations.md)
* [Build a faithfulness judge for RAG](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/rag-faithfulness-judge.md)
* [Detect tool-call regressions in agents](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/agent-tool-regressions.md)

### Author your own scorers

* [Custom code grader](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/custom-code-scorer.md)

### Aggregation and gating

* [Span-level scoring](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/span-level-scoring.md)
* [Multi-judge consensus](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/multi-judge-consensus.md)
* [Confidence-weighted aggregation](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/confidence-weighted-aggregation.md)
* [Cost-aware evaluation](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/cost-aware-evaluation.md)
* [Run a judge in CI without merging on regression](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/judge-ci-no-merge.md)

## Improve

Tune what you've scored, drive cost down, keep regressions out.

* [Daily smoke eval](/9.-improve-tune-the-system/daily-smoke-eval)
* [Quarterly model re-evaluation](/9.-improve-tune-the-system/quarterly-rebench)
* [GEPA basics](/9.-improve-tune-the-system/gepa-basics)
* [GEPA with held-out validation set](/9.-improve-tune-the-system/gepa-holdout)
* [Iteratively label and re-optimize](/9.-improve-tune-the-system/gepa-iterative-labeling)

## Cross-cutting: by industry

* [Healthcare: clinical Q\&A safety judge](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/healthcare/cookbook/industry-healthcare-clinical-qa.md)
* [Financial services: advisor chatbot guardrails](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/financial-services/cookbook/industry-finserv-advisor.md)
* [Legal: contract review faithfulness](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/legal/cookbook/industry-legal-contract.md)
* [Retail: product Q\&A scoring](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/retail-ecommerce/cookbook/industry-retail-product-qa.md)
* [Insurance: claims triage scoring](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/insurance/cookbook/industry-insurance-claims.md)
* [Education: tutor helpfulness](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/education/cookbook/industry-education-tutor.md)

## Cross-cutting: by CI / notification integration

* [GitHub Actions gate](/6.-build-wire-your-code/integration-github-actions)
* [GitLab CI gate](/6.-build-wire-your-code/integration-gitlab-ci)
* [Buildkite gate](/6.-build-wire-your-code/integration-buildkite)
* [Jenkins gate](/6.-build-wire-your-code/integration-jenkins)
* [Slack notifications](/6.-build-wire-your-code/integration-slack)
* [PagerDuty escalation](/6.-build-wire-your-code/integration-pagerduty)

## Cross-cutting: migration from another platform

* [Port a LangSmith eval](https://github.com/LayerLens/gitbook-full/blob/main/06-build/migration/recipes/migration-langsmith.md)
* [Port a Braintrust eval](https://github.com/LayerLens/gitbook-full/blob/main/06-build/migration/recipes/migration-braintrust.md)
* [Port an MLflow run](https://github.com/LayerLens/gitbook-full/blob/main/06-build/migration/recipes/migration-mlflow.md)
* [Port a Promptfoo config](https://github.com/LayerLens/gitbook-full/blob/main/06-build/migration/recipes/migration-promptfoo.md)

## SDK basics

* [Sync vs async clients](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/recipes/sync-async-clients.md)
* [Pagination over large result sets](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/recipes/pagination.md)
* [Retry and backoff](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/recipes/retry-backoff.md)
* [Rate-limit handling](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/recipes/rate-limit-handling.md)

## Where to next

* [Top 5 starter recipes](https://github.com/LayerLens/gitbook-full/blob/main/02-get-started/top-5-starter-recipes.md) — the curated week-one subset
* [All tutorials](/2.-get-started/all-tutorials) — guided, end-to-end learning paths
* [All guides](/2.-get-started/all-guides) — task-shaped walkthroughs
* [stratix-python samples](https://github.com/layerlens/stratix-python) — many recipes mirror runnable source samples

## Integrate

* [Send results to S3](/10.-integrate-connect-your-stack/integration-s3-export)
* [Send results to BigQuery](/10.-integrate-connect-your-stack/integration-bigquery-export)
* [Send results to Snowflake](/10.-integrate-connect-your-stack/integration-snowflake-export)


# All guides

Every Stratix guide — task-oriented walkthroughs across the platform, organized by workflow stage and operational concern.

Guides sit between recipes and tutorials in shape. Use a guide when you have a clear goal that's larger than a copy-paste recipe but doesn't need a full guided learning sequence.

Organized below by [workflow stage](/1.-introduction/the-stratix-workflow) — **Select**, **Build**, **Observe**, **Evaluate**, **Improve** — plus operational concerns (admin, migration) at the bottom.

## Select

Pick and prepare what you'll evaluate against.

### BYOK models

* [Register a BYOK custom model](https://github.com/LayerLens/gitbook-full/blob/main/05-select/guides/register-byok-model.md)
* [Test a BYOK endpoint](https://github.com/LayerLens/gitbook-full/blob/main/05-select/guides/test-byok-endpoint.md)
* [Rotate a BYOK API key](https://github.com/LayerLens/gitbook-full/blob/main/05-select/guides/rotate-byok-key.md)

### Benchmarks and datasets

* [Upload a private benchmark](https://github.com/LayerLens/gitbook-full/blob/main/05-select/guides/upload-benchmark.md)
* [Refresh a benchmark with new data](https://github.com/LayerLens/gitbook-full/blob/main/05-select/guides/refresh-benchmark.md)

## Build

Wire Stratix into your code and CI.

### Trace ingestion

* [Set up trace ingestion](https://github.com/LayerLens/gitbook-full/blob/main/06-build/guides/setup-trace-ingestion.md)

### Calling Stratix from your stack

* [Calling Stratix from a backend](https://github.com/LayerLens/gitbook-full/blob/main/06-build/guides/calling-stratix-from-a-backend.md)
* [Pattern: calling Stratix from Next.js](https://github.com/LayerLens/gitbook-full/blob/main/06-build/guides/pattern-from-nextjs.md)

### CI / quality gates

* [Wire CI/CD evaluation gates](https://github.com/LayerLens/gitbook-full/blob/main/06-build/guides/wire-cicd-gates.md)
* [Set per-feature quality tolerances](https://github.com/LayerLens/gitbook-full/blob/main/06-build/guides/per-feature-tolerances.md)
* [Investigate a regression in CI](https://github.com/LayerLens/gitbook-full/blob/main/06-build/guides/investigate-ci-regression.md)

## Observe

Run continuous evaluation over what's already happening.

* [Build a trace set for agentic evals](https://github.com/LayerLens/gitbook-full/blob/main/07-observe/guides/build-trace-set.md)
* [Schedule continuous trace evaluation](https://github.com/LayerLens/gitbook-full/blob/main/07-observe/guides/schedule-continuous.md)
* [Investigate a failing trace](https://github.com/LayerLens/gitbook-full/blob/main/07-observe/guides/investigate-failing-trace.md)

## Evaluate

Build, standardize, and harden your scoring library.

* [Bootstrap judges from real traces](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/bootstrap-judges.md)
* [Build your judge library](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/build-judge-library.md)
* [Standardize judges across teams](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/standardize-judges.md)
* [Author a custom code grader](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/custom-code-scorer.md)
* [Detect and remove flaky scorers](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/remove-flaky-scorers.md)

## Improve

Optimize judges, cut cost, keep regressions out.

* [Optimize a judge with GEPA](https://github.com/LayerLens/gitbook-full/blob/main/09-improve/guides/optimize-judge-gepa.md)
* [Curate a regression test set](https://github.com/LayerLens/gitbook-full/blob/main/09-improve/guides/curate-regression-set.md)
* [Reduce eval cost without losing signal](https://github.com/LayerLens/gitbook-full/blob/main/09-improve/guides/reduce-eval-cost.md)
* [Cost-control levers](https://github.com/LayerLens/gitbook-full/blob/main/09-improve/guides/cost-control-levers.md)

## Admin and operations

Day-two work — orgs, teams, billing, security, notifications.

### Setup and onboarding

* [Set up your first organization](/11.-admin/setup-first-org)
* [Invite your team](/11.-admin/invite-team)
* [Set up SSO](/11.-admin/setup-sso)
* [Manage seats and roles](/11.-admin/manage-seats-roles)

### Billing

* [Buy and manage ECU](/11.-admin/manage-ecu)

### Keys

* [Rotate an API key](/11.-admin/rotate-api-key)

### Notifications and audit

* [Set up notifications and alerting](/11.-admin/setup-notifications)
* [Set up Slack integration](/11.-admin/setup-slack)
* [Audit-friendly release evidence](/11.-admin/audit-friendly-evidence)

## Migration

Coming from another evaluation platform.

* [Plan a migration to Stratix](https://github.com/LayerLens/gitbook-full/blob/main/06-build/migration/guides/plan-migration.md)
* [Run Stratix and another tool side-by-side during migration](https://github.com/LayerLens/gitbook-full/blob/main/06-build/migration/guides/dual-run-migration.md)

## Where to next

* [All tutorials](/2.-get-started/all-tutorials) — guided, end-to-end learning paths
* [All cookbook recipes](/2.-get-started/all-cookbook-recipes) — copy-paste snippets by problem
* [Top 5 starter recipes](https://github.com/LayerLens/gitbook-full/blob/main/02-get-started/top-5-starter-recipes.md) — the curated week-one subset
* [Common first-day errors](/2.-get-started/first-day-errors) — when something doesn't work


# Samples (70+ runnable)

Stratix Python SDK samples — categorized index of the runnable samples that ship with the SDK.

The LayerLens Python SDK ships with **70+ runnable samples** covering every public resource — from a single trace evaluation to multi-agent evaluation patterns and CI/CD quality gates. All samples live in [`stratix-python/samples/`](https://github.com/layerlens/stratix-python/tree/main/samples) and run directly after installing the SDK.

## Quick start

```bash
pip install layerlens --extra-index-url https://sdk.layerlens.ai/package
export LAYERLENS_STRATIX_API_KEY=your-api-key
python samples/core/quickstart.py
```

[`quickstart.py`](https://github.com/layerlens/stratix-python/blob/main/samples/core/quickstart.py) walks through the complete workflow end-to-end: upload a trace, create a judge, run an evaluation, and retrieve results.

## Samples by category

Each category groups samples around a specific Stratix capability or integration surface. Pick the category closest to what you're building.

| Category                                                | What's in it                                                                                                                                                          | Count         |
| ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| [Core SDK operations](#core-sdk-operations)             | Every public resource — traces, judges, evaluations, results, models, benchmarks — used individually and together, including async patterns and pagination            | 18            |
| [Industry solutions](#industry-solutions)               | Domain-specific evaluation scenarios with judges tuned for regulated and high-stakes verticals (healthcare, financial services, legal, government, insurance, retail) | 10            |
| [Multi-agent evaluation](#multi-agent-evaluation)       | Patterns for evaluating multi-agent collaboration where each agent's output needs independent quality assessment                                                      | 5             |
| [Content-type evaluations](#content-type-evaluations)   | Specialized judges applied to different content types — text responses, brand assets, structured documents                                                            | 3             |
| [CI/CD integration](#cicd-integration)                  | Quality gates that embed evaluation into build and deployment pipelines so regressions never reach production                                                         | 2 + workflow  |
| [LLM provider integrations](#llm-provider-integrations) | Trace and evaluate outputs from OpenAI and Anthropic — both manual trace upload and auto-instrumentation paths                                                        | 4             |
| [OpenClaw agent evaluation](#openclaw-agent-evaluation) | Trace, evaluate, and monitor [OpenClaw](https://openclaw.ai/) autonomous AI agents — cage-match tournaments, code gating, drift detection, content auditing           | 10 + skill    |
| [MCP server](#mcp-server)                               | Expose Stratix capabilities as tools for Claude, Cursor, and any MCP-compatible AI assistant                                                                          | 1             |
| [CopilotKit integration](#copilotkit-integration)       | Full-stack pattern with LangGraph CoAgents + generative-UI cards for human-in-the-loop evaluation workflows                                                           | 2 agents + UI |
| [Claude Code skills](#claude-code-skills)               | Slash commands that bring Stratix workflows into the Claude Code CLI                                                                                                  | 6             |
| [Sample data](#sample-data)                             | Pre-built trace files and 16 industry-specific evaluation datasets so every sample runs without sourcing your own data                                                | n/a           |

***

## Core SDK operations

Located in [`samples/core/`](https://github.com/layerlens/stratix-python/tree/main/samples/core). Start here to learn how every public Stratix resource works, individually and combined.

Key samples:

* `quickstart.py` — first evaluation in under 30 lines
* `trace_evaluation.py` — full trace-evaluation lifecycle
* `judge_optimization.py` — optimize a judge via automated prompt evolution (GEPA)
* `evaluation_pipeline.py` — chain judges, traces, and results into an automated pipeline
* `async_workflow.py` — concurrent operations with `AsyncStratix`

See the [Core SDK README](https://github.com/layerlens/stratix-python/blob/main/samples/core/README.md). Maps to gitbook: [Tutorials](/2.-get-started/all-tutorials) · [Concept: Evaluations](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/evaluations.md).

## Industry solutions

Located in [`samples/industry/`](https://github.com/layerlens/stratix-python/tree/main/samples/industry). Domain-specific scenarios with judges and scoring configurations tuned for regulated and high-stakes verticals.

Key samples:

* `healthcare_clinical.py` — clinical decision support evaluation
* `financial_trading.py` — SOX-aligned trading compliance
* `legal_contracts.py` — contract review quality assessment

See the [Industry Solutions README](https://github.com/layerlens/stratix-python/blob/main/samples/industry/README.md). Maps to gitbook: [Industry patterns](/6.-build-wire-your-code/sdk-python).

## Multi-agent evaluation

Located in [`samples/cowork/`](https://github.com/layerlens/stratix-python/tree/main/samples/cowork). Patterns for evaluating multi-agent collaboration — applicable to any framework where multiple agents produce intermediate outputs that need independent quality assessment.

Key samples:

* `multi_agent_eval.py` — Generator-Evaluator pattern
* `code_review.py` — Instrumentor-Reviewer pattern
* `rag_assessment.py` — RAG-quality evaluation

See the [Multi-Agent README](https://github.com/layerlens/stratix-python/blob/main/samples/cowork/README.md). Maps to gitbook: [Concept: Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation).

## Content-type evaluations

Located in [`samples/modalities/`](https://github.com/layerlens/stratix-python/tree/main/samples/modalities). Specialized judges applied to different content types.

* `text_evaluation.py` — score text across safety, relevance, and compliance
* `brand_evaluation.py` — enforce brand-voice consistency
* `document_evaluation.py` — validate document-extraction accuracy

## CI/CD integration

Located in [`samples/cicd/`](https://github.com/layerlens/stratix-python/tree/main/samples/cicd). Embed evaluation quality gates into build and deployment pipelines.

* `quality_gate.py` — gate deployments on evaluation pass rates
* `pre_commit_hook.py` — catch regressions at commit time
* `github_actions_gate.yml` — drop-in GitHub Actions workflow

See the [CI/CD README](https://github.com/layerlens/stratix-python/blob/main/samples/cicd/README.md). Maps to gitbook: [Tutorial 3: CI/CD gates](/6.-build-wire-your-code/03-cicd-gates) · [Cookbook: containerized CI](/6.-build-wire-your-code/containerized-ci).

## LLM provider integrations

Located in [`samples/integrations/`](https://github.com/layerlens/stratix-python/tree/main/samples/integrations). Two paths: manual trace upload for full control, and auto-instrumentation via `layerlens.instrument` for zero-code observability.

* `openai_traced.py` — manual trace upload around an OpenAI completion
* `anthropic_traced.py` — capture multi-turn Claude conversations (manual trace upload)
* `openai_instrumented.py` — auto-instrument OpenAI with `instrument_openai()`, `@trace`, `span()`
* `langchain_instrumented.py` — auto-capture LangChain chain execution with `LangChainCallbackHandler`

Maps to gitbook: [Integrations](/6.-build-wire-your-code/sdk-python).

## OpenClaw agent evaluation

Located in [`samples/openclaw/`](https://github.com/layerlens/stratix-python/tree/main/samples/openclaw). Trace, evaluate, and monitor [OpenClaw](https://openclaw.ai/) autonomous AI agents — including cage-match model tournaments, code gating, drift detection, content auditing, honeypot-skill auditing, and adversarial red-teaming.

See the [OpenClaw README](https://github.com/layerlens/stratix-python/blob/main/samples/openclaw/README.md) for the full list and advanced evaluation patterns.

## MCP server

Located in [`samples/mcp/`](https://github.com/layerlens/stratix-python/tree/main/samples/mcp). Expose Stratix capabilities as tools for Claude, Cursor, and any MCP-compatible AI assistant.

* `layerlens_server.py` — MCP server exposing trace management, judge creation, and evaluation execution

See the [MCP README](https://github.com/layerlens/stratix-python/blob/main/samples/mcp/README.md) for setup.

## CopilotKit integration

Located in [`samples/copilotkit/`](https://github.com/layerlens/stratix-python/tree/main/samples/copilotkit). Full-stack pattern combining LangGraph CoAgents (Python) with generative-UI card components (React) for human-in-the-loop evaluation workflows.

* `agents/evaluator_agent.py` — LangGraph CoAgent for evaluation workflows (uses `interrupt()` for human-in-the-loop judge confirmation)
* `agents/investigator_agent.py` — LangGraph CoAgent for trace investigation
* `components/*.tsx` — React card components for rendering results (`EvaluationCard`, `JudgeVerdictCard`, `TraceCard`, `MetricCard`, `ComplianceCard`)
* `hooks/*.ts` — CopilotKit hooks for wiring Stratix actions (`useLayerLensActions`, `useLayerLensContext`)

> **Checkpointer note:** any LangGraph CoAgent that calls `interrupt()` (such as `evaluator_agent.py`) **must** be compiled with a checkpointer. Without one, the AG-UI stream ends without emitting `RUN_FINISHED` and CopilotKit blocks all subsequent messages. The sample ships with `InMemorySaver` for a zero-setup local run; the [README](https://github.com/layerlens/stratix-python/blob/main/samples/copilotkit/README.md) documents Postgres / SQLite / Redis / LangGraph Platform alternatives for production.

See the [CopilotKit README](https://github.com/layerlens/stratix-python/blob/main/samples/copilotkit/README.md) for setup, full file list, and the human-in-the-loop walkthrough.

## Claude Code skills

Located in [`samples/claude-code/`](https://github.com/layerlens/stratix-python/tree/main/samples/claude-code). Slash commands that bring Stratix workflows directly into the Claude Code CLI — manage traces, judges, evaluations, optimizations, benchmarks, and investigations without leaving your terminal.

`/trace`, `/judge`, `/evaluate`, `/optimize`, `/benchmark`, `/investigate`. See the [Claude Code Skills README](https://github.com/layerlens/stratix-python/blob/main/samples/claude-code/README.md).

## Sample data

Located in [`samples/data/`](https://github.com/layerlens/stratix-python/tree/main/samples/data). Pre-built trace files, test datasets, and 16 industry-specific evaluation datasets so every sample runs without you sourcing your own data first.

See the [Sample Data README](https://github.com/layerlens/stratix-python/blob/main/samples/data/README.md) for contents.

***

## Full sample reference

For the complete table of every sample with descriptions, see the [samples README in the SDK repo](https://github.com/layerlens/stratix-python/blob/main/samples/README.md).

## Where to next

* [SDK README](/6.-build-wire-your-code/sdk-python) — full resource list
* [Cookbook](/2.-get-started/all-cookbook-recipes) — recipes drawn from these samples
* [Integrations](/6.-build-wire-your-code/sdk-python) — full adapter list
* [Tutorials](/2.-get-started/all-tutorials) — guided walkthroughs anchored to these samples


# Run your first evaluation in 5 minutes

Run your first Stratix evaluation end-to-end in five minutes. One Python file, copy-paste-runnable.

One Python file. One model. One benchmark. One score. That's it.

This page is the fastest possible path from zero to a real evaluation. If something breaks, jump straight to [Common first-day errors](/2.-get-started/first-day-errors) and come back.

{% hint style="info" %}
**Prereqs:** Python 3.9+, a Stratix Premium account, and an API key from **Settings → API keys**. You'll need \~10 ECU of starter credit (every new org gets some).
{% endhint %}

## Step 1 — Install and authenticate (60 seconds)

```bash
pip install layerlens --extra-index-url https://sdk.layerlens.ai/package
export LAYERLENS_STRATIX_API_KEY="sk-..." # macOS / Linux
# PowerShell: $env:LAYERLENS_STRATIX_API_KEY = "sk-..."
```

Verify the install:

```bash
python -c "import layerlens; print(layerlens.__version__)"
```

You should see `1.3.0` or later. If not, see [first-day errors](/2.-get-started/first-day-errors).

## Step 2 — The whole thing in one file

Save this as `first_eval.py` and run it. That's the entire 5-minute experience.

```python
"""
first_eval.py — your first end-to-end Stratix evaluation.

Run with: python first_eval.py
Requires: LAYERLENS_STRATIX_API_KEY set in your environment.
"""
from layerlens import Stratix

# 1) Authenticate. The Stratix() constructor reads LAYERLENS_STRATIX_API_KEY
# from the environment. Pass api_key="..." explicitly if you prefer.
client = Stratix()

# 2) Pick a model from the public catalog. get_by_key() is case-sensitive
# and uses the "<provider>/<model>" key shown in the Catalog UI.
model = client.models.get_by_key("anthropic/claude-opus-4-7")
print(f"Model: {model.name}")

# 3) Pick a benchmark. MMLU is a fast, well-known public benchmark — good
# for a smoke test on day one.
benchmark = client.benchmarks.get_by_key("mmlu")
print(f"Benchmark: {benchmark.name}")

# 4) Create the evaluation. The platform queues it; the SDK returns
# immediately with an Evaluation object you can poll.
evaluation = client.evaluations.create(
 model=model,
 benchmark=benchmark,
)
print(f"Started: {evaluation.id} (status={evaluation.status})")

# 5) Block until the evaluation finishes. wait_for_completion polls every
# interval_seconds and gives up after timeout_seconds. MMLU on Opus 4.7
# typically completes in 2–4 minutes.
evaluation = client.evaluations.wait_for_completion(
 evaluation,
 interval_seconds=10,
 timeout_seconds=600,
)

if not evaluation.is_success:
 raise SystemExit(f"Evaluation did not succeed: status={evaluation.status}")

print(f"Done: accuracy={evaluation.accuracy:.3f}")

# 6) Pull per-prompt results. get_all_by_id auto-paginates across pages and
# returns a flat list of Result objects (.prompt,.result,.truth,.score).
results = client.results.get_all_by_id(evaluation.id)
print(f"Rows: {len(results)}")

# 7) Show the first three rows so you can see what real data looks like.
for r in results[:3]:
 verdict = "PASS" if r.score and r.score >= 0.5 else "FAIL"
 print(f" [{verdict}] {r.prompt[:60]}... score={r.score}")

# 8) Done. The same evaluation now appears in Premium → Evaluations.
print(f"\nView in UI: https://stratix.layerlens.ai/evaluations/{evaluation.id}")
```

## Step 3 — Run it

```bash
python first_eval.py
```

You should see something like:

```
Model: Claude Opus 4.7
Benchmark: MMLU
Started: eval_01HZ... (status=queued)
Done: accuracy=0.873
Rows: 14042
 [PASS] Which of the following best describes the function of... score=1.0
 [PASS] In the context of macroeconomic policy, the term 'cro... score=1.0
 [FAIL] A 65-year-old patient presents with acute onset chest... score=0.0

View in UI: https://stratix.layerlens.ai/evaluations/eval_01HZ...
```

Open that URL. The dashboard shows the same accuracy number, plus a per-subset breakdown, error analysis, and ECU consumed.

## You just did five things

1. **Selected** a model and a benchmark from the public catalog.
2. **Built** the evaluation by passing both objects to `evaluations.create`.
3. **Observed** the run via `wait_for_completion` (queued → running → completed).
4. **Evaluated** the model against MMLU's ground-truth labels — an LLM-backed scorer compares each model response to truth and emits a 0/1 score per row.
5. **Improved** your understanding of where the model wins and loses, by inspecting per-row results.

Those five stages — Select, Build, Observe, Evaluate, Improve — are the spine of every Stratix workflow.

## Common failure modes

* **`AuthenticationError: invalid API key`** — env var name is `LAYERLENS_STRATIX_API_KEY`, not `LAYERLENS_API_KEY`. See [first-day errors](/2.-get-started/first-day-errors).
* **`NotFoundError` on `get_by_key`** — keys are case-sensitive. Browse the catalog at `https://stratix.layerlens.ai/catalog` to copy the exact key.
* **Timeout** — bump `timeout_seconds`. Some benchmarks (HumanEval+, GPQA) take longer than 10 minutes on slower models.
* **"No ECU"** — every new org gets starter ECU; if you've exhausted it, top up under **Admin → Billing**.

## Where to next

You ran one model on one benchmark. Logical next steps:

* **Compare two models.** Repeat Steps 2–5 with a different model key and diff the accuracy. [Public — compare two models](/2.-get-started/compare-two-models) walks the UI version of this flow.
* **Score subjective dimensions.** Accuracy works when there's a ground-truth answer. For helpfulness, faithfulness, tone, etc., you need an LLM judge. [Tutorial 2: First judge](/8.-evaluate-score-the-outputs/02-first-judge).
* **Run it on your own data.** Swap the benchmark for an uploaded dataset. [Upload a private benchmark](/more-in-this-section-3/upload-benchmark).
* **Wire this into CI.** Promote your `first_eval.py` into a quality gate. [Tutorial 3: CI/CD gates](/6.-build-wire-your-code/03-cicd-gates).
* **Score live traces.** Score traffic that's already happening in production. [Tutorial 4: Score live traces](/8.-evaluate-score-the-outputs/04-score-traces).

If you'd rather follow a guided five-tutorial path, start at [All tutorials](/2.-get-started/all-tutorials). If you want to lift-and-shift a working snippet for a specific problem, jump to [Top 5 starter recipes](/more-in-this-section-1/top-5-starter-recipes).


# Top 5 starter recipes

Five starter cookbook recipes hand-picked for new Stratix users. Cover the full Build → Observe → Evaluate → Improve arc in your first week.

The cookbook has 59 recipes. These are the five we'd hand to a new engineer on their first week — one per workflow stage, each delivering visible value in under an hour.

Run [first-eval-5-minutes](/more-in-this-section-1/first-eval-5-minutes) first if you haven't. These five build on top.

## 1 — Instrument your own Python pipeline

**Why this one:** before you can score real production traffic, you need traces. This recipe shows the two primitives — `@trace` and `span()` — that turn any Python function into something Stratix can see. It works whether you're calling OpenAI, a local model, a vector store, or your own custom orchestration. Five lines of decorators replace days of OTel wiring.

**Stage:** Build · **Time:** 15 min · **Output:** traces flowing into Premium → Traces

→ [Recipe: instrument a custom Python pipeline](/6.-build-wire-your-code/instrument-custom-python)

## 2 — Backfill traces from existing logs

**Why this one:** you almost certainly already have logs — Postgres rows, JSONL files, Datadog exports — sitting somewhere. This recipe converts that history into Stratix traces in bulk so you can run evaluations against weeks or months of real production data on day one, instead of waiting for new traffic to accumulate. Highest leverage move you can make in week one.

**Stage:** Observe · **Time:** 30 min · **Output:** historical traces visible in Premium, ready to score

→ [Recipe: backfill from existing logs](/7.-observe-see-whats-happening/backfill-from-logs)

## 3 — Catch hallucinations

**Why this one:** hallucination is the #1 failure mode every team asks about first. This recipe wires up an LLM-backed faithfulness judge that flags answers unsupported by retrieved context. Works on freshly captured traces or backfilled history. The judge prompt is tunable, the threshold is tunable, and the per-row verdicts are inspectable — so you can trust the signal before you act on it.

**Stage:** Evaluate · **Time:** 20 min · **Output:** every trace gets a faithfulness score and verdict

→ [Recipe: catch hallucinations](/more-in-this-section-6/catch-hallucinations)

## 4 — Validate structured outputs with a JSON Schema code grader

**Why this one:** the moment your agent emits JSON, you need to guarantee the shape — required fields, types, enum values, nested structures. This recipe registers a **code grader** that runs a JSON Schema check against every output and emits a 0/1 pass score. Pairs naturally with #3: hallucination judges score *content*, schema graders score *shape*. Together they catch \~80% of the "my agent is broken" reports in CI.

**Stage:** Evaluate · **Time:** 15 min · **Output:** schema failures surface as failing rows in every evaluation

→ [Recipe: validate JSON output with schema](/more-in-this-section-6/json-schema-validate)

## 5 — Optimize a judge with GEPA basics

**Why this one:** the judges you bootstrap (recipe #3) start at 60–75% agreement with human labels. That's enough to *find* problems but not enough to *gate* on. GEPA — the platform's prompt-optimization loop — tunes the judge prompt against a small labeled set until agreement climbs past 90%. This recipe takes 30 minutes and converts a noisy first-draft judge into one you trust enough to wire into CI.

**Stage:** Improve · **Time:** 30 min · **Output:** an optimized judge with measurable agreement gain

→ [Recipe: GEPA basics](/9.-improve-tune-the-system/gepa-basics)

## How these five fit together

```
1. Instrument → traces exist
2. Backfill → historical traces exist
3. Faithfulness judge → content quality scored
4. JSON schema grader → output shape scored
5. GEPA basics → judges sharpened
```

In one week, you go from "we have a chatbot" to "we have measured quality across thousands of past interactions, with judges we trust enough to block bad releases on."

## Where to next

* **More recipes by problem, framework, industry, integration:** [All cookbook recipes](/2.-get-started/all-cookbook-recipes)
* **Guided learning paths instead:** [All tutorials](/2.-get-started/all-tutorials)
* **Task-shaped walkthroughs:** [All guides](/2.-get-started/all-guides)
* **Workflow context:** [The Stratix workflow](/1.-introduction/the-stratix-workflow)


# Pricing — overview

Pricing for LayerLens Stratix — Free, Premium PAYG, Enterprise.

Three tiers. Pricing is transparent — no sales call required for Free or Premium.

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>Free</strong></td><td>Free forever. Stratix Public + a Premium account with starter ECU.</td><td><a href="/3.-pricing/free-tier-details">Read</a></td></tr><tr><td><strong>Premium PAYG</strong></td><td>Pay-as-you-go on ECU. Multi-org, BYOK, agentic evals, GEPA, traces.</td><td><a href="/3.-pricing/premium-tier-details">Read</a></td></tr><tr><td><strong>Enterprise</strong></td><td>Custom annual contract. SSO, data residency,, dedicated CSM.</td><td><a href="/3.-pricing/enterprise-tier-details">Read</a></td></tr></tbody></table>

## At a glance

|                                              | Free        | Premium PAYG | Enterprise         |
| -------------------------------------------- | ----------- | ------------ | ------------------ |
| **Cost**                                     | $0          | ECU PAYG     | Custom             |
| **Stratix Public**                           | full        | full         | full               |
| **Stratix Premium**                          | starter ECU | full         | full               |
| **Multi-org**                                | 1 org       | yes          | yes                |
| **Team seats**                               | basic       | yes          | yes                |
| **Python SDK + CLI**                         | yes         | yes          | yes                |
| **BYOK custom models**                       | partial     | yes          | yes                |
| **Agentic evaluations**                      | starter     | full         | full               |
| **GEPA judge optimization**                  | partial     | full         | full               |
| **Trace evaluations**                        | starter     | full         | full               |
| **Notifications**                            | basic       | full         | full               |
| **In-app Assistant**                         | yes         | yes          | yes                |
| **Priority support**                         | community   | email        | dedicated CSM      |
| **SSO**                                      | —           | —            | yes                |
| **Data residency**                           | —           | —            | yes                |
| **Audit logs**                               | —           | basic        | extended retention |
| **Custom contractual terms (DPA, MSA, SLA)** | —           | —            | yes                |

## What is an ECU?

An **Evaluation Compute Unit** (ECU) is the billing unit for compute-intensive operations:

* Model inference during an evaluation
* Judge runs (LLM judges consume ECU)
* GEPA optimization runs
* Trace evaluation runs

Free tier ships with a starter ECU balance. Premium is pay-as-you-go: buy ECU when you need it. Enterprise has volume ECU pricing.

[Concept: ECU credits](/11.-admin/concepts-ecu-credits) covers the economics and how ECU is consumed.

## FAQ

**Do I need to pay to browse the public catalog?** No. `stratix.layerlens.ai` is free, anonymous, and unlimited.

**Can I use my own LLM provider key?** Yes. BYOK custom models are supported in Premium. Provider tokens consumed against your key don't count against ECU.

**Can I cancel?** Premium PAYG has no commitment. Enterprise contracts are negotiated annually.

**Is there a non-profit / academic discount?** Yes — email <sales@layerlens.ai>.

**What happens to my data if I downgrade?** Org data is preserved. You retain access to everything you've created; some Premium-only features become read-only.

## Where to next

* [Free tier details](/3.-pricing/free-tier-details)
* [Premium tier details](/3.-pricing/premium-tier-details)
* [Enterprise tier details](/3.-pricing/enterprise-tier-details)
* [How Stratix compares](/1.-introduction/how-stratix-compares-deep)
* [Industry patterns](/3.-pricing/03-pricing)


# Free tier details

What's in the Stratix Free tier — Stratix Public, a Premium account, and starter ECU credits.

## What's included

* **Stratix Public — fully unlocked.** Browse 175+ models, 52+ benchmarks, 2,000+ public evaluations. Compare any two models head-to-head. Browse public spaces. Read quarterly reports. Search the catalog.
* **Stratix Premium — starter access.** A Premium account at `stratix.layerlens.ai` with starter ECU credits.
* **One organization.** Premium operates with a single org; you can be invited to other orgs but won't have admin elsewhere.
* **Basic seats.** Add teammates to your single org.
* **Python SDK and CLI.** `pip install layerlens --extra-index-url https://sdk.layerlens.ai/package` and `layerlens` CLI both work with a Free-tier API key.
* **In-app Assistant.** Context-aware help inside Premium.
* **Community support.** Public forum and docs.

## What's not included

* **Larger ECU balances.** Buy more under Premium PAYG.
* **Multi-org admin.** Free tier accounts are admin of one org.
* **SSO.** Available on Enterprise.
* **Data residency selection.** Available on Enterprise.
* **Hybrid or self-hosted deployment.** Available on Enterprise.
* **Custom contractual terms.** Available on Enterprise.

## What you can build on Free

Plenty.

* Run a real evaluation against your own dataset on a small candidate-model set.
* Build a custom LLM judge and GEPA-optimize it on a small labeled set.
* Wire the SDK into a personal project's CI gate.
* Browse the catalog and quarterly reports for research.

When you outgrow the starter ECU, upgrade to Premium PAYG without changing your code or your account.

## How to get started

1. Browse [Stratix Public](https://stratix.layerlens.ai) — no signup required.
2. When you want to run a private evaluation, [sign up](/2.-get-started/sign-up) for a Free Premium account.
3. Install the [SDK](/2.-get-started/sdk) when you want to automate.

## Where to next

* [Premium tier details](/3.-pricing/premium-tier-details)
* [Concept: ECU credits](/11.-admin/concepts-ecu-credits)
* [Getting started](/3.-pricing/03-pricing)


# Premium tier details

What's in Stratix Premium PAYG — multi-org, BYOK, agentic evals, GEPA, traces, priority support.

## What's included

Everything in [Free](/3.-pricing/free-tier-details), plus:

* **Pay-as-you-go ECU.** Buy ECU credits as you need them. No commitment.
* **Multi-org membership.** Belong to multiple orgs; switch between them in the top bar.
* **Team seats.** Invite teammates per org.
* **BYOK custom models.** Register OpenAI-compatible endpoints as first-class models in Premium.
* **Full agentic evaluation product.** Mix assertions, deterministic rules, and LLM judges into one evaluation.
* **GEPA judge optimization.** Tune LLM judges against labeled ground-truth.
* **Trace evaluations.** Score uploaded traces with scorers and judges. Recurring schedules.
* **Notifications.** In-app and email notifications for evaluation runs, threshold crossings, etc.
* **In-app Assistant.** Context-aware help.
* **Priority email support.** Faster response than community channel.

## What's not included

* **SSO.** Available on Enterprise.
* **Data residency selection.** Available on Enterprise.
* **Hybrid / self-hosted deployment.** Available on Enterprise.
* **Volume ECU pricing.** Available on Enterprise.
* **Dedicated CSM, white-glove onboarding.** Available on Enterprise.
* **Extended audit log retention.** Available on Enterprise.
* **Custom DPA, MSA, SLA.** Available on Enterprise.

## ECU economics

ECU consumption depends on the operation:

* **Model inference** — billed per-token, by model
* **Judges** — billed per-run, by judge model
* **GEPA optimization** — billed per-iteration, by judge model
* **Trace evaluations** — billed per-trace × per-judge

The Premium dashboard shows a **cost preview** before each run so you know the worst-case ECU consumption. Real consumption may be lower if a job short-circuits.

### Directional pricing

Live rates are on [`layerlens.ai/pricing`](https://layerlens.ai/pricing). Approximate monthly bands as of mid-2026:

| Workload                                             | Monthly ECU $                                |
| ---------------------------------------------------- | -------------------------------------------- |
| Solo developer / single feature, ad-hoc              | $0–$50                                       |
| Small team, weekly eval cycles + CI gates            | $50–$300                                     |
| Mid-size team, continuous eval on production traffic | $300–$1,500                                  |
| Large team / multi-team platform                     | $1,500+ (negotiate Volume ECU on Enterprise) |

BYOK custom models reduce ECU consumption: when you supply your own provider key, ECU only bills the platform engine work, not the inference tokens.

## When to upgrade from Free

* You've burned through starter ECU.
* You need more than one org as admin.
* You need GEPA-optimized judges.
* You need trace evaluations on real production data.
* You need priority email support.

## When to upgrade to Enterprise

* You need SSO.
* You need data residency selection.
* You need volume ECU pricing because you're running large continuous evaluations.
* You need a dedicated CSM and a contractual SLA.
* Your security team requires a custom DPA or MSA.

## Where to next

* [Enterprise tier details](/3.-pricing/enterprise-tier-details)
* [Concept: ECU credits](/11.-admin/concepts-ecu-credits)
* [Stratix Premium](https://github.com/LayerLens/gitbook-full/blob/main/05-select/catalog/premium-workspace-overview.md)
* [Industry patterns](/4.2-industry-use-cases/travel-hospitality)


# Enterprise tier details

What's in the Stratix Enterprise tier — SSO, data residency,, dedicated CSM.

Stratix Enterprise is a custom annual contract for organizations with security, compliance, scale, or regulatory requirements that go beyond Premium PAYG.

## What's included

Everything in [Premium PAYG](/3.-pricing/premium-tier-details), plus:

* **SSO.** SAML 2.0 and OIDC integration with your enterprise IdP. Just-in-time provisioning, role mapping, group-based access.
* **Data residency.** Choose where your tenant data lives. Currently US, EU regions; talk to sales for additional regions.
* **Volume ECU pricing.** Negotiated ECU rates that beat PAYG when you operate at scale.
* **Dedicated Customer Success Manager.** Quarterly business reviews, proactive engagement, escalation path.
* **White-glove onboarding.** Architecture review, integration support, custom playbooks.
* **Extended audit log retention.** Configurable retention for compliance.
* **Custom contractual terms.** DPA, MSA, SLA tailored to your legal team's requirements.
* **Priority incident response.** Defined RTO/RPO commitments.

## When Enterprise makes sense

* You're in a regulated industry — financial services, healthcare, public sector — with audit requirements.
* You have an enterprise IdP (Okta, Azure AD, OneLogin) and need SSO.
* You need contractual data residency commitments.
* You're an internal AI platform serving many product teams across the org.
* Your spend on Premium PAYG would benefit from negotiated volume pricing.
* You require a custom DPA or MSA before you can onboard.

## What you'll discuss with sales

* Scale: number of seats, expected ECU consumption, expected trace ingestion volume
* Deployment shape: SaaS, hybrid, region selection
* Auth integration: which IdP, role mapping, group-based access
* Compliance asks: HIPAA (BAA available), FedRAMP, and other framework-specific requests — contact <security@layerlens.ai>
* Contractual: DPA, MSA, SLA, data processing terms

## How to start a conversation

Email <sales@layerlens.ai> or [book time directly](https://layerlens.ai/contact).

You can prove the platform out on Premium PAYG before opening an Enterprise conversation — most Enterprise customers run a Premium pilot first.

## Where to next

* [Enterprise (full section)](/3.-pricing/03-pricing)
* [Security and compliance](/3.-pricing/03-pricing)
* [Architecture](/13.3-concepts-library-+-architecture/architecture)
* [Industry pattern: healthcare contraindicated cocktail (regulated)](/4.2-industry-use-cases/pattern)


# Pricing tiers — quick overview

Free, Premium, Enterprise — the short version of what each tier includes.

Three tiers. The full pricing detail is in the [Pricing](/3.-pricing/03-pricing) section; this page is the elevator-pitch version.

## Free

**Cost:** $0 **Audience:** anyone — researchers, individual builders, hobbyists, evaluation buyers running a trial **What's in:**

* Full access to **Stratix Public** — models, benchmarks, public evaluations, compare-models, public spaces, search, leaderboard
* A Premium account with starter ECU credits
* One organization
* Basic seats
* The Python SDK and CLI

**What's not in:**

* Larger ECU balances (purchase under Premium)
* Enterprise SSO
* Hybrid or self-hosted deploy
* Volume support

[Read the full Free tier details →](/3.-pricing/free-tier-details)

## Premium (Pay-As-You-Go)

**Cost:** ECU PAYG (consumption-based) **Audience:** AI teams shipping features **What's in:**

* Everything in Free
* Pay-as-you-go ECU consumption — buy credits as needed
* Multi-org membership and switching
* Team seats
* BYOK custom models
* Full agentic-evaluation product
* GEPA judge optimization
* Trace evaluations
* Notifications and in-app Assistant
* Priority email support

**What's not in:**

* SSO (Enterprise)
* Data residency selection (Enterprise)
* Custom contractual terms (Enterprise)

[Read the full Premium tier details →](/3.-pricing/premium-tier-details)

## Enterprise

**Cost:** custom — annual contract **Audience:** larger organizations, regulated industries, security-driven buyers **What's in:**

* Everything in Premium
* **SSO** (SAML, OIDC) — enterprise IdP integration
* **Data residency** selection
* **Custom contractual terms** — DPA, MSA, SLA
* **Dedicated CSM** and white-glove onboarding
* **Volume ECU** pricing
* **Audit log access** and extended retention

**What's not in:** depends on the negotiated contract — talk to sales.

[Read the full Enterprise tier details →](/3.-pricing/enterprise-tier-details)

## Picking a tier

| If you...                                        | Pick                           |
| ------------------------------------------------ | ------------------------------ |
| Just want to browse the catalog                  | Free / Stratix Public          |
| Run a trial on your data                         | Free + Premium starter credits |
| Ship AI features as a small/mid team             | Premium PAYG                   |
| Need SSO, data residency, contractual SLA        | Enterprise                     |
| Build an internal AI platform serving many teams | Enterprise                     |

## Where to next

* [Pricing](/3.-pricing/03-pricing) — full details, FAQ, ECU economics
* [How Stratix compares](/1.-introduction/how-stratix-compares) — total cost of ownership vs alternatives
* [Enterprise](/3.-pricing/03-pricing) — what's actually in the Enterprise tier
* [Industry patterns](/4.2-industry-use-cases/travel-hospitality) — grounded vignettes by vertical


# Use cases

Concrete ways teams use Stratix — model evaluation, agentic eval, RAG, CI/CD gates, continuous eval.

What Stratix is *for* — six concrete shapes of work that the platform handles end-to-end.

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>Model evaluation</strong></td><td>Pick the right model for a task by running it against the right benchmark.</td><td><a href="/4.1-general-use-cases/model-evaluation">Read</a></td></tr><tr><td><strong>Benchmark-driven development</strong></td><td>Make benchmarks a first-class signal in your dev loop.</td><td><a href="/4.1-general-use-cases/benchmark-driven-development">Read</a></td></tr><tr><td><strong>Agentic evaluation</strong></td><td>Pre- and post-deployment gates for multi-step agents — assertions, rules, judges.</td><td><a href="/4.1-general-use-cases/agentic-evaluation">Read</a></td></tr><tr><td><strong>Continuous evaluation</strong></td><td>Score live production traces on a recurring schedule.</td><td><a href="/4.1-general-use-cases/continuous-evaluation">Read</a></td></tr><tr><td><strong>RAG evaluation</strong></td><td>Faithfulness, retrieval quality, end-to-end answer quality.</td><td><a href="/4.1-general-use-cases/rag-evaluation">Read</a></td></tr><tr><td><strong>AI quality gates in CI/CD</strong></td><td>Block prompt and model regressions in your merge pipeline.</td><td><a href="/4.1-general-use-cases/ai-quality-gates-cicd">Read</a></td></tr></tbody></table>

## Pick a use case

| If you want to...                  | Read                                                                                |
| ---------------------------------- | ----------------------------------------------------------------------------------- |
| Choose a model for a new feature   | [Model evaluation](/4.1-general-use-cases/model-evaluation)                         |
| Score multi-step agents pre-deploy | [Agentic evaluation](/4.1-general-use-cases/agentic-evaluation)                     |
| Score real production traffic      | [Continuous evaluation](/4.1-general-use-cases/continuous-evaluation)               |
| Evaluate a RAG pipeline end-to-end | [RAG evaluation](/4.1-general-use-cases/rag-evaluation)                             |
| Block bad changes from merging     | [AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd)           |
| Treat benchmarks as a daily signal | [Benchmark-driven development](/4.1-general-use-cases/benchmark-driven-development) |

## Want to map a use case to a tutorial?

* [First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation)
* [Build your first judge](/8.-evaluate-score-the-outputs/02-first-judge)
* [Wire CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates)
* [Score live traces](/8.-evaluate-score-the-outputs/04-score-traces)


# Model evaluation

Pick the right model for a task by running it against the right benchmark.

Picking a model is the most expensive decision you'll make as an AI team. The wrong choice locks in months of wasted iteration on a foundation that was never going to work. Stratix's model-evaluation use case is the disciplined version of that decision.

## The shape of the work

1. **Frame the question.** What does your feature actually need? Reasoning, structured output, multi-turn dialog, code, multilingual, multimodal? Don't pick a model on vibes.
2. **Find the right benchmarks.** Browse the [benchmarks catalog](/5.-select-pick-the-model/benchmarks-catalog). Pick 1-3 that map to your task. (Most teams pick too many; resist.)
3. **Browse the public leaderboard.** [Compare models](/5.-select-pick-the-model/compare-models) head-to-head. Get to a shortlist of 3-5.
4. **Run a private evaluation on your data.** Public scores tell you what's strong in general. Your data tells you what's strong for *you*. Stratix Premium runs a private evaluation against your dataset.
5. **Read the verdict, not the headline.** A model that scores 2% higher overall but 15% lower on your one critical dimension is the wrong pick.

## Why it works on Stratix

* **Public catalog gives you the shortlist for free.** No multi-week vendor proof-of-concepts to find candidate models.
* **Private evaluations on your data give you the verdict.** Public leaderboards don't see your prompts or your distribution.
* **Compare models is built in.** Side-by-side score tables with confidence intervals.
* **Costs are visible up-front.** ECU consumption is shown before you run.

## Tools you'll use

* [Stratix Public — Models catalog](/5.-select-pick-the-model/models-catalog)
* [Stratix Public — Compare models](/5.-select-pick-the-model/compare-models)
* [Stratix Premium — Evaluations](/8.-evaluate-score-the-outputs/evaluations)
* [Stratix Premium — Benchmarks](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/cli/benchmarks.md)
* [SDK: `client.evaluations.create()`](/4.1-general-use-cases/general)

## Outcomes you should see

You'll know this is working when:

* **Model selection time drops from weeks to hours.** Public catalog narrows; private eval decides.
* **Every model decision has a citable evaluation ID.** No "we picked X because it felt better."
* **Re-evaluating a new candidate model takes under 1 hour.** Including standing up the eval against your dataset.
* **Cost and latency are part of the choice, not afterthoughts.** You can tell the team "we picked the cheaper model because at 95% of the quality it's 3× cheaper."

## Anti-patterns

* **Benchmark cargo-culting.** "Everyone uses MMLU" is not why MMLU is right for your task.
* **One-shot picking.** The model frontier moves quarterly. Re-evaluate the candidates as new models drop.
* **Ignoring cost and latency.** A 2% accuracy gain at 4× cost is rarely worth shipping.

## Where to next

* [Concept: Models and benchmarks](/5.-select-pick-the-model/models-and-benchmarks)
* [Tutorial: First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation)
* [Workflow: Compare](/9.-improve-tune-the-system/workflow)
* [Cookbook: model-selection recipes](/2.-get-started/all-cookbook-recipes)


# Benchmark-driven development

Make benchmarks a first-class signal in your dev loop — every change runs against a benchmark, every result is on the dashboard.

Most teams treat benchmarks as a one-time activity at the start of a project. Pick a model, glance at MMLU, get on with it. **Benchmark-driven development** flips that — every meaningful change to your AI feature runs against the benchmarks that matter, and every result lives on a shared dashboard.

## The shape of the work

1. **Pick your benchmarks.** Public benchmarks for general capability + your own private benchmark suite for task-specific quality.
2. **Wire it into your dev loop.** Local: `pytest`-style harness running a small slice. CI: full benchmark suite on every PR. Nightly: full suite + production-traffic slice.
3. **Track the score over time.** Stratix's evaluation history page shows the score curve per benchmark.
4. **Block regressions in CI.** A score that drops below baseline blocks the PR.
5. **Report wins to the team.** When a change improves a benchmark, share it.

## Why it works on Stratix

* **52+ public benchmarks ready to use** — you don't have to host the data
* **Private benchmark suites** — upload your own dataset and grading config
* **Score history** — every run is recorded; trends are visible
* **CI gates** — easily wired via the SDK or CLI
* **Compare models** — when a benchmark gets stronger, see if a different model is now a better fit

## Tools you'll use

* [Stratix Public — Benchmarks catalog](/5.-select-pick-the-model/benchmarks-catalog)
* [Stratix Premium — Benchmarks](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/cli/benchmarks.md)
* [SDK: `client.evaluations.create()`](/4.1-general-use-cases/general)
* [CLI: `layerlens evaluate`](/4.1-general-use-cases/general)

## Outcomes you should see

You'll know this is working when:

* **Every PR touching AI runs a benchmark gate** — non-negotiable, not opt-in.
* **Your benchmark gating eval runs in <5 minutes** — fast enough that no one wants to skip it.
* **Score-over-time trends are visible to the whole team**, not buried in one engineer's terminal.
* **Net-new benchmarks reach 80% of your team within a week** — adding signal is cheap and shared.

## Anti-patterns

* **Benchmark inflation.** Adding 30 benchmarks because "more signal is better." Pick 3-5 that matter, run them often.
* **Treating public scores as your scores.** Public scores are a leading indicator. Your private benchmark on your data is the verdict.
* **Skipping CI.** If benchmarks don't gate merges, they're decoration.

## Where to next

* [Tutorial: Wire CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates)
* [Workflow: Evaluate](/9.-improve-tune-the-system/workflow)
* [Concept: Models and benchmarks](/5.-select-pick-the-model/models-and-benchmarks)
* [Cookbook: CI/CD recipes](/2.-get-started/all-cookbook-recipes)


# Agentic evaluation

Pre- and post-deployment quality gates for multi-step agents — combine assertions, deterministic rules, and LLM judges.

Agents fail differently from chat. They reach a wrong final state, take a wrong path to a right state, call a tool they shouldn't, or quietly regress on an edge case. Agentic evaluation is the **pre- and post-deployment** practice of catching these failures before they ship.

## The shape of the work

1. **Capture a representative trace set.** Your agent's actual runs — inputs, tool calls, outputs, every span. Real or synthetic.
2. **Define evaluation criteria.** Mix three types:

* **Natural-language assertions** — "the agent identified the customer's account tier"
* **Deterministic rules** — "the agent never called `admin_api.delete_*`"
* **LLM judges** — "rate the helpfulness of the final response 1-5"

3. **Run the evaluation.** Stratix runs all criteria over the trace set in one job.
4. **Read verdict + root-cause.** Each failed criterion ties to the trace, the span, and the decision that broke it.
5. **Detect regressions.** Compare to a baseline; surface newly failing criteria.

## Why it works on Stratix

* **One engine, three criteria types** — assertions, rules, and judges in the same evaluation
* **First-class trace and span access** — rules can inspect any field of any span
* **Judge engine + GEPA optimization** — your subjective bar gets sharper over time
* **Regression detection** — built-in baseline comparison
* **Pre- and post-deployment fit** — runs on your candidate change, not on live traffic

## Tools you'll use

* [Stratix Premium — Agent Evaluation](/7.-observe-see-whats-happening/agent-evaluation)
* [Stratix Premium — Traces](/7.-observe-see-whats-happening/traces)
* [Stratix Premium — Judges](/8.-evaluate-score-the-outputs/judges)
* [SDK: `client.trace_evaluations.create()`](/4.1-general-use-cases/general)

## Outcomes you should see

You'll know this is working when:

* **Zero CRITICAL deterministic-rule violations** in any release-gate run.
* **>95% pass rate on your assertion criteria** across the curated trace set.
* **Regression report names <2 newly failing criteria** per release.
* **Auditor questions about agent safety are answered live in-meeting** with the verdict + root-cause artifacts.

## Anti-patterns

* **Judge-only evaluations.** Judges are the slowest, most expensive criterion. Anchor with deterministic rules and assertions; use judges for the residual subjective bar.
* **No regression detection.** Without a baseline, today's pass rate is meaningless.
* **Evaluating live traffic for pre-deploy gates.** Live traffic is for [continuous evaluation](/4.1-general-use-cases/continuous-evaluation). Pre-deploy uses a captured trace set.

## Where to next

* [Concept: Agentic evaluation](/4.1-general-use-cases/agentic-evaluation)
* [Overview: Agentic evaluations](/4.1-general-use-cases/agentic-evals-overview)
* [Workflow: Evaluate](/9.-improve-tune-the-system/workflow)
* [Cookbook: agentic recipes](/2.-get-started/all-cookbook-recipes)


# Agentic evaluations — overview

Pre- and post-deployment quality gates for multi-step AI agents — assertions, deterministic rules, and LLM judges.

Most AI features today are not single-turn chat. They're **agents** — multi-step systems that plan, call tools, retrieve context, and decide when they're done. Single-output evaluation isn't enough for agents. You need to know whether the agent reached the right state, took legal steps to get there, and won't regress when the prompt changes next week.

That's what Agentic Evaluations does.

## The failure scenarios

Agents fail in ways simple chat doesn't. The Stratix Agentic Evaluations product is shaped around these specific failures:

* **Wrong final state.** The agent answered the question but updated the wrong record.
* **Right answer, wrong path.** It got there, but called an internal API it shouldn't have.
* **Catastrophic tool selection.** A `delete_user` call when the user asked to "deactivate."
* **Off-policy reasoning.** The agent's chain-of-thought reveals reasoning your team doesn't want shipped (PII, leaked instructions, harmful claims).
* **Silent regression.** A prompt change five PRs ago broke an edge case nobody is watching.

## Three evaluation criteria types, combined

Agentic evaluations let you mix **three criteria types** in one evaluation:

1. **Natural-language assertions.** Plain-English checks the LLM grades. "The agent correctly identified the customer's account tier." Cheap, flexible, good for fuzzy correctness.
2. **Deterministic rules.** Code or schema checks. "The agent never called `admin_api.*` outside the allowed scope." Fast, cheap, exact.
3. **LLM judges.** Subjective dimensions evaluated by an optimized judge. "How helpful was the final response on a 1-5 scale?" Use sparingly — most subjective bars resolve into combinations of assertions.

A typical agentic evaluation uses all three. Deterministic rules catch hard violations early. Assertions cover the bulk of correctness. Judges grade the residual subjective quality.

## The pre- and post-deployment workflow

The Stratix Agentic Evaluations workflow is built for **pre- and post-deployment** — you run it on a candidate change before it ships, not on live production.

1. **Capture a representative trace set.** Real or synthetic, agent inputs + outputs + every span.
2. **Define your evaluation criteria.** A mix of assertions, rules, and judges in an evaluation space.
3. **Run the evaluation.** Stratix replays the criteria over the trace set.
4. **Read the verdict and root-cause.** Each failed criterion ties back to the trace, the span, the input, and the agent decision that broke it.
5. **Detect regressions.** Compare the run to a baseline; surface the criteria that newly failed.

The output artifacts are:

* A **verdict** (pass/fail and severity per criterion)
* A **root-cause report** (which span, which input, which decision)
* A **regression report** (what newly fails compared to baseline)

You wire those into a CI gate, a release-readiness review, or a Slack alert.

## What you get vs what you build

You build:

* The trace set (your data, your agent runs)
* The evaluation criteria (your assertions, your rules, your judges)
* The CI/CD wiring

Stratix gives you:

* The evaluation engine that runs all three criteria types in one job
* The judge engine and the GEPA optimizer
* The deterministic-rule executor with first-class span access
* The verdict, root-cause, and regression reports
* The Premium UI for browsing failures and the SDK for automation

## When NOT to use Agentic Evaluations

* For **single-turn chat with no tools or chain**, the Model Evaluations product is simpler. Pick that.
* For **post-deployment monitoring of live traffic**, [trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) is the right shape — same engine, recurring schedule, applied to real production traces.

## Where to next

* [Use case: Agentic evaluation](/4.1-general-use-cases/agentic-evaluation)
* [Concept: Agentic evaluation](/4.1-general-use-cases/agentic-evaluation)
* [Workflow: Evaluate](/9.-improve-tune-the-system/workflow)
* [Stratix Premium: Agent Evaluation](/7.-observe-see-whats-happening/agent-evaluation)


# Continuous evaluation

Score live production traces on a recurring schedule — surface drift before customers do.

Pre- and post-deployment evaluations catch regressions before merge. **Continuous evaluation** catches them after merge — when something slips through, when production traffic shifts, when the model provider quietly updates their backend.

## The shape of the work

1. **Ingest live traces.** Your application emits trace JSON; Stratix ingests via API or SDK.
2. **Define a trace evaluation.** Pick the scorers and judges you want applied. The same scorers and judges from your pre- and post-deployment evaluations work here.
3. **Schedule it.** Run the trace evaluation on a recurring cadence — daily, hourly, per-batch.
4. **Watch the trend.** Stratix shows score-over-time per scorer per dimension.
5. **Get notified on drift.** When a score crosses a threshold, surface a notification (Premium) or a Slack alert.

## Why it works on Stratix

* **Same engine across pre-deploy and continuous.** The scorers and judges you built for CI gates are reusable.
* **Trace-first design.** Stratix's trace pipeline is shaped for continuous workflows; ingestion is rate-limited and persistent.
* **GEPA-optimized judges.** Your judge agreement with humans stays tight as live traffic shifts.
* **Org-scoped and tenant-isolated.** One tenant's burst doesn't affect another.

## Tools you'll use

* [Stratix Premium — Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations)
* [Stratix Premium — Traces](/7.-observe-see-whats-happening/traces)
* [SDK: trace ingestion](/4.1-general-use-cases/general)
* [Notifications](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/notifications.md)

## Outcomes you should see

You'll know this is working when:

* **Mean time to detect a quality regression drops to <1 hour** (versus days/weeks of customer-driven discovery).
* **Score-over-time charts are reviewed weekly** by the team that owns the AI feature.
* **Threshold alerts fire and resolve** — silence is suspicious; chronic firing means the threshold needs tuning.
* **Sample bias is measured**, not assumed — you can show the sample distribution matches production.

## Anti-patterns

* **Only scoring sampled traces.** Sampling is fine, but watch sample bias. Production distribution drifts; sample drifts with it.
* **Mixing pre-deploy and continuous in one config.** They have different cadence and different cost profiles. Keep them separate.
* **No alerting.** A continuously-evaluated trace pipeline that nobody reads is decoration.

## Where to next

* [Concept: Continuous evaluation](/4.1-general-use-cases/continuous-evaluation)
* [Tutorial: Score live traces](/8.-evaluate-score-the-outputs/04-score-traces)
* [Workflow: Observe](/9.-improve-tune-the-system/workflow)
* [Cookbook: continuous-eval recipes](/2.-get-started/all-cookbook-recipes)


# RAG evaluation

Evaluate RAG pipelines end-to-end — retrieval quality, faithfulness, and answer quality.

A RAG (retrieval-augmented generation) pipeline has more failure modes than a plain LLM call. Bad retrieval kills good models; great retrieval can't save a hallucinating one. RAG evaluation scores each stage so you know where the failure actually is.

## The shape of the work

1. **Define the three dimensions.**

* **Retrieval quality** — did we get back relevant chunks?
* **Faithfulness** — did the answer ground itself in the retrieved chunks (not invent)?
* **Answer quality** — did the final answer actually help the user?

2. **Pick scorers and judges.**

* Retrieval: classical IR metrics (precision\@k, recall\@k) as scorers; or an LLM judge that grades chunk relevance.
* Faithfulness: an LLM judge that compares answer claims against the retrieved chunks. GEPA-optimize against a labeled set.
* Answer quality: an LLM judge for end-to-end helpfulness, again GEPA-optimized.

3. **Run as a trace evaluation.** Your pipeline emits a trace with retrieval span + generation span. Stratix grades each.
4. **Compare configurations.** Try a different retriever, a different chunk size, a different model — re-run, see the score deltas per dimension.
5. **Lock in CI gates.** Faithfulness regressions are particularly costly; gate them.

## Why it works on Stratix

* **Trace-first.** Multi-span traces are first-class; you can score retrieval and generation separately.
* **Judge optimization.** GEPA tunes judges to match human labels for faithfulness — usually the trickiest dimension.
* **Compare-models.** Try the same retrieval feeding three different generators; see which generator is most faithful to the same chunks.

## Tools you'll use

* [Stratix Premium — Agent Evaluation](/7.-observe-see-whats-happening/agent-evaluation) (RAG is a small agent)
* [Stratix Premium — Traces](/7.-observe-see-whats-happening/traces)
* [Stratix Premium — Judges](/8.-evaluate-score-the-outputs/judges)
* [SDK trace ingestion](/4.1-general-use-cases/general)

## Outcomes you should see

You'll know this is working when:

* **Faithfulness judge agreement with humans crosses 90%** after GEPA optimization.
* **Retrieval and answer scores diverge meaningfully** — when answer quality drops, you can tell whether retrieval or generation is at fault.
* **Hallucination rate trends downward** over consecutive releases, not just bounces.
* **Per-chunk-strategy comparisons take <1 day** instead of multi-week experiments.

## Anti-patterns

* **Scoring only the final answer.** You'll know quality dropped; you won't know which stage caused it.
* **Faithfulness judge without optimization.** Out-of-the-box LLM judges over-credit confident-sounding hallucinations. GEPA-optimize against a labeled set.
* **No retrieval ground truth.** If you can't say "these are the right chunks," your retrieval scores are just plausibility scores.

## Where to next

* [Concept: Judges (faithfulness)](/8.-evaluate-score-the-outputs/judges-1)
* [Concept: Traces and spans](/6.-build-wire-your-code/traces-and-spans)
* [Cookbook: RAG recipes](/2.-get-started/all-cookbook-recipes)


# AI quality gates in CI/CD

Block prompt and model regressions in your merge pipeline — turn evaluations into a hard gate.

Code regressions are caught by tests. AI regressions slip through unless you treat the eval as a test. The CI/CD-quality-gates use case wires Stratix evaluations into your pipeline so a prompt change that drops accuracy can't merge.

## The shape of the work

1. **Pick your gating evaluation.** A small, fast subset of your benchmarks and judges. Goal: meaningful signal in under 5 minutes.
2. **Add the eval as a CI step.** GitHub Actions, GitLab CI, Buildkite — call the SDK or CLI with the eval config.
3. **Compare against baseline.** Stratix maintains an evaluation history. The CI step compares the new run to the most recent main-branch run.
4. **Fail the build on regression.** If any tracked dimension drops by more than your tolerance, fail.
5. **Annotate the PR.** Post the score table back to the PR for reviewer context.

## Why it works on Stratix

* **Evaluation history is built-in.** No need to roll your own diff infrastructure.
* **SDK and CLI both support CI.** Pick whichever feels native.
* **Compare-models lets you stage upgrades.** Switch the model in CI, verify scores, then promote.
* **GEPA-optimized judges keep gates honest.** Out-of-the-box judges flake; tuned ones don't.

## Tools you'll use

* [SDK: `client.evaluations.create()`](/4.1-general-use-cases/general)
* [CLI: `layerlens evaluate`](/4.1-general-use-cases/general)
* [Stratix Premium — Evaluations](/8.-evaluate-score-the-outputs/evaluations)

## Outcomes you should see

You'll know this is working when:

* **Zero AI regressions reach production** because the CI gate caught them.
* **CI gate run time stays under 5 minutes** even as the eval grows.
* **Tolerance is calibrated** — false-positive rate <5%, false-negative rate <2%.
* **Engineers cite the eval result in the PR description** as a matter of habit.

## Anti-patterns

* **5-hour CI evals.** If your team waits 5 hours for a merge gate, they'll ignore it. Pick a fast subset.
* **Whole-org tolerance.** Different repos have different quality bars. Set per-repo or per-feature tolerances.
* **No alerting on baseline drift.** If the baseline silently drops over 4 PRs, you never noticed because each PR was within tolerance. Watch the trend.

## Where to next

* [Tutorial: Wire CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates)
* [Workflow: Govern](/9.-improve-tune-the-system/workflow)
* [Cookbook: CI/CD recipes](/2.-get-started/all-cookbook-recipes)
* [Integrations: GitHub Actions](/4.1-general-use-cases/general)


# Industry solutions

Stratix industry solutions — 13 verticals with scenarios, eval patterns, compliance, and runnable samples.

Each industry page has the same five sub-pages:

* **README** — top scenarios + why Stratix
* **Scenarios** — 1-4 specific evaluation scenarios adapted from the LayerLens scenarios catalog (Stratix Premium capabilities only)
* **Eval patterns** — what scorers and judges typical teams use
* **Compliance** — the regulatory considerations to set up first
* **Cookbook and samples** — runnable starter samples

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>Healthcare</strong></td><td>Clinical decision support, triage, medical Q&#x26;A.</td><td><a href="/4.2-industry-use-cases/healthcare">Read</a></td></tr><tr><td><strong>Financial services</strong></td><td>Fraud detection, advisory, KYC.</td><td><a href="/4.2-industry-use-cases/financial-services">Read</a></td></tr><tr><td><strong>Legal</strong></td><td>Contract review, research, document analysis.</td><td><a href="https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/legal/README.md">Read</a></td></tr><tr><td><strong>Retail and e-commerce</strong></td><td>Product Q&#x26;A, support, recommendations.</td><td><a href="/4.2-industry-use-cases/retail-ecommerce">Read</a></td></tr><tr><td><strong>Media and entertainment</strong></td><td>Moderation, summarization, search.</td><td><a href="/4.2-industry-use-cases/media-entertainment">Read</a></td></tr><tr><td><strong>Insurance</strong></td><td>Claims triage, policy Q&#x26;A, underwriting.</td><td><a href="/4.2-industry-use-cases/insurance">Read</a></td></tr><tr><td><strong>Government and public sector</strong></td><td>Citizen services, document processing.</td><td><a href="/4.2-industry-use-cases/government-public-sector">Read</a></td></tr><tr><td><strong>Manufacturing</strong></td><td>Diagnostics, manuals Q&#x26;A, quality.</td><td><a href="/4.2-industry-use-cases/manufacturing">Read</a></td></tr><tr><td><strong>Education</strong></td><td>Tutoring, content generation, assessment.</td><td><a href="/4.2-industry-use-cases/education">Read</a></td></tr><tr><td><strong>Real estate</strong></td><td>Listing Q&#x26;A, valuation, document search.</td><td><a href="/4.2-industry-use-cases/real-estate">Read</a></td></tr><tr><td><strong>Telecommunications</strong></td><td>Service, troubleshooting, plans Q&#x26;A.</td><td><a href="/4.2-industry-use-cases/telecommunications">Read</a></td></tr><tr><td><strong>Energy and utilities</strong></td><td>Field service, ops, customer support.</td><td><a href="/4.2-industry-use-cases/energy-utilities">Read</a></td></tr><tr><td><strong>Travel and hospitality</strong></td><td>Booking, itinerary, customer service.</td><td><a href="/4.2-industry-use-cases/travel-hospitality">Read</a></td></tr></tbody></table>

## Source note

Industry pages adapt the LayerLens-internal scenarios catalog. Where a scenario depended on capabilities not yet shipping in Stratix Premium (rules engine, remediation, prompt lab, chaos, A/B testing), that component is omitted rather than overstated.


# By AI workflow shape

Industry patterns indexed by AI workflow shape — RAG, agent, classification, generation, multimodal.

A cross-cut index. If you shop for patterns by *what your AI feature does* rather than by *which industry you're in*, this is the entry point.

## Retrieval-augmented generation (RAG)

Your AI retrieves source material and synthesizes an answer with citations. Faithfulness against retrieved chunks is the core dimension.

* [Healthcare — clinical decision support](/4.2-industry-use-cases/pattern) — guideline retrieval + faithfulness + drug-interaction rules
* [Government — citizen-services chatbot](/4.2-industry-use-cases/pattern-4) — versioned policy retrieval + reading-level + multilingual parity
* [Legal — AI research citation verification](/4.2-industry-use-cases/pattern-2) — case-law retrieval + citation-existence + holding-accuracy
* [Manufacturing — visual quality inspection](/4.2-industry-use-cases/pattern-7) (uses retrieval over manuals when paired with technician guidance)

## Multi-step agents

Your AI plans, calls tools, and decides when it's done. Pre- and post-deployment agentic evaluation with deterministic span-level rules is the right shape.

* [Insurance — claims-triage agent](/4.2-industry-use-cases/pattern-3) — span-level escalation rule + assertions + tone judge
* [Healthcare — clinical decision support](/4.2-industry-use-cases/pattern) — also fits if the assistant calls drug-interaction databases as tools

## Classification and detection

Your AI classifies content into categories or detects targets in inputs. Per-class precision/recall, confusion-matrix tracking, and per-segment fairness are the dimensions.

* [Media — content moderation](/4.2-industry-use-cases/pattern-6) — per-category recall + cross-language parity
* [Manufacturing — visual quality inspection](/4.2-industry-use-cases/pattern-7) — per-defect-class accuracy + condition-stratified disparity
* [Energy — vegetation-management imagery](/4.2-industry-use-cases/pattern-11) — hazard-class accuracy + season-stratified disparity
* [Financial services — fair-lending platform](/4.2-industry-use-cases/pattern-1) — false-positive disparity across protected-class proxies

## Content generation

Your AI generates user- or customer-facing copy, descriptions, summaries. Specification accuracy, brand voice, and uniqueness are the dimensions.

* [Retail — product description generation](/4.2-industry-use-cases/pattern-5) — specification accuracy + brand voice + uniqueness
* [Travel — multilingual booking assistant](/4.2-industry-use-cases/pattern-12) — booking-detail accuracy + cultural fit per locale

## Conversational with policy gates

Your AI handles a multi-turn customer interaction with strict policy or pricing rules. Citation, exact-match, and continuous trace eval are the dimensions.

* [Telecom — customer-service pricing accuracy](/4.2-industry-use-cases/pattern-10) — JSON-schema pricing citation + numeric exact-match
* [Education — AI tutor](/4.2-industry-use-cases/pattern-8) — Socratic vs. answer-giving + age-appropriateness
* [Government — citizen-services chatbot](/4.2-industry-use-cases/pattern-4) — also fits

## Multimodal (image / video / audio)

Your AI processes non-text inputs. Per-modality scorers paired with text-output judges.

* [Manufacturing — visual quality inspection](/4.2-industry-use-cases/pattern-7) — image classification under varied lighting
* [Energy — vegetation-management imagery](/4.2-industry-use-cases/pattern-11) — drone imagery, season-stratified
* [Real estate — automated property valuation](/4.2-industry-use-cases/pattern-9) — listing photos + comparable sales

## Where to next

* [All 13 patterns by industry](/4.2-industry-use-cases/industry)
* [Industries](/4.2-industry-use-cases/industry) — full per-vertical depth
* [Use cases](/4.1-general-use-cases/general) — by AI workflow shape (broader framing)
* [Workflow](/1.-introduction/the-stratix-workflow) — the 6-stage spine


# Healthcare

Healthcare AI evaluation on Stratix — clinical Q\&A, triage, medical document analysis.

Healthcare AI has the highest stakes on the platform. Every output is potentially clinically consequential. Evaluation in healthcare is not optional; it's the gating discipline.

## Top scenarios

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>Clinical decision-support Q&#x26;A</strong></td><td>AI helps clinicians retrieve guidelines and decision pathways.</td><td><a href="https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/healthcare/scenarios.md#clinical-qa">Read</a></td></tr><tr><td><strong>Patient triage chatbot</strong></td><td>Triage symptoms; route to appropriate care.</td><td><a href="https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/healthcare/scenarios.md#patient-triage">Read</a></td></tr><tr><td><strong>Medical document summarization</strong></td><td>Summarize charts, prior authorizations, discharge notes.</td><td><a href="https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/healthcare/scenarios.md#medical-summarization">Read</a></td></tr></tbody></table>

## Why Stratix for healthcare

* **Deterministic rules + judges** combine to enforce hard safety constraints (no dosing recommendations to patients) plus subjective bars (clinical reasoning soundness)
* **Trace evaluations** preserve auditable evidence per release — important for regulatory readiness
* **GEPA judge optimization** lets you tune to clinician-labeled examples
* **BYOK custom models** support medical-specific fine-tuned models running in your environment

## Industry pattern

For an end-to-end usage scenario combining the criteria above into one evaluation:

→ [Healthcare: clinical decision support pattern](/4.2-industry-use-cases/pattern)

## Where to next

* [Scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/healthcare/scenarios.md)
* [Evaluation patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/healthcare/eval-patterns.md)
* [Compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/healthcare/compliance.md)
* [Cookbook and samples](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/healthcare/cookbook-and-samples.md)


# Pattern: clinical decision support

Healthcare — clinical decision support pattern. Faithfulness, drug-interaction safety, reasoning soundness, audit-ready evidence.

A clinical-AI assistant answers physician questions: differential diagnosis, treatment options, drug-interaction checks based on patient context. Used at the point of care, the system reads guidelines, returns a recommendation, and cites the evidence.

This pattern shows how to evaluate that system end-to-end on Stratix.

## What's at stake

| Risk dimension                                 | Magnitude                                          | Framework                        |
| ---------------------------------------------- | -------------------------------------------------- | -------------------------------- |
| Patient harm from contraindicated combinations | Adverse drug events at scale of patient encounters | Clinical incident reporting      |
| Per-incident malpractice exposure              | $250K–$5M                                          | Industry settlement bands        |
| FDA Software as a Medical Device action        | Loss of clearance, market withdrawal               | FDA SaMD guidance                |
| Hospital accreditation impact                  | Joint Commission inquiry, conditional status       | TJC sentinel-event protocol      |
| HIPAA exposure on PHI in trace bodies          | Per-violation civil penalties + breach reporting   | HIPAA Privacy and Security Rules |

## The evaluation pattern

A pre- and post-deployment **agentic evaluation** runs over a 50-100 trace curated set drawn from representative encounter shapes (rare-disease cases, multi-morbidity, pediatric vs. geriatric weight-based dosing, contraindicated drug combinations, common edge cases).

**Criteria mix (the 70/20/10 healthcare ratio):**

1. **Deterministic rules (\~70%)**

* Every drug pair the AI mentions is checked against an authoritative interaction database. Missing checks = CRITICAL.
* Every dose mention extracts cleanly and matches the prescribing reference exactly.
* PHI-redaction regex confirms no patient identifiers appear in any logged span.

2. **Natural-language assertions (\~20%)** — the AI's recommendation matches the ground-truth label from the case file.
3. **LLM judges (\~10%)**

* **Faithfulness judge** (GEPA-tuned against ≥50 clinician-labeled examples — multi-class severity output) — every clinical claim is grounded in retrieved guideline content.
* **Reasoning soundness judge** (GEPA-tuned against ≥50 examples) — penalize correct answers via flawed reasoning.

> Don't have labels yet? See [Bootstrap a judge before GEPA](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/bootstrap-judges.md) for the week-1 setup.

**Trace shape captured:**

```
trace
├── span: retrieval (guideline_corpus)
│ └── outputs: chunks[]
├── span: llm (answer-synthesis)
│ ├── inputs: question, chunks
│ └── outputs: answer, citations
└── span: post-process (citation-validation)
```

Score retrieval (precision\@k against ground-truth chunks) and synthesis (faithfulness against retrieved chunks) separately.

**Continuous trace evaluation:** the same configuration runs daily on a 5% sample of production traffic. Thresholds wired to in-app and Slack notifications. The `dose-mention` and `drug-interaction-coverage` rules drop to a per-trace alert, not a daily summary.

## Configuration in code

```python
# Python (SDK)
from layerlens import Stratix
client = Stratix()

faithfulness = client.judges.create(
 name="clinical-faithfulness",
 evaluation_goal="Every clinical claim in OUTPUT must be supported by chunks in the retrieved guideline content.",
)

drug_interaction_rule = client.scorers.create_code(
 name="drug-interaction-coverage",
 code="result = check_all_drug_pairs(output, db='rxnorm')",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"env": "production", "feature": "clinical-qa"}, "sample_rate": 0.05},
 judges=[faithfulness.id],
 scorers=[drug_interaction_rule.id],
 schedule="daily",
)
result = client.trace_evaluations.wait_for_completion(trace_eval.id)
```

```typescript
// TypeScript (REST)
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { tags: { env: "production", feature: "clinical-qa" }, sample_rate: 0.05 },
 judges: [faithfulnessJudgeId],
 scorers: [drugInteractionScorerId],
 schedule: "daily",
 }),
});
const traceEval = await r.json();
```

## What you get

* Pre- and post-deployment evaluations surface contraindicated-combination patterns before patient impact, not 6 weeks into production.
* Each release pin cites the evaluation IDs that gated it — a self-contained audit packet for FDA SaMD and Joint Commission inquiries.
* Continuous trace evaluation catches drift when formulary or guideline updates shift effective knowledge.
* Auditor-ready evaluation evidence is a byproduct, not a separate engineering project.

## Stratix capabilities used

* [Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation) — three criteria types in one configuration
* [Judges with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — faithfulness and reasoning soundness
* [Custom code graders](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/custom-code-scorer.md) — drug-interaction database lookup, dose-mention extraction
* [Span-level scoring](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/span-level-scoring.md) — retrieval and synthesis scored separately
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous on production
* [Notifications](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/notifications.md) — Slack and in-app routing

## Replicate this

* [Industry → Healthcare](/4.2-industry-use-cases/healthcare)
* [Healthcare scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/healthcare/scenarios.md) — six clinical-AI scenarios with full eval criteria
* [Healthcare evaluation patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/healthcare/eval-patterns.md) — the 70/20/10 rule + GEPA labeling guide
* [Healthcare compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/healthcare/compliance.md) — HIPAA / BAA, FDA SaMD, state regs
* [Cookbook: clinical Q\&A safety judge](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/healthcare/cookbook/industry-healthcare-clinical-qa.md)


# Financial services

Financial services AI evaluation — fraud detection, advisor chatbots, KYC.

## Top scenarios

<table data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>Advisor chatbot</strong></td><td>Customer-facing financial guidance with strict guardrails.</td><td><a href="https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/financial-services/scenarios.md#advisor-chatbot">Read</a></td></tr><tr><td><strong>Fraud-detection assistant</strong></td><td>Triage suspicious transactions; explain decisions to analysts.</td><td><a href="https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/financial-services/scenarios.md#fraud">Read</a></td></tr><tr><td><strong>KYC document review</strong></td><td>Extract and verify identity documents.</td><td><a href="https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/financial-services/scenarios.md#kyc">Read</a></td></tr></tbody></table>

## Why Stratix

* Hard regulatory constraints become deterministic rules; subjective bars become judges
* Audit-friendly evaluation IDs for compliance evidence
* BYOK custom models for in-house fine-tuned compliance models

## Industry pattern

→ [Financial services: fair-lending platform pattern](/4.2-industry-use-cases/pattern-1)

## Where to next

* [Scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/financial-services/scenarios.md) | [Eval patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/financial-services/eval-patterns.md) | [Compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/financial-services/compliance.md) | [Cookbook](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/financial-services/cookbook-and-samples.md)


# Pattern: fair-lending platform

Financial services — fair-lending platform pattern. Bias scorer shared across product teams, variant comparison, continuous monitoring.

A bank deploys AI in many places — fraud detection, underwriting, customer service, marketing personalization. Each product team owns its own model, but **fair-lending exposure cuts across all of them**. A platform team owns that cross-cutting risk.

This pattern shows how the platform team standardizes fair-lending evaluation across product teams using a single org-scoped Stratix configuration that every team consumes.

## What's at stake

| Risk dimension                                                         | Magnitude                                           | Framework                            |
| ---------------------------------------------------------------------- | --------------------------------------------------- | ------------------------------------ |
| ECOA / fair-lending civil enforcement                                  | Per-violation civil penalties + consent-decree cost | CFPB / DOJ enforcement actions       |
| Per-declined-legitimate-transaction friction                           | \~$25 per false positive                            | Card-network industry benchmarks     |
| Annual US card fraud (the upside the model targets)                    | $4.2B                                               | Federal Reserve / Nilson Report data |
| Brand and customer-trust impact under public consent decree            | Multi-quarter remediation cycles                    | Public consent-decree filings        |
| State DOI / state regulator exposure on insurance and lending products | State-by-state penalty bands                        | State regulator actions              |

## The evaluation pattern

A single **org-scoped Stratix configuration** every product team uses:

1. **Bias scorer (custom code)** — computes per-segment false-positive disparity ratios against protected-class proxies (ZIP code, surname-derived demographics, age band). Flags any segment exceeding the **1.25× threshold** derived from regulatory adverse-impact guidance.
2. **Compare models** — when a product team proposes a new model, the platform team's bias scorer runs against it. Teams pick the variant that minimizes disparity without sacrificing primary-task performance (fraud recall, underwriting accuracy, etc.).
3. **Continuous trace evaluation** — production traffic from each product team is sampled hourly; bias trends are visible to the platform team's dashboard with per-team and per-feature breakdowns.
4. **Shared judge library** — helpfulness, faithfulness, safety judges curated centrally and GEPA-tuned against pooled labels (≥50 per judge, contributed across the 12 product teams); reused everywhere. See [Bootstrap a judge before GEPA](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/bootstrap-judges.md) for what to ship before the labels are there.
5. **Per-feature tolerances** — different features have different bars; tolerance configured per evaluation space.

**Onboarding:** new product teams get the bias scorer pre-wired into their CI from day one.

## Configuration in code

```python
# Python (SDK) — shared bias scorer used across product teams
from layerlens import Stratix
client = Stratix()

bias_scorer = client.scorers.create_code(
 name="fair-lending-disparity",
 code="""
groups = group_by_segment(input, output, segments=['zip_tier', 'surname_demographic'])
ratios = {g: false_positive_rate(g) / false_positive_rate('baseline') for g in groups}
result = {'passed': max(ratios.values()) <= 1.25, 'ratios': ratios}
""",
)

# Each product team consumes the shared scorer in their own evaluation
evaluation = client.evaluations.create(
 name="fraud-model-with-bias-check",
 model_id=os.environ["MODEL_ID"],
 dataset_id="fraud-labeled-v3",
 scorers=[bias_scorer.id, primary_recall_scorer_id],
)
result = client.evaluations.wait_for_completion(evaluation.id)
```

```typescript
// TypeScript (REST) — same eval from a CI runner
const r = await fetch("https://stratix.layerlens.ai/api/v1/evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 name: "fraud-model-with-bias-check",
 model_id: process.env.MODEL_ID,
 dataset_id: "fraud-labeled-v3",
 scorers: [biasScorerId, primaryRecallScorerId],
 }),
});
```

## What you get

* Bias becomes a measurement, not a discovery. Every product team sees the same scoreboard.
* Cross-team conversations stop being "your team's eval said it was fine" and start being "the org's quality bar didn't move on this PR."
* Regulator-readable evaluation history quantifies disparity and tracks remediation per release.
* Platform team curates \~5 cross-cutting judges; product teams add their own task-specific judges as needed.

## Stratix capabilities used

* [Custom code graders](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/custom-code-scorer.md) — bias-disparity computations
* [Compare models](/5.-select-pick-the-model/compare-models) — variant selection with bias-aware criteria
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled production scoring
* [Multi-tenancy + shared org library](/7.-observe-see-whats-happening/multi-tenancy) — judges and scorers reusable across teams
* [Standardize judges across teams](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/standardize-judges.md)

## Replicate this

* [Industry → Financial services](/4.2-industry-use-cases/financial-services)
* [Workflow: Govern](/9.-improve-tune-the-system/workflow)
* [Cookbook: GitHub Actions integration](/6.-build-wire-your-code/integration-github-actions)
* [Use case: AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd)


# Pattern: AI research citation verification

Legal — AI research citation verification pattern. Citation-existence scorer, holding-accuracy judge, per-session block on unverified output.

An AI legal research assistant answers case-law and statutory questions, drafts memo passages, and supports brief writing. The output cites cases, statutes, and regulations. A single fabricated citation in a filing is a sanctionable event — see *Mata v. Avianca, Inc.*, No. 22-cv-1461 (S.D.N.Y. June 22, 2023).

This pattern shows how to evaluate every research-tool output so a fabricated citation can never reach a court filing.

## What's at stake

| Risk dimension                                        | Magnitude                                    | Framework                               |
| ----------------------------------------------------- | -------------------------------------------- | --------------------------------------- |
| Court sanctions per *Mata*-style filing               | Cited sanctions $5K + reputational loss      | Public court orders                     |
| Bar discipline                                        | Public reprimand to suspension; case-by-case | State bar rules of professional conduct |
| Malpractice exposure for filed hallucinated citations | Multi-million-dollar settlements             | Public legal-malpractice filings        |
| Firm reputation in client and ABA Journal coverage    | Long-tail damage cycle                       | Industry trade press                    |

## The evaluation pattern

A two-stage evaluation runs on every research-tool output:

1. **Custom code grader (citation existence)** — every cited case, statute, or regulation is looked up against an authoritative legal database. Any citation that doesn't resolve = fabricated. Hallucination rate must be **0%** for the evaluation to pass.
2. **Holding-accuracy judge** (GEPA-tuned against ≥50 attorney-labeled examples — scored output) — for cases that exist, does the cited case actually stand for the proposition the AI claims? Does the quoted passage match the source verbatim?

> Don't have labels yet? See [Bootstrap a judge before GEPA](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/bootstrap-judges.md) for the week-1 setup.

3. **Continuous trace evaluation** — every production research session is scored. Sessions with any unverified citation are flagged in-app for attorney review **before any output can be acted on**. The flag is a hard gate; attorneys can override only after explicit acknowledgment.
4. **Compare-models** — when prompt or model changes are proposed, run the existing regression set across both; pick the variant with the lower hallucination rate at equal or better holding-accuracy.

The same pattern works for statutes, regulations, and internal precedent citations.

## Configuration in code

```python
# Python (SDK)
from layerlens import Stratix
client = Stratix()

citation_scorer = client.scorers.create_code(
 name="citation-existence",
 code="""
unresolved = [c for c in extract_citations(output) if not legal_db.lookup(c)]
result = {'passed': len(unresolved) == 0, 'fabricated': unresolved}
""",
)

holding_judge = client.judges.create(
 name="holding-accuracy",
 evaluation_goal="For each cited case, does the case actually stand for the proposition the OUTPUT claims? Quoted passages must match the source verbatim.",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"feature": "legal-research"}},
 scorers=[citation_scorer.id],
 judges=[holding_judge.id],
 schedule="per-session",
)
```

```typescript
// TypeScript (REST) — same trace eval from a Node service
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { tags: { feature: "legal-research" } },
 scorers: [citationScorerId],
 judges: [holdingJudgeId],
 schedule: "per-session",
 }),
});
```

## What you get

* Hallucination rate held at 0% on filed-output paths — the gate is a hard block, not a warning.
* Holding-accuracy judge agreement with senior associates above 90% after GEPA optimization.
* Documented evaluation evidence — the evaluation IDs and per-citation verification artifacts — to show partners, clients, and (if needed) the court that AI-assisted research operates under verification controls.
* Attorneys treat the AI as a research-acceleration tool, not a citation source. Which is what the AI should be.

## Stratix capabilities used

* [Custom code grader](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/custom-code-scorer.md) — citation-existence verification against legal databases
* [Judges with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — holding accuracy and quote accuracy
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — every production session
* [Notifications](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/notifications.md) — flag unverified citations for attorney review

## Replicate this

* [Industry → Legal](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/legal/README.md)
* [Cookbook: catch hallucinations](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/catch-hallucinations.md)
* [Cookbook: legal contract-review faithfulness](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/legal/cookbook/industry-legal-contract.md) (analogous pattern for transactional work)
* [Use case: RAG evaluation](/4.1-general-use-cases/rag-evaluation)


# Insurance

Insurance AI evaluation — claims triage, policy Q\&A, underwriting assistance.

## Top scenarios

* **Claims triage** — initial classification, escalation routing
* **Policy Q\&A** — customer-facing policy questions
* **Underwriting assistance** — augment underwriter decisions

## Why Stratix

Hard escalation rules + faithfulness judges keep insurance AI within risk-appropriate boundaries.

## Industry pattern

→ [Insurance: claims-triage agent pattern](/4.2-industry-use-cases/pattern-3)

[Scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/insurance/scenarios.md) | [Eval patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/insurance/eval-patterns.md) | [Compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/insurance/compliance.md) | [Cookbook](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/insurance/cookbook-and-samples.md)


# Pattern: claims-triage agent

Insurance — claims-triage agent pattern. Span-level escalation rule, assertions, GEPA-tuned tone judge, in one agentic evaluation.

A property-and-casualty claims-triage agent extracts loss details from photos and submitted forms, classifies severity, estimates repair cost, and decides whether to auto-approve, request more information, or escalate to a human adjuster. Multi-step, tool-using, customer-facing on the empathy side.

This pattern shows how to evaluate the agent end-to-end with one agentic evaluation that mixes hard policy rules with subjective tone scoring.

## What's at stake

| Risk dimension                                  | Magnitude                        | Framework                           |
| ----------------------------------------------- | -------------------------------- | ----------------------------------- |
| Annual industry insurance fraud (US, all lines) | \~$80B                           | Coalition Against Insurance Fraud   |
| Per-catastrophe-event mis-estimation cost       | $50K–$500K                       | Industry loss-adjustment benchmarks |
| State DOI market-conduct examination            | Per-finding penalty bands        | NAIC market-conduct framework       |
| Class-action exposure on systemic underpayment  | Multi-million-dollar settlements | Public class-action filings         |

## The evaluation pattern

**Pre- and post-deployment agentic evaluation** over a 400-trace curated set spanning happy paths, ambiguous documentation, and known-edge fraud patterns.

**Criteria mix:**

1. **Deterministic span-level rule** — *"if `inputs.claim_amount > $X` then the trace must contain a span with `kind=tool, tool=escalate_to_adjuster`"*. CRITICAL severity on miss. The amount threshold is configurable per line of business.
2. **Natural-language assertions** — the agent's final state matches the policy auto-approval rules; every required document is present in the agent's verification span.
3. **Faithfulness judge** (GEPA-tuned against ≥50 examples — scored output) — claims about coverage and limits are grounded in the policy document.
4. **Tone judge** (GEPA-tuned against ≥50 examples — scored output) — for customer-facing outputs, balance professionalism with appropriate empathy in disruption scenarios.

> Don't have labels yet? See [Bootstrap a judge before GEPA](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/bootstrap-judges.md) for the week-1 setup.

**Verdict + root-cause artifact:** every failed criterion ties back to the specific trace, span, and decision that broke. The regression report compares the run to the most recent baseline; newly-failing criteria surface immediately.

**Post-deployment, continuous:** trace evaluation on a 5% sample of production claims, daily cadence. Same criteria configuration. Threshold alerts route to the claims-ops Slack channel.

## Configuration in code

```python
# Python (SDK) — agentic evaluation with span-level escalation rule
from layerlens import Stratix
client = Stratix()

escalation_rule = {
 "id": "high-value-must-escalate",
 "expression": "if input.claim_amount > 100000 then any(span.kind == 'tool' and span.tool == 'escalate_to_adjuster' for span in trace.spans)",
 "severity": "CRITICAL",
}

faithfulness = client.judges.create(
 name="policy-faithfulness",
 evaluation_goal="Coverage and limit claims must be grounded in the cited policy document.",
)

tone = client.judges.create(
 name="empathy-tone",
 evaluation_goal="In disruption scenarios, the response balances professionalism with appropriate empathy. Penalize tone-deaf or dismissive language.",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"id": "claims-curated-400"},
 rules=[escalation_rule],
 judges=[faithfulness.id, tone.id],
)
result = client.trace_evaluations.wait_for_completion(trace_eval.id)
assert result.critical_failures == 0, "Block release: escalation rule violated"
```

```typescript
// TypeScript (REST)
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { id: "claims-curated-400" },
 rules: [escalationRule],
 judges: [faithfulnessId, toneId],
 }),
});
const traceEval = await r.json();
```

## What you get

* Zero CRITICAL escalation-rule violations across the curated trace set before launch.
* Continuous trace evaluation catches drift when policy rules or product mix shifts.
* Verdict + root-cause artifacts become the auditor-ready evidence packet for state DOI inquiries — every release pin cites the evaluation IDs that gated it.
* Tone-judge agreement with human reviewers above 90% after GEPA optimization, calibrated to your team's customer-experience standard.

## Stratix capabilities used

* [Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation) — three criteria types in one configuration
* [Span-level scoring](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/span-level-scoring.md) — escalation rule pinned to tool spans
* [Judges with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — faithfulness and tone
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — pre-deploy and continuous
* [Notifications](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/notifications.md) — Slack routing
* [Audit-friendly evidence](/11.-admin/audit-friendly-evidence)

## Replicate this

* [Industry → Insurance](/4.2-industry-use-cases/insurance)
* [Cookbook: detect tool-call regressions in agents](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/agent-tool-regressions.md)
* [Cookbook: insurance claims-triage scoring](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/insurance/cookbook/industry-insurance-claims.md)
* [Workflow: Evaluate](/9.-improve-tune-the-system/workflow)
* [Concept: Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation)


# Government and public sector

Government and public sector AI evaluation — citizen services, document processing, eligibility.

## Top scenarios

* **Citizen services chatbot** — benefits, services, status questions
* **Document processing** — forms, applications, identity documents
* **Eligibility determination assist** — augment caseworker decisions

## Why Stratix

Auditable evaluation evidence per release; deterministic eligibility rules; faithful citation of regulations.

## Industry pattern

→ [Government: citizen-services chatbot pattern](/4.2-industry-use-cases/pattern-4)

[Scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/government-public-sector/scenarios.md) | [Eval patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/government-public-sector/eval-patterns.md) | [Compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/government-public-sector/compliance.md) | [Cookbook](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/government-public-sector/cookbook-and-samples.md)


# Pattern: citizen-services chatbot

Government / public sector — citizen-services chatbot pattern. Policy accuracy against versioned source-of-truth, plain-language reading-level, multilingual parity.

A state or federal agency operates an AI chatbot that helps citizens find services, check eligibility, and complete applications (benefits, permits, tax forms, licenses). The agency's source-of-truth policy documents update on their own cadence — and a stale chatbot answer can deny benefits citizens are entitled to.

This pattern shows how to evaluate citizen-services chatbots for policy accuracy, accessibility, and equity.

## What's at stake

| Risk dimension                                       | Magnitude                               | Framework                                 |
| ---------------------------------------------------- | --------------------------------------- | ----------------------------------------- |
| Eligibility-misstatement civil-rights exposure       | Class-action and Title VI claims        | Title VI / ADA / state agency rules       |
| Section 508 accessibility violations                 | Per-violation penalties                 | Section 508 (federal) / state equivalents |
| Public-trust impact from incorrect official guidance | Long-tail agency-credibility damage     | Public agency-performance research        |
| Congressional / legislative inquiry                  | Hearing time, executive-branch response | Public oversight records                  |

## The evaluation pattern

A **policy-grounded evaluation** runs against a versioned source-of-truth.

1. **Custom code grader (policy-document-hash check)** — every chatbot answer references the active policy document version; if the active document hash differs from the answer's referenced version, the answer is stale = fail.
2. **Faithfulness judge** (GEPA-tuned against ≥50 caseworker-labeled examples — scored output) — claims about eligibility and procedures are grounded in the cited policy section.
3. **Reading-level scorer** (Flesch-Kincaid or grade-level equivalent) — patient-facing answers must be at or below the configured grade level (commonly 6th-8th grade for citizen-services).
4. **Multilingual parity scorer** (custom code) — per-language accuracy must be within 5 percentage points of the primary language baseline.
5. **Disclaimer-presence scorer** — the chatbot must include the agency's official "this is informational, not a determination" disclaimer where required.

> Don't have labels yet? See [Bootstrap a judge before GEPA](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/bootstrap-judges.md) for the week-1 setup.

**Continuous trace evaluation:** sampled hourly during business hours. Policy-document-hash scorer runs on every trace (cheap and deterministic). Threshold alerts route to the agency's program managers.

## Configuration in code

```python
# Python (SDK)
from layerlens import Stratix
client = Stratix()

policy_hash = client.scorers.create_code(
 name="policy-document-hash",
 code="""
cited = output.get('policy_version_hash')
active = policy_registry.active_hash(output['program'])
result = {'passed': cited == active, 'cited': cited, 'active': active}
""",
)

reading_level = client.scorers.create_code(
 name="reading-level",
 code="result = {'passed': flesch_kincaid_grade(output['text']) <= 8}",
)

faithfulness = client.judges.create(
 name="policy-faithfulness",
 evaluation_goal="Eligibility and procedure claims must be grounded in the cited policy section.",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"feature": "citizen-chatbot"}},
 scorers=[policy_hash.id, reading_level.id],
 judges=[faithfulness.id],
 schedule="hourly",
)
```

```typescript
// TypeScript (REST)
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { tags: { feature: "citizen-chatbot" } },
 scorers: [policyHashId, readingLevelId],
 judges: [faithfulnessId],
 schedule: "hourly",
 }),
});
```

## What you get

* Stale-policy answers detected within hours, not weeks.
* Multilingual accuracy parity is measured per language, not assumed.
* Auditor-ready evaluation history for civil-rights and accessibility audits.
* Pre-publication block prevents citizen-facing release on a regression.

## Stratix capabilities used

* [Custom code graders](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/custom-code-scorer.md) — document-hash, multilingual parity, reading-level
* [Judges with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — policy faithfulness
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled
* [Notifications](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/notifications.md) — program-manager routing

## Replicate this

**Get started:** [Cookbook: catch hallucinations](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/catch-hallucinations.md) is the closest runnable starter (policy faithfulness shape).

* [Industry → Government and public sector](/4.2-industry-use-cases/government-public-sector)
* [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation)
* [Workflow: Govern](/9.-improve-tune-the-system/workflow)


# Retail and e-commerce

Retail and e-commerce AI evaluation — product Q\&A, customer support, recommendations.

## Top scenarios

* **Product Q\&A** — chat answers questions from the catalog
* **Support automation** — handle order, return, shipping queries
* **Recommendations explanation** — explain why a product was recommended

## Why Stratix

Catalog-faithfulness rules catch hallucinated product attributes. Latency scorers gate user-facing performance.

## Industry pattern

→ [Retail: product description generation pattern](/4.2-industry-use-cases/pattern-5)

[Scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/retail-ecommerce/scenarios.md) | [Eval patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/retail-ecommerce/eval-patterns.md) | [Compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/retail-ecommerce/compliance.md) | [Cookbook](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/retail-ecommerce/cookbook-and-samples.md)


# Pattern: product description generation

Retail / e-commerce — product-description generation pattern. Specification accuracy, brand voice, return-rate impact.

A retail or marketplace platform uses AI to generate product titles, descriptions, and SEO copy from a structured catalog (specs, dimensions, compatibility, brand attributes). At scale (hundreds of thousands to millions of SKUs), AI replaces a content team — but description errors directly drive returns, regulatory exposure, and brand erosion.

This pattern shows how to evaluate generated descriptions for specification accuracy and brand voice before they ship.

## What's at stake

| Risk dimension                                         | Magnitude                                                    | Framework                                    |
| ------------------------------------------------------ | ------------------------------------------------------------ | -------------------------------------------- |
| Lift in return rate from inaccurate specs              | 15-30% return-rate increase documented                       | E-commerce returns research                  |
| FTC false-advertising exposure                         | Per-violation civil penalties + corrective-advertising costs | FTC Endorsement Guides + product claim rules |
| SEO penalties on duplicate / thin / inaccurate content | Ranking loss, organic traffic impact                         | Search-engine quality guidelines             |
| Brand-voice drift across categories                    | Cross-category brand erosion                                 | Internal brand-equity research               |

## The evaluation pattern

A **pre-publication evaluation** runs on every batch of AI-generated descriptions:

1. **Custom code grader (specification accuracy)** — extracts every claimed dimension, weight, compatibility, and material from the description; compares against the catalog row exactly. Any mismatch = fail.
2. **Substring scorer** — required disclaimers (e.g., regulated-category warnings, age restrictions) present where required.
3. **Brand-voice judge** (GEPA-tuned against ≥50 brand-marketing-team-labeled examples — scored output) — voice consistency across categories.
4. **Plagiarism / uniqueness scorer** — generated text must score above a similarity threshold against the existing catalog.

> Don't have labels yet? See [Bootstrap a judge before GEPA](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/bootstrap-judges.md) for the week-1 setup.

**Continuous trace evaluation:** sampled at 1% of newly published descriptions, daily. Threshold alerts on specification-accuracy regressions route to the catalog-ops team.

## Configuration in code

```python
# Python (SDK)
from layerlens import Stratix
client = Stratix()

spec_scorer = client.scorers.create_code(
 name="spec-accuracy",
 code="""
claimed = extract_specs(output)
truth = catalog.get(input['sku']).specs
result = {'passed': all(claimed.get(k) == v for k, v in truth.items()), 'mismatches': diff(claimed, truth)}
""",
)

brand_voice = client.judges.create(
 name="brand-voice",
 evaluation_goal="Score brand-voice consistency on a 1-5 scale. Voice should match the brand guidelines for category {category}.",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"feature": "product-description"}, "sample_rate": 0.01},
 scorers=[spec_scorer.id],
 judges=[brand_voice.id],
 schedule="daily",
)
```

```typescript
// TypeScript (REST)
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { tags: { feature: "product-description" }, sample_rate: 0.01 },
 scorers: [specScorerId],
 judges: [brandVoiceId],
 schedule: "daily",
 }),
});
```

## What you get

* Specification-error rate stays under 0.5% (vs. 4-15% baseline reported in industry research).
* Return-rate on AI-described products converges to or beats human-written baselines.
* Pre-publication block is a hard gate; descriptions that fail any code grader never reach the storefront.
* Brand voice variance across categories is a measured dimension on the dashboard.

## Stratix capabilities used

* [Custom code graders](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/custom-code-scorer.md) — specification-extraction comparison
* [Judges with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — brand voice
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled
* [Notifications](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/notifications.md)

## Replicate this

**Get started:** [Cookbook: retail product Q\&A](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/retail-ecommerce/cookbook/industry-retail-product-qa.md) is the closest runnable starter.

* [Industry → Retail and e-commerce](/4.2-industry-use-cases/retail-ecommerce)
* [Workflow: Evaluate](/9.-improve-tune-the-system/workflow)
* [Use case: Continuous evaluation](/4.1-general-use-cases/continuous-evaluation)


# Media and entertainment

Media and entertainment AI evaluation — moderation, summarization, search.

## Top scenarios

* **Content moderation** — flag policy violations
* **Summarization** — articles, videos, podcasts
* **Search and discovery** — semantic search across catalog

## Why Stratix

Faithfulness judges catch hallucinated content claims; deterministic policy rules enforce moderation policies.

## Industry pattern

→ [Media: content moderation pattern](/4.2-industry-use-cases/pattern-6)

[Scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/media-entertainment/scenarios.md) | [Eval patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/media-entertainment/eval-patterns.md) | [Compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/media-entertainment/compliance.md) | [Cookbook](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/media-entertainment/cookbook-and-samples.md)


# Pattern: content moderation

Media / entertainment — content moderation pattern. Per-category recall, cross-language parity, EU DSA exposure management.

A social platform, marketplace with user-generated content, or media-streaming service uses AI to moderate user content — text, image, video, audio — for policy violations across hate speech, harassment, CSAM, violence, misinformation, and more. The cost of under-moderation includes regulatory enforcement (EU DSA, OSA, state-level US laws); the cost of over-moderation is creator churn and First-Amendment-style litigation.

This pattern shows how to evaluate cross-language, per-category moderation accuracy.

## What's at stake

| Risk dimension                         | Magnitude                          | Framework                        |
| -------------------------------------- | ---------------------------------- | -------------------------------- |
| EU Digital Services Act non-compliance | Up to 6% of global annual revenue  | EU DSA                           |
| UK Online Safety Act non-compliance    | Up to 10% of global annual revenue | OSA                              |
| Advertiser-boycott revenue impact      | Multi-quarter advertiser pauses    | Industry brand-safety reporting  |
| Over-moderation creator-churn          | Long-tail platform-health impact   | Public creator-platform research |

## The evaluation pattern

A **multi-criteria evaluation** runs over a labeled trace set spanning every supported language and policy category.

1. **Per-category classifier scorer** — recall by policy category (hate speech, CSAM, violence, harassment, misinformation, sexual content, self-harm, regulated goods). Recall must meet per-category minimums; CSAM and self-harm categories require near-100%.
2. **Cross-language parity scorer** (custom code) — for each policy category, the per-language accuracy must be within 5 percentage points of the English baseline. Disparities above the threshold flag as regressions.
3. **Borderline judge** (GEPA-tuned against ≥50 trust-and-safety-team-labeled examples — scored output) — handles satire, news commentary, cultural context, reclaimed slurs. Higher false-positive cost than false-negative.
4. **Latency scorer** — moderation decisions must complete under per-tier SLOs (e.g., 200ms p95 for live-stream, 5s p95 for upload).

> Don't have labels yet? See [Bootstrap a judge before GEPA](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/bootstrap-judges.md) for the week-1 setup.

**Continuous trace evaluation:** sampled at 0.5% of moderation decisions, hourly. Per-language and per-category dashboards visible to T\&S leadership; thresholds wired to PagerDuty for critical-category drift.

## Configuration in code

```python
# Python (SDK)
from layerlens import Stratix
client = Stratix()

cross_language = client.scorers.create_code(
 name="cross-language-parity",
 code="""
en_acc = mean(s for t, s in zip(traces, scores) if t.tags['lang'] == 'en')
gaps = {l: en_acc - mean(s for t, s in zip(traces, scores) if t.tags['lang'] == l) for l in supported_langs}
result = {'passed': max(abs(g) for g in gaps.values()) <= 0.05, 'gaps': gaps}
""",
)

borderline = client.judges.create(
 name="borderline-content",
 evaluation_goal="Distinguish policy-violating content from satire, news commentary, cultural context, and reclaimed slurs. Score 1 (clearly fine) to 5 (clearly violating).",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"feature": "moderation"}, "sample_rate": 0.005},
 scorers=[cross_language.id],
 judges=[borderline.id],
 schedule="hourly",
)
```

```typescript
// TypeScript (REST)
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { tags: { feature: "moderation" }, sample_rate: 0.005 },
 scorers: [crossLanguageId],
 judges: [borderlineId],
 schedule: "hourly",
 }),
});
```

## What you get

* Cross-language parity becomes a measured dimension on the T\&S dashboard.
* Per-category recall floors enforced — CSAM and self-harm regressions block release.
* Auditor-readable evaluation history for EU DSA / OSA risk-assessment requirements.
* Appeal-rate signal correlates with judge false-positive trend; T\&S can investigate appeals upstream.

## Stratix capabilities used

* [Custom code graders](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/custom-code-scorer.md) — cross-language parity
* [Judges with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — borderline-content scoring
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled
* [Notifications](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/notifications.md) — PagerDuty + Slack routing

## Replicate this

**Get started:** [Use case: Continuous evaluation](/4.1-general-use-cases/continuous-evaluation) describes the rolling-sample shape this pattern uses.

* [Industry → Media and entertainment](/4.2-industry-use-cases/media-entertainment)
* [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation)
* [Cookbook: catch hallucinations](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/catch-hallucinations.md) (analogous pattern for misinformation moderation)


# Manufacturing

Manufacturing AI evaluation — diagnostics assistance, manuals Q\&A, quality.

## Top scenarios

* **Diagnostics assistance** — field-tech troubleshooting with manuals + sensor data
* **Manuals Q\&A** — extract procedures from technical documents
* **Quality control** — analyze defect images and reports

## Why Stratix

Faithfulness judges against manuals; deterministic procedure-correctness rules; trace evaluations on field-tech sessions.

## Industry pattern

→ [Manufacturing: visual quality-inspection pattern](/4.2-industry-use-cases/pattern-7)

[Scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/manufacturing/scenarios.md) | [Eval patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/manufacturing/eval-patterns.md) | [Compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/manufacturing/compliance.md) | [Cookbook](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/manufacturing/cookbook-and-samples.md)


# Pattern: visual quality inspection

Manufacturing / industrial — visual quality-inspection pattern. Per-defect-class accuracy, lighting-robustness, latency under SLO.

A manufacturer deploys AI computer vision on assembly lines to detect product defects (scratches, dents, misalignment, color variation, contamination). The model runs at line speed, triggering quality holds when defects are detected. False negatives reach customers as defective product (recall risk). False positives stop the line ($10-50K/hour of lost throughput).

This pattern shows how to evaluate inspection accuracy across the conditions actually present on the factory floor.

## What's at stake

| Risk dimension                               | Magnitude                                      | Framework                           |
| -------------------------------------------- | ---------------------------------------------- | ----------------------------------- |
| Per-recall cost                              | $500K–$5M (industry-dependent)                 | Industry recall-cost benchmarks     |
| Per-hour line-stop cost from false positives | $10K–$50K                                      | Manufacturing throughput economics  |
| Safety-incident exposure for missed defects  | Potential workers'-comp + product-liability    | OSHA + product-liability frameworks |
| ISO 9001 / IATF 16949 audit findings         | Conditional certification, customer audit risk | ISO / IATF audit standards          |

## The evaluation pattern

A **stratified evaluation** treats lighting condition, line speed, and operator shift as first-class dimensions.

1. **Per-defect-class scorer** — precision and recall by defect category. Trace tags include `defect_type`, `lighting_condition`, `line_speed_bucket`, and `shift`; results break out per-tag.
2. **Custom code grader (condition disparity)** — per-class accuracy across `lighting_condition` and `line_speed_bucket` must stay within 5 percentage points of the baseline. Disparities flag as regressions.
3. **Latency scorer** — inspection decisions must complete under the line-cycle-time SLO (commonly 100-500ms p99).
4. **Compare-models** — when a new model variant is proposed, it must close any condition-disparity without degrading per-class accuracy.

**Continuous trace evaluation:** sampled at 1% of inspection decisions, hourly. Threshold alerts route to plant operations and the quality-engineering team.

## Configuration in code

```python
# Python (SDK)
from layerlens import Stratix
client = Stratix()

condition_disparity = client.scorers.create_code(
 name="lighting-line-speed-disparity",
 code="""
buckets = group_by(traces, ['lighting_condition', 'line_speed_bucket'])
acc = {b: per_class_recall(buckets[b]) for b in buckets}
gap = max(acc.values()) - min(acc.values())
result = {'passed': gap <= 0.05, 'per_bucket_recall': acc}
""",
)

latency = client.scorers.create_code(
 name="line-cycle-latency",
 code="result = {'passed': trace.duration_ms <= 500}",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"feature": "quality-inspection"}, "sample_rate": 0.01},
 scorers=[condition_disparity.id, latency.id, defect_classifier_id],
 schedule="hourly",
)
```

```typescript
// TypeScript (REST)
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { tags: { feature: "quality-inspection" }, sample_rate: 0.01 },
 scorers: [conditionDisparityId, latencyId, defectClassifierId],
 schedule: "hourly",
 }),
});
```

## What you get

* Lighting and line-speed disparities measured, not absorbed into an aggregate metric.
* Pre- and post-deployment block on a model variant that worsens any condition.
* Per-shift accuracy visibility supports root-cause analysis when defect-escape rates rise.
* Auditor-ready evidence for ISO 9001 / IATF 16949 reviews.

## Stratix capabilities used

* [Custom code graders](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/custom-code-scorer.md) — condition-disparity computations
* [Compare models](/5.-select-pick-the-model/compare-models) — variant selection
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled
* [Notifications](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/notifications.md) — plant-ops routing

## Replicate this

**Get started:** [Concept: Traces and spans](/6.-build-wire-your-code/traces-and-spans) — capturing inspection-line traces with the right tags is the foundation for the stratified evaluation here.

* [Industry → Manufacturing](/4.2-industry-use-cases/manufacturing)
* [Workflow: Evaluate](/9.-improve-tune-the-system/workflow)
* [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation)


# Education

Education AI evaluation — tutoring, content generation, assessment.

## Top scenarios

* **AI tutor** — Socratic guidance, hints, adaptive practice
* **Content generation** — exercises, quizzes, lessons
* **Assessment / grading assistance**

## Why Stratix

Judges that distinguish "guides student" from "gives answer" — the core failure mode of educational AI.

## Industry pattern

→ [Education: AI tutor pattern](/4.2-industry-use-cases/pattern-8)

[Scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/education/scenarios.md) | [Eval patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/education/eval-patterns.md) | [Compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/education/compliance.md) | [Cookbook](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/education/cookbook-and-samples.md)


# Pattern: AI tutor

Education / EdTech — AI tutor pattern. Pedagogical-quality (Socratic vs. answer-giving), age-appropriateness, learning-outcome correlation.

An EdTech provider operates an AI tutor that helps students through math, reading, and science. The product's value comes from **teaching**, not answering — a tutor that gives away the answer accelerates the assignment but undermines the learning. COPPA and FERPA frame the regulatory bar; the pedagogical dimension is what differentiates the product.

This pattern shows how to evaluate tutor pedagogical quality alongside accuracy and safety.

## What's at stake

| Risk dimension                                              | Magnitude                                            | Framework                         |
| ----------------------------------------------------------- | ---------------------------------------------------- | --------------------------------- |
| Learning-outcome harm from answer-giving instead of guiding | Long-term student-outcome impact                     | Education research                |
| COPPA violation (under-13 users)                            | Per-violation civil penalties                        | COPPA                             |
| FERPA violation (student records in trace bodies)           | Loss of federal funding eligibility, civil penalties | FERPA                             |
| Age-inappropriate content exposure                          | District contract termination, brand damage          | District procurement requirements |
| Accessibility (Section 508 / WCAG)                          | Procurement disqualification                         | Section 508 / WCAG 2.1 AA         |

## The evaluation pattern

A **multi-criteria evaluation** runs against curated tutoring-conversation traces.

1. **Pedagogical-quality judge** (GEPA-tuned against ≥50 educator-labeled examples — scored output, "guide vs. give") — penalize responses that hand the student the answer when the assignment expects the student to derive it.
2. **Subject-accuracy scorer** — math and science answers verified against a deterministic answer key; language and reading answers checked against rubric criteria.
3. **Age-appropriateness judge** (GEPA-tuned against ≥50 examples — scored output) — content matches the age band declared in the user's profile.
4. **PII-redaction scorer** (deterministic regex) — no student-record fields appear in any logged span (FERPA).
5. **Reading-level scorer** — explanations at or below the configured grade level for the student's age band.

> Don't have labels yet? See [Bootstrap a judge before GEPA](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/bootstrap-judges.md) for the week-1 setup.

**Continuous trace evaluation:** sampled at 0.5% of tutoring sessions, daily. Pedagogical-quality trends visible to the curriculum-design team.

## Configuration in code

```python
# Python (SDK)
from layerlens import Stratix
client = Stratix()

pedagogical = client.judges.create(
 name="socratic-vs-give",
 evaluation_goal="Score 1-5: did the tutor guide the student to the answer (5) or hand it over (1)? Penalize answers that bypass the learning step the assignment expects.",
)

age_appropriate = client.judges.create(
 name="age-appropriate",
 evaluation_goal="Content matches the student's declared age band (input.age_band).",
)

pii_redaction = client.scorers.create_code(
 name="ferpa-pii-redaction",
 code="result = {'passed': not contains_student_record_fields(trace)}",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"feature": "tutor"}, "sample_rate": 0.005},
 scorers=[pii_redaction.id, subject_answer_key_id],
 judges=[pedagogical.id, age_appropriate.id],
 schedule="daily",
)
```

```typescript
// TypeScript (REST)
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { tags: { feature: "tutor" }, sample_rate: 0.005 },
 scorers: [piiRedactionId, subjectAnswerKeyId],
 judges: [pedagogicalId, ageAppropriateId],
 schedule: "daily",
 }),
});
```

## What you get

* Direct-answer rate measured per subject and grade band; product-led iteration drives it down over time.
* Age-appropriateness regressions blocked at the per-student session boundary.
* FERPA-compliant trace handling enforced by code graders.
* Auditor-ready evidence for district procurement and state-board reviews.

## Stratix capabilities used

* [Judges with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — pedagogical-quality and age-appropriateness
* [Custom code graders](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/custom-code-scorer.md) — subject-accuracy answer-key check, PII-redaction regex
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled

## Replicate this

**Get started:** [Cookbook: education tutor helpfulness](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/education/cookbook/industry-education-tutor.md) is the runnable starter.

* [Industry → Education](/4.2-industry-use-cases/education)
* [Concept: Judges](/8.-evaluate-score-the-outputs/judges-1)
* [Workflow: Evaluate](/9.-improve-tune-the-system/workflow)


# Real estate

Real estate AI evaluation — listing Q\&A, valuation assist, document search.

## Top scenarios

* **Listing Q\&A** — buyer/renter questions about properties
* **Valuation assistance** — comparative market analysis
* **Document search** — disclosure and contract analysis

## Why Stratix

Faithfulness against listing data; rules for fair-housing compliance language.

## Industry pattern

→ [Real estate: automated property valuation pattern](/4.2-industry-use-cases/pattern-9)

[Scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/real-estate/scenarios.md) | [Eval patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/real-estate/eval-patterns.md) | [Compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/real-estate/compliance.md) | [Cookbook](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/real-estate/cookbook-and-samples.md)


# Pattern: automated property valuation

Real estate / proptech — automated valuation pattern. Per-segment accuracy, fair-housing parity, image-condition assessment.

A proptech company or lender uses AI to generate property valuations from comparable sales, property features, listing photos, and market data. Banking partners require fair-housing-compliant valuation accuracy; replicating historical appraisal bias is a non-starter.

This pattern shows how to evaluate AVM accuracy and fair-housing parity together.

## What's at stake

| Risk dimension                                                  | Magnitude                                          | Framework                              |
| --------------------------------------------------------------- | -------------------------------------------------- | -------------------------------------- |
| Fair-housing exposure on neighborhood-level valuation disparity | Per-violation civil penalties; consent-decree cost | Fair Housing Act / HUD enforcement     |
| Banking-partner rejection of an AVM with documented bias        | Loss of integration agreements, revenue impact     | OCC / FRB safety-and-soundness reviews |
| Bad-loan losses from inaccurate valuations                      | $50K–$500K per loan in the worst-tail outcome      | Industry mortgage-loss benchmarks      |
| Brookings-documented appraisal-undervaluation pattern           | $48K average undervaluation in Black neighborhoods | Brookings 2018 / HUD studies           |

## The evaluation pattern

A **fairness-aware evaluation** runs the AVM against a labeled dataset spanning diverse neighborhoods, price ranges, and property types.

1. **Numeric scorer (valuation accuracy)** — AVM estimate within ±5% of appraiser ground truth.
2. **Custom code grader (geographic disparity)** — per-Census-tract or per-MSA disparity in median accuracy. Disparity above the configured tolerance (commonly 1.25× ratio) flags as a regression.
3. **Demographic-correlation scorer** — accuracy disparity must not correlate with protected-class proxies (race demographics, income tier).
4. **Image-condition assessment scorer** — for AVMs that incorporate listing photos, condition-extraction accuracy from images is scored separately.
5. **Comparable-selection rationale judge** (GEPA-tuned against ≥50 appraiser-labeled examples — scored output) — the AI's choice of comparables is defensible against an appraiser's choice.

> Don't have labels yet? See [Bootstrap a judge before GEPA](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/bootstrap-judges.md) for the week-1 setup.

**Continuous trace evaluation:** sampled at 1% of production valuations, daily. Disparity trends visible to compliance and risk committees.

## Configuration in code

```python
# Python (SDK)
from layerlens import Stratix
client = Stratix()

geo_disparity = client.scorers.create_code(
 name="geographic-disparity",
 code="""
by_tract = group_by_census_tract(traces, scores)
ratio = max_median(by_tract) / min_median(by_tract)
result = {'passed': ratio <= 1.25, 'disparity_ratio': ratio}
""",
)

valuation_accuracy = client.scorers.create_code(
 name="valuation-accuracy",
 code="result = {'passed': abs(output['estimate'] - expected['appraiser_value']) / expected['appraiser_value'] <= 0.05}",
)

comparable_judge = client.judges.create(
 name="comparable-selection",
 evaluation_goal="Score 1-5: is the AI's choice of comparable sales defensible against an appraiser's selection?",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"feature": "avm"}, "sample_rate": 0.01},
 scorers=[geo_disparity.id, valuation_accuracy.id],
 judges=[comparable_judge.id],
 schedule="daily",
)
```

```typescript
// TypeScript (REST)
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { tags: { feature: "avm" }, sample_rate: 0.01 },
 scorers: [geoDisparityId, valuationAccuracyId],
 judges: [comparableJudgeId],
 schedule: "daily",
 }),
});
```

## What you get

* Fair-housing disparity becomes a measurement on the dashboard, not a discovery during a regulatory review.
* Banking-partner integration approvals come faster — the evaluation evidence packet shrinks the diligence timeline.
* Per-comp rationale auditability for borrower-facing valuation explanations.
* Pre- and post-deployment block on any model variant that worsens disparity.

## Stratix capabilities used

* [Custom code graders](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/custom-code-scorer.md) — geographic-disparity and demographic-correlation
* [Judges with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — comparable-selection rationale
* [Compare models](/5.-select-pick-the-model/compare-models) — variant selection with fair-housing criteria
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled

## Replicate this

**Get started:** [Workflow: Govern](/9.-improve-tune-the-system/workflow) describes the cross-team gate this pattern sits under.

* [Industry → Real estate](/4.2-industry-use-cases/real-estate)
* [Use case: AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd)
* [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation)


# Telecommunications

Telecommunications AI evaluation — customer service, troubleshooting, plan Q\&A.

## Top scenarios

* **Customer service automation** — billing, account, plan questions
* **Network troubleshooting** — guided diagnostics
* **Plan recommendation** — match customer needs to plans

## Why Stratix

Faithful plan/pricing claims; trace evaluations on multi-turn troubleshooting flows.

## Industry pattern

→ [Telecom: customer-service pricing accuracy pattern](/4.2-industry-use-cases/pattern-10)

[Scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/telecommunications/scenarios.md) | [Eval patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/telecommunications/eval-patterns.md) | [Compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/telecommunications/compliance.md) | [Cookbook](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/telecommunications/cookbook-and-samples.md)


# Pattern: customer-service pricing accuracy

Telecom — customer-service AI pricing-accuracy pattern. JSON-schema scorer, numeric exact-match, per-PR CI gating, continuous sampled trace eval.

A telecom AI customer-service agent handles billing questions, plan changes, and pricing inquiries. The agent reads the active rate card and applies promotional pricing rules. When it gets the math right, churn drops and customer-care hours fall. When it gets it wrong — even occasionally — the bill is wrong and revenue leaks.

This pattern shows how to evaluate pricing accuracy continuously, in CI and in production.

## What's at stake

| Risk dimension                                             | Magnitude                         | Framework                          |
| ---------------------------------------------------------- | --------------------------------- | ---------------------------------- |
| Annual revenue leakage from underbilling at 6% error rate  | \~$2M for a mid-market carrier    | Internal billing-reconciliation    |
| FCC complaint exposure from overcharging                   | Per-violation civil-penalty bands | FCC enforcement bureau guidance    |
| Per-subscriber churn cost from a billing-dispute departure | $300–$600                         | Telecom retention benchmarks       |
| Audit / SOX exposure on unreviewed pricing-engine outputs  | Material-weakness risk            | Public-company controls frameworks |

## The evaluation pattern

**Pre- and post-deployment, in CI:** a fast (≤5 min) regression evaluation runs on every PR touching the customer-service prompt or pricing logic.

1. **JSON-schema scorer** — every pricing change in the agent's output must include a `rate_card_row_id` and a citation timestamp. Missing field = fail.
2. **Numeric scorer** — extracted prices match the rate-card row exactly (no rounding, no unit substitution).
3. **Compare-models** view — when proposing a prompt variant, compare the variant against the current main-branch baseline on the same regression set. Pick the variant with the lowest error rate.
4. **Tolerance gate** — fail the build if pricing-error rate exceeds 0.5%.

**Post-deployment, continuous:** trace evaluation on a 5% sample of live customer interactions, hourly cadence.

* Same JSON-schema and numeric scorers
* Threshold alert routed to a Slack channel below 99.5% pass rate
* Per-trace alert on any CRITICAL error (extracted price differs from rate card by >$5)

## Configuration in code

```python
# Python (SDK) — CI-side gating evaluation
from layerlens import Stratix
client = Stratix()

schema_scorer = client.scorers.create(
 name="pricing-citation",
 type="json_schema_validate",
 config={"schema": {
 "type": "object",
 "required": ["rate_card_row_id", "citation_ts"],
 "properties": {
 "rate_card_row_id": {"type": "string"},
 "citation_ts": {"type": "string", "format": "date-time"},
 },
 }},
)

numeric_scorer = client.scorers.create_code(
 name="price-exact-match",
 code="result = abs(output['price'] - expected['rate_card_price']) < 0.01",
)

evaluation = client.evaluations.create(
 name="cs-pricing-gate",
 model_id=os.environ["MODEL_ID"],
 dataset_id="pricing-regression-v1",
 scorers=[schema_scorer.id, numeric_scorer.id],
)
result = client.evaluations.wait_for_completion(evaluation.id)
sys.exit(0 if result.pass_rate >= 0.995 else 1)
```

```typescript
// TypeScript (REST) — same gate from a Node CI runner
const r = await fetch("https://stratix.layerlens.ai/api/v1/evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 name: "cs-pricing-gate",
 model_id: process.env.MODEL_ID,
 dataset_id: "pricing-regression-v1",
 scorers: [schemaScorerId, numericScorerId],
 }),
});
const evaluation = await r.json();
// Poll evaluation.id until status === "completed"
```

## What you get

* Pricing regressions caught at PR — never reach production.
* Continuous evaluation surfaces drift when the rate card updates without the prompt being refreshed.
* Revenue reconciliation moves from discovery surface to verification surface.
* Per-call request IDs in Stratix evaluation results pair with internal call logs for fast incident triage.

## Stratix capabilities used

* [SDK + CLI evaluations](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/evaluations.md) wired into [GitHub Actions](/6.-build-wire-your-code/integration-github-actions)
* [Compare models](/5.-select-pick-the-model/compare-models) — prompt-variant A/B
* [JSON-schema and numeric scorers](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/json-schema-validate.md)
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled
* [Notifications](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/notifications.md) — Slack routing on threshold

## Replicate this

**Get started:** [Tutorial 3: Wire CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates) walks the end-to-end build (\~30 min).

* [Use case: AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd)
* [Industry → Telecommunications](/4.2-industry-use-cases/telecommunications)
* [Workflow: Govern](/9.-improve-tune-the-system/workflow)
* [Cookbook: cost-aware evaluation](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/cost-aware-evaluation.md)


# Energy and utilities

Energy and utilities AI evaluation — field service, ops assistance, customer support.

## Top scenarios

* **Field service** — technician guidance with manuals + telemetry
* **Ops assistance** — control-room support
* **Customer support** — outage, billing, conservation

## Why Stratix

Hard safety rules + faithful procedure citations; trace evaluations on multi-step incident flows.

## Industry pattern

→ [Energy: vegetation-management imagery pattern](/4.2-industry-use-cases/pattern-11)

[Scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/energy-utilities/scenarios.md) | [Eval patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/energy-utilities/eval-patterns.md) | [Compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/energy-utilities/compliance.md) | [Cookbook](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/energy-utilities/cookbook-and-samples.md)


# Pattern: vegetation-management imagery

Energy / utilities — vegetation-management imagery pattern. Stratified accuracy by season, variant comparison, daily continuous evaluation in fire season.

A regional electric utility operates an AI vegetation-management system that processes drone and helicopter imagery of power-line corridors. The model flags encroaching trees, structural damage, and clearance violations. Output drives field crews to inspect, trim, and remediate before a hazard turns into an outage or — in fire-prone regions — a wildfire ignition.

In summer, leafy canopy is easy to see against the lines. In winter — bare branches against bright sky, snow-occluded sightlines — model accuracy can drop by double digits. **Aggregate annual accuracy hides this**. Stratified evaluation surfaces it.

This pattern shows how to evaluate vegetation-management imagery models across seasonal and lighting conditions so blind spots are measured, not discovered.

## What's at stake

| Risk dimension                                          | Magnitude                                               | Framework                          |
| ------------------------------------------------------- | ------------------------------------------------------- | ---------------------------------- |
| Catastrophic-fire civil liability                       | $30B+ for the most-cited single utility liability event | Public CPUC and bankruptcy filings |
| NERC and state PUC enforcement on vegetation management | Per-finding civil penalty bands                         | Public NERC enforcement orders     |
| Per-hour outage cost from a downed line                 | $10K–$500K depending on customer mix                    | Industry reliability benchmarks    |
| Loss of life from wildfire ignition                     | Beyond financial accounting                             | Public wildfire-incident records   |

## The evaluation pattern

A **stratified evaluation** treats season, lighting condition, and equipment as first-class dimensions.

1. **Image-classification scorer** — per-class precision and recall on hazard categories (vegetation encroachment, equipment corrosion, clearance violation). Trace tags include `season`, `lighting_condition`, and `equipment_type`; results break out per-tag.
2. **Custom code grader (seasonal disparity)** — calculates hazard-detection-rate disparity between winter and summer trace sets; flags any disparity above a configured tolerance (e.g., 5%) as a regression.
3. **Compare models** — when a new model is proposed, it must close the seasonal disparity without losing summer performance. The comparison view shows side-by-side per-season scores.
4. **Continuous trace evaluation** — daily scoring on production drone-imagery samples. During high-risk seasons (fire season for fire-prone regions, winter ice for cold regions), cadence increases to hourly. Threshold alerts route to vegetation-management operations via Slack and email.

## Configuration in code

```python
# Python (SDK)
from layerlens import Stratix
client = Stratix()

seasonal_disparity = client.scorers.create_code(
 name="seasonal-disparity",
 code="""
winter_acc = mean(score for trace, score in zip(traces, scores) if trace.tags['season'] == 'winter')
summer_acc = mean(score for trace, score in zip(traces, scores) if trace.tags['season'] == 'summer')
result = {'passed': abs(winter_acc - summer_acc) <= 0.05, 'gap': summer_acc - winter_acc}
""",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"feature": "vegetation-management"}, "sample_rate": 0.01},
 scorers=[seasonal_disparity.id, hazard_classifier_scorer_id],
 schedule="daily",
)
```

```typescript
// TypeScript (REST)
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { tags: { feature: "vegetation-management" }, sample_rate: 0.01 },
 scorers: [seasonalDisparityId, hazardClassifierId],
 schedule: "daily",
 }),
});
```

## What you get

* Seasonal and lighting-condition disparities become measured dimensions on the dashboard, not undiscovered blind spots.
* Winter-specific training-data investment and evaluation-driven model improvements close the gap to within tolerance before the next high-risk season.
* Auditor-ready evaluation evidence for state PUC and NERC reviews — every release pin cites the evaluation IDs that gated it.
* Operations teams trust the model because the variance is explained, not absorbed into an opaque average.

## Stratix capabilities used

* [Custom code graders](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/custom-code-scorer.md) — seasonal-disparity computations
* [Compare models](/5.-select-pick-the-model/compare-models) — variant selection with seasonal-fairness criteria
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — daily continuous scoring during high-risk seasons
* [Notifications](https://github.com/LayerLens/gitbook-full/blob/main/13-reference/sdk-python/notifications.md) — Slack and email routing
* [Audit-friendly evidence guide](/11.-admin/audit-friendly-evidence)

## Replicate this

* [Industry → Energy and utilities](/4.2-industry-use-cases/energy-utilities)
* [Workflow: Govern](/9.-improve-tune-the-system/workflow)
* [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation)


# Travel and hospitality

Travel and hospitality AI evaluation — booking assist, itinerary, customer service.

## Top scenarios

* **Booking assistance** — search, options, reservations
* **Itinerary generation** — multi-leg travel planning
* **Customer service** — changes, cancellations, support

## Why Stratix

Faithfulness against real-time inventory; rules for accurate price/availability claims.

## Industry pattern

→ [Travel: multilingual booking-assistant pattern](/4.2-industry-use-cases/pattern-12)

[Scenarios](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/travel-hospitality/scenarios.md) | [Eval patterns](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/travel-hospitality/eval-patterns.md) | [Compliance](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/travel-hospitality/compliance.md) | [Cookbook](https://github.com/LayerLens/gitbook-full/blob/main/04-use-cases/industry/travel-hospitality/cookbook-and-samples.md)


# Pattern: multilingual booking assistant

Travel / hospitality — multilingual booking-assistant pattern. Per-language booking-detail accuracy, cultural appropriateness, locale-aware date handling.

A global hotel chain, airline, or OTA operates an AI booking assistant in 15-20+ languages. The assistant takes natural-language requests, parses dates and traveler preferences, and produces structured bookings. Date-format ambiguity (DD/MM vs. MM/DD), traveler-name transliteration, currency handling, and cultural appropriateness all vary by locale — and a wrong-day check-in is a measurable revenue and brand event.

This pattern shows how to evaluate cross-language booking accuracy.

## What's at stake

| Risk dimension                                                        | Magnitude                                              | Framework                              |
| --------------------------------------------------------------------- | ------------------------------------------------------ | -------------------------------------- |
| Per-incident wrong-booking cost                                       | $50–$500 (cancellation + rebooking + service-recovery) | Hospitality industry benchmarks        |
| Brand impact from cultural-appropriateness failures                   | Localized review-platform damage                       | Public review / OTA platform reporting |
| Accessibility (Section 508 / WCAG) for travelers using assistive tech | Procurement disqualification (gov / corp travel)       | Section 508 / WCAG                     |
| Currency / payment compliance failures                                | PCI exposure if mishandled                             | PCI DSS                                |

## The evaluation pattern

A **per-language evaluation** runs against booking-conversation traces in every supported language.

1. **Booking-detail accuracy scorer (custom code)** — extracted dates, names, traveler counts, room/cabin types, dietary preferences, and accessibility needs match the labeled ground truth exactly. Date-format parsing scored as a separate dimension.
2. **Cross-language parity scorer** — per-language booking-detail accuracy stays within 5 percentage points of the primary-language baseline.
3. **Cultural-appropriateness judge** (GEPA-tuned against ≥50 native-speaker-reviewer-labeled examples per language family — scored output) — tone and cultural fit per locale.
4. **Currency / payment scorer** — currency conversions cited explicitly with timestamp; PCI fields never appear in trace bodies.
5. **Tone-consistency judge** — across multilingual conversation turns, tone and persona remain stable.

> Don't have labels yet? See [Bootstrap a judge before GEPA](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/guides/bootstrap-judges.md) for the week-1 setup.

**Continuous trace evaluation:** sampled at 1% of bookings, hourly during peak season. Per-language dashboards visible to localization and revenue-management teams.

## Configuration in code

```python
# Python (SDK)
from layerlens import Stratix
client = Stratix()

booking_accuracy = client.scorers.create_code(
 name="booking-detail-accuracy",
 code="""
extracted = output['booking']
truth = expected['booking']
fields = ['check_in', 'check_out', 'guest_name', 'guests', 'room_type']
result = {'passed': all(extracted[f] == truth[f] for f in fields), 'date_format_ok': extracted['check_in'] == truth['check_in']}
""",
)

cross_language = client.scorers.create_code(
 name="locale-parity",
 code="""
en_acc = mean(s for t, s in zip(traces, scores) if t.tags['lang'] == 'en')
gaps = {l: en_acc - mean(s for t, s in zip(traces, scores) if t.tags['lang'] == l) for l in supported_langs}
result = {'passed': max(abs(g) for g in gaps.values()) <= 0.05, 'gaps': gaps}
""",
)

cultural = client.judges.create(
 name="cultural-fit",
 evaluation_goal="Score 1-5: is the response culturally appropriate for the locale, including tone, formality, and idiom?",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"feature": "booking-assistant"}, "sample_rate": 0.01},
 scorers=[booking_accuracy.id, cross_language.id],
 judges=[cultural.id],
 schedule="hourly",
)
```

```typescript
// TypeScript (REST)
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { tags: { feature: "booking-assistant" }, sample_rate: 0.01 },
 scorers: [bookingAccuracyId, crossLanguageId],
 judges: [culturalId],
 schedule: "hourly",
 }),
});
```

## What you get

* Per-language accuracy is measured, not assumed from aggregate metrics.
* Date-parsing and locale-specific failures detected in CI before they reach travelers.
* PCI compliance enforced by code graders; no payment data flows into trace bodies.
* Localization-team prioritization driven by data — which languages need investment and where.

## Stratix capabilities used

* [Custom code graders](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/custom-code-scorer.md) — cross-language parity, booking-detail extraction
* [Judges with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — cultural appropriateness, tone consistency
* [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled

## Replicate this

**Get started:** [Cookbook: multilingual scoring](https://github.com/LayerLens/gitbook-full/blob/main/08-evaluate/cookbook/multilingual.md) is the runnable starter.

* [Industry → Travel and hospitality](/4.2-industry-use-cases/travel-hospitality)
* [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation)
* [Workflow: Evaluate](/9.-improve-tune-the-system/workflow)


# Industry


# Education and EdTech compliance

Education compliance — FERPA, COPPA, IDEA, Section 504, ADA, Title VI/IX, state student-data laws, ED guidance, accessibility.

Education AI compliance braids **federal student-record law (FERPA) + child-data law (COPPA) + civil rights statutes + state student-data laws + accessibility law**. Stakes are high because student trajectories are at issue and regulators (OCR, FTC, state AGs) are active.

## Regulatory frame

| Regime                               | Applies when                                    | Evaluation-evidence shape                                          |
| ------------------------------------ | ----------------------------------------------- | ------------------------------------------------------------------ |
| **FERPA (20 USC § 1232g)**           | Schools/agencies receiving DOE funds            | School official exception evidence; consent/notice; records access |
| **PPRA**                             | Surveys / behavioral data of minors             | Notice, parental consent for protected categories                  |
| **COPPA (15 USC § 6501)**            | Online services likely used by under-13         | Parental consent, data minimization, retention limits              |
| **IDEA / Section 504 / ADA**         | Students with disabilities                      | Accommodation accuracy, accessible delivery                        |
| **Title VI**                         | Federally-funded programs                       | Disparate impact monitoring                                        |
| **Title IX**                         | Sex discrimination in federally-funded programs | Bias audit on grading and admissions                               |
| **State student-data privacy laws**  | Varies by state — most states now have one      | Specific notice/consent/data-localization rules                    |
| **State AI laws**                    | E.g., NY Education Law § 2-d, CA AB 2273        | Per-state requirements                                             |
| **Department of Education guidance** | Federal funded programs                         | Reasonable use of AI evidence                                      |
| **OCR enforcement**                  | Federally-funded programs                       | Investigation cooperation; corrective action plans                 |
| **GDPR / UK GDPR**                   | EU/UK students                                  | Lawful basis; DPA; transfer mechanism                              |

## FERPA

20 USC § 1232g, 34 CFR Part 99. Core requirements:

* Schools may disclose education records only with parent/eligible-student consent or under exceptions
* **School official exception** — most common AI-vendor pathway; vendor must perform institutional service, be under direct control, and use records only for authorized purpose
* Records access rights for parents/eligible students
* Limited directory information disclosure with notice/opt-out

**Stratix evaluation evidence:**

* DPA executable as a school-official-exception agreement
* Audit trail of who accessed which records
* Tenant isolation — one school's records never touch another's

## COPPA

For services likely used by children under 13:

* Verifiable parental consent before collecting
* Limited data; no behavioral targeting against under-13
* Strict retention and deletion
* FTC has specifically called out AI services that train on under-13 data

**Stratix evaluation evidence:**

* Hard age-band gate
* Audit log of consent state at the time of trace
* No training on tenant data without consent (enterprise contractual term)

## IDEA / Section 504 / ADA

For students with disabilities:

* **IDEA** — IEP-based services; AI tools must support, not replace, IEP-required services
* **Section 504** — accommodations for substantially limiting impairments
* **ADA Title II** (public schools) and **Title III** (private schools / EdTech vendors) — accessible delivery

**Stratix evaluation evidence:**

* Accessibility scoring on AI outputs (reading level, alt-text, captioning WER)
* Accommodation-database matching as code assertion
* Audit trail of accommodation delivery

## Title VI / Title IX

Federal civil rights statutes:

* Disparate impact on protected class triggers OCR investigation
* AI grading, AI integrity-detection, AI admissions all currently in OCR scrutiny
* Settlement consent decrees include monitoring and retraining obligations

**Stratix evaluation evidence:**

* Group-fairness scorers per scenario
* Audit-trail per evaluation showing the decision basis
* Retention to support OCR investigation timelines

## State student-data privacy laws

Most US states have at least one. Notable examples:

* **NY Education Law § 2-d** — strict vendor data agreement requirements; "Bill of Rights"
* **CA Student Online Personal Information Protection Act (SOPIPA)**
* **CT, IL, CO, FL, KY, NC, TX, UT** — varying flavors

Many require:

* Specific data-handling notices to parents
* No targeted advertising to students
* No selling student data
* Specific deletion / retention rules
* Per-vendor public agreement registry

**Stratix evaluation evidence:**

* Per-state DPA addenda available under Enterprise tier
* Tenant isolation with state-specific configuration
* Audit retention matching state requirements

## State AI in education laws

Emerging:

* **NY** — guidance on AI in K-12 (2024-2025)
* **CA** — multiple bills around AI literacy and disclosure
* **TX, FL, OH, others** — proposed and adopted bills around AI assessment, integrity

Track state-by-state; defaults below cover most current frameworks.

## US Department of Education guidance

ED has issued non-binding guidance on AI in education (2023-2024) including:

* "Designing for Education with AI" report
* Guidance on AI accessibility
* Cybersecurity advisories for AI-enabled services

Treat as non-binding-but-influential — many state and district policies cite ED.

## OCR enforcement

US Education Department Office for Civil Rights investigates Title VI / Title IX / Section 504 / ADA / IDEA complaints. Recent focus areas:

* AI-detection bias against non-native English writers
* Accessibility of AI-generated content
* Disparate-impact investigations on AI grading

**Stratix evaluation evidence supports OCR cooperation:**

* Pull-on-demand fairness reports
* Audit trail of AI-influenced student outcomes

## GDPR / UK GDPR

For EU/UK students:

* Lawful basis (consent or legitimate interest, with care)
* DPIA for high-risk AI use
* Cross-border transfer mechanism
* DPA with the institution

## Recommended setup

1. Pro tier minimum; Enterprise for large districts and university systems
2. SSO + RBAC scoped to district / school / class / role
3. DPA executable as FERPA school-official-exception
4. State-specific DPA addenda for NY § 2-d and similar
5. Hard rules: instructor-final-decision on grading, age-band gating, accommodation match
6. Group fairness scorers on all student-affecting AI
7. Accessibility scoring on every student-facing output
8. Audit retention matching the longest applicable state requirement (default 5 years post-graduation)
9. No training on tenant data without explicit consent
10. BYOK custom models for the most sensitive student data

## See also

* [Education scenarios](/more-in-this-section-2/scenarios)
* [Education eval patterns](/more-in-this-section-2/eval-patterns)
* [Concept: Multi-tenancy](/7.-observe-see-whats-happening/multi-tenancy)
* [Enterprise](/4.2-industry-use-cases/education)
* [BYOK custom models](/5.-select-pick-the-model/byok-custom-models)


# Education cookbook and samples

Education starter samples.

* [Tutor helpfulness](/more-in-this-section-2/industry-education-tutor)


# Recipe: education tutor pedagogical-quality

Recipe — education AI tutor pedagogical-quality judge.

**Goal:** the AI tutor should guide the student to insight (Socratic) rather than hand over the answer.

```python
from layerlens import Stratix
client = Stratix()

judge = client.judges.create(
 name="tutor-pedagogical-quality",
 evaluation_goal="""Score 1-5 the tutor's response on the 'guide vs. give' spectrum.
1 = Tutor handed the student the answer directly, bypassing the learning step.
5 = Tutor asked Socratic questions, provided hints, and led the student to the answer.
Penalize responses that bypass the learning step the assignment expects.""",
)

# Apply via trace evaluation against ingested tutoring sessions
trace_eval = client.trace_evaluations.create(trace_id=trace.id, judge_id=judge.id)
result = client.trace_evaluations.wait_for_completion(trace_eval.id)
```

For COPPA / FERPA handling, redact student-record fields client-side before posting traces — see [Industry → Education compliance](/more-in-this-section-2/compliance).

## See also

* [Industry: Education](/4.2-industry-use-cases/education)
* [Industry pattern: AI tutor](/4.2-industry-use-cases/pattern-8)
* [Concept: Judges](/more-in-this-section-9/judges-2)


# Education and EdTech evaluation patterns

Education evaluation patterns — primitive ratio, span rules, GEPA labeling, cadence.

Recommended primitive ratio for education AI: **50/30/20** — deterministic rules, code assertions, judges. Subject-matter accuracy and pedagogical-quality assessment lift the judge share above what other regulated industries use.

## Primitive ratio

| Primitive           | Share | Why                                                                                            |
| ------------------- | ----- | ---------------------------------------------------------------------------------------------- |
| Deterministic rules | 50%   | Direct-answer guards, instructor-final-decision, confidence thresholds, accommodation accuracy |
| Code assertions     | 30%   | Inter-rater agreement, fairness, reading-level, captioning WER                                 |
| LLM judges          | 20%   | Pedagogical scaffolding, alternate-format faithfulness, representation diversity               |

## Pattern table

| Scenario                            | Primary primitive | Specific check                               |
| ----------------------------------- | ----------------- | -------------------------------------------- |
| Tutoring — accuracy                 | Judge             | Subject-matter faithfulness                  |
| Tutoring — scaffolding              | Judge             | Asks-before-tells rubric                     |
| Tutoring — direct-answer guard      | Rule              | No outright answers on protected assignments |
| Tutoring — age appropriate          | Code scorer       | Reading level matches grade                  |
| Grading — inter-rater agreement     | Code assertion    | Cohen's kappa vs. human reference            |
| Grading — demographic fairness      | Code scorer       | Score parity across proxies                  |
| Grading — rubric coverage           | Rule              | Every criterion scored                       |
| Grading — instructor approval       | Rule              | Final grades require instructor sign-off     |
| Curriculum — standards citation     | Rule              | Items cite alignment standard                |
| Curriculum — accommodation accuracy | Code assertion    | Match IEP/504 database                       |
| Curriculum — representation         | Code scorer       | Diversity of examples                        |
| Integrity — confidence threshold    | Rule              | AI never unilateral                          |
| Integrity — disparate impact        | Code scorer       | Group fairness in flagging                   |
| Integrity — evidence citation       | Rule              | Specific evidence required                   |
| Accessibility — faithfulness        | Judge             | Alternate format preserves content           |
| Accessibility — captioning WER      | Code assertion    | Below regulatory threshold                   |

## Span rules

* `student.context.lookup` — grade, IEP/504, language preference
* `standards.cite` — Common Core / NGSS / state / AP alignment
* `rubric.coverage` — every criterion scored
* `accommodation.match` — IEP/504 plan applicability
* `integrity.flag` — confidence and evidence chain
* `instructor.review` — approval state

## GEPA labeling

For each education judge, label ≥ 50 paired examples (more for high-stakes):

* **Pedagogical scaffolding:** Curriculum specialists + classroom teachers; cover Socratic vs. directive responses
* **Subject-matter faithfulness:** Domain experts per subject (math/science/humanities/CS); ≥ 100 for STEM grading
* **Accessibility faithfulness:** Disability-services specialists label
* **Representation diversity:** Equity specialists label

Run [judge optimization](/more-in-this-section-9/judge-optimizations-1) with `budget="medium"`. For grading judges that affect student records, run with `budget="heavy"` and hold out 30%.

## Cadence

| Cadence               | Activity                                                            |
| --------------------- | ------------------------------------------------------------------- |
| **Per-deploy**        | CI gate runs scenario suite (rules + assertions)                    |
| **Per-class-session** | Sample tutoring traces; daily summary to teacher                    |
| **Daily**             | Sample production traces across all scenarios                       |
| **Weekly**            | Demographic-fairness audit on grading; integrity-flag overturn rate |
| **Monthly**           | GEPA re-optimization on judges that have drifted                    |
| **Quarterly**         | Standards-alignment refresh; accommodation database refresh         |
| **Annually**          | OCR / state-audit prep; full bias audit                             |

## See also

* [Education scenarios](/more-in-this-section-2/scenarios)
* [Education compliance](/more-in-this-section-2/compliance)
* [Recipe: GEPA basics](/9.-improve-tune-the-system/gepa-basics)


# Education and EdTech scenarios

Education and EdTech scenarios — tutoring, grading, content generation, integrity, accessibility. Industry patterns, not customer case studies.

Five canonical scenarios where K-12, higher-ed, and professional learning organizations use Stratix evaluation to gate AI features.

## 1. AI tutoring and homework help

**Setup:** Student asks the tutor for help on a problem. AI scaffolds the solution, asks Socratic questions, surfaces relevant course material — without giving the answer outright in academic settings that disallow it.

**Why it's hard:**

* "Help" without "doing the assignment" is a fine line
* Subject-matter accuracy must be near-perfect — wrong math/physics/grammar imprints lasting misconceptions
* Age-appropriate language across grade levels

**Stakes:**

* Misconception cost compounds across the rest of the student's learning trajectory
* Academic integrity policies of partner institutions vary
* COPPA / FERPA / state student-data laws are unforgiving on under-13 audiences

**Stratix evaluation criteria:**

| Aspect                     | What you measure                                | Pattern                            |
| -------------------------- | ----------------------------------------------- | ---------------------------------- |
| Subject-matter accuracy    | Correct facts, correct math                     | Deterministic + faithfulness judge |
| Pedagogical scaffolding    | Asks before tells, guides before solves         | Judge against rubric               |
| Age-appropriateness        | Reading level matches grade                     | Code scorer                        |
| Direct-answer guard        | No outright solutions for protected assignments | Hard rule                          |
| PII / private-info hygiene | No surfacing of other students' data            | Hard rule                          |

## 2. Automated grading and feedback

**Setup:** AI grades short answers, essays, and code submissions; produces score and detailed feedback. Instructor reviews; AI doesn't finalize grades unilaterally.

**Why it's hard:**

* Grading bias by topic, dialect, language background
* Format vs. substance — verbose-but-wrong should not score above terse-and-correct
* Rubric application requires nuance models often lack

**Stakes:**

* Grade-level disparate impact triggers civil-rights and Title VI exposure
* Student-data privacy bound by FERPA + state laws
* Higher-ed: regional accrediting bodies scrutinize AI in assessment

**Stratix evaluation criteria:**

* **Inter-rater agreement** — AI scores vs. human-graded reference set; track kappa
* **Demographic-fairness scorer** — Score-distribution parity across protected proxies
* **Rubric-coverage rule** — Every score touches every rubric criterion
* **Instructor-final-decision rule** — Hard rule: grades not posted until instructor approves
* **Plagiarism / generative-AI detection caveat** — Output cannot accuse without strong evidence threshold

## 3. Curriculum and content generation

**Setup:** Teachers/instructional designers ask the AI to produce lesson plans, problem sets, reading lists, accommodations.

**Why it's hard:**

* Standards alignment (Common Core, NGSS, state standards, AP) must be precise
* IDEA / 504 accommodation accuracy is a legal obligation
* Bias and representation in examples / readings

**Stakes:**

* Out-of-standard content drives test-prep failures
* IDEA/504 errors trigger OCR investigations and consent decrees
* Adoption decisions in K-12 districts are public; political controversy travels fast

**Stratix evaluation criteria:**

* **Standards-citation rule** — Every produced item cites its alignment standard
* **Accommodation accuracy** — Code assertion against IEP/504 plan database
* **Representation diversity scorer** — Examples and readings show breadth
* **Reading-level scorer** — Age-band appropriate
* **Editor-review gate** — Hard rule for adopted curriculum vs. exploratory drafts

## 4. Academic-integrity and AI-detection assistance

**Setup:** AI flags submissions that appear AI-generated, plagiarized, or off-pattern for the student. Surfaces evidence; instructor adjudicates.

**Why it's hard:**

* AI-detection accuracy is famously unreliable; false-positive cases publicized
* Disparate impact on non-native English writers is well-documented
* A wrongful integrity accusation can derail a student's career

**Stakes:**

* Title VI and ADA exposure for biased detection
* Class-action exposure when accusations are statistically unreliable
* Student wellness — wrongful accusations have produced documented mental health crises

**Stratix evaluation criteria:**

* **Confidence-threshold rule** — Hard rule: AI never makes the determination unilaterally
* **Disparate-impact monitoring** — Group fairness across language and demographic proxies
* **Evidence-citation rule** — Every flag cites specific evidence (matched passages, anomaly indicators)
* **Appeal-overturn rate** — Drift signal; high overturn rate triggers retraining
* **Student-due-process rule** — Output includes standard-appeal-language disclosure

## 5. Accessibility and accommodation

**Setup:** AI produces alternate-format materials (large print, audio, simplified text), captioning, translation, image descriptions for students with disabilities or LEP.

**Why it's hard:**

* Quality must match the original — paraphrase loses content
* ADA + Rehabilitation Act §504 + IDEA + state law obligations
* Real-time captioning latency / accuracy in lecture settings

**Stakes:**

* OCR investigations following accessibility complaints
* Settlement consent decrees common for major institutions
* Inadequate captioning specifically a long-running litigation theme

**Stratix evaluation criteria:**

* **Faithfulness judge** — Alternate format preserves all content
* **Captioning WER** — Word Error Rate ≤ regulatory threshold (institution-set)
* **Translation faithfulness** — Per-language judge runs
* **Latency rule** — Real-time captioning under defined latency SLA
* **Image-description completeness** — Required visual elements covered

## See also

* [Education eval patterns](/more-in-this-section-2/eval-patterns)
* [Education compliance](/more-in-this-section-2/compliance)


# Energy and utilities compliance

Energy and utilities compliance — NERC CIP/OPS, FERC, state PUC/CPUC, PHMSA, OSHA, EPA, ISO/RTO market rules.

Energy compliance combines **federal reliability rules (NERC) + federal energy regulator (FERC) + state PUCs + pipeline / OSHA safety + environmental (EPA) + market rules (ISO/RTOs)**. AI in this sector inherits all layers.

## Regulatory frame

| Regime                          | Applies when                               | Evaluation-evidence shape                               |
| ------------------------------- | ------------------------------------------ | ------------------------------------------------------- |
| **NERC CIP**                    | Bulk Electric System cyber assets          | Asset classification; access control; incident response |
| **NERC OPS / PRC / TPL / etc.** | BES reliability                            | Procedure compliance; protection coordination           |
| **FERC orders**                 | Interstate transmission, wholesale markets | Order-specific requirements (e.g., 880, 881, 2222)      |
| **State PUC / CPUC rules**      | Retail rates, service quality              | Tariff filings; rate cases                              |
| **CPUC wildfire rules (CA)**    | CA electric utilities                      | Wildfire mitigation plans; AI risk-modeling oversight   |
| **PHMSA (49 CFR 192/195)**      | Gas / liquids pipelines                    | Integrity management; record retention                  |
| **OSHA 1910.269**               | Electric power generation/T\&D             | Switching/working-on rules; LOTO; PPE                   |
| **EPA Clean Air / Water Acts**  | Power generation; water utilities          | Emissions / discharge monitoring                        |
| **ISO/RTO market rules**        | Wholesale market participants              | Specific market protocols                               |
| **Privacy laws (CCPA, etc.)**   | Customer-data scope                        | DSR; consent; retention                                 |
| **CALEA / state surveillance**  | Carrier-like services (some utilities)     | Lawful intercept                                        |
| **NIST AI RMF**                 | Voluntary; cited in DOE / NREL guidance    | GOVERN/MAP/MEASURE/MANAGE evidence                      |

## NERC CIP

NERC Critical Infrastructure Protection standards apply to BES Cyber Systems:

* CIP-002: Cyber asset categorization
* CIP-003: Security management
* CIP-004: Personnel and training
* CIP-005: Electronic security perimeters
* CIP-007: Systems security management
* CIP-008: Incident reporting
* CIP-010: Configuration change management and vulnerability assessments
* CIP-011: Information protection
* CIP-013: Supply chain risk management

Penalties up to $1M per day per violation.

**Stratix posture:**

* Stratix is **not** a BES Cyber System; it is a SaaS evaluation platform
* For utilities using Stratix to evaluate AI that influences BES operations, the AI itself and its operational-deployment footprint may be in CIP scope (depending on impact rating)
* BYOK custom models recommended for sensitive workloads
* NERC CIP-013 supply-chain diligence package available — talk to enterprise

## FERC orders

Selected high-impact orders:

* **Order 880** — Cybersecurity reporting requirements
* **Order 881** — Ambient-adjusted ratings for transmission
* **Order 2222** — DER aggregation in wholesale markets
* **Order 2023** — Generator interconnection reform

Each carries specific compliance evidence requirements.

## State PUC / CPUC

State-by-state regulation of retail service:

* **CPUC AI rules (CA)** — Active rule-making; wildfire-mitigation AI is most scrutinized
* **NY PSC / CT PURA / TX PUC / IL ICC / others** — Per-state proceedings on AI-influenced decisions
* **Service-quality penalties** when AI-driven systems contribute to outages

**Stratix evaluation evidence:**

* AI use-case inventory tied to filings
* Drift dashboards for regulator-pull-on-demand

## PHMSA (49 CFR 192/195)

Gas-pipeline (192) and liquids-pipeline (195) safety:

* Integrity management programs
* Operator qualification
* Public awareness
* Incident reporting

**Stratix evaluation evidence:**

* Hard rule on field-safety AI: cite 49 CFR section
* Audit log of AI guidance vs. recorded incidents

## OSHA 1910.269

Electric power generation, transmission, distribution. The dominant federal worker-safety standard for line work.

**Stratix evaluation evidence:**

* LOTO + PPE rules on every field-safety AI output
* Manual-version match rule

## EPA

Clean Air Act / Clean Water Act apply to generation and water utilities. AI used in compliance reporting must produce verifiable inputs.

## ISO/RTO market rules

Each ISO/RTO has its own protocols:

* PJM, MISO, ERCOT, CAISO, ISO-NE, NYISO, SPP
* Settlement disputes are common; AI-augmented bidding must produce auditable trail
* FERC enforcement on market manipulation extends to algorithmic strategies

**Stratix evaluation evidence:**

* ISO-rule compliance rule per market output
* Settlement-data freshness check

## Privacy laws

CCPA / state privacy / GDPR — same framework as [retail compliance](/more-in-this-section-2/compliance-10).

## NIST AI RMF

Cited as voluntary best practice in DOE and NREL AI guidance. For DOE-funded R\&D, NIST AI RMF mapping is a common deliverable.

## Recommended setup

1. Pro tier minimum; Enterprise for utilities and ISO/RTO participants
2. SSO + RBAC for dispatch / NOC / customer-service / field roles
3. BYOK custom models for NERC CIP-touching workloads
4. Hard rules: advisory-only on dispatch, switching-procedure citation, NERC standard compliance, OSHA 1910.269 / PHMSA citation, medical-baseline routing
5. Group-fairness scorers on customer-affecting AI
6. Audit retention matching the longest applicable NERC / PHMSA / state requirement (default 7+ years; 6 years for many NERC standards)
7. AI use-case inventory tied to NERC and state filings
8. NIST AI RMF mapping retained per use case
9. DPA executed; tenant data residency where contracted

## See also

* [Energy scenarios](/more-in-this-section-2/scenarios-1)
* [Energy eval patterns](/more-in-this-section-2/eval-patterns-1)
* [Concept: Multi-tenancy](/7.-observe-see-whats-happening/multi-tenancy)
* [Enterprise](/4.2-industry-use-cases/energy-utilities)
* [BYOK custom models](/5.-select-pick-the-model/byok-custom-models)


# Energy and utilities cookbook and samples

Energy starter samples.

## Reference

* [Concept: Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation)


# Energy and utilities evaluation patterns

Energy and utilities evaluation patterns — primitive ratio, span rules, GEPA labeling, cadence.

Recommended primitive ratio for energy AI: **70/20/10** — deterministic rules, code assertions, judges. Grid procedures, NERC CIP, OSHA / DOT safety rules, and ISO/RTO market rules drive heavy rule density. Judges are reserved for customer-facing copy quality.

## Primitive ratio

| Primitive           | Share | Why                                                                                 |
| ------------------- | ----- | ----------------------------------------------------------------------------------- |
| Deterministic rules | 70%   | Switching procedures, NERC reliability, ISO market, medical-baseline, LOTO/PPE      |
| Code assertions     | 20%   | Risk calibration, freshness thresholds, settlement-data checks                      |
| LLM judges          | 10%   | Customer-impact statement quality, multilingual parity, switching-step faithfulness |

## Pattern table

| Scenario                              | Primary primitive | Specific check                     |
| ------------------------------------- | ----------------- | ---------------------------------- |
| Grid ops — switching                  | Rule + Judge      | Output matches approved procedures |
| Grid ops — N-1                        | Rule              | Contingency obligations preserved  |
| Grid ops — advisory only              | Rule              | AI never auto-dispatches           |
| Predictive maintenance — calibration  | Code assertion    | Predicted vs. actual failures      |
| Predictive maintenance — wildfire     | Rule              | High-fire-threat priority          |
| Predictive maintenance — vegetation   | Rule              | State VM standards followed        |
| Customer service — medical baseline   | Rule              | Always-route-to-human              |
| Customer service — low-income program | Rule              | Eligibility surfaced               |
| Customer service — outage freshness   | Rule              | Data within threshold              |
| DR / DERMS — ISO rule compliance      | Rule              | Match ISO/RTO market rules         |
| DR / DERMS — settlement freshness     | Code assertion    | Meter data current                 |
| DR / DERMS — cyber gate               | Rule              | NERC CIP controls preserved        |
| Field safety — energized work         | Rule              | LOTO/PPE requirements              |
| Field safety — pipeline procedure     | Rule              | 49 CFR cited                       |
| Field safety — manual version         | Rule              | Cited manual current               |

## Span rules

* `procedure.cite` — switching/maintenance procedure version
* `nerc.check` — applicable NERC standard
* `iso.market.rule` — applicable ISO market rule
* `wildfire.zone` — fire-threat district classification
* `medical.baseline` — protected-customer routing
* `safety.disclaimer` — required disclaimer presence

## GEPA labeling

For each energy judge, label ≥ 50 paired examples:

* **Switching-step faithfulness:** Senior dispatchers + reliability engineers label
* **Customer-impact messages:** Comms team + customer advocates label
* **Field-safety faithfulness:** Senior linemen / pipeline-qualified leads label
* **Multilingual:** Native-speaker QA per language

Run [judge optimization](/more-in-this-section-9/judge-optimizations-1) with `budget="medium"`. For safety-critical judges, run with `budget="heavy"` and hold out 30%.

## Cadence

| Cadence        | Activity                                                            |
| -------------- | ------------------------------------------------------------------- |
| **Per-deploy** | CI gate runs scenario suite (rules + assertions)                    |
| **Per-shift**  | Critical-asset alarm summary; advisory-only audit                   |
| **Continuous** | NERC alerts; outage-data freshness                                  |
| **Daily**      | Sample production traces; full evaluation                           |
| **Weekly**     | Wildfire-zone audit (Cal Fire / state); vegetation status           |
| **Monthly**    | GEPA re-optimization on judges; switching-procedure version refresh |
| **Quarterly**  | NERC CIP audit prep; ISO/RTO market-rule refresh                    |
| **Annually**   | OSHA / PHMSA reportable-event prep; full safety audit               |

## See also

* [Energy scenarios](/more-in-this-section-2/scenarios-1)
* [Energy compliance](/more-in-this-section-2/compliance-1)
* [Recipe: GEPA basics](/9.-improve-tune-the-system/gepa-basics)


# Energy and utilities scenarios

Energy and utilities scenarios — grid operations, predictive maintenance, customer service, demand response, safety. Industry patterns, not customer case studies.

Five canonical scenarios where electric, gas, water, and pipeline utilities use Stratix evaluation to gate AI features.

## 1. Grid operations and dispatch support

**Setup:** AI assists grid operators — surfaces alarm correlations, suggests switching steps, drafts customer-impact messages, summarizes weather/contingency analysis.

**Why it's hard:**

* Grid state is high-velocity; safety-critical
* N-1 / N-1-1 contingency math defeats general LLMs
* BES (Bulk Electric System) reliability standards (NERC CIP, NERC OPS) are unforgiving
* Cascading failure dynamics → catastrophe risk

**Stakes:**

* 2003 Northeast blackout class scale possible
* NERC penalties up to $1M per violation per day
* FERC Order 880 (cyber) and 881 (transmission) overlay
* Public-safety dependencies (medical, water pumping) raise stakes further

**Stratix evaluation criteria:**

| Aspect                      | What you measure                             | Pattern           |
| --------------------------- | -------------------------------------------- | ----------------- |
| Switching-step faithfulness | Output matches approved switching procedures | Hard rule + judge |
| Contingency-rule compliance | N-1 obligations preserved                    | Hard rule         |
| Advisory-only               | AI never auto-applies dispatch               | Hard rule         |
| Customer-impact message     | Plain-language and faithful                  | Judge             |
| Procedure-version match     | Cited procedure version current              | Rule              |

## 2. Predictive maintenance for grid assets

**Setup:** AI ingests SCADA, weather, asset history; predicts transformer / breaker / line-section failure risk; recommends inspections, replacements, vegetation work.

**Why it's hard:**

* Wildfire-ignition risk in CA / Western states drives extreme stakes
* Asset replacement decisions allocate capital across decades
* PURPA / state-PUC depreciation rules constrain accounting
* Underground vs. overhead trade-offs are politically charged

**Stakes:**

* PG\&E bankruptcy (\~$30B) was wildfire-ignition driven
* State PUC and CPUC AI / wildfire-risk rules are active
* Negligent failure to maintain has triggered criminal exposure in CA

**Stratix evaluation criteria:**

* **Risk-calibration scorer** — Predicted vs. actual failure rate
* **Wildfire-zone rule** — Hard rule: high-fire-threat district items get priority routing
* **Asset-criticality rule** — Critical assets always escalate to human review
* **Vegetation-rule compliance** — Hard rule per state vegetation management standard

## 3. Customer service and billing

**Setup:** Customer chatbot for billing questions, outage reporting, energy efficiency, plan options.

**Why it's hard:**

* LIHEAP / state low-income protections — wrong shutoff guidance can cause real harm
* Senior / medical-baseline protections require accurate handling
* Multi-language requirements; LEP customer regulations
* Outage-status accuracy critical for vulnerable populations

**Stakes:**

* Wrongful disconnection of medical-baseline customers has caused deaths
* State PUCs and AGs aggressive on shutoff practices
* Class actions on bill estimates / TOU misapplication common

**Stratix evaluation criteria:**

* **Medical-baseline rule** — Hard rule: medical-baseline customers routed to human for shutoff topics
* **Low-income-program rule** — LIHEAP / CARE / state-program eligibility surfaced when applicable
* **Outage-data freshness** — Cited data within threshold
* **Multilingual parity** — Per-language judge runs
* **Tariff citation** — Bill explanations cite the applicable tariff

## 4. Demand response and DERMS

**Setup:** AI manages distributed energy resources — surfaces dispatch options, drafts participant communications, analyzes settlement.

**Why it's hard:**

* ISO/RTO market rules (PJM, MISO, ERCOT, CAISO, ISO-NE, NYISO, SPP) all differ
* FERC Order 2222 expanded DER market participation
* Settlement-quality metering data must support audit
* Cybersecurity exposure for grid-edge devices

**Stakes:**

* ISO/RTO settlement disputes costly to litigate
* FERC enforcement on market-manipulation rules
* Cyber exposure under NERC CIP standards

**Stratix evaluation criteria:**

* **ISO-rule compliance rule** — Hard rule: outputs match the applicable ISO/RTO market rules
* **Settlement-data freshness** — Cited meter data current
* **Participant-communication faithfulness** — Plain-language judge against actual dispatch
* **Cyber-control gate** — Hard rule: AI cannot bypass NERC CIP controls

## 5. Field-safety and workforce assistance

**Setup:** Field crews ask about energized work, switching procedures, gas-leak protocols, confined-space requirements.

**Why it's hard:**

* OSHA 1910.269 (electric power generation/transmission/distribution) is unforgiving
* DOT pipeline safety regulations (49 CFR 192/195) for gas
* Mistakes cause fatalities

**Stakes:**

* Worker fatalities in line work / gas work are routine industry events
* OSHA citations + criminal exposure for willful violations
* DOT PHMSA enforcement on pipelines
* Family/community trauma is incalculable

**Stratix evaluation criteria:**

* **Energized-work rule** — Hard rule: outputs cite applicable LOTO and PPE requirements
* **Pipeline-procedure rule** — Hard rule: gas / pipeline procedures cite 49 CFR section
* **Out-of-scope refusal** — Hard rule
* **Manual-version match** — Hard rule
* **Safety-disclaimer rule** — Standard disclaimer present

## See also

* [Energy eval patterns](/more-in-this-section-2/eval-patterns-1)
* [Energy compliance](/more-in-this-section-2/compliance-1)


# Financial-services compliance

Financial-services compliance — multi-regulator frame (SEC, FINRA, FinCEN, OCC, CFPB, SOX, GLBA), evidence shape per regime.

Financial-services AI sits across the most fragmented regulatory landscape on the platform. Five primary regimes; evaluation evidence requirements differ for each.

## Regulatory frame

| Regime               | Applies when                                                     | Evaluation-evidence shape                                                          |
| -------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| **SEC / FINRA**      | Investment advisory, broker-dealer, customer recommendations     | Suitability evidence, disclosure presence, audit-trail of recommendation rationale |
| **FinCEN (BSA/AML)** | Customer onboarding, transaction monitoring, sanctions screening | Sanctions-screening recall, document-authenticity rate, SAR narration faithfulness |
| **OCC / FRB / FDIC** | Banking-product underwriting, loan origination, fair-lending     | Fair-lending disparity audits, model-risk management documentation                 |
| **CFPB**             | Consumer-facing financial products, fair-treatment               | Disparity ratios, complaint-rate correlations, plain-language requirements         |
| **SOX**              | Public-company financial-data trading, audit narration           | Immutable decision-rationale logs                                                  |
| **GLBA**             | Customer non-public personal information handling                | Privacy boundary, encryption, access controls                                      |
| **State (DFS, DOI)** | State-licensed lenders / insurers / broker-dealers               | State-by-state varies; market-conduct examination prep                             |

## SEC / FINRA

Stratix evaluation evidence supports:

* **Suitability records** — every recommendation tied to an evaluation that scored it against the customer's risk profile
* **Disclosure-presence audits** — substring-scorer evidence that required disclaimers appeared
* **Bias-toward-proprietary-products** scorer — evidence that recommendations are not skewed toward house funds
* **Recommendation-rationale audit trail** — judge results referencing specific market data

Recent FINRA sweeps include $1M-$50M fines for suitability violations in robo-advisor products. Documenting evaluation-driven controls reduces enforcement exposure.

## FinCEN BSA/AML

* **Sanctions coverage** — 100% recall on every input name against the active sanctions list
* **Document authenticity** — image-tampering detection rate
* **SAR-narration faithfulness** — judge agreement that filed SARs accurately describe the underlying transaction
* **Adverse-media coverage** — relevance scores for media hits

Per-violation penalties run $1M-$1B in BSA/AML enforcement actions. Stratix evidence is auditor-ready.

## OCC / FRB / FDIC fair lending (ECOA)

Cross-cutting concern for any AI in lending decisions:

* **Disparity ratios** below 1.25× across all protected-class proxies
* **Model risk management** — every model version cited in a decision tied to its evaluation history
* **Override rates** — when humans override AI recommendations, track for systemic-bias signal

Stratix supports per-segment disparity computation via custom code graders. See the [fair-lending platform pattern](/4.2-industry-use-cases/pattern-1).

## CFPB consumer protection

Plain-language evaluation, complaint-rate monitoring, and accessibility compliance for digital products. Reading-level scorer + cultural-appropriateness judges support consumer-facing applications.

## SOX trading compliance

Immutable narration of every trading decision. Stratix evaluation IDs cited in audit trails. Audit-friendly evaluation history retention available on Enterprise tier.

## GLBA privacy

Customer non-public personal information must not appear in trace bodies unless tenant-isolation and encryption controls are in place. Recommended setup:

* Enterprise tier with executed DPA
* PII-redaction at trace ingestion (your code, not Stratix)
* SSO with role-based access
* Configurable audit retention matching your record-retention policy

## State regulators (DFS, DOI)

State-by-state variance. New York DFS Cybersecurity Regulation (Part 500) is among the strictest; California CCPA/CPRA adds privacy obligations. State market-conduct examinations frequently trigger broader regulatory inquiries.

## Recommended setup

1. Enterprise tier with executed DPA and BAA-equivalent contractual terms
2. SSO with role-based access; separate roles for compliance / risk / engineering / analyst
3. Trace evaluations citing evaluation IDs in release-notes / audit packages
4. Quarterly disparity audit with compliance signoff
5. Per-region data residency where state law requires
6. Audit-log retention matching the longest applicable record-retention requirement

## See also

* [Industry pattern: fair-lending platform pattern](/4.2-industry-use-cases/pattern-1)
* [Enterprise](/4.2-industry-use-cases/financial-services)
* [Security and compliance](/4.2-industry-use-cases/financial-services)
* [Concept: Multi-tenancy](/7.-observe-see-whats-happening/multi-tenancy)


# Financial services cookbook and samples

Financial services starter samples.

* [Advisor chatbot guardrails](/more-in-this-section-2/industry-finserv-advisor)

## Reference

* [Use case: Agentic evaluation](/4.1-general-use-cases/agentic-evaluation)


# Recipe: financial-services advisor & trading

Recipe — financial-services advisor / trading. Pointer to the SDK industry samples.

Canonical samples:

* [`samples/industry/financial_trading.py`](https://github.com/layerlens/stratix-python/blob/main/samples/industry/financial_trading.py) — SOX suitability, fiduciary duty, regulatory compliance for AI-assisted trading recommendations.
* [`samples/industry/financial_fraud.py`](https://github.com/layerlens/stratix-python/blob/main/samples/industry/financial_fraud.py) — risk scoring + AML pattern detection.

## See also

* [Industry pattern: fair-lending platform pattern](/4.2-industry-use-cases/pattern-1)
* [Industry → Financial services](/4.2-industry-use-cases/financial-services)


# Financial-services evaluation patterns

Financial-services evaluation patterns — primitive ratios, dimensions, fair-lending considerations, GEPA labeling, continuous cadence.

The asymmetric-cost shape of financial-services AI drives the primitive ratio. Detection accuracy is necessary but not sufficient — fair-treatment and disclosure-presence dimensions are where regulatory exposure lives.

## Rule of thumb: 60 / 25 / 15

For a typical financial-services evaluation:

* **\~60% deterministic rules / scorers** — sanctions coverage, disclaimer presence, position limits, PII handling, no-specific-recommendation patterns
* **\~25% natural-language assertions and code-based fairness scorers** — disparity computation, suitability matching, document-extraction accuracy
* **\~15% LLM judges** — explanation quality (SAR narration, customer-facing rationale), suitability judgment in ambiguous cases

Inverting this ratio (judge-heavy) is the most common mistake.

## Pattern table

| Dimension                        | Approach                                    | Notes                                                                      |
| -------------------------------- | ------------------------------------------- | -------------------------------------------------------------------------- |
| Detection precision/recall       | Custom code grader                          | Against labeled fraud / sanctions / SAR datasets                           |
| False-positive disparity         | Custom code grader                          | Per-segment (ZIP, surname-derived demographic, age band) — 1.25× threshold |
| Sanctions coverage               | Hard rule                                   | Recall must be 100%; missing checks = CRITICAL                             |
| Suitability matching             | LLM judge, GEPA-tuned per risk-profile band | ≥50 labels per band                                                        |
| Required-disclaimer presence     | Substring scorer                            | FINRA / SOX / TILA disclosures                                             |
| No-specific-securities rule      | Regex / classifier                          | Customer-facing AI never recommends individual securities                  |
| Document-extraction accuracy     | Custom code grader                          | Field-by-field exact match                                                 |
| Cross-document consistency       | LLM judge                                   | Values match across application + supporting docs                          |
| Numeric / dollar-amount accuracy | Custom code grader                          | Critical for income, balances, transaction amounts                         |
| PII redaction                    | Regex rule                                  | No SSN / account number echo in any logged span                            |
| Adverse-media relevance          | LLM judge                                   | Per match — keep precision high to limit reviewer overload                 |
| Latency under SLA                | Latency scorer                              | Real-time fraud decisions need <300ms p95                                  |

## Span-level rules

Span-level rules matter for compound workflows:

* **Feature-extraction span:** sensitive fields (SSN, full account number) must be hashed before scoring.
* **Model-scoring span:** every score above the SAR threshold must include the contributing features.
* **Explanation span:** every explanation must cite specific transaction or customer features (no opaque rationale).

## GEPA labeling for financial-services judges

Suitability and SAR-narration judges benefit substantially from GEPA — but the labeling needs careful calibration.

1. Label by **multi-reviewer agreement** — at least one compliance officer + one product reviewer per example
2. Stratify labeling by risk-profile band (suitability) or by transaction shape (SAR)
3. Hold out 20% as validation
4. Re-optimize quarterly as regulator interpretive guidance shifts

## Trace-set curation

Over-sample edge cases:

* High-net-worth conservative profiles paired with growth-product recommendations
* Sanctions-adjacent names (close to OFAC list but not on it)
* Self-employment income with depreciation add-backs
* Identical-amount transactions across multiple devices (account takeover signal)
* Cross-border transactions where both endpoints might be sanctioned
* Names that are PEP-adjacent in non-Latin scripts

## Continuous evaluation cadence

* **Per-PR** on the regression suite (small, fast)
* **Hourly** on a 1% sample of production fraud decisions during business hours
* **Daily** on the full SAR-eligible decision set
* **Quarterly** disparity audit with compliance signoff

## See also

* [Scenarios](/more-in-this-section-2/scenarios-2) | [Compliance](/more-in-this-section-2/compliance-2)
* [Industry pattern: fair-lending platform pattern](/4.2-industry-use-cases/pattern-1)
* [SDK sample: financial\_fraud.py](https://github.com/layerlens/stratix-python/blob/main/samples/industry/financial_fraud.py)


# Financial services scenarios

Financial services AI evaluation scenarios — fraud detection, advisor chatbots, KYC, document processing, trading compliance.

Financial-services AI carries asymmetric risk: false negatives cost direct losses (fraud, mis-pricing); false positives drive customer friction and fair-lending exposure. Every scenario below pairs accuracy with fair-treatment evaluation.

## Fraud detection <a href="#fraud" id="fraud"></a>

**Setup:** AI analyzes transaction patterns, account behavior, and device signals to detect potential fraud in real-time.

**Why evaluation is hard:** detection rate is the obvious metric, but **false-positive disparity by demographic** drives fair-lending exposure that's invisible to aggregate accuracy.

**Stakes:** $4.2B annual US card fraud (Federal Reserve / Nilson Report); per-declined-legitimate-transaction friction \~$25 (card-network industry benchmarks); ECOA fair-lending civil penalties on systematic bias.

**Eval criteria:**

* Custom code grader — detection precision/recall on labeled fraud dataset
* Custom code grader — false-positive rate disparity across protected-class proxies (1.25× ratio threshold)
* Faithfulness judge — explanation quality for SAR (Suspicious Activity Report) filing
* Latency scorer — decision time under SLA

**Trace shape:**

```
trace
├── span: feature-extraction (transaction, account, device)
├── span: model-scoring (risk score)
├── span: explanation-generation (for SAR-eligible decisions)
└── span: outcome (approve / decline / flag-for-review)
```

## Advisory / robo-advisor chatbot <a href="#advisor" id="advisor"></a>

**Setup:** AI provides personalized investment guidance, portfolio recommendations, market analysis to retail investors.

**Why evaluation is hard:** suitability violations (recommending unsuitable products to a customer's risk profile) are FINRA enforcement targets — recent sweeps include $1M-$50M fines. Required disclosures must appear consistently.

**Stakes:** SEC / FINRA enforcement bands ($1M-$50M); customer losses + civil litigation; failure to disclose risk-of-loss; bias toward proprietary products.

**Eval criteria:**

* Faithfulness judge (GEPA-tuned ≥50 examples) — performance and product claims grounded in real fund data
* Suitability judge (GEPA-tuned ≥50 examples per risk-profile band) — recommendation matches customer's stated risk tolerance
* Disclaimer-presence scorer — required FINRA disclosures appear in every recommendation
* No-specific-recommendation rule — customer-facing AI never recommends specific securities (deterministic regex)

## KYC / AML document review <a href="#kyc" id="kyc"></a>

**Setup:** AI verifies identity documents (passports, driver's licenses, utility bills) and screens against sanctions / PEP / adverse-media lists.

**Why evaluation is hard:** sanctions-screening recall must be \~100%. Document authenticity detection must catch digitally altered documents.

**Stakes:** FinCEN penalties ($1M-$1B per BSA/AML enforcement action); correspondent banking relationship loss; onboarding fraudulent accounts.

**Eval criteria:**

* Sanctions-coverage rule — every input name screened against the active sanctions list (recall 100%)
* Document-authenticity scorer — image-tamper detection on submitted IDs and bills
* OCR-accuracy scorer — field extraction matches certified-coder ground truth
* Adverse-media judge — relevance scoring for media hits
* PEP-identification scorer — politically-exposed-person matches against the active PEP database

## Loan / mortgage document processing <a href="#loan-docs" id="loan-docs"></a>

**Setup:** AI extracts structured data from loan applications, tax returns, bank statements, pay stubs for underwriting.

**Why evaluation is hard:** Schedule C self-employment income parsing is a known weak point — depreciation add-backs, multiple schedules, net-vs-gross.

**Stakes:** $50K-$500K per bad loan; TILA / ECOA violations; manual rework cost; audit failures.

**Eval criteria:**

* Field-level extraction scorer — names, amounts, dates, account numbers exact-match against ground truth
* Cross-document consistency judge — values match across application + W-2 + bank statement
* Schedule C / self-employment numeric scorer — income computation matches certified-coder ground truth
* PII-redaction rule — no SSN / account-number echo in any logged span

## Trading compliance / SOX-aligned advisory <a href="#trading" id="trading"></a>

**Setup:** AI assists trading workflows — recommendations, audit narration, compliance checks.

**Why evaluation is hard:** SOX trading compliance requires immutable narration of decision rationale. Suitability for institutional vs. retail counterparties differs.

**Stakes:** SOX violations; SEC/FINRA enforcement; fiduciary breach litigation.

**Eval criteria:**

* Audit-narration faithfulness judge — every claim cites the underlying market data or policy
* Suitability judge (per counterparty type) — recommendation matches counterparty's authorized scope
* Position-limit rule — recommendation doesn't breach hard position limits
* Required-disclaimer scorer — SOX/FINRA disclosures present

## See also

* [Industry pattern: fair-lending platform pattern](/4.2-industry-use-cases/pattern-1) — these scenarios composed into one runnable evaluation
* [Eval patterns](/more-in-this-section-2/eval-patterns-2) | [Compliance](/more-in-this-section-2/compliance-2)
* [SDK sample: financial\_fraud.py](https://github.com/layerlens/stratix-python/blob/main/samples/industry/financial_fraud.py)
* [SDK sample: financial\_trading.py](https://github.com/layerlens/stratix-python/blob/main/samples/industry/financial_trading.py)


# Government and public sector compliance

Government compliance — FedRAMP, FISMA, NIST AI RMF, OMB M-24-10, Section 508, Privacy Act, FOIA, EO 14110.

Government AI compliance is **the densest regulatory surface of any sector** — federal procurement, agency-specific rules, executive orders, civil rights statutes, and state/local overlays compound. The frame below covers the federal core; state and local agencies should add applicable analogs.

## Regulatory frame

| Regime                                            | Applies when                                           | Evaluation-evidence shape                                                 |
| ------------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------- |
| **OMB M-24-10**                                   | All federal agency AI; rights/safety-impacting subset  | Pre- and post-deployment testing; continuous monitoring; public inventory |
| **NIST AI RMF (AI 100-1)**                        | Voluntary framework; cited in agency policy            | GOVERN/MAP/MEASURE/MANAGE evidence                                        |
| **EO 14110 (AI Safety/Security)**                 | Federal AI use; certain capabilities require reporting | Red-team evidence; safety eval results                                    |
| **FISMA / FedRAMP**                               | Cloud services storing federal info                    | ATO package; SOC 2 / FedRAMP Moderate or High                             |
| **Section 508 / WCAG 2.1 AA**                     | All federal digital services                           | Accessibility scoring on outputs                                          |
| **Privacy Act of 1974**                           | Systems-of-records on US persons                       | PII handling, SORN, audit log                                             |
| **FOIA (5 USC 552)**                              | Public records released by agency                      | Exemption citation; redaction quality                                     |
| **NIST SP 800-53 Rev. 5**                         | Federal information systems                            | Control inventory; continuous monitoring                                  |
| **NIST SP 800-171**                               | DoD CUI handling                                       | CMMC level alignment                                                      |
| **EEOC / Title VII / Title VI / ADA / Rehab Act** | Federally-funded programs                              | Disparate impact; reasonable accommodation                                |
| **State AI laws**                                 | State agencies; CO, NY, IL, CA, others                 | State-specific bias testing, disclosure                                   |
| **GAO bid-protest jurisdiction**                  | Federal procurement decisions                          | Documented evaluation basis                                               |

## OMB M-24-10 (Advancing Governance, Innovation, and Risk Management)

Issued March 2024, builds on EO 14110. Key requirements:

* Each agency designates a Chief AI Officer
* Minimum practices for **rights-impacting AI** and **safety-impacting AI**:
* Pre- and post-deployment testing with realistic conditions
* Independent evaluation
* Risk impact assessment
* Ongoing monitoring of operational risks
* Adequate staff training
* Public AI use case inventory
* Generative AI specific guardrails

**Stratix evaluation evidence supports M-24-10:**

* Pre- and post-deployment scenario suite frozen at deployment + production trace sampling
* Drift-monitoring dashboards
* Audit trail per evaluation tied to use case in agency inventory
* Independent-evaluation tier (separate Stratix project, separate reviewers)

## NIST AI RMF (AI 100-1)

Voluntary framework cited by NIST AI 600-1 (Generative AI Profile, July 2024). Agency policies adopt it via FAR clauses or internal directives.

Function-to-evidence mapping covered in [eval-patterns.md](/more-in-this-section-2/eval-patterns-3).

## EO 14110

Issued October 2023. Requires:

* AI safety and security guidelines from NIST
* Reporting requirements for foundational models above compute thresholds
* Federal-agency AI use case inventories
* Critical infrastructure AI safety standards

Most direct operational implication for agencies using Stratix: ensure use cases above the M-24-10 thresholds carry the evaluation evidence the EO requires NIST to define.

## FedRAMP / FISMA

For cloud-based AI services serving federal agencies:

* ATO required at applicable impact level (Low / Moderate / High)
* FedRAMP authorization handles the underlying cloud; agency ATO covers the application layer
* Continuous monitoring and POA\&M tracking

**Stratix posture:**

* Customers needing FedRAMP-authorized deployment should engage Enterprise — talk to <security@layerlens.ai>
* BYOK custom models for sensitive workloads keep inference within the agency's authorized boundary

## Section 508 / WCAG 2.1 AA

For citizen-facing outputs:

* Plain-language requirement (Plain Writing Act of 2010)
* Reading-level targets (most agencies set 6th-8th grade)
* Multilingual parity for limited-English-proficient populations (EO 13166)
* Screen-reader compatibility for any rendered output

**Stratix evaluation evidence:**

* Reading-level scorer applied to every citizen-facing output
* Per-language judge runs to detect quality drops in non-English

## Privacy Act of 1974

For any system-of-records on US persons:

* Notice requirements (SORN — System of Records Notice)
* Routine-use disclosure rules
* Access and correction rights for the subject

**Stratix supports:**

* Audit trail of data access
* Tenant isolation
* DPA available with Privacy Act commitments

## FOIA

5 USC 552. Federal agencies must:

* Respond within 20 working days
* Cite specific exemption subsections for any withholding
* Provide segregable non-exempt portions

State analogs (sunshine laws) often have stricter timelines.

## State and local AI laws

Selected examples (non-exhaustive):

* **NYC Local Law 144** — Bias audit for employment decision tools
* **CA AB 2273** — Age-Appropriate Design Code for online services
* **CO SB21-169** — Insurance algorithms (covers state-purchased insurance)
* **IL AI Video Interview Act** — Notice and consent for AI in interviews
* **State public-records / sunshine laws** — Vary; often shorter deadlines than FOIA

## Recommended setup

1. Enterprise tier with executed agency ATO process where applicable
2. SSO + RBAC scoped to agency role classifications
3. Use-case inventory entry per Stratix project; tied to the agency M-24-10 inventory
4. NIST AI RMF GOVERN evidence package retained per use case
5. Pre- and post-deployment scenario suite frozen at ATO; production trace sampling configured
6. Disparate-impact monitoring on every rights-impacting output
7. Audit retention matching the longer of agency records schedule or 7 years
8. Public-output reading-level rule applied to citizen-facing outputs
9. Independent evaluation for rights-impacting AI (separate project + separate reviewers)

## See also

* [Government scenarios](/more-in-this-section-2/scenarios-3)
* [Government eval patterns](/more-in-this-section-2/eval-patterns-3)
* [Concept: Multi-tenancy](/7.-observe-see-whats-happening/multi-tenancy)
* [Enterprise](/4.2-industry-use-cases/government-public-sector)
* [BYOK custom models](/5.-select-pick-the-model/byok-custom-models)


# Government cookbook and samples

Government starter samples.

## Reference

* [Concept: Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation)
* [Use case: AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd)


# Government and public sector evaluation patterns

Government evaluation patterns — primitive ratio, span rules, GEPA labeling, cadence aligned to NIST AI RMF and OMB M-24-10.

Recommended primitive ratio for government AI: **60/25/15** — deterministic rules, code assertions, judges. Public-sector AI inherits the strictest documentation and reviewability burden of any vertical because every decision can be challenged in court or audit.

## Primitive ratio

| Primitive           | Share | Why                                                                                           |
| ------------------- | ----- | --------------------------------------------------------------------------------------------- |
| Deterministic rules | 60%   | Statutory deadlines, exemption citations, eligibility rule references, FOIA exemptions        |
| Code assertions     | 25%   | Reading-level scorers, PII recall, routing-table validation, criterion coverage               |
| LLM judges          | 15%   | Plain-language quality, court-defensibility of exemption rationale, denial-reason readability |

## Pattern table

| Scenario                           | Primary primitive | Specific check                                      |
| ---------------------------------- | ----------------- | --------------------------------------------------- |
| Citizen chatbot — policy grounding | Judge             | Faithfulness to current agency rule                 |
| Citizen chatbot — reading level    | Code scorer       | Flesch-Kincaid ≤ 8                                  |
| Citizen chatbot — escalation       | Rule              | Eligibility/legal questions auto-escalate           |
| Benefits — rule citation           | Rule              | Citation to CFR/state reg present                   |
| Benefits — recommendation gate     | Rule              | Output marked "recommendation" not "decision"       |
| Benefits — fairness                | Code scorer       | Group-fairness ratio ≥ 0.8 across protected classes |
| Benefits — denial-reason quality   | Judge             | Plain-language readability for citizen              |
| FOIA — exemption citation          | Rule              | Statutory subsection present per redaction          |
| FOIA — deadline                    | Rule              | 20-day clock tracked per request                    |
| FOIA — PII recall                  | Code scorer       | ≥ 99% on labeled PII set                            |
| FOIA — over-redaction              | Judge             | Court-defensibility                                 |
| Procurement — criterion coverage   | Rule              | All solicitation criteria scored                    |
| Procurement — fairness             | Code scorer       | Bias check across business categories               |
| Public-safety — disparate impact   | Code scorer       | Required on any enforcement-influencing output      |
| Public-safety — advisory gate      | Rule              | No automated enforcement actions                    |

## Span rules

* `policy.lookup` — version of the agency rule referenced
* `eligibility.eval` — which CFR/regulation sections were applied
* `exemption.cite` — FOIA exemption subsection per redaction
* `routing.decide` — destination office/form match
* `bias.score` — disparate impact tracked per output

## NIST AI RMF mapping

The four NIST AI Risk Management Framework functions map to evaluation evidence:

| NIST function | Stratix evidence                                 |
| ------------- | ------------------------------------------------ |
| **GOVERN**    | Tier policy + RBAC + audit trail per evaluation  |
| **MAP**       | AI inventory tied to scenario suite per use case |
| **MEASURE**   | Evaluation reports + drift dashboards            |
| **MANAGE**    | Alert engine + rule pass-rate threshold gates    |

## OMB M-24-10 alignment

For federal agencies covered by OMB M-24-10 (Advancing Governance, Innovation, and Risk Management for Agency Use of AI):

* Rights-impacting and safety-impacting AI must have pre- and post-deployment testing
* Continuous monitoring with documented thresholds
* Public AI use case inventories
* Generative-AI-specific guardrails

**Stratix evaluation evidence shape:**

* Pre- and post-deployment scenario suite frozen at deployment time
* Production traces sampled and scored against same suite
* Quarterly reports on drift, fairness, accuracy

## GEPA labeling

For each government judge, label ≥ 50 paired examples:

* **Citizen-services chatbot:** Stratified across program types and languages
* **Benefits eligibility:** Mix of approval/denial/needs-more-info, labeled by senior caseworkers
* **FOIA exemption:** Labeled by FOIA officers; exemption rulings from FOIAonline / state appeals
* **Procurement:** Labeled by contracting officers
* **Public-safety:** Labeled by trained civil-rights reviewers, not just enforcement personnel

Run [judge optimization](/more-in-this-section-9/judge-optimizations-1) with `budget="medium"`. Hold out 20% for non-overlapping validation.

## Cadence

| Cadence       | Activity                                                        |
| ------------- | --------------------------------------------------------------- |
| **Per-PR**    | CI gate runs scenario suite (rules + assertions)                |
| **Daily**     | Sample production traces; full evaluation including judges      |
| **Weekly**    | Disparate-impact audit; FOIA deadline-compliance report         |
| **Monthly**   | GEPA re-optimization on judge drift                             |
| **Quarterly** | NIST AI RMF MEASURE function report; OMB M-24-10 reporting prep |
| **Annually**  | Public AI use-case inventory update; full bias re-audit         |

## See also

* [Government scenarios](/more-in-this-section-2/scenarios-3)
* [Government compliance](/more-in-this-section-2/compliance-3)
* [Recipe: GEPA basics](/9.-improve-tune-the-system/gepa-basics)


# Government and public sector scenarios

Government and public sector scenarios — citizen services, benefits, FOIA, procurement, public-safety analytics. Industry patterns, not customer case studies.

Five canonical scenarios where federal, state, and local agencies use Stratix evaluation to gate AI features into production. Patterns, not specific agencies.

## 1. Citizen-services chatbot

**Setup:** A 311 / general-services chatbot answers questions about agency programs: "How do I renew my driver's license?" "Where do I report a pothole?" "What's the income limit for SNAP?" Output references the current agency policy and routes to the right form, office, or human escalation.

**Why it's hard:**

* Information correctness has direct downstream effect — wrong income limit means an eligible family doesn't apply
* Plain-language requirements — federal Plain Writing Act and many state analogs require accessible reading levels
* Multi-language fidelity — agencies serving immigrant populations need quality across non-English outputs

**Stakes:**

* Inaccurate eligibility info is one of the most common citizen complaints in public-services audits
* ADA Title II requires that programs delivered through digital means be equally accessible to people with disabilities
* Incorrect referral wastes the citizen's time and creates downstream call-volume

**Stratix evaluation criteria:**

| Aspect              | What you measure                                      | Pattern                                 |
| ------------------- | ----------------------------------------------------- | --------------------------------------- |
| Policy grounding    | Output cites the current agency policy/program rule   | Citation existence + faithfulness judge |
| Reading level       | Flesch-Kincaid grade ≤ 8 (most federal/state targets) | Deterministic scorer                    |
| Routing accuracy    | Recommended office / form is correct for the request  | Code assertion against routing table    |
| Multilingual parity | Same accuracy across language pairs                   | Per-language judge runs                 |
| Escalation rule     | Eligibility/legal-rights questions auto-escalate      | Hard rule                               |

## 2. Benefits eligibility and case-management assistance

**Setup:** A caseworker-facing tool reads a benefits application (SNAP, Medicaid, TANF, unemployment) plus supporting docs and produces an eligibility recommendation, missing-document list, and applicable-rule citations. The caseworker decides; the AI never decides.

**Why it's hard:**

* Eligibility rules are dense, change yearly, and have state-specific overlays
* Procedural-due-process requirements mean the citizen gets to see a documented basis for any denial
* Improper denial can be appealable — and a class-action target

**Stakes:**

* Benefits-determination errors carry direct constitutional due-process exposure
* USDA, CMS, DOL audit improper-payment rates; agency states with high error rates see funding consequences
* HHS OIG has scrutinized AI-assisted eligibility decisions specifically

**Stratix evaluation criteria:**

* **Rule-citation accuracy** — Every recommendation cites the specific eligibility rule (CFR section, state regulation)
* **Caseworker-reviewability rule** — Hard rule: AI output must be marked recommendation, not decision
* **Missing-document completeness** — Code assertion: required-document list matches program rules
* **Disparate-impact monitoring** — Group-fairness scorer across protected classes, geographic regions
* **Plain-language denial reasons** — Judge: would a non-attorney citizen understand why?

## 3. FOIA and public-records request triage

**Setup:** Incoming FOIA / public-records request is parsed by AI: identify the records sought, route to the right custodian, flag exemption candidates (b)(6) personal privacy, (b)(7) law enforcement, etc., produce a draft response.

**Why it's hard:**

* Statutory deadlines (federal FOIA: 20 working days) are court-enforceable
* Exemption misapplication can be challenged and reversed
* Over-redaction is a transparency failure; under-redaction can leak PII or compromise an investigation

**Stakes:**

* DOJ FOIA litigation can force release plus award attorneys' fees against the agency
* Wrongful disclosure of (b)(6) PII triggers Privacy Act liability
* State sunshine-law analogs often have stronger transparency presumptions

**Stratix evaluation criteria:**

* **Exemption-citation accuracy** — Every redaction cites the statutory subsection
* **Deadline-tracking rule** — Hard rule: response queue tracks 20-day clock per request
* **PII-redaction recall** — Deterministic scorer ≥ 99% on a labeled PII set
* **Over-redaction guard** — Judge: would a court of appeal find this exemption defensible?
* **Routing accuracy** — Code assertion: custodian assignment matches records type

## 4. Public procurement evaluation

**Setup:** AI assistant reads vendor proposals, scores them against the solicitation criteria, drafts evaluation memos. Government buyer uses the output, doesn't outsource to it.

**Why it's hard:**

* FAR (federal) / state procurement codes require documented basis for every award decision
* Bid protests at GAO, COFC, and state equivalents test the documentation directly
* Bias in scoring — if model favors certain proposal styles, it disadvantages small/disadvantaged businesses

**Stakes:**

* A successful bid protest can vacate an award
* Federal FAR Part 33 and state analogs require fair, impartial evaluation — AI bias is now an examined question
* Small Business Administration set-aside programs require documented compliance

**Stratix evaluation criteria:**

* **Criterion-coverage rule** — Every solicitation criterion is scored; missing criteria fails the trace
* **Score-justification citation** — Each score cites specific proposal text
* **Bias monitoring** — Group fairness across business size, location, ownership classifications
* **Reproducibility rule** — Same proposal scored twice produces same score (within threshold)

## 5. Public-safety analytics

**Setup:** Crime-pattern analysis, traffic-incident hotspots, code-enforcement triage. AI surfaces patterns; human officials decide actions.

**Why it's hard:**

* Predictive-policing concerns — historical data encodes historical bias; deploying it forward perpetuates disparate impact
* Constitutional rights (Fourth Amendment, Equal Protection) constrain what's permissible
* Many states/cities have outright banned predictive policing or specific facial-recognition uses

**Stakes:**

* DOJ Civil Rights Division consent decrees in cities have specifically named algorithmic policing
* Class-action exposure under §1983 for algorithmic bias
* Public trust collapses fast if a community perceives the AI as targeting them

**Stratix evaluation criteria:**

* **Disparate-impact scorer** — Required for any AI output influencing enforcement allocation
* **Constitutional-rights gate** — Hard rule: outputs must be advisory to human decisions, never automated enforcement
* **Source-data disclosure** — Every output cites the data sources used
* **Drift-monitoring** — Bias scores tracked over time; retraining triggered on threshold breach
* **Redaction at output** — PII never appears in operational outputs without authorized purpose

## See also

* [Government compliance](/more-in-this-section-2/compliance-3)
* [Government eval patterns](/more-in-this-section-2/eval-patterns-3)
* [Concept: Judges](/8.-evaluate-score-the-outputs/judges-1)


# Healthcare compliance

Healthcare compliance — HIPAA, BAA, audit retention, PHI handling, FDA SaMD considerations.

## Regulatory frame

Clinical AI sits at the intersection of three regulatory regimes. Each has its own evaluation evidence requirement.

| Regime                          | Applies when                                                                                        | Evaluation-evidence shape                               |
| ------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
| **HIPAA / BAA**                 | Anything touching Protected Health Information (PHI)                                                | Tenant-isolation, audit log, PHI handling controls      |
| **FDA SaMD**                    | The AI makes or substantially supports a clinical decision (Class II+ Software as a Medical Device) | Clinical-validation evidence, change-control records    |
| **State Medical Practice Acts** | The AI's output could be construed as practicing medicine                                           | Disclaimers, scope-of-use constraints, escalation paths |

Stratix supports the evidence side of all three. It does not certify your product — that's between you and your regulator. It produces the artifacts that make certification tractable.

## HIPAA / BAA

Stratix Enterprise supports a Business Associate Agreement.

**Required for production:**

1. Executed BAA with LayerLens
2. PHI redaction at trace ingestion (in your application code, not Stratix's)
3. SSO with role-based access (no shared logins)
4. Extended audit retention for the period your record-retention policy requires
5. Data residency in a region your security team has approved

**Discuss with** [**sales@layerlens.ai**](mailto:sales@layerlens.ai) **and** [**legal@layerlens.ai**](mailto:legal@layerlens.ai) to start the BAA workflow. See **DPA process** (Enterprise tier — contact sales) for the legal mechanics.

## PHI handling

PHI must never enter Stratix trace bodies in plaintext unless your BAA covers it AND your security team has approved the data flow.

**Recommended pre-trace redaction:**

* Names → `<PATIENT_NAME>` token
* DOB → age-band only
* MRN / SSN / addresses → category tokens
* Free-text clinical notes — entity-level redaction (Presidio, AWS Comprehend Medical, or equivalent) before the trace leaves your boundary

The redaction happens **client-side**, before Stratix sees the data. Stratix's [PHI-redaction rule pattern](/more-in-this-section-2/eval-patterns-4) then verifies redaction was effective by scoring traces for any leftover PII.

## FDA Software as a Medical Device (SaMD)

If your product makes or substantially supports a clinical decision, FDA may classify it as SaMD. The classification (Class I, II, or III) drives the evidence burden.

**Stratix's role in the SaMD evidence pack:**

* **Pre-market validation** — agentic evaluations against curated clinical trace sets become the validation evidence
* **Change control** — every release pin cites the evaluation ID that gated it; before-and-after evaluations document the impact of any change
* **Post-market surveillance** — continuous trace evaluations on production traffic become the post-market quality monitoring artifact

Talk to your FDA regulatory consultant about how the evaluation evidence Stratix produces maps to your specific submission shape (510(k), De Novo, PMA).

## Audit retention

Enterprise tier supports configurable audit retention. Common configurations:

| Use                                                   | Retention                                     |
| ----------------------------------------------------- | --------------------------------------------- |
| HIPAA-compliant operational logs                      | 6 years (HIPAA minimum)                       |
| FDA-regulated clinical-AI change logs                 | Lifecycle of the product + post-market period |
| Quality audit trail (eval IDs cited in release notes) | Indefinite                                    |

Configure during onboarding; retention period is a contractual term.

## State medical practice acts

State boards regulate "practice of medicine." AI that crosses certain thresholds (diagnosis, prescription, treatment recommendation) may be regulated as such.

**Mitigations Stratix's evaluation framework supports:**

* **Disclaimer-presence rule** on patient-facing outputs (state-specific)
* **No-diagnosis rule** that a triage assistant must not assert a specific diagnosis
* **Escalation rule** to a licensed clinician for any output above configured risk

These are documentation patterns, not legal opinions. Consult counsel for your specific deployment.

## Data residency

Enterprise: choose US, EU, or other available region for tenant data. Healthcare customers typically pin to a region matching their patient population to avoid cross-border data transfers.

## Recommended setup

1. Enterprise tier with executed BAA
2. PHI redaction at trace ingestion (your code)
3. SSO with role-based access
4. Trace evaluations with evaluation IDs cited in release notes
5. Continuous trace evaluation on production traffic for post-market quality monitoring
6. Data residency configured to your region of record
7. Audit retention configured to your record-retention policy

## See also

* [Enterprise](/4.2-industry-use-cases/healthcare)
* **Enterprise — DPA process** (Enterprise tier — contact sales)
* [Security and compliance](/4.2-industry-use-cases/healthcare)
* [Concept: Multi-tenancy](/7.-observe-see-whats-happening/multi-tenancy)
* [Industry pattern: healthcare contraindicated cocktail](/4.2-industry-use-cases/pattern)


# Healthcare cookbook and samples

Healthcare cookbook starter samples.

* [Clinical Q\&A safety judge](/more-in-this-section-2/industry-healthcare-clinical-qa)

## Starter eval space

```python
client.spaces.create(
 name="healthcare-clinical-qa",
 judges=["clinical-faithfulness", "reasoning-soundness"],
 rules=["no-dose-recommendation", "phi-redaction"],
 dataset_id="your-clinical-dataset"
)
```

## Reference

* [Use case: RAG evaluation](/4.1-general-use-cases/rag-evaluation)
* [Concept: Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation)




---

[Next Page](https://docs.layerlens.ai/llms-full.txt/1)