> This is part 1 of 5 of the full documentation (pages 1–100 of 426). > The content is paginated: fetch every part to see all of it. > Next part: https://docs.layerlens.ai/llms-full.txt/1 > Page index: https://docs.layerlens.ai/llms.txt # LayerLens Documentation Welcome to LayerLens documentation. One platform — pick a path and ship faster. LayerLens is the AI evaluation platform for teams that need to know — before customers do — whether a model, prompt, or agent is good enough to ship. Run private evaluations on your own data, score traces with deterministic rules and LLM graders, and ship AI you can trust. ## Customer-facing experiences {% tabs %} {% tab title="LayerLens workspace" %} **`app.layerlens.ai` — after you sign in.** The logged-in workspace where teams run private evaluations, build graders, score traces, manage graders, run agentic evaluations, and govern AI quality across an organization. [Start with the LayerLens workspace →](/2.-get-started/premium) {% endtab %} {% tab title="LayerLens SDK" %} **`pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]"`** — the Python client SDK for programmatic evaluation, grader orchestration, and trace ingestion. 138 sample programs. [Start with the SDK →](/2.-get-started/sdk) {% endtab %} {% endtabs %} ## Pick a path
I'm a builderShip faster with evaluations wired into your dev loop.Builder path
I'm an operatorRun live AI in production with quality gates and continuous evaluation.Operator path
I'm a researcherExplore 175+ models against 52+ benchmarks.Researcher path
I'm an admin or buyerSet up your org, manage seats and credits, evaluate enterprise readiness.Admin/Buyer path
## The LayerLens Workflow Five stages from raw model to governed production. Every capability maps to one of them. {% hint style="info" %} **Select → Build → Observe → Evaluate → Improve.** This is the spine. Every concept, how-to, tutorial, and recipe in this documentation pins to one of these stages. [Learn the workflow →](/1.-introduction/the-layerlens-workflow) {% endhint %} ## What's new * **Agentic evaluations** — pre- and post-deployment quality gates for multi-step agents. [Read the announcement](/4.1-general-use-cases/agentic-evals-overview). * **GEPA grader optimization** — automatically tune your LLM graders against ground-truth labels. [Read the concept](/8.-evaluate-score-the-outputs/judges-1#grader-optimization-gepa). ## Need help? * **In-app:** click the Assistant icon for context-aware help. * **Email:** * **Docs feedback:** open an issue or reach out via the in-app feedback form. ## Where to next * New here? → [What is LayerLens?](/1.-introduction/what-is-layerlens) * Want to ship today? → [Getting Started](/2.-get-started/02-get-started) * Researching for purchase? → [How LayerLens compares](/1.-introduction/how-layerlens-compares-deep) * Building a business case? → [Pricing](/) # Quickstart 5-minute quickstart — pick a path and ship. You're in a hurry. Pick the line that matches you. ## "I want to evaluate my AI on my data" ```bash pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]" export LAYERLENS_STRATIX_API_KEY="..." ``` ```python from layerlens import Client client = Client() model = client.models.get_by_key("openai/gpt-4o") benchmark = client.benchmarks.get_by_key("mmlu") e = client.evaluations.create(model=model, benchmark=benchmark) print(client.evaluations.wait_for_completion(e).accuracy) ``` [Sign up](/2.-get-started/sign-up) → [First evaluation](/2.-get-started/first-evaluation). ## "I just want to verify my key works (curl, language-agnostic)" ```bash curl -H "X-API-Key: $LAYERLENS_STRATIX_API_KEY" \ "https://api.layerlens.ai/api/v1/models" ``` JSON back → you're good. 401 → see [Authentication](/more-in-this-section-9/auth). ## "I want to gate AI changes in CI" [Tutorial 3: CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates). 30 minutes from "no gate" to "regression blocks merge." ## "I want to score live production traces" [Tutorial 4: Score live traces](/8.-evaluate-score-the-outputs/04-score-traces). Continuous evaluation on real traffic. ## "I'm evaluating LayerLens for my company" * [What is LayerLens?](/1.-introduction/what-is-layerlens) * [How LayerLens compares](/1.-introduction/how-layerlens-compares-deep) * [Pricing](/3.-pricing/03-pricing) * [Industry patterns](/4.2-industry-use-cases/travel-hospitality) ## Got 30 seconds? LayerLens is the AI evaluation platform. Private workspace at `app.layerlens.ai`. Python SDK on the LayerLens package index. Read the [README](/). # Glossary Definitions for every LayerLens term — models, evaluations, judges, scorers, traces, and more. ## Agentic evaluation Pre- and post-deployment evaluation of multi-step AI agents. Combines natural-language assertions, deterministic rules, and LLM graders to verify that an agent reaches a correct state, takes legal actions along the way, and satisfies regression criteria. Surfaced in the LayerLens workspace under **Agent Evaluations**. See [Agentic evaluations](/4.1-general-use-cases/agentic-evals-overview). ## Benchmark A standardized dataset and grading procedure used to measure model performance. LayerLens hosts 52+ benchmarks (MMLU, HumanEval, GSM8K, etc.) in its catalog. See [Benchmarks](/5.-select-pick-the-model/benchmarks). ## BYOK (Bring Your Own Key) The ability to use your own LLM provider API keys with LayerLens. Two scopes: * **BYOK custom models** — register an OpenAI-compatible endpoint as a custom model the workspace can target. * **BYOK provider keys** — supply your own OpenAI/Anthropic/Google keys for evaluations that consume tokens. See [BYOK custom models](/5.-select-pick-the-model/byok-custom-models) and [BYOK models concept](/5.-select-pick-the-model/byok-models). ## Compare models A LayerLens workspace feature that puts two models head-to-head across a benchmark or your own dataset. See [Compare evaluations](/8.-evaluate-score-the-outputs/evaluations). ## Continuous evaluation The practice of running evaluations on a recurring schedule against live traffic — not only at release time. LayerLens supports continuous evaluation via trace-evaluations on real production traces. See [Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation). ## ECU (Evaluation Compute Unit) The LayerLens billing unit for compute-intensive operations: model inference, grader runs, GEPA optimization, etc. ECU is pay-as-you-go. See [ECU credits](/11.-admin/concepts-ecu-credits). ## Evaluation A run that scores model output against a benchmark or dataset. Results are stored, browsable, and shareable, and scoped to your organization in `app.layerlens.ai`. See [Evaluations](/8.-evaluate-score-the-outputs/evaluations-1). ## GEPA grader optimization Automatic tuning of an LLM grader's prompt against a labeled ground-truth set. GEPA finds the prompt that best matches your labels, raising grader agreement with humans. See [Grader optimization (GEPA)](/8.-evaluate-score-the-outputs/judges-1#grader-optimization-gepa). ## Grader Anything that scores an output. Judges and scorers are being merged into one concept, the grader. LLM graders score subjective dimensions (helpfulness, accuracy, faithfulness) with a model and a rubric; you can version them, optimize them with GEPA, and apply them to evaluations and traces. Code graders run deterministic checks (regex, JSON-schema validation, exact match) and suit objective dimensions. See [Graders](/8.-evaluate-score-the-outputs/judges-1). In the API, SDK and CLI the resources keep their current names for now: `judges` and `scorers`. ## Model An LLM (e.g., GPT-5.3, Claude Opus 4.6, Gemini 3.1 Pro). LayerLens catalogs 175+ models with metadata, benchmark scores, and per-context information. See [Models and benchmarks](/5.-select-pick-the-model/models-and-benchmarks). ## Multi-org / Organization LayerLens supports users belonging to multiple organizations. After sign-in, the user picks an active organization. Org-switch updates the session and reloads org-scoped data. See [Organizations](/more-in-this-section-9/organizations-1). ## Quarterly Reports Long-form research reports (Q1-Q4) summarizing model performance, benchmark winners, and AI evaluation trends. See [Quarterly reports](/5.-select-pick-the-model/quarterly-reports). ## Span A child unit of work within a trace. A span represents one tool call, one LLM call, or one logical step. See [Traces and spans](/6.-build-wire-your-code/traces-and-spans). ## LayerLens workspace The logged-in workspace at `app.layerlens.ai`. Run private evaluations, build graders, score traces, manage graders, run agentic evals, govern AI quality. ## Trace A record of an AI call (or a chain of calls) — inputs, outputs, latencies, costs, errors, and any spans. Traces are uploaded to the LayerLens workspace for inspection and trace-evaluation. See [Traces and spans](/6.-build-wire-your-code/traces-and-spans). ## Trace evaluation A scoring run applied to ingested traces using one or more graders. Lets you grade real production traffic. See [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations). *** Seeded from the [LayerLens "25 AI evaluation terms" blog post](https://layerlens.ai/blog) and the live product surface. If a term is missing, open an issue or PR. # Search & Persona Index Persona and goal index — find the right page in seconds. ## By persona ### Builder *"I'm building an AI feature and want it to be good."* * [Your first evaluation](/2.-get-started/first-evaluation) * [Your first grader](/2.-get-started/first-judge) * [Tutorial: First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation) * [Workflow: Instrument](/9.-improve-tune-the-system/workflow) * [Cookbook recipes](/2.-get-started/all-cookbook-recipes) * [Python SDK reference](/6.-build-wire-your-code/sdk-python) * [Integrations](/10.-integrate-connect-your-stack/10-integrate) ### Operator *"I run live AI in production and need to keep it healthy."* * [Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation) * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) * [Workflow: Observe](/9.-improve-tune-the-system/workflow) * [Workflow: Evaluate](/9.-improve-tune-the-system/workflow) * [AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd) * [Tutorial: Wire CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates) * [Status and reliability](/13.5-operational/status) ### Researcher *"I want to know which model is best for my use case."* * [Models](/5.-select-pick-the-model/models) * [Compare evaluations](/8.-evaluate-score-the-outputs/evaluations) * [Quarterly reports](/5.-select-pick-the-model/quarterly-reports) * [Benchmarks](/5.-select-pick-the-model/benchmarks) ### Admin *"I'm setting up LayerLens for my team."* * [Sign up](/2.-get-started/sign-up) * [Organizations and multi-org](/more-in-this-section-9/organizations-1) * [Team management](/11.-admin/team-management) * [ECU credits and billing](/11.-admin/ecu-credits-billing) * [Settings](/11.-admin/settings) * [Enterprise](/more-in-this-section-9/screenshots-1) * [Security and compliance](/more-in-this-section/trust) ### Buyer *"I'm evaluating LayerLens for purchase."* * [What is LayerLens?](/1.-introduction/what-is-layerlens) * [Three experiences](/1.-introduction/three-experiences) * [How LayerLens compares](/1.-introduction/how-layerlens-compares-deep) * [Pricing](/3.-pricing/03-pricing) * [Industry patterns](/4.2-industry-use-cases/travel-hospitality) * [Use cases](/4.1-general-use-cases/general) * [Enterprise](/more-in-this-section-9/screenshots-1) * [Security and compliance](/more-in-this-section/trust) ## By goal | I want to... | Go to | | ------------------------------- | ------------------------------------------------------------------ | | Browse 175+ models | [Models](/5.-select-pick-the-model/models) | | Compare two models head-to-head | [Compare evaluations](/8.-evaluate-score-the-outputs/evaluations) | | Run my first evaluation | [Tutorial 1](/8.-evaluate-score-the-outputs/01-first-evaluation) | | Build a custom LLM grader | [Tutorial 2](/8.-evaluate-score-the-outputs/02-first-judge) | | Optimize a grader with GEPA | [Tutorial 5](/9.-improve-tune-the-system/05-gepa-optimize) | | Wire evals into CI/CD | [Tutorial 3](/6.-build-wire-your-code/03-cicd-gates) | | Evaluate live traces | [Tutorial 4](/8.-evaluate-score-the-outputs/04-score-traces) | | Use my own LLM key | [BYOK custom models](/5.-select-pick-the-model/byok-custom-models) | | Migrate from LangSmith | [From LangSmith](/6.-build-wire-your-code/from-langsmith) | | Migrate from Braintrust | [From Braintrust](/6.-build-wire-your-code/from-braintrust) | | Set up SSO | **Enterprise SSO** (contact sales) | | Understand pricing | [Pricing](/3.-pricing/03-pricing) | | File a support ticket | [Support](/12.-support/12-support) | ## By source surface | Surface | Section | | ------------------------------ | ------------------------------------------------------------------------- | | `app.layerlens.ai` (logged-in) | [LayerLens workspace](/more-in-this-section-3/premium-workspace-overview) | | Python SDK | [Python SDK](/6.-build-wire-your-code/sdk-python) | | REST API | [API reference](/13.1-sdk-and-apis/api) | | CLI | [CLI reference](/13.1-sdk-and-apis/cli) | | VS Code extension | [VS Code](/6.-build-wire-your-code/vscode-extension) | # What is LayerLens? LayerLens is the AI evaluation platform — private workspace and SDK. LayerLens is a complete platform for evaluating AI — from the moment you pick a model, through the build, into production, and across an organization. The platform is structured as **two product lines** (Model Evaluations and Agentic Evaluations) delivered through **two customer-facing experiences** (the LayerLens workspace and the Python SDK) that share one catalog, one grader engine, and one trace pipeline. If you came in via the marketing site, you've seen the two product lines. If you're starting from this documentation, you'll mostly hear about the two experiences. They reconcile: the products are *what* you can do; the experiences are *where* you do it. ## High Level Overview Most AI teams ship by intuition. They pick a model because it's popular. They iterate prompts because the last few examples felt better. They watch CSAT in production because that's the only signal that survives. By the time they know something's wrong, customers have already noticed. LayerLens gives you a real signal. Two of them, actually: 1. **Private workspace** — run evaluations on your own data, build LLM graders that match your team's quality bar, and score live traces. Stop guessing whether the new model is better; measure it. 2. **CI/CD gates** — wire evaluations into your pipeline. Block regressions. Promote winners. Ship with the same confidence you ship traditional code. ## The two product lines (what you can do) * **Model Evaluations** — model + benchmark + score. Use it to pick the right model and to track how the frontier is moving. * **Agentic Evaluations** — pre- and post-deployment quality gates for multi-step AI agents: assertions, deterministic rules, and LLM graders combined into a verdict + root-cause report. Both share the same trace pipeline, the same graders. You don't pick "one product" — you pick the workflow. ## What ships today * **Models:** 200+ in the catalog, including GPT-5.3, Claude Opus 4.6, Gemini 3.1 Pro/Flash. * **Benchmarks:** 52+ public benchmarks (MMLU, HumanEval, GSM8K, AGIEval, MATH, etc.). * **Compare models:** head-to-head model comparisons across any benchmark. * **Graders:** code graders for objective dimensions, and LLM graders for subjective dimensions with the GEPA optimizer. * **Traces:** upload trace JSON; run trace-evaluations against graders. * **ECU credits:** pay-as-you-go billing for compute-intensive runs. * **BYOK custom models:** point at your own OpenAI-compatible endpoint. * **Multi-org:** users belong to multiple orgs; switch from the picker. * **Auth:** GitHub OAuth, Google OAuth, email/password. * **Search, notifications, onboarding wizard.** ## Where to next * [Workspace and SDK](/1.-introduction/three-experiences) — side-by-side * [Agentic evaluations](/4.1-general-use-cases/agentic-evals-overview) — the pre- and post-deployment workflow * [Getting started](/2.-get-started/02-get-started) — pick a path and ship # Overview Overview of LayerLens — the workspace and SDK, the workflow spine, what's in the platform. LayerLens is the AI evaluation platform for teams who need to know — before customers do — whether a model, a prompt, or an agent is good enough to ship. Two customer-facing experiences cover the full evaluation lifecycle: the **LayerLens workspace** for team workflows and the **Python SDK** for programmatic access. {% hint style="info" %} **Want to see LayerLens in action?** [Schedule a 30-minute demo →](/more-in-this-section/schedule-a-demo) — or jump straight into the [LayerLens workspace](/2.-get-started/sign-up). {% endhint %}
What is it?High Level Overview and the "two products" framing.Read
Workspace and SDKHow the workspace and the SDK fit together.Read
Agentic evaluationsPre- and post-deployment quality gates for multi-step agents.Read
Platform architectureThe components: catalog, evaluation engine, graders, traces.Read
Personas and pathsBuilder, Operator, Researcher, Admin, Buyer — pick yours.Read
Quarterly reportsQ1-Q4 research reports on model performance and benchmark trends.Read
How LayerLens comparesLayerLens vs Braintrust, LangSmith, Galileo, Promptfoo, Arize.Read
Trust and securityEncryption, compliance, tenant isolation, audit, BYOK.Read
Schedule a demoBook a 30-minute walk-through — or start hands-on today.Read
Brand assetsProduct names, logos, voice, and boilerplate for partners.Read
## The LayerLens Workflow Six stages from picking a model to governing it in production: 1. **Select** — review leaderboards and benchmarks; optionally compare candidates side-by-side. 2. **Build** — wire your code or agent to emit traces. 3. **Observe** — see real production behavior across surfaces. 4. **Evaluate** — score outputs with LLM graders and code graders. 5. **Improve** — tune prompts and graders (GEPA); re-run. 6. **Govern** — enforce gates in CI/CD and across your org. Every concept, how-to, tutorial, and recipe in these docs maps to one of these stages. [Learn the workflow →](/1.-introduction/the-layerlens-workflow) # Why evaluate? Why AI teams need evaluation discipline — the cost of shipping unmeasured AI. Most teams ship AI by intuition. They pick a model because it's popular. They iterate prompts because the last few examples felt better. They watch CSAT in production because that's the only signal that survives. By the time they know something's wrong, customers have noticed. ## The cost of unmeasured AI * **Wrong model from the start.** A 4-week feature build gets thrown out when the foundation model can't do the task. * **Silent regressions.** A prompt change five PRs ago broke an edge case nobody noticed; customers find it first. * **Vendor migrations driven by hunches.** Switching from one provider to another without quantitative justification. * **Eval-by-eyeballing.** A handful of cherry-picked examples drive ship/no-ship decisions. Evaluation discipline gets you out of all four traps. ## What evaluation discipline looks like 1. **Quantitative model selection.** You pick a model because it scored higher on the benchmarks closest to your task. Not because it was on the front page of HN. 2. **Versioned regression suites.** Every prompt change runs against a known set with a known baseline. 3. **Continuous scoring on production traffic.** When live behavior drifts, you know before customers do. 4. **Auditable evidence.** Releases cite the evaluation that gated them. ## What it takes * A shared **mental model** for evaluation (this documentation) * A **catalog** to anchor your model choices * A **workspace** to run private evaluations ([LayerLens workspace](/more-in-this-section-3/premium-workspace-overview)) * **Automation** to wire it into CI/CD ([SDK](/6.-build-wire-your-code/sdk-python), [CLI](/13.1-sdk-and-apis/cli)) ## Three failure modes LayerLens is shaped to prevent 1. **The "hopeful" launch.** Shipping without quantitative evidence the AI works. 2. **The slow regression.** Quality erosion over a quarter that nobody catches because nobody's watching the trend. 3. **The vendor lock-in mistake.** Picking a model based on superficial features and discovering, six months later, that switching is now expensive. ## Where to next * [What is LayerLens?](/1.-introduction/what-is-layerlens) * [Workflow](/1.-introduction/the-layerlens-workflow) * [Use cases](/4.1-general-use-cases/general) * [Industry patterns](/4.2-industry-use-cases/travel-hospitality) # Personas and paths Five personas, five paths — Builder, Operator, Researcher, Admin, Buyer. LayerLens is read by five kinds of people. Each one has a different first question and a different "ah-ha" moment. Pick your path; we'll get you there fast. ## Builder > *"I'm shipping an AI feature and I want it to be good."* You write code. Your name is on the PR that adds the new prompt or the new agent. You want a tight feedback loop: write a change, run an eval, see the score change. **First questions:** "How do I write my first eval? How do I score outputs my eyes can't grade?" **Path:** 1. [Sign up](/2.-get-started/sign-up) 2. [Tutorial 1: First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation) 3. [Tutorial 2: Build your first grader](/8.-evaluate-score-the-outputs/02-first-judge) 4. [Workflow: Instrument](/9.-improve-tune-the-system/workflow) → [Evaluate](/9.-improve-tune-the-system/workflow) 5. [SDK quickstart](/2.-get-started/sdk) 6. [Cookbook](/2.-get-started/all-cookbook-recipes) — recipes by problem, framework, and integration **You'll know you're winning when:** every prompt change has an eval result attached and you've stopped trusting your gut. ## Operator > *"I run live AI in production and I need to keep it healthy."* You own the on-call. You watch dashboards. You ship rollback scripts. AI quality is your latest reliability surface. **First questions:** "How do I score real production traffic? How do I get paged when quality drifts?" **Path:** 1. [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation) 2. [Concept: Traces and spans](/6.-build-wire-your-code/traces-and-spans) 3. [Tutorial 4: Score live traces](/8.-evaluate-score-the-outputs/04-score-traces) 4. [Workflow: Observe](/9.-improve-tune-the-system/workflow) → [Govern](/9.-improve-tune-the-system/workflow) 5. [Use case: AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd) 6. [Status and reliability](/13.5-operational/status) **You'll know you're winning when:** quality regressions surface to you before they surface to customers. ## Researcher > *"I want to know which model is best for my use case."* You don't (yet) need to wire anything up. You need ground truth — what's the frontier, what's commoditized, what's punching above its weight. **First questions:** "Which model is best at structured-output tasks? How is open-weight catching up?" **Path:** 1. [Sign up](/2.-get-started/sign-up) 2. [Models](/5.-select-pick-the-model/models) 3. [Benchmarks](/5.-select-pick-the-model/benchmarks) 4. [Compare evaluations](/8.-evaluate-score-the-outputs/evaluations) — head-to-head 5. [Quarterly reports](/5.-select-pick-the-model/quarterly-reports) 6. [Run a private evaluation](/2.-get-started/first-evaluation) on your own data **You'll know you're winning when:** you can answer "is X actually better than Y for our task" in under 5 minutes. ## Admin > *"I'm setting up LayerLens for my team."* You provision seats, manage SSO, watch the bill, write the IT-approval memo. **First questions:** "How does auth work? How do I manage multiple orgs and seats? What's the data residency story?" **Path:** 1. [Sign up](/2.-get-started/sign-up) 2. [Organizations and multi-org](/more-in-this-section-9/organizations-1) 3. [Team management](/11.-admin/team-management) 4. [ECU credits and billing](/11.-admin/ecu-credits-billing) 5. [Settings](/11.-admin/settings) 6. [Enterprise](/1.-introduction/01-introduction) — SSO, data residency, 7. [Security and compliance](/1.-introduction/01-introduction) **You'll know you're winning when:** auth, billing, and access control are boring. ## Buyer > *"I'm evaluating LayerLens for purchase."* You may not be the user. You're the budget owner, the AI-platform PM, the head of engineering. You need to know whether LayerLens is the right bet — and whether your team will adopt it. **First questions:** "How does LayerLens compare to Braintrust/LangSmith/etc? What's the ROI? Who else uses it?" **Path:** 1. [What is LayerLens?](/1.-introduction/what-is-layerlens) 2. [Workspace and SDK](/1.-introduction/three-experiences) 3. [How LayerLens compares](/1.-introduction/how-layerlens-compares-deep) 4. [Pricing](/1.-introduction/01-introduction) 5. [Industry patterns](/4.2-industry-use-cases/travel-hospitality) 6. [Use cases](/4.1-general-use-cases/general) 7. [Enterprise](/1.-introduction/01-introduction) 8. [Security and compliance](/1.-introduction/01-introduction) **You'll know you're winning when:** you have a one-page rationale and your team has run a real evaluation in the trial. ## Where to next * [Getting started](/2.-get-started/02-get-started) — pick a path and ship * [Workflow](/1.-introduction/the-layerlens-workflow) — the spine that ties all five personas together # Workspace and SDK The LayerLens workspace and the Python SDK — what each one is for and how they work together. LayerLens is delivered as two customer-facing experiences. They share one catalog, one grader engine, and one trace pipeline — but each one is shaped for a specific kind of user and a specific kind of question. ## Side by side | | LayerLens workspace | Python SDK | | ---------------------------- | -------------------------------------- | --------------------------------------------------------------------------------- | | **Where** | `app.layerlens.ai` — after you sign in | `pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]"` | | **Auth** | OAuth (Google, GitHub) + email | API key (`X-API-Key`) | | **Audience** | builders, operators, admins | builders, automation, CI/CD | | **What you do** | build, evaluate, govern | automate, integrate, batch | | **Catalog visibility** | full public + private | full public + private | | **Private evaluations** | yes | yes | | **Build graders** | yes | yes | | **Trace upload** | yes | yes | | **GEPA grader optimization** | yes | yes | | **Org and team management** | yes | n/a | | **Cost** | ECU PAYG | ECU PAYG | ## LayerLens workspace Logged-in. Multi-org. Pay-as-you-go on compute. Built for teams shipping AI features. You can: * Run **private evaluations** on your own data * Build **LLM graders** for subjective dimensions and optimize them with **GEPA** * Upload **traces** and run **trace evaluations** against graders * Manage **graders** (code graders) at the org level * Run **agentic evaluations** — pre- and post-deployment quality gates for multi-step agents * Use **BYOK custom models** — register your own OpenAI-compatible endpoint * Manage **organizations**, multi-org membership, and team seats * Buy and consume **ECU credits** The left rail is: Dashboard, Traces, Model Evaluations, Agent Evaluations, Environments, Models, Benchmarks, Graders, Admin (Collaborators, Billing, Integrations, API Keys, Providers), Notifications. The in-app **Assistant** offers context-aware help. ## Python SDK — the automation surface `pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]"`. 11 resources, 138 sample programs. You can: * Programmatically create evaluations, graders, traces * Run trace evaluations as part of CI/CD * Sync results into your own data warehouse or BI * Automate grader GEPA optimization * Integrate with frameworks (LangChain, LlamaIndex, Haystack, etc.) via instrumentation samples The SDK is sync-and-async. `Client().evaluations.create()` and `await AsyncClient().evaluations.create()` both work. ## How they work together A typical team uses both: 1. **A builder** logs into the workspace, runs a private evaluation on the team's own dataset, and chooses a winner. They build a grader that captures the team's quality bar. 2. **A platform engineer** wires the SDK into CI/CD: every PR that changes a prompt re-runs the evaluation. Graders gate the merge. 3. **An operator** ingests live traces and runs trace evaluations on a daily cadence. Quality regressions surface in a dashboard before they surface to customers. ## Where to next * [LayerLens workspace](/more-in-this-section-3/premium-workspace-overview) * [Python SDK](/6.-build-wire-your-code/sdk-python) * [Getting started](/2.-get-started/02-get-started) — choose a path and ship # The LayerLens Workflow — the six intents The LayerLens Workflow — six stages from picking a model to governing it in production. Six stages from "I need to pick a model" to "this AI is governed in production." Every concept, how-to, tutorial, and recipe in these docs maps to one of them.
1. SelectReview leaderboards and benchmarks; optionally compare candidates side-by-side.Read
2. InstrumentWire your code to emit traces.Read
3. ObserveSee real production behavior.Read
4. EvaluateScore with LLM graders and code graders.Read
5. ImproveTune prompts, graders, models. Re-run.Read
6. GovernEnforce gates in CI/CD and across the org.Read
## Why a workflow spine Without a spine, evaluation is a grab-bag of features. With a spine: * Each stage has a clear question it answers * The path from "picking a model" to "shipping confidently" is named * Documentation links naturally between stages ## How the stages relate The stages aren't strictly sequential; they're a cycle. Your AI is always at some stage, and the cycle goes around as it matures: ``` Select → Build → Observe → Evaluate → Improve → Govern ↑ └─ back to Observe ``` The exception: **Select** can also reappear mid-cycle when a new model generation drops or your evaluation surfaces a model that's outgrown its envelope. Returning to Select means re-running the side-by-side and confirming your incumbent is still the right choice. ## Pick a starting stage | If you're... | Start at | | ------------------------ | --------------------------------------------------------------------------- | | Thinking about adding AI | [Select](/9.-improve-tune-the-system/workflow) — pick a model | | Already shipping | [Observe](/9.-improve-tune-the-system/workflow) — what's it actually doing? | | Mid-build | [Evaluate](/9.-improve-tune-the-system/workflow) — score it | | In production | [Govern](/9.-improve-tune-the-system/workflow) — gate it | ## Where to next * [Select](/9.-improve-tune-the-system/workflow) * [Concepts](/13.3-concepts-library-+-architecture/concepts-library) * [Tutorials](/2.-get-started/all-tutorials) # How LayerLens compares LayerLens vs Braintrust, LangSmith, Galileo, Promptfoo, Arize — the short version. The AI evaluation space is crowded. This page is the short, honest comparison. The deep version with feature-by-feature tables lives in [How LayerLens compares (deep)](/1.-introduction/how-layerlens-compares-deep). ## What's distinctly LayerLens These are the things you can do on LayerLens today that are **not** equivalent on the alternatives: * **Quarterly research reports.** Long-form, dated, methodologically transparent. * **Industry pages.** 13 industries, scenario-mapped to platform capabilities. * **Migration guides** from LangSmith, Braintrust, Galileo, Promptfoo, MLflow. ## What the alternatives do that we don't (yet) Honesty up front: * **Some have larger out-of-the-box framework integrations.** We ship a focused set; see [Integrations](/1.-introduction/01-introduction). When we don't have a turnkey integration, we have a sample. ## Honest competitive table | Capability | LayerLens | Braintrust | LangSmith | Galileo | Promptfoo | Arize | | ------------------- | ---------- | ---------- | --------- | ------- | --------- | ------- | | Quarterly research | yes | no | no | no | no | no | | Migration guides | 6 | minimal | minimal | minimal | minimal | minimal | | LLM-as-a-judge | yes | yes | yes | yes | yes | yes | | Grader optimization | yes (GEPA) | partial | no | partial | no | no | | Trace ingestion | yes | yes | yes | yes | partial | yes | | Trace evaluation | yes | yes | yes | yes | partial | yes | | Compare models | yes | partial | no | partial | yes | partial | | Industry pages | 13 | no | no | no | no | partial | | SDKs | yes | yes | yes | yes | yes | yes | ## Where to next * [How LayerLens compares (deep)](/1.-introduction/how-layerlens-compares-deep) — feature-by-feature, with citations * [Migration guides](/6.-build-wire-your-code/migration) — from LangSmith, Braintrust, Galileo, Promptfoo, MLflow, Arize * [Pricing](/1.-introduction/01-introduction) * [Industry patterns](/4.2-industry-use-cases/travel-hospitality) # Deep comparison LayerLens vs Braintrust, LangSmith, Galileo, Promptfoo, Arize — feature-by-feature comparison. The short version is on [Overview → How LayerLens compares](/1.-introduction/how-layerlens-compares). This page is the long version: feature-by-feature tables and the qualitative trade-offs. ## What's distinctly LayerLens These are capabilities LayerLens offers today that the alternatives don't match: 1. **Quarterly research reports.** Long-form, dated, methodologically transparent reports on the model frontier. 2. **Industry pages.** 13 industries with scenario mappings to platform capabilities. 3. **Migration guides** from LangSmith, Braintrust, Galileo, Promptfoo, MLflow, Arize. ## Feature-by-feature ### Research | Feature | LayerLens | Braintrust | LangSmith | Galileo | Promptfoo | Arize | | ------------------ | --------- | ---------- | --------- | ------- | --------- | ----- | | Quarterly research | yes | no | no | no | no | no | ### Evaluation and judging | Feature | LayerLens | Braintrust | LangSmith | Galileo | Promptfoo | Arize | | ------------------------------------- | ---------- | ---------- | --------- | ------- | --------- | ------- | | LLM-as-a-judge | yes | yes | yes | yes | yes | yes | | Built-in grader optimization | yes (GEPA) | partial | no | partial | no | no | | LLM-backed graders | yes | yes | yes | yes | yes | yes | | Code graders for deterministic checks | yes | yes | yes | yes | yes | yes | | Compare-models head-to-head | yes | partial | no | partial | yes | partial | | Trace ingestion | yes | yes | yes | yes | partial | yes | | Trace evaluation | yes | yes | yes | yes | partial | yes | | Pre-deploy agentic evaluation | yes | partial | partial | partial | partial | partial | ### Onboarding | Feature | LayerLens | Braintrust | LangSmith | Galileo | Promptfoo | Arize | | ------------------ | --------- | ---------- | --------- | ------- | --------- | ------- | | Self-serve sign-up | yes | yes | yes | partial | yes | partial | ## Migration paths Coming from another tool? See: * [From LangSmith](/6.-build-wire-your-code/from-langsmith) * [From Braintrust](/6.-build-wire-your-code/from-braintrust) * [From Galileo](/6.-build-wire-your-code/from-galileo) * [From Promptfoo](/6.-build-wire-your-code/from-promptfoo) * [From MLflow](/6.-build-wire-your-code/from-mlflow) * [From Arize](/6.-build-wire-your-code/from-arize) ## Where to next * [Pricing](/3.-pricing/03-pricing) * [Industry patterns](/4.2-industry-use-cases/industry) # Brand assets Brand assets, product names, voice, and boilerplate for partners and press covering LayerLens. Use this page to reference LayerLens correctly in partner materials, press, conference programs, and integrations. For anything not on this page — extended logo lockups, motion assets, talking points for a specific announcement — email . ## Product naming Use the name that matches the level of context the reader already has. | Form | When to use | | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **LayerLens** | First mention in any document, headline, press release, or partner page. | | **LayerLens** | Second and subsequent mentions inside the same document. Also fine in product UI and conversational copy where the LayerLens parent brand is already established. | | **LayerLens workspace**, **LayerLens SDK** | When the specific experience matters — logged-in workspace or Python client. | Do **not** refer to the product as "LayerLens" on its own when the context is the evaluation platform. LayerLens is the parent brand; LayerLens is the product. Calling the product "LayerLens" creates ambiguity for partners and analysts. Also avoid the legacy internal codename ATEAM in customer-facing materials. ## Logos and isotype The canonical logo files live in the `docs/logos/` directory of this repository. | Asset | File | | ----------------------- | -------------------------------- | | Wordmark, color | `docs/logos/LL-logo.svg` | | Wordmark, light-on-dark | `docs/logos/LL-logo-white.svg` | | Isotype (symbol only) | `docs/logos/isotype.png` | | Isotype, 512x512 PNG | `docs/logos/isotype-512x512.png` | For social avatars, favicons, and small-format placements, use the isotype. For everywhere else, use the wordmark. Keep at least the height of the "L" as clear space on every side. For high-resolution vector assets, EPS, or extended lockups, contact . ## Color palette These are placeholder reference values for documentation. The authoritative palette — including AA / AAA contrast pairings, dark-mode tokens, and gradient stops — is owned by the design team and ships with the design system. Confirm any production use with before publishing. | Role | Token | Hex | | --------------- | ------------------------- | --------- | | Primary | `layerlens.primary` | `#5046E5` | | Secondary | `layerlens.secondary` | `#1A1A2E` | | Accent | `layerlens.accent` | `#22D3EE` | | Surface — light | `layerlens.surface.light` | `#FFFFFF` | | Surface — dark | `layerlens.surface.dark` | `#0B0B1A` | | Text — primary | `layerlens.text.primary` | `#0F172A` | | Text — muted | `layerlens.text.muted` | `#475569` | ## Voice The LayerLens voice is declarative and technical. We respect the reader's time and assume they are smart, busy, and skeptical of marketing claims. * **Declarative, not hyped.** State what the product does. Skip the adjectives. * **No exclamation marks.** None. * **No superlatives** ("the best", "world-class", "revolutionary", "game-changing"). Replace with what the product actually does. * **Specific over vague.** "175+ models, 52+ benchmarks" beats "comprehensive coverage." * **Honest about scope.** If a capability is on the roadmap, say "on the roadmap." Do not imply it ships today. * **Active voice.** "LayerLens scores your traces" not "Your traces are scored by LayerLens." * **Sentence case** for headings. When in doubt, lean toward the tone of a senior engineer explaining the system to a peer. ## Boilerplate — one paragraph > LayerLens is an AI evaluation platform built for teams shipping AI to customers. The workspace lets teams run private evaluations on their own data, build LLM graders that capture their team's quality bar, ingest live traces, and wire all of it into CI/CD. LayerLens is delivered as two experiences — a logged-in workspace with pay-as-you-go compute, and a Python SDK — sharing one catalog, one grader engine, and one trace pipeline. ## Boilerplate — one sentence > LayerLens is the AI evaluation platform that lets teams pick the right model, score it on their own data, and govern its quality in production. ## Press kit For the full press kit — extended copy, headshots, screenshots, customer quotes (where permitted), and the latest fact sheet — contact . ## Where to next * [What is LayerLens?](/1.-introduction/what-is-layerlens) * [Trust and security](/more-in-this-section/trust) * [Schedule a demo](/more-in-this-section/schedule-a-demo) * [How LayerLens compares](/1.-introduction/how-layerlens-compares) # Schedule a demo Book a 30-minute LayerLens walk-through, or start in the product today — for free. See LayerLens on real data, with real models, in 30 minutes. A solutions engineer walks you through picking a model from the catalog, instrumenting a sample agent, running an evaluation with an LLM grader, and tightening that grader with GEPA — end-to-end. You leave with a working playbook for your team's first evaluation, not a sales pitch. If you'd rather start hands-on right now, the LayerLens workspace is open — no calendar invite required. ## Two ways to start ### 1. Schedule a 30-minute walk-through Email with a couple of times that work for you and a one-line description of what you're trying to evaluate. We'll send a calendar invite, a short pre-call questionnaire, and a copy of the demo dataset so you can follow along live. Bring a teammate. The demo is more useful when the builder, the operator, and the buyer are all in the room. ### 2. Sign up for the LayerLens workspace Create a workspace, invite a teammate, run your first private evaluation on your own data. You only pay for compute-heavy runs in ECU credits once you're ready. * [Sign up](/2.-get-started/sign-up) * [Your first evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation) ## What we cover in a 30-minute demo The demo follows the LayerLens workflow. We pick whichever of your real use cases is closest to "we need to know if this is good enough to ship" and walk it through four of the six stages live. 1. **Select** — open the catalog, filter to the model family relevant to your problem, compare two finalists head-to-head, and pull up the latest quarterly report on that family. * Reference: [Select — workflow stage](/5.-select-pick-the-model/workflow) 2. **Build** — instrument a sample agent (LangChain, LlamaIndex, or your stack of choice) with the Python SDK so it emits traces into the workspace. * Reference: [Build — workflow stage](/6.-build-wire-your-code/workflow) 3. **Evaluate** — run an evaluation on a small dataset, score the outputs with a code grader plus an LLM grader, and read the verdict + root-cause panel. * Reference: [Evaluate — workflow stage](/8.-evaluate-score-the-outputs/workflow) 4. **Improve** — tighten the grader with GEPA against a labelled fixture set, re-run the evaluation, and watch the regression / improvement land in the dashboard. * Reference: [Improve — workflow stage](/9.-improve-tune-the-system/workflow) If your audience includes a buyer or an IT reviewer, we also cover the [trust and security posture](/more-in-this-section/trust), pricing, and what a typical Enterprise rollout looks like. ## FAQ ### Can I bring my own data? Yes. Most teams do. You can upload a dataset in CSV / JSONL, register a private benchmark on top of it, and run any model in the catalog — or a BYOK custom model — against it. See [BYOK custom models](/5.-select-pick-the-model/byok-custom-models). ### How long is the trial? There is no trial countdown. LayerLens charges only for compute-heavy runs in ECU credits — you pay for what you use, when you use it. Details: [Pricing — overview](/3.-pricing/03-pricing). ### Do I need to install anything before the demo? No. The 30-minute demo runs entirely in the browser plus a shared notebook. If you'd like to follow along in your own environment, `pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]"` ahead of time and we'll wire your local SDK into the demo workspace. ### Can my whole team join? Yes. We recommend pulling in at least one builder, one operator (or on-call lead), and one decision-maker so the conversation covers the full lifecycle in one pass. ### What if I'm evaluating LayerLens for procurement? Send a note to and we'll route you to a longer technical session plus a security review track with . See [Trust and security](/more-in-this-section/trust). ## Where to next * [Trust and security](/more-in-this-section/trust) * [How LayerLens compares](/1.-introduction/how-layerlens-compares) * [Pricing — overview](/3.-pricing/03-pricing) * [Personas and paths](/1.-introduction/personas-paths) # Trust and security Trust, security, and compliance posture for LayerLens — encryption, tenancy, audit, and how buyers run a security review. LayerLens is built for teams whose AI ships in front of paying customers. That means evaluation data, traces, grader outputs, and audit history are treated with the same care as the production systems they govern. This page is the consolidated security and compliance summary for buyers, IT, and security reviewers. For an in-depth review, contact . For procurement and contract questions, contact . ## Encryption * **In transit:** All traffic between clients, the SDK, and the workspace is TLS 1.2 or higher. Internal service-to-service traffic is mutually authenticated. * **At rest:** All persistent data — evaluation results, traces, grader configurations, datasets, customer-supplied content — is encrypted with AES-256. * **Customer-managed keys:** Available on the Enterprise tier. ## Compliance commitments * **GDPR-aligned data handling** — DPA available; sub-processor list and data-flow diagram available on request. * **CCPA** — consumer-data subject requests are supported through the same flow as GDPR data-subject requests. * **HIPAA** — available on the Enterprise tier with an executable Business Associate Agreement (BAA). PHI workloads must be provisioned under a BAA-enabled tenant before any PHI is sent to the platform. For the current compliance posture and timeline, request the latest summary from . ## Tenant isolation Every record — evaluations, graders, traces, datasets, API keys, audit events — is scoped to the organization that created it. * **Per-org data scoping** at the database layer with row-level access controls. No application code path returns another organization's data. * **No cross-tenant flow:** A grader built in Org A cannot read Org B's traces. A grader registered in Org A is not visible to Org B. * **Workspace isolation:** Each organization is provisioned with its own logical workspace, including its own catalog overlay (private models, private benchmarks, private evaluations). * **Cache and queue isolation:** All caches and processing queues are keyed by organization so that one tenant's burst cannot affect another tenant's latency or quality of service. ## Authentication Three customer-facing authentication paths, all always active: * **Workspace login** — managed identity with email + password, Google OAuth, and GitHub OAuth out of the box. Sessions issued as short-lived JWTs. * **API keys** (SDK, CLI, automation) — created from the workspace, rotated by an admin, scoped to one organization. Old keys remain valid for 24 hours after rotation so deployments can roll forward without downtime. * **Single Sign-On (SSO)** — Enterprise tier. Okta, Azure AD / Entra ID, and any SAML 2.0 or OIDC identity provider. Group-to-role mapping is configurable per organization. Multi-organization users switch between organizations from the workspace picker; every API call carries the active organization context. ## BYOK custom models Customers can register their own OpenAI-compatible endpoints and bring their own provider keys (BYOK). When BYOK is configured: * Keys are encrypted at rest within the customer's tenant boundary. * Keys are used **only** to fulfill that customer's own evaluation, grader, or trace workload. * Keys are never logged and never used outside the requesting organization. * A customer can rotate or revoke a BYOK key from the workspace at any time; in-flight runs that depend on it will fail closed. ## Audit trail Every meaningful action on the platform is recorded to an append-only audit log: * Evaluation created, started, completed, cancelled * Grader created, updated, run, optimized * Trace ingested * Dataset uploaded or deleted * API key created, rotated, revoked * Organization, team, or role change * BYOK key registered or rotated ## Data residency * **Default:** US region. * **Enterprise:** Additional regional residency options on request, including dedicated-tenant deployments and hybrid configurations where evaluation execution runs inside the customer's own VPC. The data-residency story is part of every Enterprise procurement conversation. Reach out to for the current set of supported regions. ## Sub-processors LayerLens relies on a short list of sub-processors for hosting, identity, and operational telemetry. The current sub-processor list is maintained as part of the DPA and is available to current and prospective Enterprise customers from . The list is versioned; customers under contract are notified before any material change. ## Vulnerability disclosure We welcome responsible disclosure of security issues. Report findings to . Please include: * A description of the issue and the affected surface (catalog, workspace, SDK, API). * Steps to reproduce. * Impact, if known. We will acknowledge receipt within two business days, work with you on a remediation timeline, and credit you once a fix is shipped if you would like to be named. ## Run a security review For a guided security review — questionnaire response, architecture walk-through, sub-processor list, DPA, BAA, contact . For a product demo that includes the security and compliance surface, [schedule a demo](/more-in-this-section/schedule-a-demo). ## Where to next * [Schedule a demo](/more-in-this-section/schedule-a-demo) * [Workspace and SDK](/1.-introduction/three-experiences) * [Brand assets](/more-in-this-section/brand-assets) # Getting started Getting started with LayerLens — go from zero to your first evaluation, or pick the entry point that fits your role. Welcome. This section gets you from "I just heard of LayerLens" to "I have a real evaluation running against my data" — fast. The page you're on is a launchpad; the links below are the actual ride. {% hint style="warning" %} **Hit an error already?** Skip ahead to [Common first-day errors](/2.-get-started/first-day-errors) — install, auth, env-var, and network problems most people see in their first hour, with fixes. {% endhint %} ## Fastest path: 5 minutes to a real score If you have a Python environment and 5 minutes, do this first. One file, copy-paste-runnable, ends with a real accuracy number and a UI link.
Run your first evaluation in 5 minutesOne Python script. Install, auth, fetch a model, fetch a benchmark, run, inspect results.Start
Did the 5-minute run. Now what?
Top 5 starter recipesThe five highest-ROI cookbook recipes for week one — one per workflow stage.Open
Choose your pathMatch your role and goal to the right entry point: the workspace or the SDK.Open
## Prefer to start in the UI? Not everyone wants to start in Python. Pick whichever entry point matches what you're trying to learn.
LayerLens workspaceSign up, run your first private evaluation, build a grader in the UI.Sign up
Python SDKpip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]" and run an eval from code.Install
Most teams use both together. A typical first day: the **workspace** to run a private eval on your data, the **SDK** to wire the winner into CI. ## Browse by depth Once you've run your first eval, dig into whichever shape of content matches how you learn. | Format | When to use it | Index | | ------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------ | | **Tutorials** | Guided, end-to-end, sequenced. Take an evening, finish a working pipeline. | [All tutorials](/2.-get-started/all-tutorials) | | **Cookbook** | Problem-shaped recipes. "I have task X, give me code." | [All cookbook recipes](/2.-get-started/all-cookbook-recipes) | | **Guides** | Task-oriented walkthroughs. Bigger than a recipe, smaller than a tutorial. | [All guides](/2.-get-started/all-guides) | ## The workflow under the hood Everything in LayerLens maps to five stages. Knowing the stage helps you find the right doc fast. | Stage | What you're doing | Where it lives | | ------------ | ------------------------------------------------------ | ------------------------------------------------------ | | **Select** | Pick models, benchmarks, datasets to evaluate against | [§5 Select](/5.-select-pick-the-model/workflow) | | **Build** | Instrument your code so LayerLens can see what it does | [§6 Build](/6.-build-wire-your-code/workflow) | | **Observe** | Capture traces, route them to where they're scored | [§7 Observe](/7.-observe-see-whats-happening/workflow) | | **Evaluate** | Score runs with graders and code graders | [§8 Evaluate](/8.-evaluate-score-the-outputs/workflow) | | **Improve** | Optimize graders, cut cost, regression-protect | [§9 Improve](/9.-improve-tune-the-system/workflow) | See [The LayerLens workflow](/1.-introduction/the-layerlens-workflow) for the full picture. ## When something breaks * [Common first-day errors](/2.-get-started/first-day-errors) — auth, install, network, env-var problems. * [SDK troubleshooting](/more-in-this-section-9/troubleshooting) — deeper SDK-specific issues. * [Support](/12.-support/contact) — how to reach a human. * [Status and reliability](/13.5-operational/status) — is it us or you? ## Where to next The honest answer depends on what you're trying to do. If you're not sure: 1. Run [Your first evaluation in 5 minutes](/more-in-this-section-1/first-eval-5-minutes). 2. Pick one of the [Top 5 starter recipes](/more-in-this-section-1/top-5-starter-recipes) that matches a real problem you have. 3. When you hit something specific you need to learn, return to [All tutorials](/2.-get-started/all-tutorials) / [All cookbook recipes](/2.-get-started/all-cookbook-recipes) / [All guides](/2.-get-started/all-guides). # Choose your path Pick the LayerLens entry point that fits what you're trying to do. | If you... | Start here | | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | ...want to run a private evaluation on your own data | [LayerLens workspace — Sign up](/2.-get-started/sign-up) → [First evaluation](/2.-get-started/first-evaluation) | | ...want to build an LLM grader | [LayerLens workspace — First grader](/2.-get-started/first-judge) | | ...want to upload a trace and score it | [LayerLens workspace — First trace](/2.-get-started/first-trace) | | ...want to integrate evaluations into code or CI/CD | [SDK quickstart](/2.-get-started/sdk) | | ...are setting up for a team | [LayerLens workspace — Sign up](/2.-get-started/sign-up) | ## Both at once Many teams use both surfaces. A typical first day looks like: 1. **Workspace** — sign up, run a private evaluation against your data on a shortlist of models. 2. **SDK** — wire the winning evaluation into your code or CI gate. ## Common pitfalls * **Building a grader before you have a labeled set.** Graders only work as well as the labels you tune them against. Even 30 labels makes a meaningful difference. * **Wiring CI gates before you've run the eval interactively.** Start in the workspace UI; promote to CI once you trust the result. ## Where to next * [LayerLens workspace quickstart](/2.-get-started/premium) * [SDK quickstart](/2.-get-started/sdk) # First-day errors First-day errors — the auth, install, and network problems most people hit on day 1, with fixes. Most new users hit one of these in their first hour. Find your symptom, apply the fix. ## "command not found: layerlens" after pip install **Cause:** the `layerlens` CLI installed but isn't on your PATH. **Fix:** * On macOS/Linux: confirm `~/.local/bin` (or your venv `bin/`) is on PATH * In a venv: activate it (`source.venv/bin/activate`) * Reinstall in the active venv: `pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]"` ## `pip install` returns "Could not find a version" **Cause:** missing the `--extra-index-url` flag. **Fix:** always use the LayerLens package index: ```bash pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]" ``` ## `AuthenticationError: invalid API key` **Cause:** wrong key, wrong env-var name, or expired key. **Fix:** 1. Verify env var name is `LAYERLENS_STRATIX_API_KEY` (the SDK and the CLI both read it) 2. Echo it: `echo $LAYERLENS_STRATIX_API_KEY` — should show your key (or its prefix) 3. Generate a fresh key in **Admin → API Keys** 4. If using a CI runner, check the secret name matches ## `ConnectionError` / `httpx.ConnectError` to `api.layerlens.ai` **Cause:** firewall, proxy, or DNS. **Fix:** 1. Confirm DNS: `nslookup api.layerlens.ai` should resolve 2. Confirm reachability: `curl -I https://app.layerlens.ai/health` 3. If behind a corporate proxy: set `HTTPS_PROXY` env var; ensure CA bundle is configured 4. If using `--extra-index-url`, confirm `sdk.layerlens.ai` is also reachable ## `ImportError: cannot import name 'Client' from 'layerlens'` **Cause:** old SDK version, or `layerlens` namespace collision. **Fix:** 1. `pip show layerlens` — verify version ≥ 1.3.0 2. Upgrade: `pip install --upgrade --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]"` 3. If still failing, check for a local file or directory named `layerlens.py` shadowing the install ## `RateLimitError` on first 5 calls **Cause:** rare on day 1; usually concurrent retries from a flaky network. **Fix:** the SDK retries with backoff automatically. If it persists, drop concurrency and confirm your API key isn't shared across multiple processes. ## "I don't see my evaluation in the UI" **Cause:** wrong active org. **Fix:** the API key is scoped to one org; the dashboard's active org may differ. Switch active org in the top bar to match the key's org. See [Organizations](/more-in-this-section-9/organizations-1). ## CI: `LAYERLENS_API_KEY` not set **Cause:** secret not configured on the runner. **Fix (GitHub Actions):** ```yaml env: LAYERLENS_API_KEY: ${{ secrets.LAYERLENS_API_KEY }} ``` For other CIs, see [CI/CD integrations](/more-in-this-section-4/cicd). ## Still stuck * [SDK troubleshooting](/more-in-this-section-9/troubleshooting) * [Support — contact](/12.-support/contact) * [Status and reliability](/13.5-operational/status) # LayerLens workspace LayerLens workspace quickstart — sign up, run your first evaluation, build your first judge. Four steps to a working LayerLens account with a real evaluation on your data.
1. Sign upCreate an account with GitHub, Google, or email.Start
2. First evaluationRun a private evaluation on your data.Start
3. First traceUpload a trace and inspect it.Start
4. First graderBuild an LLM grader for a subjective dimension.Start
## What you'll have at the end * A LayerLens account at `app.layerlens.ai` * One organization with starter ECU credits * One private evaluation run with results * One uploaded trace, inspected * One custom grader ready to use Total time: \~30 minutes if you already have data; \~60 minutes from scratch. ## Where to next * [Sign up](/2.-get-started/sign-up) * [LayerLens workspace reference](/more-in-this-section-3/premium-workspace-overview) # Sign up Create a LayerLens account with GitHub, Google, or email. Three auth methods. Pick whichever your team uses today. ## Steps ### 1. Open the sign-up page Go to [`app.layerlens.ai/sign-up`](https://app.layerlens.ai/sign-up). ### 2. Pick an auth method * **Continue with GitHub** — OAuth, fastest if you're a developer * **Continue with Google** — OAuth via your Google account * **Email** — email + password; verification email sent ### 3. Verify (email only) If you signed up with email, check your inbox for the verification link. Click it. You're back in the app. ### 4. Create your organization Create a new org. You can be invited to additional orgs later and switch between them. ## Verify You should see the workspace dashboard at `app.layerlens.ai//home`. The left rail shows: Dashboard, Traces, Model Evaluations, Agent Evaluations, Environments, Models, Benchmarks, Graders, Admin, Notifications. ## Multi-org and switching If you're invited to a second org later: 1. Accept the invitation from the email 2. Use the **Organizations** menu to flip between orgs 3. The dashboard data refreshes for the active org ## Common pitfalls * **Email signup but no verification email.** Check spam. If still missing, see [Troubleshooting auth](/2.-get-started/premium). * **GitHub/Google OAuth blocked by org.** Some enterprise GitHub orgs require admin approval before third-party OAuth apps work. Ask your admin to allow the LayerLens app. ## Where to next * [First evaluation](/2.-get-started/first-evaluation) * [Organizations and multi-org](/more-in-this-section-9/organizations-1) # Your first evaluation Run your first private evaluation in the LayerLens workspace — pick a model, pick a dataset, see scores in minutes. A private evaluation runs a model against a dataset and scores the outputs. Here's the fastest way to do one. ## Prerequisites * [ ] A LayerLens account ([sign up](/2.-get-started/sign-up)) * [ ] Either: a small dataset (CSV or JSONL with input → expected output columns), or willingness to use a starter dataset ## Steps ### 1. Open Evaluations In the left rail, click **Model Evaluations**. Click **Run Evaluation**. ### 2. Pick a model Search and select. The full catalog is searchable, plus any BYOK custom models you've registered. ### 3. Pick a dataset Two options: * **Upload a dataset** — drag a CSV/JSONL onto the upload zone. LayerLens infers the schema; you confirm input and expected-output columns. * **Use a starter dataset** — pick a benchmark from the catalog (e.g., MMLU subset). ### 4. Pick scoring For your first eval, pick a code grader that fits your data: * **Exact match** — outputs equal to expected * **Semantic similarity** — fuzzy match using an embedding model * **Regex match** — outputs match a pattern * **JSON schema** — outputs validate against a schema Skip graders for now — you'll build one later. ### 5. Preview cost LayerLens shows the worst-case ECU consumption before you run. Approve. ### 6. Run Click **Run**. The evaluation queues; results stream in row-by-row. ### 7. Read the results The results page shows: * Overall score * Per-row scores (input → output → score) * Score distribution * Latency and cost per row ## Verify You should have a completed evaluation with a top-line score. Click any failing row to see the input, the model output, and the expected output side-by-side. ## What just happened LayerLens called the model on each input row, ran the grader against each output, and aggregated the result. Every row is stored — you can re-open this evaluation later or compare it against a future run. ## What to try next * **Run the same evaluation against a different model.** LayerLens's compare-models view shows both runs side-by-side. * **Iterate on the prompt.** If your dataset includes a prompt template column, change the template and rerun. * **Add a grader.** See [First grader](/2.-get-started/first-judge) for subjective dimensions. ## Where to next * [First grader](/2.-get-started/first-judge) * [First trace](/2.-get-started/first-trace) * [Tutorial: First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation) * [LayerLens workspace — Evaluations](/8.-evaluate-score-the-outputs/evaluations) # Your first trace Upload a trace to the LayerLens workspace and inspect it — see every span, every cost, every error. A trace is a record of an AI call (or a chain of calls). Uploading a trace into the workspace lets you inspect it in the UI and run trace evaluations against it. ## Prerequisites * [ ] A LayerLens account ([sign up](/2.-get-started/sign-up)) * [ ] Either: a JSON file with one trace, or willingness to use the SDK to ingest a sample trace ## Option A — Upload via UI ### 1. Open Traces Left rail: **Traces**. Click **Import traces**. ### 2. Drop the JSON file A valid trace has at least: ```json { "id": "trace-1", "name": "answer-question", "inputs": {"prompt": "What is the capital of France?"}, "outputs": {"response": "Paris."}, "spans": [ {"name": "llm-call", "kind": "llm", "model": "claude-opus-4-7",...} ] } ``` ### 3. Confirm LayerLens parses the JSON, infers the schema, and shows a preview. Click **Save**. ### 4. Inspect The trace page shows: * The full span tree * Per-span model, latency, cost, errors * Inputs and outputs at every level ## Option B — Ingest via SDK ```python import os from layerlens import Client client = Client(api_key=os.environ["LAYERLENS_STRATIX_API_KEY"]) client.traces.upload("trace.json") ``` The trace appears in **Traces** within seconds. ## Verify Open the trace page; you should see your trace's span tree, inputs, and outputs. ## What to try next * **Score this trace.** See [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations). * **Ingest more traces.** A representative trace set is the starting point for agentic evaluations. ## Where to next * [First grader](/2.-get-started/first-judge) * [LayerLens workspace — Traces](/7.-observe-see-whats-happening/traces) * [Concept: Traces and spans](/6.-build-wire-your-code/traces-and-spans) * [Tutorial: Score live traces](/8.-evaluate-score-the-outputs/04-score-traces) # Your first grader Build your first LLM grader in the LayerLens workspace — define a rubric, test it, optionally GEPA-optimize. A grader is an LLM that scores subjective dimensions — helpfulness, faithfulness, tone, safety — that a code grader can't grade. Building one takes about 15 minutes. ## Prerequisites * [ ] A LayerLens account ([sign up](/2.-get-started/sign-up)) * [ ] A clear definition of what you want to grade (write it in plain English first) * [ ] At least 10 example outputs labeled "good" or "bad" — more is better ## Steps ### 1. Open Graders Left rail: **Graders**. Click **Create judge**. ### 2. Name and describe * **Name:** e.g., "helpfulness-customer-support" * **Description:** what this grader grades, in one sentence * **Output type:** binary (pass/fail), score (e.g., 1-5), or labeled (e.g., "helpful" / "neutral" / "unhelpful") ### 3. Pick a judging model The model that runs the rubric. Default is a balanced choice; pick a stronger model for harder rubrics. ### 4. Write the rubric The rubric is the prompt the judging model uses. Best practices: * State the dimension you're grading * Describe what "good" looks like with examples * Describe what "bad" looks like with examples * Show the output format you want back The workspace provides a starter template you can adapt. ### 5. Test on a few examples Paste 3-5 sample outputs and run the grader. Read the verdicts. If they don't match your intuition, iterate the rubric. ### 6. Optional: GEPA-optimize If you have ≥30 labeled examples, run **GEPA optimization**. LayerLens tunes the rubric to better match your labels. Agreement with humans typically rises 10-20 percentage points. [More about GEPA optimization](/8.-evaluate-score-the-outputs/judges-1#grader-optimization-gepa) ### 7. Save The grader is now reusable in any evaluation, trace evaluation, or agentic evaluation in your org. ## Verify You should be able to apply your grader to a single sample input/output and get back a verdict. ## What to try next * **Apply the grader to your first evaluation.** Open the eval, add the grader, rerun. * **Apply the grader to a trace evaluation.** Real production data, judged. * **Build a second grader** for a different dimension. LayerLens evaluations let you stack many graders. ## Where to next * [Tutorial: Build your first grader](/8.-evaluate-score-the-outputs/02-first-judge) * [Tutorial: Optimize a grader with GEPA](/9.-improve-tune-the-system/05-gepa-optimize) * [Concept: Graders](/8.-evaluate-score-the-outputs/judges-1) * [LayerLens workspace — Graders](/8.-evaluate-score-the-outputs/judges) # SDK LayerLens Python SDK quickstart — install, authenticate, run an evaluation in code. The `layerlens` package is the Python client SDK for LayerLens. Install, set an API key, and you can do everything the workspace UI does — programmatically.
1. InstallOne pip command, three minutes.Start
2. First evaluation via SDKRun a benchmark evaluation from Python.Start
## Why use the SDK * **Automate.** CI/CD gates, scheduled trace evaluations, batch jobs. * **Integrate.** Wire LayerLens into your existing data pipeline. * **Share recipes.** Sample programs across every major AI agent surface — copy, paste, ship. ## Samples for the AI agent ecosystem LayerLens ships first-class samples for the platforms your team is most likely already on:
MCP ServerEvaluate Model Context Protocol servers and tool-use trajectories.Browse
Claude Code SkillsScore Claude Code skill invocations, deltas, and policy gates.Browse
Claude CoWorkTrace multi-agent CoWork sessions and compare runs.Browse
CopilotKitWire CopilotKit-powered agents into LayerLens traces, graders, and CI gates.Browse
OpenAI Agents SDKEnd-to-end coverage for the OpenAI Agents pattern.Browse
LangChain / LlamaIndex / Haystack / DSPyFramework-native instrumentation samples.Browse
## What's the SDK shape ```python from layerlens import Client client = Client() # picks up LAYERLENS_STRATIX_API_KEY from env # 11 resource handlers client.models # public model catalog client.benchmarks # public benchmark catalog client.evaluations # private evaluations client.scorers # code graders client.judges # LLM judges client.judge_optimizations # GEPA runs client.traces # trace ingest + browse client.trace_evaluations # score traces client.results # result browsing client.public # public-only convenience namespace client.integrations # integrations ``` Every method has an async counterpart on `AsyncClient`: `await AsyncClient().evaluations.create(...)`. ## Where to next * [Install](/2.-get-started/install) * [First evaluation via SDK](/2.-get-started/first-eval-via-sdk) * [Python SDK reference](/6.-build-wire-your-code/sdk-python) * [Samples overview](/2.-get-started/samples-overview) — MCP, Claude Code Skills, CoWork, CopilotKit, OpenAI Agents, and framework-native examples * [Cookbook](/2.-get-started/all-cookbook-recipes) — recipes ported from the SDK's `samples/` folder # Install Install the LayerLens Python SDK from the LayerLens package registry. ## Requirements * Python 3.8 or higher * pip ## Install ```bash pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]" ``` ## Verify ```python import layerlens print(layerlens.__version__) ``` You should see `1.3.0` (or later). ## Set your API key Get an API key from **Admin → API Keys** in the workspace. Then export: ```bash export LAYERLENS_STRATIX_API_KEY="your-api-key" ``` Or pass it explicitly when creating the client: ```python from layerlens import Client client = Client(api_key="your-api-key") ``` ## Virtual environment (recommended) ```bash python -m venv.venv # Linux/macOS: source.venv/bin/activate # Windows PowerShell: #.venv\Scripts\Activate.ps1 pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]" ``` ## Upgrade ```bash pip install --upgrade --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]" ``` ## Dependencies installed * `httpx` — HTTP client * `pydantic` — data validation ## Where to next * [First evaluation via SDK](/2.-get-started/first-eval-via-sdk) * [SDK authentication reference](/2.-get-started/sdk) * [SDK API reference](/2.-get-started/sdk) # First evaluation via SDK Run your first benchmark evaluation programmatically via the LayerLens Python SDK. Twenty lines of Python that run a real evaluation. ## Prerequisites * [ ] [SDK installed](/2.-get-started/install) * [ ] `LAYERLENS_STRATIX_API_KEY` exported * [ ] A LayerLens account with starter ECU ## Sync version ```python from layerlens import Client client = Client() # Pick a model and a benchmark from the public catalog model = client.models.get_by_key("openai/gpt-4o") benchmark = client.benchmarks.get_by_key("arc-agi-2") # Create the evaluation evaluation = client.evaluations.create( model=model, benchmark=benchmark, ) # Wait for completion (polling) result = client.evaluations.wait_for_completion(evaluation) print(f"Accuracy: {result.accuracy}") ``` ## Async version Every method has an async counterpart on `AsyncClient`: ```python import asyncio from layerlens import AsyncClient async def main(): client = AsyncClient() model = await client.models.get_by_key("openai/gpt-4o") benchmark = await client.benchmarks.get_by_key("arc-agi-2") evaluation = await client.evaluations.create(model=model, benchmark=benchmark) result = await client.evaluations.wait_for_completion(evaluation) print(f"Accuracy: {result.accuracy}") asyncio.run(main()) ``` ## What just happened 1. The SDK authenticated with your API key. 2. It fetched the model and benchmark records from the public catalog. 3. It created an evaluation in your org. 4. The platform queued the work; the SDK polled for completion. 5. The result was returned; you got an accuracy score. ## Verify The same evaluation should appear in **Model Evaluations**. Same record, same ID, same scores. ## Try variations ### Different model ```python model = client.models.get_by_key("anthropic/claude-opus-4-7") ``` ### Different benchmark ```python benchmark = client.benchmarks.get_by_key("mmlu") ``` ### Browse public results before running ```python public_models = client.public.models.get() for m in public_models.models[:5]: print(m.key, m.name) ``` ## Error handling ```python from layerlens import Client, NotFoundError, AuthenticationError, APIError try: model = client.models.get_by_id("nonexistent-id") except NotFoundError as e: print(f"Not found: {e.message}") except AuthenticationError as e: print(f"Auth failed: {e.message}") except APIError as e: print(f"API error ({e.status_code}): {e.message}") ``` ## Where to next * [Python SDK reference](/2.-get-started/sdk) * [Tutorial: First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation) * [Cookbook recipes](/2.-get-started/all-cookbook-recipes) # All tutorials Every LayerLens tutorial — guided, end-to-end walkthroughs organized by workflow stage. Tutorials are the guided learning path. Each one is end-to-end, sequenced, and produces something working. If you want the fastest possible first-touch instead, see [Run your first evaluation in 5 minutes](/more-in-this-section-1/first-eval-5-minutes). Tutorials are organized along the five stages of [the LayerLens workflow](/1.-introduction/the-layerlens-workflow): **Select**, **Build**, **Observe**, **Evaluate**, **Improve**. ## Select Pick what you're evaluating — models, benchmarks, datasets. No standalone tutorial in this stage today; for guided model-selection flows, see the cookbook: * [Pick a model for a task](/more-in-this-section-3/pick-a-model) * [Compare prompts on a small test set](/more-in-this-section-3/compare-prompts) ## Build Get LayerLens wired into your code or CI. | # | Tutorial | Time | What you'll have at the end | | - | ------------------------------------------------------------- | ------ | ---------------------------------------------------------- | | 3 | [CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates) | 30 min | A GitHub Actions workflow that blocks merges on regression | ## Observe Capture and route traces so they can be scored. No standalone tutorial in this stage today. For trace ingestion, start with: * [Set up trace ingestion](/more-in-this-section-4/setup-trace-ingestion) (guide) * [Backfill from existing logs](/7.-observe-see-whats-happening/backfill-from-logs) (recipe) * [Schedule continuous trace evaluation](/more-in-this-section-5/schedule-continuous) (guide) ## Evaluate Score model runs and traces with graders and code graders. | # | Tutorial | Time | What you'll have at the end | | - | ------------------------------------------------------------------------------------ | ------ | ----------------------------------------------------- | | 1 | [First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation) | 10 min | A scored model vs. benchmark, in UI and SDK | | 2 | [First grader](/8.-evaluate-score-the-outputs/02-first-judge) | 20 min | An LLM grader scoring a subjective dimension | | 4 | [Score live traces](/8.-evaluate-score-the-outputs/04-score-traces) | 30 min | Continuous evaluation running over production traffic | ## Improve Tune graders, cut costs, harden regression protection. | # | Tutorial | Time | What you'll have at the end | | - | ---------------------------------------------------------------------- | ------ | ---------------------------------------------------- | | 5 | [GEPA-optimize a grader](/9.-improve-tune-the-system/05-gepa-optimize) | 45 min | An LLM grader with measurably higher human agreement | ## Reading order If you're new, take them in numeric order — each builds on what came before. 1. [First evaluation](/8.-evaluate-score-the-outputs/01-first-evaluation) — Evaluate 2. [First grader](/8.-evaluate-score-the-outputs/02-first-judge) — Evaluate 3. [CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates) — Build 4. [Score live traces](/8.-evaluate-score-the-outputs/04-score-traces) — Evaluate / Observe 5. [GEPA-optimize](/9.-improve-tune-the-system/05-gepa-optimize) — Improve Total time investment for all five: \~2.5 hours. ## Where to next * [Top 5 starter recipes](/more-in-this-section-1/top-5-starter-recipes) — same week-one outcomes, recipe-shaped instead of tutorial-shaped * [All cookbook recipes](/2.-get-started/all-cookbook-recipes) — 59 recipes by problem, framework, industry, integration * [All guides](/2.-get-started/all-guides) — task-oriented walkthroughs * [The LayerLens workflow](/1.-introduction/the-layerlens-workflow) — the spine these tutorials map to # All cookbook recipes Every LayerLens cookbook recipe — runnable snippets organized by the workflow stage they belong to. Recipes are the "give me code I can paste" shape. Each one is small, self-contained, and runnable. They're organized below by the [workflow stage](/1.-introduction/the-layerlens-workflow) they map to — **Select**, **Build**, **Observe**, **Evaluate**, **Improve** — plus cross-cutting groupings (industry, integration, migration, SDK basics) at the bottom. If you're new, start with [Top 5 starter recipes](/more-in-this-section-1/top-5-starter-recipes). Otherwise, browse below. ## Select Pick the model, benchmark, or prompt before you commit ECU. * [Pick a model for a task](/more-in-this-section-3/pick-a-model) * [Compare prompts on a small test set](/more-in-this-section-3/compare-prompts) ## Build Instrument your code and ship the wiring. ### Framework adapters * [Instrument a LangChain agent](/6.-build-wire-your-code/instrument-langchain) * [Instrument a LlamaIndex pipeline](/6.-build-wire-your-code/instrument-llamaindex) * [Instrument a Haystack pipeline](/6.-build-wire-your-code/instrument-haystack) * [Instrument an OpenAI Assistants run](/6.-build-wire-your-code/instrument-openai-assistants) * [Instrument an Anthropic SDK call](/6.-build-wire-your-code/instrument-anthropic-sdk) * [Instrument a custom Python pipeline](/6.-build-wire-your-code/instrument-custom-python) * [OTel-style trace ingestion](/6.-build-wire-your-code/otel-trace-ingestion) ### CI / release * [Containerized CI runner](/6.-build-wire-your-code/containerized-ci) * [Pre-release readiness eval](/6.-build-wire-your-code/prerelease-readiness) ## Observe Capture, route, and warehouse what your system is doing. * [Backfill from existing logs](/7.-observe-see-whats-happening/backfill-from-logs) * [Post-release continuous eval](/7.-observe-see-whats-happening/postrelease-continuous) ## Evaluate Score what you've captured. ### Score by output shape * [Score a structured-output prompt](/more-in-this-section-6/score-structured-output) * [Score a free-form chat prompt](/more-in-this-section-6/score-chat) * [Score code outputs (HumanEval-style)](/more-in-this-section-6/score-code) * [Score reasoning chains](/more-in-this-section-6/score-reasoning) * [Score retrieval quality (precision@k)](/more-in-this-section-6/score-retrieval) * [Score multilingual outputs](/more-in-this-section-6/multilingual) * [Validate JSON output with schema](/more-in-this-section-6/json-schema-validate) ### Score by failure mode * [Catch hallucinations](/more-in-this-section-6/catch-hallucinations) * [Build a faithfulness grader for RAG](/more-in-this-section-6/rag-faithfulness-judge) * [Detect tool-call regressions in agents](/more-in-this-section-6/agent-tool-regressions) ### Author your own graders * [Custom code grader](/more-in-this-section-6/custom-code-scorer) ### Aggregation and gating * [Span-level scoring](/more-in-this-section-6/span-level-scoring) * [Multi-grader consensus](/more-in-this-section-6/multi-judge-consensus) * [Confidence-weighted aggregation](/more-in-this-section-6/confidence-weighted-aggregation) * [Cost-aware evaluation](/more-in-this-section-6/cost-aware-evaluation) * [Run a grader in CI without merging on regression](/more-in-this-section-6/judge-ci-no-merge) ## Improve Tune what you've scored, drive cost down, keep regressions out. * [Daily smoke eval](/9.-improve-tune-the-system/daily-smoke-eval) * [Quarterly model re-evaluation](/9.-improve-tune-the-system/quarterly-rebench) * [GEPA basics](/9.-improve-tune-the-system/gepa-basics) * [GEPA with held-out validation set](/9.-improve-tune-the-system/gepa-holdout) * [Iteratively label and re-optimize](/9.-improve-tune-the-system/gepa-iterative-labeling) ## Cross-cutting: by industry * [Healthcare: clinical Q\&A safety grader](/more-in-this-section-2/industry-healthcare-clinical-qa) * [Financial services: advisor chatbot guardrails](/more-in-this-section-2/industry-finserv-advisor) * [Legal: contract review faithfulness](/more-in-this-section-2/industry-legal-contract) * [Retail: product Q\&A scoring](/more-in-this-section-2/industry-retail-product-qa) * [Insurance: claims triage scoring](/more-in-this-section-2/industry-insurance-claims) * [Education: tutor helpfulness](/more-in-this-section-2/industry-education-tutor) ## Cross-cutting: by CI / notification integration * [GitHub Actions gate](/6.-build-wire-your-code/integration-github-actions) * [GitLab CI gate](/6.-build-wire-your-code/integration-gitlab-ci) * [Buildkite gate](/6.-build-wire-your-code/integration-buildkite) * [Jenkins gate](/6.-build-wire-your-code/integration-jenkins) * [Slack notifications](/6.-build-wire-your-code/integration-slack) * [PagerDuty escalation](/6.-build-wire-your-code/integration-pagerduty) ## Cross-cutting: migration from another platform * [Port a LangSmith eval](/more-in-this-section-4/migration-langsmith) * [Port a Braintrust eval](/more-in-this-section-4/migration-braintrust) * [Port an MLflow run](/more-in-this-section-4/migration-mlflow) * [Port a Promptfoo config](/more-in-this-section-4/migration-promptfoo) ## SDK basics * [Sync vs async clients](/more-in-this-section-9/sync-async-clients) * [Pagination over large result sets](/more-in-this-section-9/pagination-1) * [Retry and backoff](/more-in-this-section-9/retry-backoff) * [Rate-limit handling](/more-in-this-section-9/rate-limit-handling) ## Where to next * [Top 5 starter recipes](/more-in-this-section-1/top-5-starter-recipes) — the curated week-one subset * [All tutorials](/2.-get-started/all-tutorials) — guided, end-to-end learning paths * [All guides](/2.-get-started/all-guides) — task-shaped walkthroughs * [LayerLens Python SDK samples](https://github.com/layerlens/stratix-python) — many recipes mirror runnable source samples ## Integrate * [Send results to S3](/10.-integrate-connect-your-stack/integration-s3-export) * [Send results to BigQuery](/10.-integrate-connect-your-stack/integration-bigquery-export) * [Send results to Snowflake](/10.-integrate-connect-your-stack/integration-snowflake-export) # All guides Every LayerLens guide — task-oriented walkthroughs across the platform, organized by workflow stage and operational concern. Guides sit between recipes and tutorials in shape. Use a guide when you have a clear goal that's larger than a copy-paste recipe but doesn't need a full guided learning sequence. Organized below by [workflow stage](/1.-introduction/the-layerlens-workflow) — **Select**, **Build**, **Observe**, **Evaluate**, **Improve** — plus operational concerns (admin, migration) at the bottom. ## Select Pick and prepare what you'll evaluate against. ### BYOK models * [Register a BYOK custom model](/more-in-this-section-3/register-byok-model) * [Test a BYOK endpoint](/more-in-this-section-3/test-byok-endpoint) * [Rotate a BYOK API key](/more-in-this-section-3/rotate-byok-key) ### Benchmarks and datasets * [Upload a private benchmark](/more-in-this-section-3/upload-benchmark) * [Refresh a benchmark with new data](/more-in-this-section-3/refresh-benchmark) ## Build Wire LayerLens into your code and CI. ### Trace ingestion * [Set up trace ingestion](/more-in-this-section-4/setup-trace-ingestion) ### Calling LayerLens from your stack * [Calling LayerLens from a backend](/more-in-this-section-4/calling-layerlens-from-a-backend) ### CI / quality gates * [Wire CI/CD evaluation gates](/more-in-this-section-4/wire-cicd-gates) * [Investigate a regression in CI](/more-in-this-section-4/investigate-ci-regression) ## Observe Run continuous evaluation over what's already happening. * [Build a trace set for agentic evals](/more-in-this-section-5/build-trace-set) * [Schedule continuous trace evaluation](/more-in-this-section-5/schedule-continuous) * [Investigate a failing trace](/more-in-this-section-5/investigate-failing-trace) ## Evaluate Build, standardize, and harden your scoring library. * [Bootstrap graders from real traces](/more-in-this-section-6/bootstrap-judges) * [Build your grader library](/more-in-this-section-6/build-judge-library) * [Standardize graders across teams](/more-in-this-section-6/standardize-judges) * [Detect and remove flaky graders](/more-in-this-section-6/remove-flaky-scorers) ## Improve Optimize graders, cut cost, keep regressions out. * [Optimize a grader with GEPA](/more-in-this-section-7/optimize-judge-gepa) * [Curate a regression test set](/more-in-this-section-7/curate-regression-set) * [Reduce eval cost without losing signal](/more-in-this-section-7/reduce-eval-cost) * [Cost-control levers](/more-in-this-section-7/cost-control-levers) ## Admin and operations Day-two work — orgs, teams, billing, security, notifications. ### Setup and onboarding * [Set up your first organization](/11.-admin/setup-first-org) * [Invite your team](/11.-admin/invite-team) * [Manage seats and roles](/11.-admin/manage-seats-roles) ### Billing * [Buy and manage ECU](/11.-admin/manage-ecu) ### Keys * [Rotate an API key](/11.-admin/rotate-api-key) ### Notifications and audit * [Set up notifications and alerting](/11.-admin/setup-notifications) * [Set up Slack integration](/11.-admin/setup-slack) * [Audit-friendly release evidence](/11.-admin/audit-friendly-evidence) ## Migration Coming from another evaluation platform. * [Plan a migration to LayerLens](/more-in-this-section-4/plan-migration) * [Run LayerLens and another tool side-by-side during migration](/more-in-this-section-4/dual-run-migration) ## Where to next * [All tutorials](/2.-get-started/all-tutorials) — guided, end-to-end learning paths * [All cookbook recipes](/2.-get-started/all-cookbook-recipes) — copy-paste snippets by problem * [Top 5 starter recipes](/more-in-this-section-1/top-5-starter-recipes) — the curated week-one subset * [Common first-day errors](/2.-get-started/first-day-errors) — when something doesn't work # Samples (70+ runnable) LayerLens Python SDK samples — categorized index of the runnable samples that ship with the SDK. The LayerLens Python SDK ships with **70+ runnable samples** covering every public resource — from a single trace evaluation to multi-agent evaluation patterns and CI/CD quality gates. All samples live in [`samples/` in the LayerLens Python SDK repo](https://github.com/layerlens/stratix-python/tree/main/samples) and run directly after installing the SDK. ## Quick start ```bash pip install --extra-index-url https://sdk.layerlens.ai/package layerlens export LAYERLENS_STRATIX_API_KEY=your-api-key python samples/core/quickstart.py ``` [`quickstart.py`](https://github.com/layerlens/stratix-python/blob/main/samples/core/quickstart.py) walks through the complete workflow end-to-end: upload a trace, create a grader, run an evaluation, and retrieve results. ## Samples by category Each category groups samples around a specific LayerLens capability or integration surface. Pick the category closest to what you're building. | Category | What's in it | Count | | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | [Core SDK operations](#core-sdk-operations) | Every public resource — traces, graders, evaluations, results, models, benchmarks — used individually and together, including async patterns and pagination | 18 | | [Industry solutions](#industry-solutions) | Domain-specific evaluation scenarios with graders tuned for regulated and high-stakes verticals (healthcare, financial services, legal, government, insurance, retail) | 10 | | [Multi-agent evaluation](#multi-agent-evaluation) | Patterns for evaluating multi-agent collaboration where each agent's output needs independent quality assessment | 5 | | [Content-type evaluations](#content-type-evaluations) | Specialized graders applied to different content types — text responses, brand assets, structured documents | 3 | | [CI/CD integration](#cicd-integration) | Quality gates that embed evaluation into build and deployment pipelines so regressions never reach production | 2 + workflow | | [LLM provider integrations](#llm-provider-integrations) | Trace and evaluate outputs from OpenAI and Anthropic — both manual trace upload and auto-instrumentation paths | 4 | | [OpenClaw agent evaluation](#openclaw-agent-evaluation) | Trace, evaluate, and monitor [OpenClaw](https://openclaw.ai/) autonomous AI agents — cage-match tournaments, code gating, drift detection, content auditing | 10 + skill | | [MCP server](#mcp-server) | Expose LayerLens capabilities as tools for Claude, Cursor, and any MCP-compatible AI assistant | 1 | | [CopilotKit integration](#copilotkit-integration) | Full-stack pattern with LangGraph CoAgents + generative-UI cards for human-in-the-loop evaluation workflows | 2 agents + UI | | [Claude Code skills](#claude-code-skills) | Slash commands that bring LayerLens workflows into the Claude Code CLI | 6 | | [Sample data](#sample-data) | Pre-built trace files and 16 industry-specific evaluation datasets so every sample runs without sourcing your own data | n/a | *** ## Core SDK operations Located in [`samples/core/`](https://github.com/layerlens/stratix-python/tree/main/samples/core). Start here to learn how every public LayerLens resource works, individually and combined. Key samples: * `quickstart.py` — first evaluation in under 50 lines * `trace_evaluation.py` — full trace-evaluation lifecycle * `judge_optimization.py` — optimize a grader via automated prompt evolution (GEPA) * `evaluation_pipeline.py` — chain graders, traces, and results into an automated pipeline * `async_workflow.py` — concurrent operations with `AsyncClient` See the [Core SDK README](https://github.com/layerlens/stratix-python/blob/main/samples/core/README.md). Maps to gitbook: [Tutorials](/2.-get-started/all-tutorials) · [Concept: Evaluations](/more-in-this-section-9/evaluations-1). ## Industry solutions Located in [`samples/industry/`](https://github.com/layerlens/stratix-python/tree/main/samples/industry). Domain-specific scenarios with graders and scoring configurations tuned for regulated and high-stakes verticals. Key samples: * `healthcare_clinical.py` — clinical decision support evaluation * `financial_trading.py` — SOX-aligned trading compliance * `legal_contracts.py` — contract review quality assessment See the [Industry Solutions README](https://github.com/layerlens/stratix-python/blob/main/samples/industry/README.md). Maps to gitbook: [Industry patterns](/6.-build-wire-your-code/sdk-python). ## Multi-agent evaluation Located in [`samples/cowork/`](https://github.com/layerlens/stratix-python/tree/main/samples/cowork). Patterns for evaluating multi-agent collaboration — applicable to any framework where multiple agents produce intermediate outputs that need independent quality assessment. Key samples: * `multi_agent_eval.py` — Generator-Evaluator pattern * `code_review.py` — Instrumentor-Reviewer pattern * `rag_assessment.py` — RAG-quality evaluation See the [Multi-Agent README](https://github.com/layerlens/stratix-python/blob/main/samples/cowork/README.md). Maps to gitbook: [Concept: Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation). ## Content-type evaluations Located in [`samples/modalities/`](https://github.com/layerlens/stratix-python/tree/main/samples/modalities). Specialized graders applied to different content types. * `text_evaluation.py` — score text across safety, relevance, and compliance * `brand_evaluation.py` — enforce brand-voice consistency * `document_evaluation.py` — validate document-extraction accuracy ## CI/CD integration Located in [`samples/cicd/`](https://github.com/layerlens/stratix-python/tree/main/samples/cicd). Embed evaluation quality gates into build and deployment pipelines. * `quality_gate.py` — gate deployments on evaluation pass rates * `pre_commit_hook.py` — catch regressions at commit time * `github_actions_gate.yml` — drop-in GitHub Actions workflow See the [CI/CD README](https://github.com/layerlens/stratix-python/blob/main/samples/cicd/README.md). Maps to gitbook: [Tutorial 3: CI/CD gates](/6.-build-wire-your-code/03-cicd-gates) · [Cookbook: containerized CI](/6.-build-wire-your-code/containerized-ci). ## LLM provider integrations Located in [`samples/integrations/`](https://github.com/layerlens/stratix-python/tree/main/samples/integrations). Two paths: manual trace upload for full control, and auto-instrumentation via `layerlens.instrument` for zero-code observability. * `openai_traced.py` — manual trace upload around an OpenAI completion * `anthropic_traced.py` — capture multi-turn Claude conversations (manual trace upload) * `openai_instrumented.py` — auto-instrument OpenAI with `instrument_openai()`, `@trace`, `span()` * `langchain_instrumented.py` — auto-capture LangChain chain execution with `LangChainCallbackHandler` Maps to gitbook: [Integrations](/6.-build-wire-your-code/sdk-python). ## OpenClaw agent evaluation Located in [`samples/openclaw/`](https://github.com/layerlens/stratix-python/tree/main/samples/openclaw). Trace, evaluate, and monitor [OpenClaw](https://openclaw.ai/) autonomous AI agents — including cage-match model tournaments, code gating, drift detection, content auditing, honeypot-skill auditing, and adversarial red-teaming. See the [OpenClaw README](https://github.com/layerlens/stratix-python/blob/main/samples/openclaw/README.md) for the full list and advanced evaluation patterns. ## MCP server Located in [`samples/mcp/`](https://github.com/layerlens/stratix-python/tree/main/samples/mcp). Expose LayerLens capabilities as tools for Claude, Cursor, and any MCP-compatible AI assistant. * `layerlens_server.py` — MCP server exposing trace management, grader creation, and evaluation execution See the [MCP README](https://github.com/layerlens/stratix-python/blob/main/samples/mcp/README.md) for setup. ## CopilotKit integration Located in [`samples/copilotkit/`](https://github.com/layerlens/stratix-python/tree/main/samples/copilotkit). Full-stack pattern combining LangGraph CoAgents (Python) with generative-UI card components (React) for human-in-the-loop evaluation workflows. * `agents/evaluator_agent.py` — LangGraph CoAgent for evaluation workflows * `agents/investigator_agent.py` — LangGraph CoAgent for trace investigation * `components/*.tsx` — React card components for rendering results (`EvaluationCard`, `JudgeVerdictCard`, `TraceCard`, `MetricCard`, `ComplianceCard`) * `hooks/*.ts` — CopilotKit hooks for wiring LayerLens actions (`useLayerLensActions`, `useLayerLensContext`) See the [CopilotKit README](https://github.com/layerlens/stratix-python/blob/main/samples/copilotkit/README.md) for setup, full file list, and the human-in-the-loop walkthrough. ## Claude Code skills Located in [`samples/claude-code/`](https://github.com/layerlens/stratix-python/tree/main/samples/claude-code). Slash commands that bring LayerLens workflows directly into the Claude Code CLI — manage traces, graders, evaluations, optimizations, benchmarks, and investigations without leaving your terminal. `/trace`, `/judge`, `/evaluate`, `/optimize`, `/benchmark`, `/investigate`. See the [Claude Code Skills README](https://github.com/layerlens/stratix-python/blob/main/samples/claude-code/README.md). ## Sample data Located in [`samples/data/`](https://github.com/layerlens/stratix-python/tree/main/samples/data). Pre-built trace files, test datasets, and 16 industry-specific evaluation datasets so every sample runs without you sourcing your own data first. See the [Sample Data README](https://github.com/layerlens/stratix-python/blob/main/samples/data/README.md) for contents. *** ## Full sample reference For the complete table of every sample with descriptions, see the [samples README in the SDK repo](https://github.com/layerlens/stratix-python/blob/main/samples/README.md). ## Where to next * [SDK README](/6.-build-wire-your-code/sdk-python) — full resource list * [Cookbook](/2.-get-started/all-cookbook-recipes) — recipes drawn from these samples * [Integrations](/6.-build-wire-your-code/sdk-python) — full adapter list * [Tutorials](/2.-get-started/all-tutorials) — guided walkthroughs anchored to these samples # Run your first evaluation in 5 minutes Run your first LayerLens evaluation end-to-end in five minutes. One Python file, copy-paste-runnable. One Python file. One model. One benchmark. One score. That's it. This page is the fastest possible path from zero to a real evaluation. If something breaks, jump straight to [Common first-day errors](/2.-get-started/first-day-errors) and come back. {% hint style="info" %} **Prereqs:** Python 3.9+, a LayerLens account, and an API key from **Admin → API Keys**. You'll need \~10 ECU of starter credit (every new org gets some). {% endhint %} ## Step 1 — Install and authenticate (60 seconds) ```bash pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]" export LAYERLENS_STRATIX_API_KEY="sk-..." # macOS / Linux # PowerShell: $env:LAYERLENS_STRATIX_API_KEY = "sk-..." ``` Verify the install: ```bash python -c "import layerlens; print(layerlens.__version__)" ``` You should see `1.3.0` or later. If not, see [first-day errors](/2.-get-started/first-day-errors). ## Step 2 — The whole thing in one file Save this as `first_eval.py` and run it. That's the entire 5-minute experience. ```python """ first_eval.py — your first end-to-end LayerLens evaluation. Run with: python first_eval.py Requires: LAYERLENS_STRATIX_API_KEY set in your environment. """ from layerlens import Client # 1) Authenticate. The Client() constructor reads LAYERLENS_STRATIX_API_KEY # from the environment. Pass api_key="..." explicitly if you prefer. client = Client() # 2) Pick a model from the catalog. get_by_key() is case-sensitive # and uses the "/" key shown in the Catalog UI. model = client.models.get_by_key("anthropic/claude-opus-4-7") print(f"Model: {model.name}") # 3) Pick a benchmark. MMLU is a fast, well-known public benchmark — good # for a smoke test on day one. benchmark = client.benchmarks.get_by_key("mmlu") print(f"Benchmark: {benchmark.name}") # 4) Create the evaluation. The platform queues it; the SDK returns # immediately with an Evaluation object you can poll. evaluation = client.evaluations.create( model=model, benchmark=benchmark, ) print(f"Started: {evaluation.id} (status={evaluation.status})") # 5) Block until the evaluation finishes. wait_for_completion polls every # interval_seconds and gives up after timeout_seconds. MMLU on Opus 4.7 # typically completes in 2–4 minutes. evaluation = client.evaluations.wait_for_completion( evaluation, interval_seconds=10, timeout_seconds=600, ) if not evaluation.is_success: raise SystemExit(f"Evaluation did not succeed: status={evaluation.status}") print(f"Done: accuracy={evaluation.accuracy:.3f}") # 6) Pull per-prompt results. get_all_by_id auto-paginates across pages and # returns a flat list of Result objects (.prompt,.result,.truth,.score). results = client.results.get_all_by_id(evaluation.id) print(f"Rows: {len(results)}") # 7) Show the first three rows so you can see what real data looks like. for r in results[:3]: verdict = "PASS" if r.score and r.score >= 0.5 else "FAIL" print(f" [{verdict}] {r.prompt[:60]}... score={r.score}") # 8) Done. The same evaluation now appears in Model Evaluations. print(f"\nView in UI: https://app.layerlens.ai/{client.project_id}/evaluations/{evaluation.id}") ``` ## Step 3 — Run it ```bash python first_eval.py ``` You should see something like: ``` Model: Claude Opus 4.7 Benchmark: MMLU Started: eval_01HZ... (status=queued) Done: accuracy=0.873 Rows: 14042 [PASS] Which of the following best describes the function of... score=1.0 [PASS] In the context of macroeconomic policy, the term 'cro... score=1.0 [FAIL] A 65-year-old patient presents with acute onset chest... score=0.0 View in UI: https://app.layerlens.ai//evaluations/eval_01HZ... ``` Open that URL. The dashboard shows the same accuracy number, plus a per-subset breakdown, error analysis, and ECU consumed. ## You just did five things 1. **Selected** a model and a benchmark from the catalog. 2. **Built** the evaluation by passing both objects to `evaluations.create`. 3. **Observed** the run via `wait_for_completion` (queued → running → completed). 4. **Evaluated** the model against MMLU's ground-truth labels — an LLM-backed grader compares each model response to truth and emits a 0/1 score per row. 5. **Improved** your understanding of where the model wins and loses, by inspecting per-row results. Those five stages — Select, Build, Observe, Evaluate, Improve — are the spine of every LayerLens workflow. ## Common failure modes * **`AuthenticationError: invalid API key`** — env var name is `LAYERLENS_STRATIX_API_KEY`. See [first-day errors](/2.-get-started/first-day-errors). * **`NotFoundError` on `get_by_key`** — keys are case-sensitive. Browse the **Models** page in the workspace to copy the exact key. * **Timeout** — bump `timeout_seconds`. Some benchmarks (HumanEval+, GPQA) take longer than 10 minutes on slower models. * **"No ECU"** — every new org gets starter ECU; if you've exhausted it, top up under **Admin → Billing**. ## Where to next You ran one model on one benchmark. Logical next steps: * **Compare two models.** Repeat Steps 2–5 with a different model key and diff the accuracy. * **Score subjective dimensions.** Accuracy works when there's a ground-truth answer. For helpfulness, faithfulness, tone, etc., you need an LLM grader. [Tutorial 2: First grader](/8.-evaluate-score-the-outputs/02-first-judge). * **Run it on your own data.** Swap the benchmark for an uploaded dataset. [Upload a private benchmark](/more-in-this-section-3/upload-benchmark). * **Wire this into CI.** Promote your `first_eval.py` into a quality gate. [Tutorial 3: CI/CD gates](/6.-build-wire-your-code/03-cicd-gates). * **Score live traces.** Score traffic that's already happening in production. [Tutorial 4: Score live traces](/8.-evaluate-score-the-outputs/04-score-traces). If you'd rather follow a guided five-tutorial path, start at [All tutorials](/2.-get-started/all-tutorials). If you want to lift-and-shift a working snippet for a specific problem, jump to [Top 5 starter recipes](/more-in-this-section-1/top-5-starter-recipes). # Top 5 starter recipes Five starter cookbook recipes hand-picked for new LayerLens users. Cover the full Build → Observe → Evaluate → Improve arc in your first week. The cookbook has 59 recipes. These are the five we'd hand to a new engineer on their first week — one per workflow stage, each delivering visible value in under an hour. Run [first-eval-5-minutes](/more-in-this-section-1/first-eval-5-minutes) first if you haven't. These five build on top. ## 1 — Instrument your own Python pipeline **Why this one:** before you can score real production traffic, you need traces. This recipe shows the two primitives — `@trace` and `span()` — that turn any Python function into something LayerLens can see. It works whether you're calling OpenAI, a local model, a vector store, or your own custom orchestration. Five lines of decorators replace days of OTel wiring. **Stage:** Build · **Time:** 15 min · **Output:** traces flowing into **Traces** in the workspace → [Recipe: instrument a custom Python pipeline](/6.-build-wire-your-code/instrument-custom-python) ## 2 — Backfill traces from existing logs **Why this one:** you almost certainly already have logs — Postgres rows, JSONL files, Datadog exports — sitting somewhere. This recipe converts that history into LayerLens traces in bulk so you can run evaluations against weeks or months of real production data on day one, instead of waiting for new traffic to accumulate. Highest leverage move you can make in week one. **Stage:** Observe · **Time:** 30 min · **Output:** historical traces visible in the workspace, ready to score → [Recipe: backfill from existing logs](/7.-observe-see-whats-happening/backfill-from-logs) ## 3 — Catch hallucinations **Why this one:** hallucination is the #1 failure mode every team asks about first. This recipe wires up an LLM-backed faithfulness grader that flags answers unsupported by retrieved context. Works on freshly captured traces or backfilled history. The grader prompt is tunable, the threshold is tunable, and the per-row verdicts are inspectable — so you can trust the signal before you act on it. **Stage:** Evaluate · **Time:** 20 min · **Output:** every trace gets a faithfulness score and verdict → [Recipe: catch hallucinations](/more-in-this-section-6/catch-hallucinations) ## 4 — Validate structured outputs with a JSON Schema code grader **Why this one:** the moment your agent emits JSON, you need to guarantee the shape — required fields, types, enum values, nested structures. This recipe registers a **code grader** that runs a JSON Schema check against every output and emits a 0/1 pass score. Pairs naturally with #3: hallucination graders score *content*, schema graders score *shape*. Together they catch \~80% of the "my agent is broken" reports in CI. **Stage:** Evaluate · **Time:** 15 min · **Output:** schema failures surface as failing rows in every evaluation → [Recipe: validate JSON output with schema](/more-in-this-section-6/json-schema-validate) ## 5 — Optimize a grader with GEPA basics **Why this one:** the graders you bootstrap (recipe #3) start at 60–75% agreement with human labels. That's enough to *find* problems but not enough to *gate* on. GEPA — the platform's prompt-optimization loop — tunes the grader prompt against a small labeled set until agreement climbs past 90%. This recipe takes 30 minutes and converts a noisy first-draft grader into one you trust enough to wire into CI. **Stage:** Improve · **Time:** 30 min · **Output:** an optimized grader with measurable agreement gain → [Recipe: GEPA basics](/9.-improve-tune-the-system/gepa-basics) ## How these five fit together ``` 1. Instrument → traces exist 2. Backfill → historical traces exist 3. Faithfulness judge → content quality scored 4. JSON schema grader → output shape scored 5. GEPA basics → judges sharpened ``` In one week, you go from "we have a chatbot" to "we have measured quality across thousands of past interactions, with graders we trust enough to block bad releases on." ## Where to next * **More recipes by problem, framework, industry, integration:** [All cookbook recipes](/2.-get-started/all-cookbook-recipes) * **Guided learning paths instead:** [All tutorials](/2.-get-started/all-tutorials) * **Task-shaped walkthroughs:** [All guides](/2.-get-started/all-guides) * **Workflow context:** [The LayerLens workflow](/1.-introduction/the-layerlens-workflow) # Pricing — overview Pricing for LayerLens — pay-as-you-go ECU credits. ## What is an ECU? An **Evaluation Compute Unit** (ECU) is the billing unit for compute-intensive operations: * Model inference during an evaluation * Grader runs (LLM graders consume ECU) * GEPA optimization runs * Trace evaluation runs ECU is pay-as-you-go: buy ECU when you need it from **Admin → Billing**. [Concept: ECU credits](/11.-admin/concepts-ecu-credits) covers the economics and how ECU is consumed. ## FAQ **Can I use my own LLM provider key?** Yes. BYOK custom models are supported. Provider tokens consumed against your key don't count against ECU. **Can I cancel?** ECU is pay-as-you-go with no commitment. **Is there a non-profit / academic discount?** Yes — email . ## Where to next * [How LayerLens compares](/1.-introduction/how-layerlens-compares-deep) * [Industry patterns](/3.-pricing/03-pricing) # Use cases Concrete ways teams use LayerLens — model evaluation, agentic eval, RAG, CI/CD gates, continuous eval. What LayerLens is *for* — six concrete shapes of work that the platform handles end-to-end.
Model evaluationPick the right model for a task by running it against the right benchmark.Read
Benchmark-driven developmentMake benchmarks a first-class signal in your dev loop.Read
Agentic evaluationPre- and post-deployment gates for multi-step agents — assertions and graders.Read
Continuous evaluationScore live production traces on a recurring schedule.Read
RAG evaluationFaithfulness, retrieval quality, end-to-end answer quality.Read
AI quality gates in CI/CDBlock prompt and model regressions in your merge pipeline.Read
## Pick a use case | If you want to... | Read | | ---------------------------------- | ----------------------------------------------------------------------------------- | | Choose a model for a new feature | [Model evaluation](/4.1-general-use-cases/model-evaluation) | | Score multi-step agents pre-deploy | [Agentic evaluation](/4.1-general-use-cases/agentic-evaluation) | | Score real production traffic | [Continuous evaluation](/4.1-general-use-cases/continuous-evaluation) | | Evaluate a RAG pipeline end-to-end | [RAG evaluation](/4.1-general-use-cases/rag-evaluation) | | Block bad changes from merging | [AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd) | | Treat benchmarks as a daily signal | [Benchmark-driven development](/4.1-general-use-cases/benchmark-driven-development) | ## Want to map a use case to a tutorial? * [First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation) * [Build your first grader](/8.-evaluate-score-the-outputs/02-first-judge) * [Wire CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates) * [Score live traces](/8.-evaluate-score-the-outputs/04-score-traces) # Model evaluation Pick the right model for a task by running it against the right benchmark. Picking a model is the most expensive decision you'll make as an AI team. The wrong choice locks in months of wasted iteration on a foundation that was never going to work. LayerLens's model-evaluation use case is the disciplined version of that decision. ## The shape of the work 1. **Frame the question.** What does your feature actually need? Reasoning, structured output, multi-turn dialog, code, multilingual, multimodal? Don't pick a model on vibes. 2. **Find the right benchmarks.** Browse the [benchmarks page](/5.-select-pick-the-model/benchmarks). Pick 1-3 that map to your task. (Most teams pick too many; resist.) 3. **Compare models.** [Compare evaluations](/8.-evaluate-score-the-outputs/evaluations) head-to-head on published benchmark results. Get to a shortlist of 3-5. 4. **Run a private evaluation on your data.** Public scores tell you what's strong in general. Your data tells you what's strong for *you*. LayerLens runs a private evaluation against your dataset. 5. **Read the verdict, not the headline.** A model that scores 2% higher overall but 15% lower on your one critical dimension is the wrong pick. ## Why it works on LayerLens * **Published benchmark results give you the shortlist.** No multi-week vendor proof-of-concepts to find candidate models. * **Private evaluations on your data give you the verdict.** Public leaderboards don't see your prompts or your distribution. * **Compare models is built in.** Side-by-side score tables. * **Costs are visible up-front.** The credit cost is shown before you run. ## Tools you'll use * [Models](/5.-select-pick-the-model/models) * [Compare evaluations](/8.-evaluate-score-the-outputs/evaluations) * [Evaluations](/8.-evaluate-score-the-outputs/evaluations) * [SDK: `client.evaluations.create()`](/4.1-general-use-cases/general) ## Outcomes you should see You'll know this is working when: * **Model selection time drops from weeks to hours.** Published results narrow; private eval decides. * **Every model decision has a citable evaluation ID.** No "we picked X because it felt better." * **Re-evaluating a new candidate model takes under 1 hour.** Including standing up the eval against your dataset. * **Cost and latency are part of the choice, not afterthoughts.** You can tell the team "we picked the cheaper model because at 95% of the quality it's 3× cheaper." ## Anti-patterns * **Benchmark cargo-culting.** "Everyone uses MMLU" is not why MMLU is right for your task. * **One-shot picking.** The model frontier moves quarterly. Re-evaluate the candidates as new models drop. * **Ignoring cost and latency.** A 2% accuracy gain at 4× cost is rarely worth shipping. ## Where to next * [Concept: Models and benchmarks](/5.-select-pick-the-model/models-and-benchmarks) * [Tutorial: First evaluation in 10 minutes](/8.-evaluate-score-the-outputs/01-first-evaluation) * [Workflow: Compare](/9.-improve-tune-the-system/workflow) * [Cookbook: model-selection recipes](/2.-get-started/all-cookbook-recipes) # Benchmark-driven development Make benchmarks a first-class signal in your dev loop — every change runs against a benchmark, every result is on the dashboard. Most teams treat benchmarks as a one-time activity at the start of a project. Pick a model, glance at MMLU, get on with it. **Benchmark-driven development** flips that — every meaningful change to your AI feature runs against the benchmarks that matter, and every result lives on a shared dashboard. ## The shape of the work 1. **Pick your benchmarks.** Public benchmarks for general capability + your own private benchmark suite for task-specific quality. 2. **Wire it into your dev loop.** Local: `pytest`-style harness running a small slice. CI: full benchmark suite on every PR. Nightly: full suite + production-traffic slice. 3. **Track the score over time.** LayerLens's evaluation history page shows the score curve per benchmark. 4. **Block regressions in CI.** A score that drops below baseline blocks the PR. 5. **Report wins to the team.** When a change improves a benchmark, share it. ## Why it works on LayerLens * **52+ public benchmarks ready to use** — you don't have to host the data * **Private benchmark suites** — upload your own dataset and grading config * **Score history** — every run is recorded; trends are visible * **CI gates** — easily wired via the SDK or CLI * **Compare models** — when a benchmark gets stronger, see if a different model is now a better fit ## Tools you'll use * [Benchmarks](/5.-select-pick-the-model/benchmarks) * [SDK: `client.evaluations.create()`](/4.1-general-use-cases/general) * [CLI: `layerlens evaluate`](/4.1-general-use-cases/general) ## Outcomes you should see You'll know this is working when: * **Every PR touching AI runs a benchmark gate** — non-negotiable, not opt-in. * **Your benchmark gating eval runs in <5 minutes** — fast enough that no one wants to skip it. * **Score-over-time trends are visible to the whole team**, not buried in one engineer's terminal. * **Net-new benchmarks reach 80% of your team within a week** — adding signal is cheap and shared. ## Anti-patterns * **Benchmark inflation.** Adding 30 benchmarks because "more signal is better." Pick 3-5 that matter, run them often. * **Treating public scores as your scores.** Public scores are a leading indicator. Your private benchmark on your data is the verdict. * **Skipping CI.** If benchmarks don't gate merges, they're decoration. ## Where to next * [Tutorial: Wire CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates) * [Workflow: Evaluate](/9.-improve-tune-the-system/workflow) * [Concept: Models and benchmarks](/5.-select-pick-the-model/models-and-benchmarks) * [Cookbook: CI/CD recipes](/2.-get-started/all-cookbook-recipes) # Agentic evaluation Pre- and post-deployment quality gates for multi-step agents — combine assertions and LLM graders. Agents fail differently from chat. They reach a wrong final state, take a wrong path to a right state, call a tool they shouldn't, or quietly regress on an edge case. Agentic evaluation is the **pre- and post-deployment** practice of catching these failures before they ship. ## The shape of the work 1. **Capture a representative trace set.** Your agent's actual runs — inputs, tool calls, outputs, every span. Real or synthetic. 2. **Define evaluation criteria.** Mix two types: * **Natural-language assertions** — "the agent identified the customer's account tier" * **LLM graders** — "rate the helpfulness of the final response 1-5" 3. **Run the evaluation.** LayerLens runs the criteria over the trace set. 4. **Read verdict + root-cause.** Each failed criterion ties to the trace, the span, and the decision that broke it. 5. **Detect regressions.** Compare to a baseline; surface newly failing criteria. ## Why it works on LayerLens * **Grader engine + GEPA optimization** — your subjective bar gets sharper over time * **Regression detection** — built-in baseline comparison * **Pre- and post-deployment fit** — runs on your candidate change, not on live traffic ## Tools you'll use * [Agent Evaluation](/7.-observe-see-whats-happening/agent-evaluation) * [Traces](/7.-observe-see-whats-happening/traces) * [Graders](/8.-evaluate-score-the-outputs/judges) * [SDK: `client.trace_evaluations.create()`](/4.1-general-use-cases/general) ## Outcomes you should see You'll know this is working when: * **>95% pass rate on your assertion criteria** across the curated trace set. * **Regression report names <2 newly failing criteria** per release. * **Auditor questions about agent safety are answered live in-meeting** with the verdict + root-cause artifacts. ## Anti-patterns * **Grader-only evaluations.** Graders are the slowest, most expensive criterion. Anchor with assertions; use graders for the residual subjective bar. * **No regression detection.** Without a baseline, today's pass rate is meaningless. * **Evaluating live traffic for pre-deploy gates.** Live traffic is for [continuous evaluation](/4.1-general-use-cases/continuous-evaluation). Pre-deploy uses a captured trace set. ## Where to next * [Concept: Agentic evaluation](/4.1-general-use-cases/agentic-evaluation) * [Overview: Agentic evaluations](/4.1-general-use-cases/agentic-evals-overview) * [Workflow: Evaluate](/9.-improve-tune-the-system/workflow) * [Cookbook: agentic recipes](/2.-get-started/all-cookbook-recipes) # Agentic evaluations — overview Pre- and post-deployment quality gates for multi-step AI agents — assertions and LLM graders. Most AI features today are not single-turn chat. They're **agents** — multi-step systems that plan, call tools, retrieve context, and decide when they're done. Single-output evaluation isn't enough for agents. You need to know whether the agent reached the right state, took legal steps to get there, and won't regress when the prompt changes next week. That's what Agentic Evaluations does. ## The failure scenarios Agents fail in ways simple chat doesn't. The LayerLens Agentic Evaluations product is shaped around these specific failures: * **Wrong final state.** The agent answered the question but updated the wrong record. * **Right answer, wrong path.** It got there, but called an internal API it shouldn't have. * **Catastrophic tool selection.** A `delete_user` call when the user asked to "deactivate." * **Off-policy reasoning.** The agent's chain-of-thought reveals reasoning your team doesn't want shipped (PII, leaked instructions, harmful claims). * **Silent regression.** A prompt change five PRs ago broke an edge case nobody is watching. ## Evaluation criteria Agentic evaluations combine two kinds of criteria: 1. **Natural-language assertions.** Plain-English checks the LLM grades. "The agent correctly identified the customer's account tier." Cheap, flexible, good for fuzzy correctness. 2. **LLM graders.** Subjective dimensions evaluated by an optimized grader. "How helpful was the final response on a 1-5 scale?" Use sparingly — most subjective bars resolve into combinations of assertions. Assertions cover the bulk of correctness. Graders grade the residual subjective quality. ## The pre- and post-deployment workflow The LayerLens Agentic Evaluations workflow is built for **pre- and post-deployment** — you run it on a candidate change before it ships, not on live production. 1. **Capture a representative trace set.** Real or synthetic, agent inputs + outputs + every span. 2. **Define your evaluation criteria.** A mix of assertions and graders. 3. **Run the evaluation.** LayerLens replays the criteria over the trace set. 4. **Read the verdict and root-cause.** Each failed criterion ties back to the trace, the span, the input, and the agent decision that broke it. 5. **Detect regressions.** Compare the run to a baseline; surface the criteria that newly failed. The output artifacts are: * A **verdict** (pass/fail and severity per criterion) * A **root-cause report** (which span, which input, which decision) * A **regression report** (what newly fails compared to baseline) You wire those into a CI gate, a release-readiness review, or a Slack alert. ## What you get vs what you build You build: * The trace set (your data, your agent runs) * The evaluation criteria (your assertions, your graders) * The CI/CD wiring LayerLens gives you: * The grader engine and the GEPA optimizer * The verdict, root-cause, and regression reports * The UI for browsing failures and the SDK for automation ## When NOT to use Agentic Evaluations * For **single-turn chat with no tools or chain**, the Model Evaluations product is simpler. Pick that. * For **post-deployment monitoring of live traffic**, [trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) is the right shape — same engine, recurring schedule, applied to real production traces. ## Where to next * [Use case: Agentic evaluation](/4.1-general-use-cases/agentic-evaluation) * [Concept: Agentic evaluation](/4.1-general-use-cases/agentic-evaluation) * [Workflow: Evaluate](/9.-improve-tune-the-system/workflow) * [Agent Evaluation](/7.-observe-see-whats-happening/agent-evaluation) # Continuous evaluation Score live production traces on a recurring schedule — surface drift before customers do. Pre- and post-deployment evaluations catch regressions before merge. **Continuous evaluation** catches them after merge — when something slips through, when production traffic shifts, when the model provider quietly updates their backend. ## The shape of the work 1. **Ingest live traces.** Your application emits trace JSON; LayerLens ingests via API or SDK. 2. **Define a trace evaluation.** Pick the graders you want applied. The same graders from your pre- and post-deployment evaluations work here. 3. **Automate it.** Add an automation rule to a grader so new traces matching a filter (agent, framework, model, status, tags) are judged automatically, with a sample rate and a daily cap. 4. **Watch the trend.** LayerLens shows score-over-time per grader per dimension. 5. **Get notified.** Route trace-evaluation results and grader verdicts to in-app notifications or an outbound destination such as Slack. ## Why it works on LayerLens * **Same engine across pre-deploy and continuous.** The graders you built for CI gates are reusable. * **Trace-first design.** LayerLens's trace pipeline is shaped for continuous workflows; ingestion is rate-limited and persistent. * **GEPA-optimized graders.** Your grader agreement with humans stays tight as live traffic shifts. * **Org-scoped and tenant-isolated.** One tenant's burst doesn't affect another. ## Tools you'll use * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) * [Traces](/7.-observe-see-whats-happening/traces) * [SDK: trace ingestion](/4.1-general-use-cases/general) * [Notifications](/11.-admin/notifications) ## Outcomes you should see You'll know this is working when: * **Mean time to detect a quality regression drops to <1 hour** (versus days/weeks of customer-driven discovery). * **Score-over-time charts are reviewed weekly** by the team that owns the AI feature. * **Sample bias is measured**, not assumed — you can show the sample distribution matches production. ## Anti-patterns * **Only scoring sampled traces.** Sampling is fine, but watch sample bias. Production distribution drifts; sample drifts with it. * **Mixing pre-deploy and continuous in one config.** They have different cadence and different cost profiles. Keep them separate. * **No alerting.** A continuously-evaluated trace pipeline that nobody reads is decoration. ## Where to next * [Concept: Continuous evaluation](/4.1-general-use-cases/continuous-evaluation) * [Tutorial: Score live traces](/8.-evaluate-score-the-outputs/04-score-traces) * [Workflow: Observe](/9.-improve-tune-the-system/workflow) * [Cookbook: continuous-eval recipes](/2.-get-started/all-cookbook-recipes) # RAG evaluation Evaluate RAG pipelines end-to-end — retrieval quality, faithfulness, and answer quality. A RAG (retrieval-augmented generation) pipeline has more failure modes than a plain LLM call. Bad retrieval kills good models; great retrieval can't save a hallucinating one. RAG evaluation scores each stage so you know where the failure actually is. ## The shape of the work 1. **Define the three dimensions.** * **Retrieval quality** — did we get back relevant chunks? * **Faithfulness** — did the answer ground itself in the retrieved chunks (not invent)? * **Answer quality** — did the final answer actually help the user? 2. **Pick graders.** * Retrieval: classical IR metrics (precision\@k, recall\@k) as graders; or an LLM grader that grades chunk relevance. * Faithfulness: an LLM grader that compares answer claims against the retrieved chunks. GEPA-optimize against a labeled set. * Answer quality: an LLM grader for end-to-end helpfulness, again GEPA-optimized. 3. **Run as a trace evaluation.** Your pipeline emits a trace with retrieval span + generation span. LayerLens grades each. 4. **Compare configurations.** Try a different retriever, a different chunk size, a different model — re-run, see the score deltas per dimension. 5. **Lock in CI gates.** Faithfulness regressions are particularly costly; gate them. ## Why it works on LayerLens * **Trace-first.** Multi-span traces are first-class; you can score retrieval and generation separately. * **Grader optimization.** GEPA tunes graders to match human labels for faithfulness — usually the trickiest dimension. * **Compare-models.** Try the same retrieval feeding three different generators; see which generator is most faithful to the same chunks. ## Tools you'll use * [Agent Evaluation](/7.-observe-see-whats-happening/agent-evaluation) (RAG is a small agent) * [Traces](/7.-observe-see-whats-happening/traces) * [Graders](/8.-evaluate-score-the-outputs/judges) * [SDK trace ingestion](/4.1-general-use-cases/general) ## Outcomes you should see You'll know this is working when: * **Faithfulness grader agreement with humans crosses 90%** after GEPA optimization. * **Retrieval and answer scores diverge meaningfully** — when answer quality drops, you can tell whether retrieval or generation is at fault. * **Hallucination rate trends downward** over consecutive releases, not just bounces. * **Per-chunk-strategy comparisons take <1 day** instead of multi-week experiments. ## Anti-patterns * **Scoring only the final answer.** You'll know quality dropped; you won't know which stage caused it. * **Faithfulness grader without optimization.** Out-of-the-box LLM graders over-credit confident-sounding hallucinations. GEPA-optimize against a labeled set. * **No retrieval ground truth.** If you can't say "these are the right chunks," your retrieval scores are just plausibility scores. ## Where to next * [Concept: Graders (faithfulness)](/8.-evaluate-score-the-outputs/judges-1) * [Concept: Traces and spans](/6.-build-wire-your-code/traces-and-spans) * [Cookbook: RAG recipes](/2.-get-started/all-cookbook-recipes) # AI quality gates in CI/CD Block prompt and model regressions in your merge pipeline — turn evaluations into a hard gate. Code regressions are caught by tests. AI regressions slip through unless you treat the eval as a test. The CI/CD-quality-gates use case wires LayerLens evaluations into your pipeline so a prompt change that drops accuracy can't merge. ## The shape of the work 1. **Pick your gating evaluation.** A small, fast subset of your benchmarks and graders. Goal: meaningful signal in under 5 minutes. 2. **Add the eval as a CI step.** GitHub Actions, GitLab CI, Buildkite — call the SDK or CLI with the eval config. 3. **Compare against baseline.** LayerLens maintains an evaluation history. The CI step compares the new run to the most recent main-branch run. 4. **Fail the build on regression.** If any tracked dimension drops by more than your tolerance, fail. 5. **Annotate the PR.** Post the score table back to the PR for reviewer context. ## Why it works on LayerLens * **Evaluation history is built-in.** No need to roll your own diff infrastructure. * **SDK and CLI both support CI.** Pick whichever feels native. * **Compare-models lets you stage upgrades.** Switch the model in CI, verify scores, then promote. * **GEPA-optimized graders keep gates honest.** Out-of-the-box graders flake; tuned ones don't. ## Tools you'll use * [SDK: `client.evaluations.create()`](/4.1-general-use-cases/general) * [CLI: `layerlens evaluate`](/4.1-general-use-cases/general) * [Evaluations](/8.-evaluate-score-the-outputs/evaluations) ## Outcomes you should see You'll know this is working when: * **Zero AI regressions reach production** because the CI gate caught them. * **CI gate run time stays under 5 minutes** even as the eval grows. * **Tolerance is calibrated** — false-positive rate <5%, false-negative rate <2%. * **Engineers cite the eval result in the PR description** as a matter of habit. ## Anti-patterns * **5-hour CI evals.** If your team waits 5 hours for a merge gate, they'll ignore it. Pick a fast subset. * **Whole-org tolerance.** Different repos have different quality bars. Set per-repo or per-feature tolerances. * **No alerting on baseline drift.** If the baseline silently drops over 4 PRs, you never noticed because each PR was within tolerance. Watch the trend. ## Where to next * [Tutorial: Wire CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates) * [Workflow: Govern](/9.-improve-tune-the-system/workflow) * [Cookbook: CI/CD recipes](/2.-get-started/all-cookbook-recipes) * [Integrations: GitHub Actions](/4.1-general-use-cases/general) # Industry solutions LayerLens industry solutions — 13 verticals with scenarios, eval patterns, compliance, and runnable samples. Each industry page has the same five sub-pages: * **README** — top scenarios + why LayerLens * **Scenarios** — 1-4 specific evaluation scenarios adapted from the LayerLens scenarios catalog * **Eval patterns** — what graders typical teams use * **Compliance** — the regulatory considerations to set up first * **Cookbook and samples** — runnable starter samples
HealthcareClinical decision support, triage, medical Q&A.Read
Financial servicesFraud detection, advisory, KYC.Read
LegalContract review, research, document analysis.Read
Retail and e-commerceProduct Q&A, support, recommendations.Read
Media and entertainmentModeration, summarization, search.Read
InsuranceClaims triage, policy Q&A, underwriting.Read
Government and public sectorCitizen services, document processing.Read
ManufacturingDiagnostics, manuals Q&A, quality.Read
EducationTutoring, content generation, assessment.Read
Real estateListing Q&A, valuation, document search.Read
TelecommunicationsService, troubleshooting, plans Q&A.Read
Energy and utilitiesField service, ops, customer support.Read
Travel and hospitalityBooking, itinerary, customer service.Read
## Source note Industry pages adapt the LayerLens-internal scenarios catalog. Where a scenario depended on capabilities not yet shipping in LayerLens (rules engine, remediation, prompt lab, chaos, A/B testing), that component is omitted rather than overstated. # By AI workflow shape Industry patterns indexed by AI workflow shape — RAG, agent, classification, generation, multimodal. A cross-cut index. If you shop for patterns by *what your AI feature does* rather than by *which industry you're in*, this is the entry point. ## Retrieval-augmented generation (RAG) Your AI retrieves source material and synthesizes an answer with citations. Faithfulness against retrieved chunks is the core dimension. * [Healthcare — clinical decision support](/4.2-industry-use-cases/pattern) — guideline retrieval + faithfulness + drug-interaction rules * [Government — citizen-services chatbot](/4.2-industry-use-cases/pattern-4) — versioned policy retrieval + reading-level + multilingual parity * [Legal — AI research citation verification](/4.2-industry-use-cases/pattern-2) — case-law retrieval + citation-existence + holding-accuracy * [Manufacturing — visual quality inspection](/4.2-industry-use-cases/pattern-7) (uses retrieval over manuals when paired with technician guidance) ## Multi-step agents Your AI plans, calls tools, and decides when it's done. Pre- and post-deployment agentic evaluation is the right shape. * [Insurance — claims-triage agent](/4.2-industry-use-cases/pattern-3) — span-level escalation rule + assertions + tone grader * [Healthcare — clinical decision support](/4.2-industry-use-cases/pattern) — also fits if the assistant calls drug-interaction databases as tools ## Classification and detection Your AI classifies content into categories or detects targets in inputs. Per-class precision/recall, confusion-matrix tracking, and per-segment fairness are the dimensions. * [Media — content moderation](/4.2-industry-use-cases/pattern-6) — per-category recall + cross-language parity * [Manufacturing — visual quality inspection](/4.2-industry-use-cases/pattern-7) — per-defect-class accuracy + condition-stratified disparity * [Energy — vegetation-management imagery](/4.2-industry-use-cases/pattern-11) — hazard-class accuracy + season-stratified disparity * [Financial services — fair-lending platform](/4.2-industry-use-cases/pattern-1) — false-positive disparity across protected-class proxies ## Content generation Your AI generates user- or customer-facing copy, descriptions, summaries. Specification accuracy, brand voice, and uniqueness are the dimensions. * [Retail — product description generation](/4.2-industry-use-cases/pattern-5) — specification accuracy + brand voice + uniqueness * [Travel — multilingual booking assistant](/4.2-industry-use-cases/pattern-12) — booking-detail accuracy + cultural fit per locale ## Conversational with policy gates Your AI handles a multi-turn customer interaction with strict policy or pricing rules. Citation, exact-match, and continuous trace eval are the dimensions. * [Telecom — customer-service pricing accuracy](/4.2-industry-use-cases/pattern-10) — JSON-schema pricing citation + numeric exact-match * [Education — AI tutor](/4.2-industry-use-cases/pattern-8) — Socratic vs. answer-giving + age-appropriateness * [Government — citizen-services chatbot](/4.2-industry-use-cases/pattern-4) — also fits ## Multimodal (image / video / audio) Your AI processes non-text inputs. Per-modality graders paired with text-output LLM graders. * [Manufacturing — visual quality inspection](/4.2-industry-use-cases/pattern-7) — image classification under varied lighting * [Energy — vegetation-management imagery](/4.2-industry-use-cases/pattern-11) — drone imagery, season-stratified * [Real estate — automated property valuation](/4.2-industry-use-cases/pattern-9) — listing photos + comparable sales ## Where to next * [All 13 patterns by industry](/4.2-industry-use-cases/industry) * [Industries](/4.2-industry-use-cases/industry) — full per-vertical depth * [Use cases](/4.1-general-use-cases/general) — by AI workflow shape (broader framing) * [Workflow](/1.-introduction/the-layerlens-workflow) — the 6-stage spine # Healthcare Healthcare AI evaluation on LayerLens — clinical Q\&A, triage, medical document analysis. Healthcare AI has the highest stakes on the platform. Every output is potentially clinically consequential. Evaluation in healthcare is not optional; it's the gating discipline. ## Top scenarios
Clinical decision-support Q&AAI helps clinicians retrieve guidelines and decision pathways.Read
Patient triage chatbotTriage symptoms; route to appropriate care.Read
Medical document summarizationSummarize charts, prior authorizations, discharge notes.Read
## Why LayerLens for healthcare * **Deterministic rules + graders** combine to enforce hard safety constraints (no dosing recommendations to patients) plus subjective bars (clinical reasoning soundness) * **Trace evaluations** preserve auditable evidence per release — important for regulatory readiness * **GEPA grader optimization** lets you tune to clinician-labeled examples * **BYOK custom models** support medical-specific fine-tuned models running in your environment ## Industry pattern For an end-to-end usage scenario combining the criteria above into one evaluation: → [Healthcare: clinical decision support pattern](/4.2-industry-use-cases/pattern) ## Where to next * [Scenarios](/more-in-this-section-2/scenarios-4) * [Evaluation patterns](/more-in-this-section-2/eval-patterns-4) * [Compliance](/more-in-this-section-2/compliance-4) * [Cookbook and samples](/more-in-this-section-2/cookbook-and-samples-4) # Pattern: clinical decision support Healthcare — clinical decision support pattern. Faithfulness, drug-interaction safety, reasoning soundness, audit-ready evidence. A clinical-AI assistant answers physician questions: differential diagnosis, treatment options, drug-interaction checks based on patient context. Used at the point of care, the system reads guidelines, returns a recommendation, and cites the evidence. This pattern shows how to evaluate that system end-to-end on LayerLens. ## What's at stake | Risk dimension | Magnitude | Framework | | ---------------------------------------------- | -------------------------------------------------- | -------------------------------- | | Patient harm from contraindicated combinations | Adverse drug events at scale of patient encounters | Clinical incident reporting | | Per-incident malpractice exposure | $250K–$5M | Industry settlement bands | | FDA Software as a Medical Device action | Loss of clearance, market withdrawal | FDA SaMD guidance | | Hospital accreditation impact | Joint Commission inquiry, conditional status | TJC sentinel-event protocol | | HIPAA exposure on PHI in trace bodies | Per-violation civil penalties + breach reporting | HIPAA Privacy and Security Rules | ## The evaluation pattern A pre- and post-deployment **agentic evaluation** runs over a 50-100 trace curated set drawn from representative encounter shapes (rare-disease cases, multi-morbidity, pediatric vs. geriatric weight-based dosing, contraindicated drug combinations, common edge cases). **Criteria mix (the 70/20/10 healthcare ratio):** 1. **Deterministic rules (\~70%)** * Every drug pair the AI mentions is checked against an authoritative interaction database. Missing checks = CRITICAL. * Every dose mention extracts cleanly and matches the prescribing reference exactly. * PHI-redaction regex confirms no patient identifiers appear in any logged span. 2. **Natural-language assertions (\~20%)** — the AI's recommendation matches the ground-truth label from the case file. 3. **LLM graders (\~10%)** * **Faithfulness grader** (GEPA-tuned against ≥50 clinician-labeled examples — multi-class severity output) — every clinical claim is grounded in retrieved guideline content. * **Reasoning soundness grader** (GEPA-tuned against ≥50 examples) — penalize correct answers via flawed reasoning. > Don't have labels yet? See [Bootstrap a grader before GEPA](/more-in-this-section-6/bootstrap-judges) for the week-1 setup. **Trace shape captured:** ``` trace ├── span: retrieval (guideline_corpus) │ └── outputs: chunks[] ├── span: llm (answer-synthesis) │ ├── inputs: question, chunks │ └── outputs: answer, citations └── span: post-process (citation-validation) ``` Score retrieval (precision\@k against ground-truth chunks) and synthesis (faithfulness against retrieved chunks) separately. **Continuous trace evaluation:** the same configuration runs daily on a 5% sample of production traffic. Results route to in-app and Slack notifications. The `dose-mention` and `drug-interaction-coverage` rules drop to a per-trace alert, not a daily summary. ## Configuration in code ```python # Python (SDK) from layerlens import Client client = Client() faithfulness = client.judges.create( name="clinical-faithfulness", evaluation_goal="Every clinical claim in OUTPUT must be supported by chunks in the retrieved guideline content.", ) ``` ## What you get * Pre- and post-deployment evaluations surface contraindicated-combination patterns before patient impact, not 6 weeks into production. * Each release pin cites the evaluation IDs that gated it — a self-contained audit packet for FDA SaMD and Joint Commission inquiries. * Continuous trace evaluation catches drift when formulary or guideline updates shift effective knowledge. * Auditor-ready evaluation evidence is a byproduct, not a separate engineering project. ## LayerLens capabilities used * [Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation) * [Graders with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — faithfulness and reasoning soundness * [Custom code graders](/more-in-this-section-6/custom-code-scorer) — drug-interaction database lookup, dose-mention extraction * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous on production * [Notifications](/11.-admin/notifications) — Slack and in-app routing ## Replicate this * [Industry → Healthcare](/4.2-industry-use-cases/healthcare) * [Healthcare scenarios](/more-in-this-section-2/scenarios-4) — six clinical-AI scenarios with full eval criteria * [Healthcare evaluation patterns](/more-in-this-section-2/eval-patterns-4) — the 70/20/10 rule + GEPA labeling guide * [Healthcare compliance](/more-in-this-section-2/compliance-4) — HIPAA / BAA, FDA SaMD, state regs * [Cookbook: clinical Q\&A safety grader](/more-in-this-section-2/industry-healthcare-clinical-qa) # Financial services Financial services AI evaluation — fraud detection, advisor chatbots, KYC. ## Top scenarios
Advisor chatbotCustomer-facing financial guidance with strict guardrails.Read
Fraud-detection assistantTriage suspicious transactions; explain decisions to analysts.Read
KYC document reviewExtract and verify identity documents.Read
## Why LayerLens * Hard regulatory constraints become deterministic rules; subjective bars become graders * Audit-friendly evaluation IDs for compliance evidence * BYOK custom models for in-house fine-tuned compliance models ## Industry pattern → [Financial services: fair-lending platform pattern](/4.2-industry-use-cases/pattern-1) ## Where to next * [Scenarios](/more-in-this-section-2/scenarios-2) | [Eval patterns](/more-in-this-section-2/eval-patterns-2) | [Compliance](/more-in-this-section-2/compliance-2) | [Cookbook](/more-in-this-section-2/cookbook-and-samples-2) # Pattern: fair-lending platform Financial services — fair-lending platform pattern. Bias scorer shared across product teams, variant comparison, continuous monitoring. A bank deploys AI in many places — fraud detection, underwriting, customer service, marketing personalization. Each product team owns its own model, but **fair-lending exposure cuts across all of them**. A platform team owns that cross-cutting risk. This pattern shows how the platform team standardizes fair-lending evaluation across product teams using a single org-scoped LayerLens configuration that every team consumes. ## What's at stake | Risk dimension | Magnitude | Framework | | ---------------------------------------------------------------------- | --------------------------------------------------- | ------------------------------------ | | ECOA / fair-lending civil enforcement | Per-violation civil penalties + consent-decree cost | CFPB / DOJ enforcement actions | | Per-declined-legitimate-transaction friction | \~$25 per false positive | Card-network industry benchmarks | | Annual US card fraud (the upside the model targets) | $4.2B | Federal Reserve / Nilson Report data | | Brand and customer-trust impact under public consent decree | Multi-quarter remediation cycles | Public consent-decree filings | | State DOI / state regulator exposure on insurance and lending products | State-by-state penalty bands | State regulator actions | ## The evaluation pattern A single **org-scoped LayerLens configuration** every product team uses: 1. **Bias grader (custom code)** — computes per-segment false-positive disparity ratios against protected-class proxies (ZIP code, surname-derived demographics, age band). Flags any segment exceeding the **1.25× threshold** derived from regulatory adverse-impact guidance. 2. **Compare models** — when a product team proposes a new model, the platform team's bias grader runs against it. Teams pick the variant that minimizes disparity without sacrificing primary-task performance (fraud recall, underwriting accuracy, etc.). 3. **Continuous trace evaluation** — production traffic from each product team is sampled hourly; bias trends are visible to the platform team's dashboard with per-team and per-feature breakdowns. 4. **Shared grader library** — helpfulness, faithfulness, safety graders curated centrally and GEPA-tuned against pooled labels (≥50 per grader, contributed across the 12 product teams); reused everywhere. See [Bootstrap a grader before GEPA](/more-in-this-section-6/bootstrap-judges) for what to ship before the labels are there. 5. **Per-feature tolerances** — different features have different bars. **Onboarding:** new product teams get the bias grader pre-wired into their CI from day one. ## What you get * Bias becomes a measurement, not a discovery. Every product team sees the same scoreboard. * Cross-team conversations stop being "your team's eval said it was fine" and start being "the org's quality bar didn't move on this PR." * Regulator-readable evaluation history quantifies disparity and tracks remediation per release. * Platform team curates \~5 cross-cutting graders; product teams add their own task-specific graders as needed. ## LayerLens capabilities used * [Custom code graders](/more-in-this-section-6/custom-code-scorer) — bias-disparity computations * [Compare evaluations](/8.-evaluate-score-the-outputs/evaluations) — variant selection with bias-aware criteria * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled production scoring * [Multi-tenancy + shared org library](/7.-observe-see-whats-happening/multi-tenancy) — graders reusable across teams * [Standardize graders across teams](/more-in-this-section-6/standardize-judges) ## Replicate this * [Industry → Financial services](/4.2-industry-use-cases/financial-services) * [Workflow: Govern](/9.-improve-tune-the-system/workflow) * [Cookbook: GitHub Actions integration](/6.-build-wire-your-code/integration-github-actions) * [Use case: AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd) # Pattern: AI research citation verification Legal — AI research citation verification pattern. Citation-existence scorer, holding-accuracy judge, per-session block on unverified output. An AI legal research assistant answers case-law and statutory questions, drafts memo passages, and supports brief writing. The output cites cases, statutes, and regulations. A single fabricated citation in a filing is a sanctionable event — see *Mata v. Avianca, Inc.*, No. 22-cv-1461 (S.D.N.Y. June 22, 2023). This pattern shows how to evaluate every research-tool output so a fabricated citation can never reach a court filing. ## What's at stake | Risk dimension | Magnitude | Framework | | ----------------------------------------------------- | -------------------------------------------- | --------------------------------------- | | Court sanctions per *Mata*-style filing | Cited sanctions $5K + reputational loss | Public court orders | | Bar discipline | Public reprimand to suspension; case-by-case | State bar rules of professional conduct | | Malpractice exposure for filed hallucinated citations | Multi-million-dollar settlements | Public legal-malpractice filings | | Firm reputation in client and ABA Journal coverage | Long-tail damage cycle | Industry trade press | ## The evaluation pattern A two-stage evaluation runs on every research-tool output: 1. **Custom code grader (citation existence)** — every cited case, statute, or regulation is looked up against an authoritative legal database. Any citation that doesn't resolve = fabricated. Hallucination rate must be **0%** for the evaluation to pass. 2. **Holding-accuracy grader** (GEPA-tuned against ≥50 attorney-labeled examples — scored output) — for cases that exist, does the cited case actually stand for the proposition the AI claims? Does the quoted passage match the source verbatim? > Don't have labels yet? See [Bootstrap a grader before GEPA](/more-in-this-section-6/bootstrap-judges) for the week-1 setup. 3. **Continuous trace evaluation** — every production research session is scored. Sessions with any unverified citation are flagged in-app for attorney review **before any output can be acted on**. The flag is a hard gate; attorneys can override only after explicit acknowledgment. 4. **Compare-models** — when prompt or model changes are proposed, run the existing regression set across both; pick the variant with the lower hallucination rate at equal or better holding-accuracy. The same pattern works for statutes, regulations, and internal precedent citations. ## Configuration in code ```python # Python (SDK) from layerlens import Client client = Client() holding_judge = client.judges.create( name="holding-accuracy", evaluation_goal="For each cited case, does the case actually stand for the proposition the OUTPUT claims? Quoted passages must match the source verbatim.", ) ``` ## What you get * Hallucination rate held at 0% on filed-output paths — the gate is a hard block, not a warning. * Holding-accuracy grader agreement with senior associates above 90% after GEPA optimization. * Documented evaluation evidence — the evaluation IDs and per-citation verification artifacts — to show partners, clients, and (if needed) the court that AI-assisted research operates under verification controls. * Attorneys treat the AI as a research-acceleration tool, not a citation source. Which is what the AI should be. ## LayerLens capabilities used * [Custom code grader](/more-in-this-section-6/custom-code-scorer) — citation-existence verification against legal databases * [Graders with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — holding accuracy and quote accuracy * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — every production session * [Notifications](/11.-admin/notifications) — flag unverified citations for attorney review ## Replicate this * [Industry → Legal](/more-in-this-section-2/legal) * [Cookbook: catch hallucinations](/more-in-this-section-6/catch-hallucinations) * [Cookbook: legal contract-review faithfulness](/more-in-this-section-2/industry-legal-contract) (analogous pattern for transactional work) * [Use case: RAG evaluation](/4.1-general-use-cases/rag-evaluation) # Insurance Insurance AI evaluation — claims triage, policy Q\&A, underwriting assistance. ## Top scenarios * **Claims triage** — initial classification, escalation routing * **Policy Q\&A** — customer-facing policy questions * **Underwriting assistance** — augment underwriter decisions ## Why LayerLens Hard escalation rules + faithfulness graders keep insurance AI within risk-appropriate boundaries. ## Industry pattern → [Insurance: claims-triage agent pattern](/4.2-industry-use-cases/pattern-3) [Scenarios](/more-in-this-section-2/scenarios-5) | [Eval patterns](/more-in-this-section-2/eval-patterns-5) | [Compliance](/more-in-this-section-2/compliance-5) | [Cookbook](/more-in-this-section-2/cookbook-and-samples-5) # Pattern: claims-triage agent Insurance — claims-triage agent pattern. Span-level escalation rule, assertions, GEPA-tuned tone judge, in one agentic evaluation. A property-and-casualty claims-triage agent extracts loss details from photos and submitted forms, classifies severity, estimates repair cost, and decides whether to auto-approve, request more information, or escalate to a human adjuster. Multi-step, tool-using, customer-facing on the empathy side. This pattern shows how to evaluate the agent end-to-end with one agentic evaluation that mixes hard policy rules with subjective tone scoring. ## What's at stake | Risk dimension | Magnitude | Framework | | ----------------------------------------------- | -------------------------------- | ----------------------------------- | | Annual industry insurance fraud (US, all lines) | \~$80B | Coalition Against Insurance Fraud | | Per-catastrophe-event mis-estimation cost | $50K–$500K | Industry loss-adjustment benchmarks | | State DOI market-conduct examination | Per-finding penalty bands | NAIC market-conduct framework | | Class-action exposure on systemic underpayment | Multi-million-dollar settlements | Public class-action filings | ## The evaluation pattern **Pre- and post-deployment agentic evaluation** over a 400-trace curated set spanning happy paths, ambiguous documentation, and known-edge fraud patterns. **Criteria mix:** 1. **Deterministic span-level rule** — *"if `inputs.claim_amount > $X` then the trace must contain a span with `kind=tool, tool=escalate_to_adjuster`"*. CRITICAL severity on miss. The amount threshold is configurable per line of business. 2. **Natural-language assertions** — the agent's final state matches the policy auto-approval rules; every required document is present in the agent's verification span. 3. **Faithfulness grader** (GEPA-tuned against ≥50 examples — scored output) — claims about coverage and limits are grounded in the policy document. 4. **Tone grader** (GEPA-tuned against ≥50 examples — scored output) — for customer-facing outputs, balance professionalism with appropriate empathy in disruption scenarios. > Don't have labels yet? See [Bootstrap a grader before GEPA](/more-in-this-section-6/bootstrap-judges) for the week-1 setup. **Verdict + root-cause artifact:** every failed criterion ties back to the specific trace, span, and decision that broke. The regression report compares the run to the most recent baseline; newly-failing criteria surface immediately. **Post-deployment, continuous:** trace evaluation on a 5% sample of production claims, daily cadence. Same criteria configuration. Results route to the claims-ops Slack channel. ## Configuration in code ```python # Python (SDK) from layerlens import Client client = Client() faithfulness = client.judges.create( name="policy-faithfulness", evaluation_goal="Coverage and limit claims must be grounded in the cited policy document.", ) tone = client.judges.create( name="empathy-tone", evaluation_goal="In disruption scenarios, the response balances professionalism with appropriate empathy. Penalize tone-deaf or dismissive language.", ) ``` ## What you get * Zero CRITICAL escalation-rule violations across the curated trace set before launch. * Continuous trace evaluation catches drift when policy rules or product mix shifts. * Verdict + root-cause artifacts become the auditor-ready evidence packet for state DOI inquiries — every release pin cites the evaluation IDs that gated it. * Tone-grader agreement with human reviewers above 90% after GEPA optimization, calibrated to your team's customer-experience standard. ## LayerLens capabilities used * [Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation) * [Graders with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — faithfulness and tone * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — pre-deploy and continuous * [Notifications](/11.-admin/notifications) — Slack routing * [Audit-friendly evidence](/11.-admin/audit-friendly-evidence) ## Replicate this * [Industry → Insurance](/4.2-industry-use-cases/insurance) * [Cookbook: detect tool-call regressions in agents](/more-in-this-section-6/agent-tool-regressions) * [Cookbook: insurance claims-triage scoring](/more-in-this-section-2/industry-insurance-claims) * [Workflow: Evaluate](/9.-improve-tune-the-system/workflow) * [Concept: Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation) # Government and public sector Government and public sector AI evaluation — citizen services, document processing, eligibility. ## Top scenarios * **Citizen services chatbot** — benefits, services, status questions * **Document processing** — forms, applications, identity documents * **Eligibility determination assist** — augment caseworker decisions ## Why LayerLens Auditable evaluation evidence per release; deterministic eligibility rules; faithful citation of regulations. ## Industry pattern → [Government: citizen-services chatbot pattern](/4.2-industry-use-cases/pattern-4) [Scenarios](/more-in-this-section-2/scenarios-3) | [Eval patterns](/more-in-this-section-2/eval-patterns-3) | [Compliance](/more-in-this-section-2/compliance-3) | [Cookbook](/more-in-this-section-2/cookbook-and-samples-3) # Pattern: citizen-services chatbot Government / public sector — citizen-services chatbot pattern. Policy accuracy against versioned source-of-truth, plain-language reading-level, multilingual parity. A state or federal agency operates an AI chatbot that helps citizens find services, check eligibility, and complete applications (benefits, permits, tax forms, licenses). The agency's source-of-truth policy documents update on their own cadence — and a stale chatbot answer can deny benefits citizens are entitled to. This pattern shows how to evaluate citizen-services chatbots for policy accuracy, accessibility, and equity. ## What's at stake | Risk dimension | Magnitude | Framework | | ---------------------------------------------------- | --------------------------------------- | ----------------------------------------- | | Eligibility-misstatement civil-rights exposure | Class-action and Title VI claims | Title VI / ADA / state agency rules | | Section 508 accessibility violations | Per-violation penalties | Section 508 (federal) / state equivalents | | Public-trust impact from incorrect official guidance | Long-tail agency-credibility damage | Public agency-performance research | | Congressional / legislative inquiry | Hearing time, executive-branch response | Public oversight records | ## The evaluation pattern A **policy-grounded evaluation** runs against a versioned source-of-truth. 1. **Custom code grader (policy-document-hash check)** — every chatbot answer references the active policy document version; if the active document hash differs from the answer's referenced version, the answer is stale = fail. 2. **Faithfulness grader** (GEPA-tuned against ≥50 caseworker-labeled examples — scored output) — claims about eligibility and procedures are grounded in the cited policy section. 3. **Reading-level grader** (Flesch-Kincaid or grade-level equivalent) — patient-facing answers must be at or below the configured grade level (commonly 6th-8th grade for citizen-services). 4. **Multilingual parity grader** (custom code) — per-language accuracy must be within 5 percentage points of the primary language baseline. 5. **Disclaimer-presence grader** — the chatbot must include the agency's official "this is informational, not a determination" disclaimer where required. > Don't have labels yet? See [Bootstrap a grader before GEPA](/more-in-this-section-6/bootstrap-judges) for the week-1 setup. **Continuous trace evaluation:** sampled hourly during business hours. Policy-document-hash grader runs on every trace (cheap and deterministic). Results route to the agency's program managers. ## Configuration in code ```python # Python (SDK) from layerlens import Client client = Client() faithfulness = client.judges.create( name="policy-faithfulness", evaluation_goal="Eligibility and procedure claims must be grounded in the cited policy section.", ) ``` ## What you get * Stale-policy answers detected within hours, not weeks. * Multilingual accuracy parity is measured per language, not assumed. * Auditor-ready evaluation history for civil-rights and accessibility audits. * Pre-publication block prevents citizen-facing release on a regression. ## LayerLens capabilities used * [Custom code graders](/more-in-this-section-6/custom-code-scorer) — document-hash, multilingual parity, reading-level * [Graders with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — policy faithfulness * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled * [Notifications](/11.-admin/notifications) — program-manager routing ## Replicate this **Get started:** [Cookbook: catch hallucinations](/more-in-this-section-6/catch-hallucinations) is the closest runnable starter (policy faithfulness shape). * [Industry → Government and public sector](/4.2-industry-use-cases/government-public-sector) * [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation) * [Workflow: Govern](/9.-improve-tune-the-system/workflow) # Retail and e-commerce Retail and e-commerce AI evaluation — product Q\&A, customer support, recommendations. ## Top scenarios * **Product Q\&A** — chat answers questions from the catalog * **Support automation** — handle order, return, shipping queries * **Recommendations explanation** — explain why a product was recommended ## Why LayerLens Catalog-faithfulness rules catch hallucinated product attributes. Latency graders gate user-facing performance. ## Industry pattern → [Retail: product description generation pattern](/4.2-industry-use-cases/pattern-5) [Scenarios](/more-in-this-section-2/scenarios-10) | [Eval patterns](/more-in-this-section-2/eval-patterns-10) | [Compliance](/more-in-this-section-2/compliance-10) | [Cookbook](/more-in-this-section-2/cookbook-and-samples-10) # Pattern: product description generation Retail / e-commerce — product-description generation pattern. Specification accuracy, brand voice, return-rate impact. A retail or marketplace platform uses AI to generate product titles, descriptions, and SEO copy from a structured catalog (specs, dimensions, compatibility, brand attributes). At scale (hundreds of thousands to millions of SKUs), AI replaces a content team — but description errors directly drive returns, regulatory exposure, and brand erosion. This pattern shows how to evaluate generated descriptions for specification accuracy and brand voice before they ship. ## What's at stake | Risk dimension | Magnitude | Framework | | ------------------------------------------------------ | ------------------------------------------------------------ | -------------------------------------------- | | Lift in return rate from inaccurate specs | 15-30% return-rate increase documented | E-commerce returns research | | FTC false-advertising exposure | Per-violation civil penalties + corrective-advertising costs | FTC Endorsement Guides + product claim rules | | SEO penalties on duplicate / thin / inaccurate content | Ranking loss, organic traffic impact | Search-engine quality guidelines | | Brand-voice drift across categories | Cross-category brand erosion | Internal brand-equity research | ## The evaluation pattern A **pre-publication evaluation** runs on every batch of AI-generated descriptions: 1. **Custom code grader (specification accuracy)** — extracts every claimed dimension, weight, compatibility, and material from the description; compares against the catalog row exactly. Any mismatch = fail. 2. **Substring grader** — required disclaimers (e.g., regulated-category warnings, age restrictions) present where required. 3. **Brand-voice grader** (GEPA-tuned against ≥50 brand-marketing-team-labeled examples — scored output) — voice consistency across categories. 4. **Plagiarism / uniqueness grader** — generated text must score above a similarity threshold against the existing catalog. > Don't have labels yet? See [Bootstrap a grader before GEPA](/more-in-this-section-6/bootstrap-judges) for the week-1 setup. **Continuous trace evaluation:** sampled at 1% of newly published descriptions, daily. Specification-accuracy results route to the catalog-ops team. ## Configuration in code ```python # Python (SDK) from layerlens import Client client = Client() brand_voice = client.judges.create( name="brand-voice", evaluation_goal="Score brand-voice consistency on a 1-5 scale. Voice should match the brand guidelines for category {category}.", ) ``` ## What you get * Specification-error rate stays under 0.5% (vs. 4-15% baseline reported in industry research). * Return-rate on AI-described products converges to or beats human-written baselines. * Pre-publication block is a hard gate; descriptions that fail any code grader never reach the storefront. * Brand voice variance across categories is a measured dimension on the dashboard. ## LayerLens capabilities used * [Custom code graders](/more-in-this-section-6/custom-code-scorer) — specification-extraction comparison * [Graders with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — brand voice * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled * [Notifications](/11.-admin/notifications) ## Replicate this **Get started:** [Cookbook: retail product Q\&A](/more-in-this-section-2/industry-retail-product-qa) is the closest runnable starter. * [Industry → Retail and e-commerce](/4.2-industry-use-cases/retail-ecommerce) * [Workflow: Evaluate](/9.-improve-tune-the-system/workflow) * [Use case: Continuous evaluation](/4.1-general-use-cases/continuous-evaluation) # Media and entertainment Media and entertainment AI evaluation — moderation, summarization, search. ## Top scenarios * **Content moderation** — flag policy violations * **Summarization** — articles, videos, podcasts * **Search and discovery** — semantic search across catalog ## Why LayerLens Faithfulness graders catch hallucinated content claims; deterministic policy rules enforce moderation policies. ## Industry pattern → [Media: content moderation pattern](/4.2-industry-use-cases/pattern-6) [Scenarios](/more-in-this-section-2/scenarios-8) | [Eval patterns](/more-in-this-section-2/eval-patterns-8) | [Compliance](/more-in-this-section-2/compliance-8) | [Cookbook](/more-in-this-section-2/cookbook-and-samples-8) # Pattern: content moderation Media / entertainment — content moderation pattern. Per-category recall, cross-language parity, EU DSA exposure management. A social platform, marketplace with user-generated content, or media-streaming service uses AI to moderate user content — text, image, video, audio — for policy violations across hate speech, harassment, CSAM, violence, misinformation, and more. The cost of under-moderation includes regulatory enforcement (EU DSA, OSA, state-level US laws); the cost of over-moderation is creator churn and First-Amendment-style litigation. This pattern shows how to evaluate cross-language, per-category moderation accuracy. ## What's at stake | Risk dimension | Magnitude | Framework | | -------------------------------------- | ---------------------------------- | -------------------------------- | | EU Digital Services Act non-compliance | Up to 6% of global annual revenue | EU DSA | | UK Online Safety Act non-compliance | Up to 10% of global annual revenue | OSA | | Advertiser-boycott revenue impact | Multi-quarter advertiser pauses | Industry brand-safety reporting | | Over-moderation creator-churn | Long-tail platform-health impact | Public creator-platform research | ## The evaluation pattern A **multi-criteria evaluation** runs over a labeled trace set spanning every supported language and policy category. 1. **Per-category classifier grader** — recall by policy category (hate speech, CSAM, violence, harassment, misinformation, sexual content, self-harm, regulated goods). Recall must meet per-category minimums; CSAM and self-harm categories require near-100%. 2. **Cross-language parity grader** (custom code) — for each policy category, the per-language accuracy must be within 5 percentage points of the English baseline. Disparities above the threshold flag as regressions. 3. **Borderline grader** (GEPA-tuned against ≥50 trust-and-safety-team-labeled examples — scored output) — handles satire, news commentary, cultural context, reclaimed slurs. Higher false-positive cost than false-negative. 4. **Latency grader** — moderation decisions must complete under per-tier SLOs (e.g., 200ms p95 for live-stream, 5s p95 for upload). > Don't have labels yet? See [Bootstrap a grader before GEPA](/more-in-this-section-6/bootstrap-judges) for the week-1 setup. **Continuous trace evaluation:** sampled at 0.5% of moderation decisions, hourly. Per-language and per-category dashboards visible to T\&S leadership; results for critical categories routed to PagerDuty. ## Configuration in code ```python # Python (SDK) from layerlens import Client client = Client() borderline = client.judges.create( name="borderline-content", evaluation_goal="Distinguish policy-violating content from satire, news commentary, cultural context, and reclaimed slurs. Score 1 (clearly fine) to 5 (clearly violating).", ) ``` ## What you get * Cross-language parity becomes a measured dimension on the T\&S dashboard. * Per-category recall floors enforced — CSAM and self-harm regressions block release. * Auditor-readable evaluation history for EU DSA / OSA risk-assessment requirements. * Appeal-rate signal correlates with grader false-positive trend; T\&S can investigate appeals upstream. ## LayerLens capabilities used * [Custom code graders](/more-in-this-section-6/custom-code-scorer) — cross-language parity * [Graders with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — borderline-content scoring * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled * [Notifications](/11.-admin/notifications) — PagerDuty + Slack routing ## Replicate this **Get started:** [Use case: Continuous evaluation](/4.1-general-use-cases/continuous-evaluation) describes the rolling-sample shape this pattern uses. * [Industry → Media and entertainment](/4.2-industry-use-cases/media-entertainment) * [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation) * [Cookbook: catch hallucinations](/more-in-this-section-6/catch-hallucinations) (analogous pattern for misinformation moderation) # Manufacturing Manufacturing AI evaluation — diagnostics assistance, manuals Q\&A, quality. ## Top scenarios * **Diagnostics assistance** — field-tech troubleshooting with manuals + sensor data * **Manuals Q\&A** — extract procedures from technical documents * **Quality control** — analyze defect images and reports ## Why LayerLens Faithfulness graders against manuals; deterministic procedure-correctness rules; trace evaluations on field-tech sessions. ## Industry pattern → [Manufacturing: visual quality-inspection pattern](/4.2-industry-use-cases/pattern-7) [Scenarios](/more-in-this-section-2/scenarios-7) | [Eval patterns](/more-in-this-section-2/eval-patterns-7) | [Compliance](/more-in-this-section-2/compliance-7) | [Cookbook](/more-in-this-section-2/cookbook-and-samples-7) # Pattern: visual quality inspection Manufacturing / industrial — visual quality-inspection pattern. Per-defect-class accuracy, lighting-robustness, latency under SLO. A manufacturer deploys AI computer vision on assembly lines to detect product defects (scratches, dents, misalignment, color variation, contamination). The model runs at line speed, triggering quality holds when defects are detected. False negatives reach customers as defective product (recall risk). False positives stop the line ($10-50K/hour of lost throughput). This pattern shows how to evaluate inspection accuracy across the conditions actually present on the factory floor. ## What's at stake | Risk dimension | Magnitude | Framework | | -------------------------------------------- | ---------------------------------------------- | ----------------------------------- | | Per-recall cost | $500K–$5M (industry-dependent) | Industry recall-cost benchmarks | | Per-hour line-stop cost from false positives | $10K–$50K | Manufacturing throughput economics | | Safety-incident exposure for missed defects | Potential workers'-comp + product-liability | OSHA + product-liability frameworks | | ISO 9001 / IATF 16949 audit findings | Conditional certification, customer audit risk | ISO / IATF audit standards | ## The evaluation pattern A **stratified evaluation** treats lighting condition, line speed, and operator shift as first-class dimensions. 1. **Per-defect-class grader** — precision and recall by defect category. Trace tags include `defect_type`, `lighting_condition`, `line_speed_bucket`, and `shift`; results break out per-tag. 2. **Custom code grader (condition disparity)** — per-class accuracy across `lighting_condition` and `line_speed_bucket` must stay within 5 percentage points of the baseline. Disparities flag as regressions. 3. **Latency grader** — inspection decisions must complete under the line-cycle-time SLO (commonly 100-500ms p99). 4. **Compare-models** — when a new model variant is proposed, it must close any condition-disparity without degrading per-class accuracy. **Continuous trace evaluation:** sampled at 1% of inspection decisions, hourly. Results route to plant operations and the quality-engineering team. ## What you get * Lighting and line-speed disparities measured, not absorbed into an aggregate metric. * Pre- and post-deployment block on a model variant that worsens any condition. * Per-shift accuracy visibility supports root-cause analysis when defect-escape rates rise. * Auditor-ready evidence for ISO 9001 / IATF 16949 reviews. ## LayerLens capabilities used * [Custom code graders](/more-in-this-section-6/custom-code-scorer) — condition-disparity computations * [Compare evaluations](/8.-evaluate-score-the-outputs/evaluations) — variant selection * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled * [Notifications](/11.-admin/notifications) — plant-ops routing ## Replicate this **Get started:** [Concept: Traces and spans](/6.-build-wire-your-code/traces-and-spans) — capturing inspection-line traces with the right tags is the foundation for the stratified evaluation here. * [Industry → Manufacturing](/4.2-industry-use-cases/manufacturing) * [Workflow: Evaluate](/9.-improve-tune-the-system/workflow) * [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation) # Education Education AI evaluation — tutoring, content generation, assessment. ## Top scenarios * **AI tutor** — Socratic guidance, hints, adaptive practice * **Content generation** — exercises, quizzes, lessons * **Assessment / grading assistance** ## Why LayerLens Graders that distinguish "guides student" from "gives answer" — the core failure mode of educational AI. ## Industry pattern → [Education: AI tutor pattern](/4.2-industry-use-cases/pattern-8) [Scenarios](/more-in-this-section-2/scenarios) | [Eval patterns](/more-in-this-section-2/eval-patterns) | [Compliance](/more-in-this-section-2/compliance) | [Cookbook](/more-in-this-section-2/cookbook-and-samples) # Pattern: AI tutor Education / EdTech — AI tutor pattern. Pedagogical-quality (Socratic vs. answer-giving), age-appropriateness, learning-outcome correlation. An EdTech provider operates an AI tutor that helps students through math, reading, and science. The product's value comes from **teaching**, not answering — a tutor that gives away the answer accelerates the assignment but undermines the learning. COPPA and FERPA frame the regulatory bar; the pedagogical dimension is what differentiates the product. This pattern shows how to evaluate tutor pedagogical quality alongside accuracy and safety. ## What's at stake | Risk dimension | Magnitude | Framework | | ----------------------------------------------------------- | ---------------------------------------------------- | --------------------------------- | | Learning-outcome harm from answer-giving instead of guiding | Long-term student-outcome impact | Education research | | COPPA violation (under-13 users) | Per-violation civil penalties | COPPA | | FERPA violation (student records in trace bodies) | Loss of federal funding eligibility, civil penalties | FERPA | | Age-inappropriate content exposure | District contract termination, brand damage | District procurement requirements | | Accessibility (Section 508 / WCAG) | Procurement disqualification | Section 508 / WCAG 2.1 AA | ## The evaluation pattern A **multi-criteria evaluation** runs against curated tutoring-conversation traces. 1. **Pedagogical-quality grader** (GEPA-tuned against ≥50 educator-labeled examples — scored output, "guide vs. give") — penalize responses that hand the student the answer when the assignment expects the student to derive it. 2. **Subject-accuracy grader** — math and science answers verified against a deterministic answer key; language and reading answers checked against rubric criteria. 3. **Age-appropriateness grader** (GEPA-tuned against ≥50 examples — scored output) — content matches the age band declared in the user's profile. 4. **PII-redaction grader** (deterministic regex) — no student-record fields appear in any logged span (FERPA). 5. **Reading-level grader** — explanations at or below the configured grade level for the student's age band. > Don't have labels yet? See [Bootstrap a grader before GEPA](/more-in-this-section-6/bootstrap-judges) for the week-1 setup. **Continuous trace evaluation:** sampled at 0.5% of tutoring sessions, daily. Pedagogical-quality trends visible to the curriculum-design team. ## Configuration in code ```python # Python (SDK) from layerlens import Client client = Client() pedagogical = client.judges.create( name="socratic-vs-give", evaluation_goal="Score 1-5: did the tutor guide the student to the answer (5) or hand it over (1)? Penalize answers that bypass the learning step the assignment expects.", ) age_appropriate = client.judges.create( name="age-appropriate", evaluation_goal="Content matches the student's declared age band (input.age_band).", ) ``` ## What you get * Direct-answer rate measured per subject and grade band; product-led iteration drives it down over time. * Age-appropriateness regressions blocked at the per-student session boundary. * FERPA-compliant trace handling enforced by code graders. * Auditor-ready evidence for district procurement and state-board reviews. ## LayerLens capabilities used * [Graders with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — pedagogical-quality and age-appropriateness * [Custom code graders](/more-in-this-section-6/custom-code-scorer) — subject-accuracy answer-key check, PII-redaction regex * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled ## Replicate this **Get started:** [Cookbook: education tutor helpfulness](/more-in-this-section-2/industry-education-tutor) is the runnable starter. * [Industry → Education](/4.2-industry-use-cases/education) * [Concept: Graders](/8.-evaluate-score-the-outputs/judges-1) * [Workflow: Evaluate](/9.-improve-tune-the-system/workflow) # Real estate Real estate AI evaluation — listing Q\&A, valuation assist, document search. ## Top scenarios * **Listing Q\&A** — buyer/renter questions about properties * **Valuation assistance** — comparative market analysis * **Document search** — disclosure and contract analysis ## Why LayerLens Faithfulness against listing data; rules for fair-housing compliance language. ## Industry pattern → [Real estate: automated property valuation pattern](/4.2-industry-use-cases/pattern-9) [Scenarios](/more-in-this-section-2/scenarios-9) | [Eval patterns](/more-in-this-section-2/eval-patterns-9) | [Compliance](/more-in-this-section-2/compliance-9) | [Cookbook](/more-in-this-section-2/cookbook-and-samples-9) # Pattern: automated property valuation Real estate / proptech — automated valuation pattern. Per-segment accuracy, fair-housing parity, image-condition assessment. A proptech company or lender uses AI to generate property valuations from comparable sales, property features, listing photos, and market data. Banking partners require fair-housing-compliant valuation accuracy; replicating historical appraisal bias is a non-starter. This pattern shows how to evaluate AVM accuracy and fair-housing parity together. ## What's at stake | Risk dimension | Magnitude | Framework | | --------------------------------------------------------------- | -------------------------------------------------- | -------------------------------------- | | Fair-housing exposure on neighborhood-level valuation disparity | Per-violation civil penalties; consent-decree cost | Fair Housing Act / HUD enforcement | | Banking-partner rejection of an AVM with documented bias | Loss of integration agreements, revenue impact | OCC / FRB safety-and-soundness reviews | | Bad-loan losses from inaccurate valuations | $50K–$500K per loan in the worst-tail outcome | Industry mortgage-loss benchmarks | | Brookings-documented appraisal-undervaluation pattern | $48K average undervaluation in Black neighborhoods | Brookings 2018 / HUD studies | ## The evaluation pattern A **fairness-aware evaluation** runs the AVM against a labeled dataset spanning diverse neighborhoods, price ranges, and property types. 1. **Numeric grader (valuation accuracy)** — AVM estimate within ±5% of appraiser ground truth. 2. **Custom code grader (geographic disparity)** — per-Census-tract or per-MSA disparity in median accuracy. Disparity above the configured tolerance (commonly 1.25× ratio) flags as a regression. 3. **Demographic-correlation grader** — accuracy disparity must not correlate with protected-class proxies (race demographics, income tier). 4. **Image-condition assessment grader** — for AVMs that incorporate listing photos, condition-extraction accuracy from images is scored separately. 5. **Comparable-selection rationale grader** (GEPA-tuned against ≥50 appraiser-labeled examples — scored output) — the AI's choice of comparables is defensible against an appraiser's choice. > Don't have labels yet? See [Bootstrap a grader before GEPA](/more-in-this-section-6/bootstrap-judges) for the week-1 setup. **Continuous trace evaluation:** sampled at 1% of production valuations, daily. Disparity trends visible to compliance and risk committees. ## Configuration in code ```python # Python (SDK) from layerlens import Client client = Client() comparable_judge = client.judges.create( name="comparable-selection", evaluation_goal="Score 1-5: is the AI's choice of comparable sales defensible against an appraiser's selection?", ) ``` ## What you get * Fair-housing disparity becomes a measurement on the dashboard, not a discovery during a regulatory review. * Banking-partner integration approvals come faster — the evaluation evidence packet shrinks the diligence timeline. * Per-comp rationale auditability for borrower-facing valuation explanations. * Pre- and post-deployment block on any model variant that worsens disparity. ## LayerLens capabilities used * [Custom code graders](/more-in-this-section-6/custom-code-scorer) — geographic-disparity and demographic-correlation * [Graders with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — comparable-selection rationale * [Compare evaluations](/8.-evaluate-score-the-outputs/evaluations) — variant selection with fair-housing criteria * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled ## Replicate this **Get started:** [Workflow: Govern](/9.-improve-tune-the-system/workflow) describes the cross-team gate this pattern sits under. * [Industry → Real estate](/4.2-industry-use-cases/real-estate) * [Use case: AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd) * [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation) # Telecommunications Telecommunications AI evaluation — customer service, troubleshooting, plan Q\&A. ## Top scenarios * **Customer service automation** — billing, account, plan questions * **Network troubleshooting** — guided diagnostics * **Plan recommendation** — match customer needs to plans ## Why LayerLens Faithful plan/pricing claims; trace evaluations on multi-turn troubleshooting flows. ## Industry pattern → [Telecom: customer-service pricing accuracy pattern](/4.2-industry-use-cases/pattern-10) [Scenarios](/more-in-this-section-2/scenarios-11) | [Eval patterns](/more-in-this-section-2/eval-patterns-11) | [Compliance](/more-in-this-section-2/compliance-11) | [Cookbook](/more-in-this-section-2/cookbook-and-samples-11) # Pattern: customer-service pricing accuracy Telecom — customer-service AI pricing-accuracy pattern. JSON-schema scorer, numeric exact-match, per-PR CI gating, continuous sampled trace eval. A telecom AI customer-service agent handles billing questions, plan changes, and pricing inquiries. The agent reads the active rate card and applies promotional pricing rules. When it gets the math right, churn drops and customer-care hours fall. When it gets it wrong — even occasionally — the bill is wrong and revenue leaks. This pattern shows how to evaluate pricing accuracy continuously, in CI and in production. ## What's at stake | Risk dimension | Magnitude | Framework | | ---------------------------------------------------------- | --------------------------------- | ---------------------------------- | | Annual revenue leakage from underbilling at 6% error rate | \~$2M for a mid-market carrier | Internal billing-reconciliation | | FCC complaint exposure from overcharging | Per-violation civil-penalty bands | FCC enforcement bureau guidance | | Per-subscriber churn cost from a billing-dispute departure | $300–$600 | Telecom retention benchmarks | | Audit / SOX exposure on unreviewed pricing-engine outputs | Material-weakness risk | Public-company controls frameworks | ## The evaluation pattern **Pre- and post-deployment, in CI:** a fast (≤5 min) regression evaluation runs on every PR touching the customer-service prompt or pricing logic. 1. **JSON-schema grader** — every pricing change in the agent's output must include a `rate_card_row_id` and a citation timestamp. Missing field = fail. 2. **Numeric grader** — extracted prices match the rate-card row exactly (no rounding, no unit substitution). 3. **Compare-models** view — when proposing a prompt variant, compare the variant against the current main-branch baseline on the same regression set. Pick the variant with the lowest error rate. 4. **Tolerance gate** — fail the build if pricing-error rate exceeds 0.5%. **Post-deployment, continuous:** trace evaluation on a 5% sample of live customer interactions, hourly cadence. * Same JSON-schema and numeric graders * Results routed to a Slack channel * Per-trace alert on any CRITICAL error (extracted price differs from rate card by >$5) ## What you get * Pricing regressions caught at PR — never reach production. * Continuous evaluation surfaces drift when the rate card updates without the prompt being refreshed. * Revenue reconciliation moves from discovery surface to verification surface. * Per-call request IDs in LayerLens evaluation results pair with internal call logs for fast incident triage. ## LayerLens capabilities used * [SDK + CLI evaluations](/more-in-this-section-9/evaluations-1) wired into [GitHub Actions](/6.-build-wire-your-code/integration-github-actions) * [Compare evaluations](/8.-evaluate-score-the-outputs/evaluations) — prompt-variant A/B * [JSON-schema and numeric graders](/more-in-this-section-6/json-schema-validate) * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled * [Notifications](/11.-admin/notifications) — Slack routing ## Replicate this **Get started:** [Tutorial 3: Wire CI/CD quality gates](/6.-build-wire-your-code/03-cicd-gates) walks the end-to-end build (\~30 min). * [Use case: AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd) * [Industry → Telecommunications](/4.2-industry-use-cases/telecommunications) * [Workflow: Govern](/9.-improve-tune-the-system/workflow) * [Cookbook: cost-aware evaluation](/more-in-this-section-6/cost-aware-evaluation) # Energy and utilities Energy and utilities AI evaluation — field service, ops assistance, customer support. ## Top scenarios * **Field service** — technician guidance with manuals + telemetry * **Ops assistance** — control-room support * **Customer support** — outage, billing, conservation ## Why LayerLens Hard safety rules + faithful procedure citations; trace evaluations on multi-step incident flows. ## Industry pattern → [Energy: vegetation-management imagery pattern](/4.2-industry-use-cases/pattern-11) [Scenarios](/more-in-this-section-2/scenarios-1) | [Eval patterns](/more-in-this-section-2/eval-patterns-1) | [Compliance](/more-in-this-section-2/compliance-1) | [Cookbook](/more-in-this-section-2/cookbook-and-samples-1) # Pattern: vegetation-management imagery Energy / utilities — vegetation-management imagery pattern. Stratified accuracy by season, variant comparison, daily continuous evaluation in fire season. A regional electric utility operates an AI vegetation-management system that processes drone and helicopter imagery of power-line corridors. The model flags encroaching trees, structural damage, and clearance violations. Output drives field crews to inspect, trim, and remediate before a hazard turns into an outage or — in fire-prone regions — a wildfire ignition. In summer, leafy canopy is easy to see against the lines. In winter — bare branches against bright sky, snow-occluded sightlines — model accuracy can drop by double digits. **Aggregate annual accuracy hides this**. Stratified evaluation surfaces it. This pattern shows how to evaluate vegetation-management imagery models across seasonal and lighting conditions so blind spots are measured, not discovered. ## What's at stake | Risk dimension | Magnitude | Framework | | ------------------------------------------------------- | ------------------------------------------------------- | ---------------------------------- | | Catastrophic-fire civil liability | $30B+ for the most-cited single utility liability event | Public CPUC and bankruptcy filings | | NERC and state PUC enforcement on vegetation management | Per-finding civil penalty bands | Public NERC enforcement orders | | Per-hour outage cost from a downed line | $10K–$500K depending on customer mix | Industry reliability benchmarks | | Loss of life from wildfire ignition | Beyond financial accounting | Public wildfire-incident records | ## The evaluation pattern A **stratified evaluation** treats season, lighting condition, and equipment as first-class dimensions. 1. **Image-classification grader** — per-class precision and recall on hazard categories (vegetation encroachment, equipment corrosion, clearance violation). Trace tags include `season`, `lighting_condition`, and `equipment_type`; results break out per-tag. 2. **Custom code grader (seasonal disparity)** — calculates hazard-detection-rate disparity between winter and summer trace sets; flags any disparity above a configured tolerance (e.g., 5%) as a regression. 3. **Compare models** — when a new model is proposed, it must close the seasonal disparity without losing summer performance. The comparison view shows side-by-side per-season scores. 4. **Continuous trace evaluation** — daily scoring on production drone-imagery samples. During high-risk seasons (fire season for fire-prone regions, winter ice for cold regions), cadence increases to hourly. Results route to vegetation-management operations via Slack and email. ## What you get * Seasonal and lighting-condition disparities become measured dimensions on the dashboard, not undiscovered blind spots. * Winter-specific training-data investment and evaluation-driven model improvements close the gap to within tolerance before the next high-risk season. * Auditor-ready evaluation evidence for state PUC and NERC reviews — every release pin cites the evaluation IDs that gated it. * Operations teams trust the model because the variance is explained, not absorbed into an opaque average. ## LayerLens capabilities used * [Custom code graders](/more-in-this-section-6/custom-code-scorer) — seasonal-disparity computations * [Compare evaluations](/8.-evaluate-score-the-outputs/evaluations) — variant selection with seasonal-fairness criteria * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — daily continuous scoring during high-risk seasons * [Notifications](/11.-admin/notifications) — Slack and email routing * [Audit-friendly evidence guide](/11.-admin/audit-friendly-evidence) ## Replicate this * [Industry → Energy and utilities](/4.2-industry-use-cases/energy-utilities) * [Workflow: Govern](/9.-improve-tune-the-system/workflow) * [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation) # Travel and hospitality Travel and hospitality AI evaluation — booking assist, itinerary, customer service. ## Top scenarios * **Booking assistance** — search, options, reservations * **Itinerary generation** — multi-leg travel planning * **Customer service** — changes, cancellations, support ## Why LayerLens Faithfulness against real-time inventory; rules for accurate price/availability claims. ## Industry pattern → [Travel: multilingual booking-assistant pattern](/4.2-industry-use-cases/pattern-12) [Scenarios](/more-in-this-section-2/scenarios-12) | [Eval patterns](/more-in-this-section-2/eval-patterns-12) | [Compliance](/more-in-this-section-2/compliance-12) | [Cookbook](/more-in-this-section-2/cookbook-and-samples-12) # Pattern: multilingual booking assistant Travel / hospitality — multilingual booking-assistant pattern. Per-language booking-detail accuracy, cultural appropriateness, locale-aware date handling. A global hotel chain, airline, or OTA operates an AI booking assistant in 15-20+ languages. The assistant takes natural-language requests, parses dates and traveler preferences, and produces structured bookings. Date-format ambiguity (DD/MM vs. MM/DD), traveler-name transliteration, currency handling, and cultural appropriateness all vary by locale — and a wrong-day check-in is a measurable revenue and brand event. This pattern shows how to evaluate cross-language booking accuracy. ## What's at stake | Risk dimension | Magnitude | Framework | | --------------------------------------------------------------------- | ------------------------------------------------------ | -------------------------------------- | | Per-incident wrong-booking cost | $50–$500 (cancellation + rebooking + service-recovery) | Hospitality industry benchmarks | | Brand impact from cultural-appropriateness failures | Localized review-platform damage | Public review / OTA platform reporting | | Accessibility (Section 508 / WCAG) for travelers using assistive tech | Procurement disqualification (gov / corp travel) | Section 508 / WCAG | | Currency / payment compliance failures | PCI exposure if mishandled | PCI DSS | ## The evaluation pattern A **per-language evaluation** runs against booking-conversation traces in every supported language. 1. **Booking-detail accuracy grader (custom code)** — extracted dates, names, traveler counts, room/cabin types, dietary preferences, and accessibility needs match the labeled ground truth exactly. Date-format parsing scored as a separate dimension. 2. **Cross-language parity grader** — per-language booking-detail accuracy stays within 5 percentage points of the primary-language baseline. 3. **Cultural-appropriateness grader** (GEPA-tuned against ≥50 native-speaker-reviewer-labeled examples per language family — scored output) — tone and cultural fit per locale. 4. **Currency / payment grader** — currency conversions cited explicitly with timestamp; PCI fields never appear in trace bodies. 5. **Tone-consistency grader** — across multilingual conversation turns, tone and persona remain stable. > Don't have labels yet? See [Bootstrap a grader before GEPA](/more-in-this-section-6/bootstrap-judges) for the week-1 setup. **Continuous trace evaluation:** sampled at 1% of bookings, hourly during peak season. Per-language dashboards visible to localization and revenue-management teams. ## Configuration in code ```python # Python (SDK) from layerlens import Client client = Client() cultural = client.judges.create( name="cultural-fit", evaluation_goal="Score 1-5: is the response culturally appropriate for the locale, including tone, formality, and idiom?", ) ``` ## What you get * Per-language accuracy is measured, not assumed from aggregate metrics. * Date-parsing and locale-specific failures detected in CI before they reach travelers. * PCI compliance enforced by code graders; no payment data flows into trace bodies. * Localization-team prioritization driven by data — which languages need investment and where. ## LayerLens capabilities used * [Custom code graders](/more-in-this-section-6/custom-code-scorer) — cross-language parity, booking-detail extraction * [Graders with GEPA optimization](/8.-evaluate-score-the-outputs/judges-1) — cultural appropriateness, tone consistency * [Trace evaluations](/8.-evaluate-score-the-outputs/trace-evaluations) — continuous sampled ## Replicate this **Get started:** [Cookbook: multilingual scoring](/more-in-this-section-6/multilingual) is the runnable starter. * [Industry → Travel and hospitality](/4.2-industry-use-cases/travel-hospitality) * [Concept: Continuous evaluation](/7.-observe-see-whats-happening/continuous-evaluation) * [Workflow: Evaluate](/9.-improve-tune-the-system/workflow) # Industry # Education and EdTech compliance Education compliance — FERPA, COPPA, IDEA, Section 504, ADA, Title VI/IX, state student-data laws, ED guidance, accessibility. Education AI compliance braids **federal student-record law (FERPA) + child-data law (COPPA) + civil rights statutes + state student-data laws + accessibility law**. Stakes are high because student trajectories are at issue and regulators (OCR, FTC, state AGs) are active. ## Regulatory frame | Regime | Applies when | Evaluation-evidence shape | | ------------------------------------ | ----------------------------------------------- | ------------------------------------------------------------------ | | **FERPA (20 USC § 1232g)** | Schools/agencies receiving DOE funds | School official exception evidence; consent/notice; records access | | **PPRA** | Surveys / behavioral data of minors | Notice, parental consent for protected categories | | **COPPA (15 USC § 6501)** | Online services likely used by under-13 | Parental consent, data minimization, retention limits | | **IDEA / Section 504 / ADA** | Students with disabilities | Accommodation accuracy, accessible delivery | | **Title VI** | Federally-funded programs | Disparate impact monitoring | | **Title IX** | Sex discrimination in federally-funded programs | Bias audit on grading and admissions | | **State student-data privacy laws** | Varies by state — most states now have one | Specific notice/consent/data-localization rules | | **State AI laws** | E.g., NY Education Law § 2-d, CA AB 2273 | Per-state requirements | | **Department of Education guidance** | Federal funded programs | Reasonable use of AI evidence | | **OCR enforcement** | Federally-funded programs | Investigation cooperation; corrective action plans | | **GDPR / UK GDPR** | EU/UK students | Lawful basis; DPA; transfer mechanism | ## FERPA 20 USC § 1232g, 34 CFR Part 99. Core requirements: * Schools may disclose education records only with parent/eligible-student consent or under exceptions * **School official exception** — most common AI-vendor pathway; vendor must perform institutional service, be under direct control, and use records only for authorized purpose * Records access rights for parents/eligible students * Limited directory information disclosure with notice/opt-out **LayerLens evaluation evidence:** * DPA executable as a school-official-exception agreement * Audit trail of who accessed which records * Tenant isolation — one school's records never touch another's ## COPPA For services likely used by children under 13: * Verifiable parental consent before collecting * Limited data; no behavioral targeting against under-13 * Strict retention and deletion * FTC has specifically called out AI services that train on under-13 data **LayerLens evaluation evidence:** * Hard age-band gate * Audit log of consent state at the time of trace * No training on tenant data without consent (enterprise contractual term) ## IDEA / Section 504 / ADA For students with disabilities: * **IDEA** — IEP-based services; AI tools must support, not replace, IEP-required services * **Section 504** — accommodations for substantially limiting impairments * **ADA Title II** (public schools) and **Title III** (private schools / EdTech vendors) — accessible delivery **LayerLens evaluation evidence:** * Accessibility scoring on AI outputs (reading level, alt-text, captioning WER) * Accommodation-database matching as code assertion * Audit trail of accommodation delivery ## Title VI / Title IX Federal civil rights statutes: * Disparate impact on protected class triggers OCR investigation * AI grading, AI integrity-detection, AI admissions all currently in OCR scrutiny * Settlement consent decrees include monitoring and retraining obligations **LayerLens evaluation evidence:** * Group-fairness graders per scenario * Audit-trail per evaluation showing the decision basis * Retention to support OCR investigation timelines ## State student-data privacy laws Most US states have at least one. Notable examples: * **NY Education Law § 2-d** — strict vendor data agreement requirements; "Bill of Rights" * **CA Student Online Personal Information Protection Act (SOPIPA)** * **CT, IL, CO, FL, KY, NC, TX, UT** — varying flavors Many require: * Specific data-handling notices to parents * No targeted advertising to students * No selling student data * Specific deletion / retention rules * Per-vendor public agreement registry **LayerLens evaluation evidence:** * Per-state DPA addenda available * Tenant isolation with state-specific configuration * Audit retention matching state requirements ## State AI in education laws Emerging: * **NY** — guidance on AI in K-12 (2024-2025) * **CA** — multiple bills around AI literacy and disclosure * **TX, FL, OH, others** — proposed and adopted bills around AI assessment, integrity Track state-by-state; defaults below cover most current frameworks. ## US Department of Education guidance ED has issued non-binding guidance on AI in education (2023-2024) including: * "Designing for Education with AI" report * Guidance on AI accessibility * Cybersecurity advisories for AI-enabled services Treat as non-binding-but-influential — many state and district policies cite ED. ## OCR enforcement US Education Department Office for Civil Rights investigates Title VI / Title IX / Section 504 / ADA / IDEA complaints. Recent focus areas: * AI-detection bias against non-native English writers * Accessibility of AI-generated content * Disparate-impact investigations on AI grading **LayerLens evaluation evidence supports OCR cooperation:** * Pull-on-demand fairness reports * Audit trail of AI-influenced student outcomes ## GDPR / UK GDPR For EU/UK students: * Lawful basis (consent or legitimate interest, with care) * DPIA for high-risk AI use * Cross-border transfer mechanism * DPA with the institution ## Recommended setup 1. SSO + RBAC scoped to district / school / class / role 2. DPA executable as FERPA school-official-exception 3. State-specific DPA addenda for NY § 2-d and similar 4. Hard rules: instructor-final-decision on grading, age-band gating, accommodation match 5. Group fairness graders on all student-affecting AI 6. Accessibility scoring on every student-facing output 7. Audit retention matching the longest applicable state requirement (default 5 years post-graduation) 8. No training on tenant data without explicit consent 9. BYOK custom models for the most sensitive student data ## See also * [Education scenarios](/more-in-this-section-2/scenarios) * [Education eval patterns](/more-in-this-section-2/eval-patterns) * [Concept: Multi-tenancy](/7.-observe-see-whats-happening/multi-tenancy) * [Enterprise](/4.2-industry-use-cases/education) * [BYOK custom models](/5.-select-pick-the-model/byok-custom-models) # Education cookbook and samples Education starter samples. * [Tutor helpfulness](/more-in-this-section-2/industry-education-tutor) # Recipe: education tutor pedagogical-quality Recipe — education AI tutor pedagogical-quality grader. **Goal:** the AI tutor should guide the student to insight (Socratic) rather than hand over the answer. ```python from layerlens import Client client = Client() judge = client.judges.create( name="tutor-pedagogical-quality", evaluation_goal="""Score 1-5 the tutor's response on the 'guide vs. give' spectrum. 1 = Tutor handed the student the answer directly, bypassing the learning step. 5 = Tutor asked Socratic questions, provided hints, and led the student to the answer. Penalize responses that bypass the learning step the assignment expects.""", ) # Apply via trace evaluation against ingested tutoring sessions trace_eval = client.trace_evaluations.create(trace_id=trace.id, judge_id=judge.id) result = client.trace_evaluations.wait_for_completion(trace_eval.id) ``` For COPPA / FERPA handling, redact student-record fields client-side before posting traces — see [Industry → Education compliance](/more-in-this-section-2/compliance). ## See also * [Industry: Education](/4.2-industry-use-cases/education) * [Industry pattern: AI tutor](/4.2-industry-use-cases/pattern-8) * [Concept: Graders](broken://pages/jiSJM7NnGeZVea1dxyBl) # Education and EdTech evaluation patterns Education evaluation patterns — primitive ratio, span rules, GEPA labeling, cadence. Recommended primitive ratio for education AI: **50/30/20** — deterministic rules, code assertions, graders. Subject-matter accuracy and pedagogical-quality assessment lift the grader share above what other regulated industries use. ## Primitive ratio | Primitive | Share | Why | | ------------------- | ----- | ---------------------------------------------------------------------------------------------- | | Deterministic rules | 50% | Direct-answer guards, instructor-final-decision, confidence thresholds, accommodation accuracy | | Code assertions | 30% | Inter-rater agreement, fairness, reading-level, captioning WER | | LLM graders | 20% | Pedagogical scaffolding, alternate-format faithfulness, representation diversity | ## Pattern table | Scenario | Primary primitive | Specific check | | ----------------------------------- | ----------------- | -------------------------------------------- | | Tutoring — accuracy | Grader | Subject-matter faithfulness | | Tutoring — scaffolding | Grader | Asks-before-tells rubric | | Tutoring — direct-answer guard | Rule | No outright answers on protected assignments | | Tutoring — age appropriate | Code grader | Reading level matches grade | | Grading — inter-rater agreement | Code assertion | Cohen's kappa vs. human reference | | Grading — demographic fairness | Code grader | Score parity across proxies | | Grading — rubric coverage | Rule | Every criterion scored | | Grading — instructor approval | Rule | Final grades require instructor sign-off | | Curriculum — standards citation | Rule | Items cite alignment standard | | Curriculum — accommodation accuracy | Code assertion | Match IEP/504 database | | Curriculum — representation | Code grader | Diversity of examples | | Integrity — confidence threshold | Rule | AI never unilateral | | Integrity — disparate impact | Code grader | Group fairness in flagging | | Integrity — evidence citation | Rule | Specific evidence required | | Accessibility — faithfulness | Grader | Alternate format preserves content | | Accessibility — captioning WER | Code assertion | Below regulatory threshold | ## Span rules * `student.context.lookup` — grade, IEP/504, language preference * `standards.cite` — Common Core / NGSS / state / AP alignment * `rubric.coverage` — every criterion scored * `accommodation.match` — IEP/504 plan applicability * `integrity.flag` — confidence and evidence chain * `instructor.review` — approval state ## GEPA labeling For each education grader, label ≥ 50 paired examples (more for high-stakes): * **Pedagogical scaffolding:** Curriculum specialists + classroom teachers; cover Socratic vs. directive responses * **Subject-matter faithfulness:** Domain experts per subject (math/science/humanities/CS); ≥ 100 for STEM grading * **Accessibility faithfulness:** Disability-services specialists label * **Representation diversity:** Equity specialists label Run [grader optimization](/more-in-this-section-9/judge-optimizations-1) with `budget="medium"`. For grading graders that affect student records, run with `budget="heavy"` and hold out 30%. ## Cadence | Cadence | Activity | | --------------------- | ------------------------------------------------------------------- | | **Per-deploy** | CI gate runs scenario suite (rules + assertions) | | **Per-class-session** | Sample tutoring traces; daily summary to teacher | | **Daily** | Sample production traces across all scenarios | | **Weekly** | Demographic-fairness audit on grading; integrity-flag overturn rate | | **Monthly** | GEPA re-optimization on graders that have drifted | | **Quarterly** | Standards-alignment refresh; accommodation database refresh | | **Annually** | OCR / state-audit prep; full bias audit | ## See also * [Education scenarios](/more-in-this-section-2/scenarios) * [Education compliance](/more-in-this-section-2/compliance) * [Recipe: GEPA basics](/9.-improve-tune-the-system/gepa-basics) # Education and EdTech scenarios Education and EdTech scenarios — tutoring, grading, content generation, integrity, accessibility. Industry patterns, not customer case studies. Five canonical scenarios where K-12, higher-ed, and professional learning organizations use LayerLens evaluation to gate AI features. ## 1. AI tutoring and homework help **Setup:** Student asks the tutor for help on a problem. AI scaffolds the solution, asks Socratic questions, surfaces relevant course material — without giving the answer outright in academic settings that disallow it. **Why it's hard:** * "Help" without "doing the assignment" is a fine line * Subject-matter accuracy must be near-perfect — wrong math/physics/grammar imprints lasting misconceptions * Age-appropriate language across grade levels **Stakes:** * Misconception cost compounds across the rest of the student's learning trajectory * Academic integrity policies of partner institutions vary * COPPA / FERPA / state student-data laws are unforgiving on under-13 audiences **LayerLens evaluation criteria:** | Aspect | What you measure | Pattern | | -------------------------- | ----------------------------------------------- | ----------------------------------- | | Subject-matter accuracy | Correct facts, correct math | Deterministic + faithfulness grader | | Pedagogical scaffolding | Asks before tells, guides before solves | Grader against rubric | | Age-appropriateness | Reading level matches grade | Code grader | | Direct-answer guard | No outright solutions for protected assignments | Hard rule | | PII / private-info hygiene | No surfacing of other students' data | Hard rule | ## 2. Automated grading and feedback **Setup:** AI grades short answers, essays, and code submissions; produces score and detailed feedback. Instructor reviews; AI doesn't finalize grades unilaterally. **Why it's hard:** * Grading bias by topic, dialect, language background * Format vs. substance — verbose-but-wrong should not score above terse-and-correct * Rubric application requires nuance models often lack **Stakes:** * Grade-level disparate impact triggers civil-rights and Title VI exposure * Student-data privacy bound by FERPA + state laws * Higher-ed: regional accrediting bodies scrutinize AI in assessment **LayerLens evaluation criteria:** * **Inter-rater agreement** — AI scores vs. human-graded reference set; track kappa * **Demographic-fairness grader** — Score-distribution parity across protected proxies * **Rubric-coverage rule** — Every score touches every rubric criterion * **Instructor-final-decision rule** — Hard rule: grades not posted until instructor approves * **Plagiarism / generative-AI detection caveat** — Output cannot accuse without strong evidence threshold ## 3. Curriculum and content generation **Setup:** Teachers/instructional designers ask the AI to produce lesson plans, problem sets, reading lists, accommodations. **Why it's hard:** * Standards alignment (Common Core, NGSS, state standards, AP) must be precise * IDEA / 504 accommodation accuracy is a legal obligation * Bias and representation in examples / readings **Stakes:** * Out-of-standard content drives test-prep failures * IDEA/504 errors trigger OCR investigations and consent decrees * Adoption decisions in K-12 districts are public; political controversy travels fast **LayerLens evaluation criteria:** * **Standards-citation rule** — Every produced item cites its alignment standard * **Accommodation accuracy** — Code assertion against IEP/504 plan database * **Representation diversity grader** — Examples and readings show breadth * **Reading-level grader** — Age-band appropriate * **Editor-review gate** — Hard rule for adopted curriculum vs. exploratory drafts ## 4. Academic-integrity and AI-detection assistance **Setup:** AI flags submissions that appear AI-generated, plagiarized, or off-pattern for the student. Surfaces evidence; instructor adjudicates. **Why it's hard:** * AI-detection accuracy is famously unreliable; false-positive cases publicized * Disparate impact on non-native English writers is well-documented * A wrongful integrity accusation can derail a student's career **Stakes:** * Title VI and ADA exposure for biased detection * Class-action exposure when accusations are statistically unreliable * Student wellness — wrongful accusations have produced documented mental health crises **LayerLens evaluation criteria:** * **Confidence-threshold rule** — Hard rule: AI never makes the determination unilaterally * **Disparate-impact monitoring** — Group fairness across language and demographic proxies * **Evidence-citation rule** — Every flag cites specific evidence (matched passages, anomaly indicators) * **Appeal-overturn rate** — Drift signal; high overturn rate triggers retraining * **Student-due-process rule** — Output includes standard-appeal-language disclosure ## 5. Accessibility and accommodation **Setup:** AI produces alternate-format materials (large print, audio, simplified text), captioning, translation, image descriptions for students with disabilities or LEP. **Why it's hard:** * Quality must match the original — paraphrase loses content * ADA + Rehabilitation Act §504 + IDEA + state law obligations * Real-time captioning latency / accuracy in lecture settings **Stakes:** * OCR investigations following accessibility complaints * Settlement consent decrees common for major institutions * Inadequate captioning specifically a long-running litigation theme **LayerLens evaluation criteria:** * **Faithfulness grader** — Alternate format preserves all content * **Captioning WER** — Word Error Rate ≤ regulatory threshold (institution-set) * **Translation faithfulness** — Per-language grader runs * **Latency rule** — Real-time captioning under defined latency SLA * **Image-description completeness** — Required visual elements covered ## See also * [Education eval patterns](/more-in-this-section-2/eval-patterns) * [Education compliance](/more-in-this-section-2/compliance) # Energy and utilities compliance Energy and utilities compliance — NERC CIP/OPS, FERC, state PUC/CPUC, PHMSA, OSHA, EPA, ISO/RTO market rules. Energy compliance combines **federal reliability rules (NERC) + federal energy regulator (FERC) + state PUCs + pipeline / OSHA safety + environmental (EPA) + market rules (ISO/RTOs)**. AI in this sector inherits all layers. ## Regulatory frame | Regime | Applies when | Evaluation-evidence shape | | ------------------------------- | ------------------------------------------ | ------------------------------------------------------- | | **NERC CIP** | Bulk Electric System cyber assets | Asset classification; access control; incident response | | **NERC OPS / PRC / TPL / etc.** | BES reliability | Procedure compliance; protection coordination | | **FERC orders** | Interstate transmission, wholesale markets | Order-specific requirements (e.g., 880, 881, 2222) | | **State PUC / CPUC rules** | Retail rates, service quality | Tariff filings; rate cases | | **CPUC wildfire rules (CA)** | CA electric utilities | Wildfire mitigation plans; AI risk-modeling oversight | | **PHMSA (49 CFR 192/195)** | Gas / liquids pipelines | Integrity management; record retention | | **OSHA 1910.269** | Electric power generation/T\&D | Switching/working-on rules; LOTO; PPE | | **EPA Clean Air / Water Acts** | Power generation; water utilities | Emissions / discharge monitoring | | **ISO/RTO market rules** | Wholesale market participants | Specific market protocols | | **Privacy laws (CCPA, etc.)** | Customer-data scope | DSR; consent; retention | | **CALEA / state surveillance** | Carrier-like services (some utilities) | Lawful intercept | | **NIST AI RMF** | Voluntary; cited in DOE / NREL guidance | GOVERN/MAP/MEASURE/MANAGE evidence | ## NERC CIP NERC Critical Infrastructure Protection standards apply to BES Cyber Systems: * CIP-002: Cyber asset categorization * CIP-003: Security management * CIP-004: Personnel and training * CIP-005: Electronic security perimeters * CIP-007: Systems security management * CIP-008: Incident reporting * CIP-010: Configuration change management and vulnerability assessments * CIP-011: Information protection * CIP-013: Supply chain risk management Penalties up to $1M per day per violation. **LayerLens posture:** * LayerLens is **not** a BES Cyber System; it is a SaaS evaluation platform * For utilities using LayerLens to evaluate AI that influences BES operations, the AI itself and its operational-deployment footprint may be in CIP scope (depending on impact rating) * BYOK custom models recommended for sensitive workloads * NERC CIP-013 supply-chain diligence package available — talk to enterprise ## FERC orders Selected high-impact orders: * **Order 880** — Cybersecurity reporting requirements * **Order 881** — Ambient-adjusted ratings for transmission * **Order 2222** — DER aggregation in wholesale markets * **Order 2023** — Generator interconnection reform Each carries specific compliance evidence requirements. ## State PUC / CPUC State-by-state regulation of retail service: * **CPUC AI rules (CA)** — Active rule-making; wildfire-mitigation AI is most scrutinized * **NY PSC / CT PURA / TX PUC / IL ICC / others** — Per-state proceedings on AI-influenced decisions * **Service-quality penalties** when AI-driven systems contribute to outages **LayerLens evaluation evidence:** * AI use-case inventory tied to filings * Drift dashboards for regulator-pull-on-demand ## PHMSA (49 CFR 192/195) Gas-pipeline (192) and liquids-pipeline (195) safety: * Integrity management programs * Operator qualification * Public awareness * Incident reporting **LayerLens evaluation evidence:** * Hard rule on field-safety AI: cite 49 CFR section * Audit log of AI guidance vs. recorded incidents ## OSHA 1910.269 Electric power generation, transmission, distribution. The dominant federal worker-safety standard for line work. **LayerLens evaluation evidence:** * LOTO + PPE rules on every field-safety AI output * Manual-version match rule ## EPA Clean Air Act / Clean Water Act apply to generation and water utilities. AI used in compliance reporting must produce verifiable inputs. ## ISO/RTO market rules Each ISO/RTO has its own protocols: * PJM, MISO, ERCOT, CAISO, ISO-NE, NYISO, SPP * Settlement disputes are common; AI-augmented bidding must produce auditable trail * FERC enforcement on market manipulation extends to algorithmic strategies **LayerLens evaluation evidence:** * ISO-rule compliance rule per market output * Settlement-data freshness check ## Privacy laws CCPA / state privacy / GDPR — same framework as [retail compliance](/more-in-this-section-2/compliance-10). ## NIST AI RMF Cited as voluntary best practice in DOE and NREL AI guidance. For DOE-funded R\&D, NIST AI RMF mapping is a common deliverable. ## Recommended setup 1. SSO + RBAC for dispatch / NOC / customer-service / field roles 2. BYOK custom models for NERC CIP-touching workloads 3. Hard rules: advisory-only on dispatch, switching-procedure citation, NERC standard compliance, OSHA 1910.269 / PHMSA citation, medical-baseline routing 4. Group-fairness graders on customer-affecting AI 5. Audit retention matching the longest applicable NERC / PHMSA / state requirement (default 7+ years; 6 years for many NERC standards) 6. AI use-case inventory tied to NERC and state filings 7. NIST AI RMF mapping retained per use case 8. DPA executed ## See also * [Energy scenarios](/more-in-this-section-2/scenarios-1) * [Energy eval patterns](/more-in-this-section-2/eval-patterns-1) * [Concept: Multi-tenancy](/7.-observe-see-whats-happening/multi-tenancy) * [Enterprise](/4.2-industry-use-cases/energy-utilities) * [BYOK custom models](/5.-select-pick-the-model/byok-custom-models) # Energy and utilities cookbook and samples Energy starter samples. ## Reference * [Concept: Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation) # Energy and utilities evaluation patterns Energy and utilities evaluation patterns — primitive ratio, span rules, GEPA labeling, cadence. Recommended primitive ratio for energy AI: **70/20/10** — deterministic rules, code assertions, graders. Grid procedures, NERC CIP, OSHA / DOT safety rules, and ISO/RTO market rules drive heavy rule density. Graders are reserved for customer-facing copy quality. ## Primitive ratio | Primitive | Share | Why | | ------------------- | ----- | ----------------------------------------------------------------------------------- | | Deterministic rules | 70% | Switching procedures, NERC reliability, ISO market, medical-baseline, LOTO/PPE | | Code assertions | 20% | Risk calibration, freshness thresholds, settlement-data checks | | LLM graders | 10% | Customer-impact statement quality, multilingual parity, switching-step faithfulness | ## Pattern table | Scenario | Primary primitive | Specific check | | ------------------------------------- | ----------------- | ---------------------------------- | | Grid ops — switching | Rule + Grader | Output matches approved procedures | | Grid ops — N-1 | Rule | Contingency obligations preserved | | Grid ops — advisory only | Rule | AI never auto-dispatches | | Predictive maintenance — calibration | Code assertion | Predicted vs. actual failures | | Predictive maintenance — wildfire | Rule | High-fire-threat priority | | Predictive maintenance — vegetation | Rule | State VM standards followed | | Customer service — medical baseline | Rule | Always-route-to-human | | Customer service — low-income program | Rule | Eligibility surfaced | | Customer service — outage freshness | Rule | Data within threshold | | DR / DERMS — ISO rule compliance | Rule | Match ISO/RTO market rules | | DR / DERMS — settlement freshness | Code assertion | Meter data current | | DR / DERMS — cyber gate | Rule | NERC CIP controls preserved | | Field safety — energized work | Rule | LOTO/PPE requirements | | Field safety — pipeline procedure | Rule | 49 CFR cited | | Field safety — manual version | Rule | Cited manual current | ## Span rules * `procedure.cite` — switching/maintenance procedure version * `nerc.check` — applicable NERC standard * `iso.market.rule` — applicable ISO market rule * `wildfire.zone` — fire-threat district classification * `medical.baseline` — protected-customer routing * `safety.disclaimer` — required disclaimer presence ## GEPA labeling For each energy grader, label ≥ 50 paired examples: * **Switching-step faithfulness:** Senior dispatchers + reliability engineers label * **Customer-impact messages:** Comms team + customer advocates label * **Field-safety faithfulness:** Senior linemen / pipeline-qualified leads label * **Multilingual:** Native-speaker QA per language Run [grader optimization](/more-in-this-section-9/judge-optimizations-1) with `budget="medium"`. For safety-critical graders, run with `budget="heavy"` and hold out 30%. ## Cadence | Cadence | Activity | | -------------- | -------------------------------------------------------------------- | | **Per-deploy** | CI gate runs scenario suite (rules + assertions) | | **Per-shift** | Critical-asset alarm summary; advisory-only audit | | **Continuous** | NERC alerts; outage-data freshness | | **Daily** | Sample production traces; full evaluation | | **Weekly** | Wildfire-zone audit (Cal Fire / state); vegetation status | | **Monthly** | GEPA re-optimization on graders; switching-procedure version refresh | | **Quarterly** | NERC CIP audit prep; ISO/RTO market-rule refresh | | **Annually** | OSHA / PHMSA reportable-event prep; full safety audit | ## See also * [Energy scenarios](/more-in-this-section-2/scenarios-1) * [Energy compliance](/more-in-this-section-2/compliance-1) * [Recipe: GEPA basics](/9.-improve-tune-the-system/gepa-basics) # Energy and utilities scenarios Energy and utilities scenarios — grid operations, predictive maintenance, customer service, demand response, safety. Industry patterns, not customer case studies. Five canonical scenarios where electric, gas, water, and pipeline utilities use LayerLens evaluation to gate AI features. ## 1. Grid operations and dispatch support **Setup:** AI assists grid operators — surfaces alarm correlations, suggests switching steps, drafts customer-impact messages, summarizes weather/contingency analysis. **Why it's hard:** * Grid state is high-velocity; safety-critical * N-1 / N-1-1 contingency math defeats general LLMs * BES (Bulk Electric System) reliability standards (NERC CIP, NERC OPS) are unforgiving * Cascading failure dynamics → catastrophe risk **Stakes:** * 2003 Northeast blackout class scale possible * NERC penalties up to $1M per violation per day * FERC Order 880 (cyber) and 881 (transmission) overlay * Public-safety dependencies (medical, water pumping) raise stakes further **LayerLens evaluation criteria:** | Aspect | What you measure | Pattern | | --------------------------- | -------------------------------------------- | ------------------ | | Switching-step faithfulness | Output matches approved switching procedures | Hard rule + grader | | Contingency-rule compliance | N-1 obligations preserved | Hard rule | | Advisory-only | AI never auto-applies dispatch | Hard rule | | Customer-impact message | Plain-language and faithful | Grader | | Procedure-version match | Cited procedure version current | Rule | ## 2. Predictive maintenance for grid assets **Setup:** AI ingests SCADA, weather, asset history; predicts transformer / breaker / line-section failure risk; recommends inspections, replacements, vegetation work. **Why it's hard:** * Wildfire-ignition risk in CA / Western states drives extreme stakes * Asset replacement decisions allocate capital across decades * PURPA / state-PUC depreciation rules constrain accounting * Underground vs. overhead trade-offs are politically charged **Stakes:** * PG\&E bankruptcy (\~$30B) was wildfire-ignition driven * State PUC and CPUC AI / wildfire-risk rules are active * Negligent failure to maintain has triggered criminal exposure in CA **LayerLens evaluation criteria:** * **Risk-calibration grader** — Predicted vs. actual failure rate * **Wildfire-zone rule** — Hard rule: high-fire-threat district items get priority routing * **Asset-criticality rule** — Critical assets always escalate to human review * **Vegetation-rule compliance** — Hard rule per state vegetation management standard ## 3. Customer service and billing **Setup:** Customer chatbot for billing questions, outage reporting, energy efficiency, plan options. **Why it's hard:** * LIHEAP / state low-income protections — wrong shutoff guidance can cause real harm * Senior / medical-baseline protections require accurate handling * Multi-language requirements; LEP customer regulations * Outage-status accuracy critical for vulnerable populations **Stakes:** * Wrongful disconnection of medical-baseline customers has caused deaths * State PUCs and AGs aggressive on shutoff practices * Class actions on bill estimates / TOU misapplication common **LayerLens evaluation criteria:** * **Medical-baseline rule** — Hard rule: medical-baseline customers routed to human for shutoff topics * **Low-income-program rule** — LIHEAP / CARE / state-program eligibility surfaced when applicable * **Outage-data freshness** — Cited data within threshold * **Multilingual parity** — Per-language grader runs * **Tariff citation** — Bill explanations cite the applicable tariff ## 4. Demand response and DERMS **Setup:** AI manages distributed energy resources — surfaces dispatch options, drafts participant communications, analyzes settlement. **Why it's hard:** * ISO/RTO market rules (PJM, MISO, ERCOT, CAISO, ISO-NE, NYISO, SPP) all differ * FERC Order 2222 expanded DER market participation * Settlement-quality metering data must support audit * Cybersecurity exposure for grid-edge devices **Stakes:** * ISO/RTO settlement disputes costly to litigate * FERC enforcement on market-manipulation rules * Cyber exposure under NERC CIP standards **LayerLens evaluation criteria:** * **ISO-rule compliance rule** — Hard rule: outputs match the applicable ISO/RTO market rules * **Settlement-data freshness** — Cited meter data current * **Participant-communication faithfulness** — Plain-language grader against actual dispatch * **Cyber-control gate** — Hard rule: AI cannot bypass NERC CIP controls ## 5. Field-safety and workforce assistance **Setup:** Field crews ask about energized work, switching procedures, gas-leak protocols, confined-space requirements. **Why it's hard:** * OSHA 1910.269 (electric power generation/transmission/distribution) is unforgiving * DOT pipeline safety regulations (49 CFR 192/195) for gas * Mistakes cause fatalities **Stakes:** * Worker fatalities in line work / gas work are routine industry events * OSHA citations + criminal exposure for willful violations * DOT PHMSA enforcement on pipelines * Family/community trauma is incalculable **LayerLens evaluation criteria:** * **Energized-work rule** — Hard rule: outputs cite applicable LOTO and PPE requirements * **Pipeline-procedure rule** — Hard rule: gas / pipeline procedures cite 49 CFR section * **Out-of-scope refusal** — Hard rule * **Manual-version match** — Hard rule * **Safety-disclaimer rule** — Standard disclaimer present ## See also * [Energy eval patterns](/more-in-this-section-2/eval-patterns-1) * [Energy compliance](/more-in-this-section-2/compliance-1) # Financial-services compliance Financial-services compliance — multi-regulator frame (SEC, FINRA, FinCEN, OCC, CFPB, SOX, GLBA), evidence shape per regime. Financial-services AI sits across the most fragmented regulatory landscape on the platform. Five primary regimes; evaluation evidence requirements differ for each. ## Regulatory frame | Regime | Applies when | Evaluation-evidence shape | | -------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | **SEC / FINRA** | Investment advisory, broker-dealer, customer recommendations | Suitability evidence, disclosure presence, audit-trail of recommendation rationale | | **FinCEN (BSA/AML)** | Customer onboarding, transaction monitoring, sanctions screening | Sanctions-screening recall, document-authenticity rate, SAR narration faithfulness | | **OCC / FRB / FDIC** | Banking-product underwriting, loan origination, fair-lending | Fair-lending disparity audits, model-risk management documentation | | **CFPB** | Consumer-facing financial products, fair-treatment | Disparity ratios, complaint-rate correlations, plain-language requirements | | **SOX** | Public-company financial-data trading, audit narration | Immutable decision-rationale logs | | **GLBA** | Customer non-public personal information handling | Privacy boundary, encryption, access controls | | **State (DFS, DOI)** | State-licensed lenders / insurers / broker-dealers | State-by-state varies; market-conduct examination prep | ## SEC / FINRA LayerLens evaluation evidence supports: * **Suitability records** — every recommendation tied to an evaluation that scored it against the customer's risk profile * **Disclosure-presence audits** — substring-grader evidence that required disclaimers appeared * **Bias-toward-proprietary-products** grader — evidence that recommendations are not skewed toward house funds * **Recommendation-rationale audit trail** — grader results referencing specific market data Recent FINRA sweeps include $1M-$50M fines for suitability violations in robo-advisor products. Documenting evaluation-driven controls reduces enforcement exposure. ## FinCEN BSA/AML * **Sanctions coverage** — 100% recall on every input name against the active sanctions list * **Document authenticity** — image-tampering detection rate * **SAR-narration faithfulness** — grader agreement that filed SARs accurately describe the underlying transaction * **Adverse-media coverage** — relevance scores for media hits Per-violation penalties run $1M-$1B in BSA/AML enforcement actions. LayerLens evidence is auditor-ready. ## OCC / FRB / FDIC fair lending (ECOA) Cross-cutting concern for any AI in lending decisions: * **Disparity ratios** below 1.25× across all protected-class proxies * **Model risk management** — every model version cited in a decision tied to its evaluation history * **Override rates** — when humans override AI recommendations, track for systemic-bias signal LayerLens supports per-segment disparity computation via custom code graders. See the [fair-lending platform pattern](/4.2-industry-use-cases/pattern-1). ## CFPB consumer protection Plain-language evaluation, complaint-rate monitoring, and accessibility compliance for digital products. Reading-level grader + cultural-appropriateness graders support consumer-facing applications. ## SOX trading compliance Immutable narration of every trading decision. LayerLens evaluation IDs cited in audit trails. ## GLBA privacy Customer non-public personal information must not appear in trace bodies unless tenant-isolation and encryption controls are in place. Recommended setup: * Executed DPA * PII-redaction at trace ingestion (your code, not LayerLens) * SSO with role-based access * Configurable audit retention matching your record-retention policy ## State regulators (DFS, DOI) State-by-state variance. New York DFS Cybersecurity Regulation (Part 500) is among the strictest; California CCPA/CPRA adds privacy obligations. State market-conduct examinations frequently trigger broader regulatory inquiries. ## Recommended setup 1. Executed DPA and BAA-equivalent contractual terms 2. SSO with role-based access; separate roles for compliance / risk / engineering / analyst 3. Trace evaluations citing evaluation IDs in release-notes / audit packages 4. Quarterly disparity audit with compliance signoff 5. Audit-log retention matching the longest applicable record-retention requirement ## See also * [Industry pattern: fair-lending platform pattern](/4.2-industry-use-cases/pattern-1) * [Enterprise](/4.2-industry-use-cases/financial-services) * [Security and compliance](/4.2-industry-use-cases/financial-services) * [Concept: Multi-tenancy](/7.-observe-see-whats-happening/multi-tenancy) # Financial services cookbook and samples Financial services starter samples. * [Advisor chatbot guardrails](/more-in-this-section-2/industry-finserv-advisor) ## Reference * [Use case: Agentic evaluation](/4.1-general-use-cases/agentic-evaluation) # Recipe: financial-services advisor & trading Recipe — financial-services advisor / trading. Pointer to the SDK industry samples. Canonical samples: * [`samples/industry/financial_trading.py`](https://github.com/layerlens/stratix-python/blob/main/samples/industry/financial_trading.py) — SOX suitability, fiduciary duty, regulatory compliance for AI-assisted trading recommendations. * [`samples/industry/financial_fraud.py`](https://github.com/layerlens/stratix-python/blob/main/samples/industry/financial_fraud.py) — risk scoring + AML pattern detection. ## See also * [Industry pattern: fair-lending platform pattern](/4.2-industry-use-cases/pattern-1) * [Industry → Financial services](/4.2-industry-use-cases/financial-services) # Financial-services evaluation patterns Financial-services evaluation patterns — primitive ratios, dimensions, fair-lending considerations, GEPA labeling, continuous cadence. The asymmetric-cost shape of financial-services AI drives the primitive ratio. Detection accuracy is necessary but not sufficient — fair-treatment and disclosure-presence dimensions are where regulatory exposure lives. ## Rule of thumb: 60 / 25 / 15 For a typical financial-services evaluation: * **\~60% deterministic rules / graders** — sanctions coverage, disclaimer presence, position limits, PII handling, no-specific-recommendation patterns * **\~25% natural-language assertions and code-based fairness graders** — disparity computation, suitability matching, document-extraction accuracy * **\~15% LLM graders** — explanation quality (SAR narration, customer-facing rationale), suitability judgment in ambiguous cases Inverting this ratio (grader-heavy) is the most common mistake. ## Pattern table | Dimension | Approach | Notes | | -------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------- | | Detection precision/recall | Custom code grader | Against labeled fraud / sanctions / SAR datasets | | False-positive disparity | Custom code grader | Per-segment (ZIP, surname-derived demographic, age band) — 1.25× threshold | | Sanctions coverage | Hard rule | Recall must be 100%; missing checks = CRITICAL | | Suitability matching | LLM grader, GEPA-tuned per risk-profile band | ≥50 labels per band | | Required-disclaimer presence | Substring grader | FINRA / SOX / TILA disclosures | | No-specific-securities rule | Regex / classifier | Customer-facing AI never recommends individual securities | | Document-extraction accuracy | Custom code grader | Field-by-field exact match | | Cross-document consistency | LLM grader | Values match across application + supporting docs | | Numeric / dollar-amount accuracy | Custom code grader | Critical for income, balances, transaction amounts | | PII redaction | Regex rule | No SSN / account number echo in any logged span | | Adverse-media relevance | LLM grader | Per match — keep precision high to limit reviewer overload | | Latency under SLA | Latency grader | Real-time fraud decisions need <300ms p95 | ## Span-level rules Span-level rules matter for compound workflows: * **Feature-extraction span:** sensitive fields (SSN, full account number) must be hashed before scoring. * **Model-scoring span:** every score above the SAR threshold must include the contributing features. * **Explanation span:** every explanation must cite specific transaction or customer features (no opaque rationale). ## GEPA labeling for financial-services graders Suitability and SAR-narration graders benefit substantially from GEPA — but the labeling needs careful calibration. 1. Label by **multi-reviewer agreement** — at least one compliance officer + one product reviewer per example 2. Stratify labeling by risk-profile band (suitability) or by transaction shape (SAR) 3. Hold out 20% as validation 4. Re-optimize quarterly as regulator interpretive guidance shifts ## Trace-set curation Over-sample edge cases: * High-net-worth conservative profiles paired with growth-product recommendations * Sanctions-adjacent names (close to OFAC list but not on it) * Self-employment income with depreciation add-backs * Identical-amount transactions across multiple devices (account takeover signal) * Cross-border transactions where both endpoints might be sanctioned * Names that are PEP-adjacent in non-Latin scripts ## Continuous evaluation cadence * **Per-PR** on the regression suite (small, fast) * **Hourly** on a 1% sample of production fraud decisions during business hours * **Daily** on the full SAR-eligible decision set * **Quarterly** disparity audit with compliance signoff ## See also * [Scenarios](/more-in-this-section-2/scenarios-2) | [Compliance](/more-in-this-section-2/compliance-2) * [Industry pattern: fair-lending platform pattern](/4.2-industry-use-cases/pattern-1) * [SDK sample: financial\_fraud.py](https://github.com/layerlens/stratix-python/blob/main/samples/industry/financial_fraud.py) # Financial services scenarios Financial services AI evaluation scenarios — fraud detection, advisor chatbots, KYC, document processing, trading compliance. Financial-services AI carries asymmetric risk: false negatives cost direct losses (fraud, mis-pricing); false positives drive customer friction and fair-lending exposure. Every scenario below pairs accuracy with fair-treatment evaluation. ## Fraud detection **Setup:** AI analyzes transaction patterns, account behavior, and device signals to detect potential fraud in real-time. **Why evaluation is hard:** detection rate is the obvious metric, but **false-positive disparity by demographic** drives fair-lending exposure that's invisible to aggregate accuracy. **Stakes:** $4.2B annual US card fraud (Federal Reserve / Nilson Report); per-declined-legitimate-transaction friction \~$25 (card-network industry benchmarks); ECOA fair-lending civil penalties on systematic bias. **Eval criteria:** * Custom code grader — detection precision/recall on labeled fraud dataset * Custom code grader — false-positive rate disparity across protected-class proxies (1.25× ratio threshold) * Faithfulness grader — explanation quality for SAR (Suspicious Activity Report) filing * Latency grader — decision time under SLA **Trace shape:** ``` trace ├── span: feature-extraction (transaction, account, device) ├── span: model-scoring (risk score) ├── span: explanation-generation (for SAR-eligible decisions) └── span: outcome (approve / decline / flag-for-review) ``` ## Advisory / robo-advisor chatbot **Setup:** AI provides personalized investment guidance, portfolio recommendations, market analysis to retail investors. **Why evaluation is hard:** suitability violations (recommending unsuitable products to a customer's risk profile) are FINRA enforcement targets — recent sweeps include $1M-$50M fines. Required disclosures must appear consistently. **Stakes:** SEC / FINRA enforcement bands ($1M-$50M); customer losses + civil litigation; failure to disclose risk-of-loss; bias toward proprietary products. **Eval criteria:** * Faithfulness grader (GEPA-tuned ≥50 examples) — performance and product claims grounded in real fund data * Suitability grader (GEPA-tuned ≥50 examples per risk-profile band) — recommendation matches customer's stated risk tolerance * Disclaimer-presence grader — required FINRA disclosures appear in every recommendation * No-specific-recommendation rule — customer-facing AI never recommends specific securities (deterministic regex) ## KYC / AML document review **Setup:** AI verifies identity documents (passports, driver's licenses, utility bills) and screens against sanctions / PEP / adverse-media lists. **Why evaluation is hard:** sanctions-screening recall must be \~100%. Document authenticity detection must catch digitally altered documents. **Stakes:** FinCEN penalties ($1M-$1B per BSA/AML enforcement action); correspondent banking relationship loss; onboarding fraudulent accounts. **Eval criteria:** * Sanctions-coverage rule — every input name screened against the active sanctions list (recall 100%) * Document-authenticity grader — image-tamper detection on submitted IDs and bills * OCR-accuracy grader — field extraction matches certified-coder ground truth * Adverse-media grader — relevance scoring for media hits * PEP-identification grader — politically-exposed-person matches against the active PEP database ## Loan / mortgage document processing **Setup:** AI extracts structured data from loan applications, tax returns, bank statements, pay stubs for underwriting. **Why evaluation is hard:** Schedule C self-employment income parsing is a known weak point — depreciation add-backs, multiple schedules, net-vs-gross. **Stakes:** $50K-$500K per bad loan; TILA / ECOA violations; manual rework cost; audit failures. **Eval criteria:** * Field-level extraction grader — names, amounts, dates, account numbers exact-match against ground truth * Cross-document consistency grader — values match across application + W-2 + bank statement * Schedule C / self-employment numeric grader — income computation matches certified-coder ground truth * PII-redaction rule — no SSN / account-number echo in any logged span ## Trading compliance / SOX-aligned advisory **Setup:** AI assists trading workflows — recommendations, audit narration, compliance checks. **Why evaluation is hard:** SOX trading compliance requires immutable narration of decision rationale. Suitability for institutional vs. retail counterparties differs. **Stakes:** SOX violations; SEC/FINRA enforcement; fiduciary breach litigation. **Eval criteria:** * Audit-narration faithfulness grader — every claim cites the underlying market data or policy * Suitability grader (per counterparty type) — recommendation matches counterparty's authorized scope * Position-limit rule — recommendation doesn't breach hard position limits * Required-disclaimer grader — SOX/FINRA disclosures present ## See also * [Industry pattern: fair-lending platform pattern](/4.2-industry-use-cases/pattern-1) — these scenarios composed into one runnable evaluation * [Eval patterns](/more-in-this-section-2/eval-patterns-2) | [Compliance](/more-in-this-section-2/compliance-2) * [SDK sample: financial\_fraud.py](https://github.com/layerlens/stratix-python/blob/main/samples/industry/financial_fraud.py) * [SDK sample: financial\_trading.py](https://github.com/layerlens/stratix-python/blob/main/samples/industry/financial_trading.py) # Government and public sector compliance Government compliance — FedRAMP, FISMA, NIST AI RMF, OMB M-24-10, Section 508, Privacy Act, FOIA, EO 14110. Government AI compliance is **the densest regulatory surface of any sector** — federal procurement, agency-specific rules, executive orders, civil rights statutes, and state/local overlays compound. The frame below covers the federal core; state and local agencies should add applicable analogs. ## Regulatory frame | Regime | Applies when | Evaluation-evidence shape | | ------------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------- | | **OMB M-24-10** | All federal agency AI; rights/safety-impacting subset | Pre- and post-deployment testing; continuous monitoring; public inventory | | **NIST AI RMF (AI 100-1)** | Voluntary framework; cited in agency policy | GOVERN/MAP/MEASURE/MANAGE evidence | | **EO 14110 (AI Safety/Security)** | Federal AI use; certain capabilities require reporting | Red-team evidence; safety eval results | | **FISMA / FedRAMP** | Cloud services storing federal info | ATO package; SOC 2 / FedRAMP Moderate or High | | **Section 508 / WCAG 2.1 AA** | All federal digital services | Accessibility scoring on outputs | | **Privacy Act of 1974** | Systems-of-records on US persons | PII handling, SORN, audit log | | **FOIA (5 USC 552)** | Public records released by agency | Exemption citation; redaction quality | | **NIST SP 800-53 Rev. 5** | Federal information systems | Control inventory; continuous monitoring | | **NIST SP 800-171** | DoD CUI handling | CMMC level alignment | | **EEOC / Title VII / Title VI / ADA / Rehab Act** | Federally-funded programs | Disparate impact; reasonable accommodation | | **State AI laws** | State agencies; CO, NY, IL, CA, others | State-specific bias testing, disclosure | | **GAO bid-protest jurisdiction** | Federal procurement decisions | Documented evaluation basis | ## OMB M-24-10 (Advancing Governance, Innovation, and Risk Management) Issued March 2024, builds on EO 14110. Key requirements: * Each agency designates a Chief AI Officer * Minimum practices for **rights-impacting AI** and **safety-impacting AI**: * Pre- and post-deployment testing with realistic conditions * Independent evaluation * Risk impact assessment * Ongoing monitoring of operational risks * Adequate staff training * Public AI use case inventory * Generative AI specific guardrails **LayerLens evaluation evidence supports M-24-10:** * Pre- and post-deployment scenario suite frozen at deployment + production trace sampling * Drift-monitoring dashboards * Audit trail per evaluation tied to use case in agency inventory * Independent-evaluation tier (separate LayerLens project, separate reviewers) ## NIST AI RMF (AI 100-1) Voluntary framework cited by NIST AI 600-1 (Generative AI Profile, July 2024). Agency policies adopt it via FAR clauses or internal directives. Function-to-evidence mapping covered in [eval-patterns.md](/more-in-this-section-2/eval-patterns-3). ## EO 14110 Issued October 2023. Requires: * AI safety and security guidelines from NIST * Reporting requirements for foundational models above compute thresholds * Federal-agency AI use case inventories * Critical infrastructure AI safety standards Most direct operational implication for agencies using LayerLens: ensure use cases above the M-24-10 thresholds carry the evaluation evidence the EO requires NIST to define. ## FedRAMP / FISMA For cloud-based AI services serving federal agencies: * ATO required at applicable impact level (Low / Moderate / High) * FedRAMP authorization handles the underlying cloud; agency ATO covers the application layer * Continuous monitoring and POA\&M tracking **LayerLens posture:** * Customers needing FedRAMP-authorized deployment should talk to * BYOK custom models for sensitive workloads keep inference within the agency's authorized boundary ## Section 508 / WCAG 2.1 AA For citizen-facing outputs: * Plain-language requirement (Plain Writing Act of 2010) * Reading-level targets (most agencies set 6th-8th grade) * Multilingual parity for limited-English-proficient populations (EO 13166) * Screen-reader compatibility for any rendered output **LayerLens evaluation evidence:** * Reading-level grader applied to every citizen-facing output * Per-language grader runs to detect quality drops in non-English ## Privacy Act of 1974 For any system-of-records on US persons: * Notice requirements (SORN — System of Records Notice) * Routine-use disclosure rules * Access and correction rights for the subject **LayerLens supports:** * Audit trail of data access * Tenant isolation * DPA available with Privacy Act commitments ## FOIA 5 USC 552. Federal agencies must: * Respond within 20 working days * Cite specific exemption subsections for any withholding * Provide segregable non-exempt portions State analogs (sunshine laws) often have stricter timelines. ## State and local AI laws Selected examples (non-exhaustive): * **NYC Local Law 144** — Bias audit for employment decision tools * **CA AB 2273** — Age-Appropriate Design Code for online services * **CO SB21-169** — Insurance algorithms (covers state-purchased insurance) * **IL AI Video Interview Act** — Notice and consent for AI in interviews * **State public-records / sunshine laws** — Vary; often shorter deadlines than FOIA ## Recommended setup 1. Executed agency ATO process where applicable 2. SSO + RBAC scoped to agency role classifications 3. Use-case inventory entry per LayerLens project; tied to the agency M-24-10 inventory 4. NIST AI RMF GOVERN evidence package retained per use case 5. Pre- and post-deployment scenario suite frozen at ATO; production trace sampling configured 6. Disparate-impact monitoring on every rights-impacting output 7. Audit retention matching the longer of agency records schedule or 7 years 8. Public-output reading-level rule applied to citizen-facing outputs 9. Independent evaluation for rights-impacting AI (separate project + separate reviewers) ## See also * [Government scenarios](/more-in-this-section-2/scenarios-3) * [Government eval patterns](/more-in-this-section-2/eval-patterns-3) * [Concept: Multi-tenancy](/7.-observe-see-whats-happening/multi-tenancy) * [Enterprise](/4.2-industry-use-cases/government-public-sector) * [BYOK custom models](/5.-select-pick-the-model/byok-custom-models) # Government cookbook and samples Government starter samples. ## Reference * [Concept: Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation) * [Use case: AI quality gates in CI/CD](/4.1-general-use-cases/ai-quality-gates-cicd) # Government and public sector evaluation patterns Government evaluation patterns — primitive ratio, span rules, GEPA labeling, cadence aligned to NIST AI RMF and OMB M-24-10. Recommended primitive ratio for government AI: **60/25/15** — deterministic rules, code assertions, graders. Public-sector AI inherits the strictest documentation and reviewability burden of any vertical because every decision can be challenged in court or audit. ## Primitive ratio | Primitive | Share | Why | | ------------------- | ----- | --------------------------------------------------------------------------------------------- | | Deterministic rules | 60% | Statutory deadlines, exemption citations, eligibility rule references, FOIA exemptions | | Code assertions | 25% | Reading-level graders, PII recall, routing-table validation, criterion coverage | | LLM graders | 15% | Plain-language quality, court-defensibility of exemption rationale, denial-reason readability | ## Pattern table | Scenario | Primary primitive | Specific check | | ---------------------------------- | ----------------- | --------------------------------------------------- | | Citizen chatbot — policy grounding | Grader | Faithfulness to current agency rule | | Citizen chatbot — reading level | Code grader | Flesch-Kincaid ≤ 8 | | Citizen chatbot — escalation | Rule | Eligibility/legal questions auto-escalate | | Benefits — rule citation | Rule | Citation to CFR/state reg present | | Benefits — recommendation gate | Rule | Output marked "recommendation" not "decision" | | Benefits — fairness | Code grader | Group-fairness ratio ≥ 0.8 across protected classes | | Benefits — denial-reason quality | Grader | Plain-language readability for citizen | | FOIA — exemption citation | Rule | Statutory subsection present per redaction | | FOIA — deadline | Rule | 20-day clock tracked per request | | FOIA — PII recall | Code grader | ≥ 99% on labeled PII set | | FOIA — over-redaction | Grader | Court-defensibility | | Procurement — criterion coverage | Rule | All solicitation criteria scored | | Procurement — fairness | Code grader | Bias check across business categories | | Public-safety — disparate impact | Code grader | Required on any enforcement-influencing output | | Public-safety — advisory gate | Rule | No automated enforcement actions | ## Span rules * `policy.lookup` — version of the agency rule referenced * `eligibility.eval` — which CFR/regulation sections were applied * `exemption.cite` — FOIA exemption subsection per redaction * `routing.decide` — destination office/form match * `bias.score` — disparate impact tracked per output ## NIST AI RMF mapping The four NIST AI Risk Management Framework functions map to evaluation evidence: | NIST function | LayerLens evidence | | ------------- | ------------------------------------------------ | | **GOVERN** | RBAC + audit trail per evaluation | | **MAP** | AI inventory tied to scenario suite per use case | | **MEASURE** | Evaluation reports + drift dashboards | | **MANAGE** | Result notifications + pass-rate gates in CI | ## OMB M-24-10 alignment For federal agencies covered by OMB M-24-10 (Advancing Governance, Innovation, and Risk Management for Agency Use of AI): * Rights-impacting and safety-impacting AI must have pre- and post-deployment testing * Continuous monitoring with documented thresholds * Public AI use case inventories * Generative-AI-specific guardrails **LayerLens evaluation evidence shape:** * Pre- and post-deployment scenario suite frozen at deployment time * Production traces sampled and scored against same suite * Quarterly reports on drift, fairness, accuracy ## GEPA labeling For each government grader, label ≥ 50 paired examples: * **Citizen-services chatbot:** Stratified across program types and languages * **Benefits eligibility:** Mix of approval/denial/needs-more-info, labeled by senior caseworkers * **FOIA exemption:** Labeled by FOIA officers; exemption rulings from FOIAonline / state appeals * **Procurement:** Labeled by contracting officers * **Public-safety:** Labeled by trained civil-rights reviewers, not just enforcement personnel Run [grader optimization](/more-in-this-section-9/judge-optimizations-1) with `budget="medium"`. Hold out 20% for non-overlapping validation. ## Cadence | Cadence | Activity | | ------------- | --------------------------------------------------------------- | | **Per-PR** | CI gate runs scenario suite (rules + assertions) | | **Daily** | Sample production traces; full evaluation including graders | | **Weekly** | Disparate-impact audit; FOIA deadline-compliance report | | **Monthly** | GEPA re-optimization on grader drift | | **Quarterly** | NIST AI RMF MEASURE function report; OMB M-24-10 reporting prep | | **Annually** | Public AI use-case inventory update; full bias re-audit | ## See also * [Government scenarios](/more-in-this-section-2/scenarios-3) * [Government compliance](/more-in-this-section-2/compliance-3) * [Recipe: GEPA basics](/9.-improve-tune-the-system/gepa-basics) # Government and public sector scenarios Government and public sector scenarios — citizen services, benefits, FOIA, procurement, public-safety analytics. Industry patterns, not customer case studies. Five canonical scenarios where federal, state, and local agencies use LayerLens evaluation to gate AI features into production. Patterns, not specific agencies. ## 1. Citizen-services chatbot **Setup:** A 311 / general-services chatbot answers questions about agency programs: "How do I renew my driver's license?" "Where do I report a pothole?" "What's the income limit for SNAP?" Output references the current agency policy and routes to the right form, office, or human escalation. **Why it's hard:** * Information correctness has direct downstream effect — wrong income limit means an eligible family doesn't apply * Plain-language requirements — federal Plain Writing Act and many state analogs require accessible reading levels * Multi-language fidelity — agencies serving immigrant populations need quality across non-English outputs **Stakes:** * Inaccurate eligibility info is one of the most common citizen complaints in public-services audits * ADA Title II requires that programs delivered through digital means be equally accessible to people with disabilities * Incorrect referral wastes the citizen's time and creates downstream call-volume **LayerLens evaluation criteria:** | Aspect | What you measure | Pattern | | ------------------- | ----------------------------------------------------- | ---------------------------------------- | | Policy grounding | Output cites the current agency policy/program rule | Citation existence + faithfulness grader | | Reading level | Flesch-Kincaid grade ≤ 8 (most federal/state targets) | Deterministic grader | | Routing accuracy | Recommended office / form is correct for the request | Code assertion against routing table | | Multilingual parity | Same accuracy across language pairs | Per-language grader runs | | Escalation rule | Eligibility/legal-rights questions auto-escalate | Hard rule | ## 2. Benefits eligibility and case-management assistance **Setup:** A caseworker-facing tool reads a benefits application (SNAP, Medicaid, TANF, unemployment) plus supporting docs and produces an eligibility recommendation, missing-document list, and applicable-rule citations. The caseworker decides; the AI never decides. **Why it's hard:** * Eligibility rules are dense, change yearly, and have state-specific overlays * Procedural-due-process requirements mean the citizen gets to see a documented basis for any denial * Improper denial can be appealable — and a class-action target **Stakes:** * Benefits-determination errors carry direct constitutional due-process exposure * USDA, CMS, DOL audit improper-payment rates; agency states with high error rates see funding consequences * HHS OIG has scrutinized AI-assisted eligibility decisions specifically **LayerLens evaluation criteria:** * **Rule-citation accuracy** — Every recommendation cites the specific eligibility rule (CFR section, state regulation) * **Caseworker-reviewability rule** — Hard rule: AI output must be marked recommendation, not decision * **Missing-document completeness** — Code assertion: required-document list matches program rules * **Disparate-impact monitoring** — Group-fairness grader across protected classes, geographic regions * **Plain-language denial reasons** — Grader: would a non-attorney citizen understand why? ## 3. FOIA and public-records request triage **Setup:** Incoming FOIA / public-records request is parsed by AI: identify the records sought, route to the right custodian, flag exemption candidates (b)(6) personal privacy, (b)(7) law enforcement, etc., produce a draft response. **Why it's hard:** * Statutory deadlines (federal FOIA: 20 working days) are court-enforceable * Exemption misapplication can be challenged and reversed * Over-redaction is a transparency failure; under-redaction can leak PII or compromise an investigation **Stakes:** * DOJ FOIA litigation can force release plus award attorneys' fees against the agency * Wrongful disclosure of (b)(6) PII triggers Privacy Act liability * State sunshine-law analogs often have stronger transparency presumptions **LayerLens evaluation criteria:** * **Exemption-citation accuracy** — Every redaction cites the statutory subsection * **Deadline-tracking rule** — Hard rule: response queue tracks 20-day clock per request * **PII-redaction recall** — Deterministic grader ≥ 99% on a labeled PII set * **Over-redaction guard** — Grader: would a court of appeal find this exemption defensible? * **Routing accuracy** — Code assertion: custodian assignment matches records type ## 4. Public procurement evaluation **Setup:** AI assistant reads vendor proposals, scores them against the solicitation criteria, drafts evaluation memos. Government buyer uses the output, doesn't outsource to it. **Why it's hard:** * FAR (federal) / state procurement codes require documented basis for every award decision * Bid protests at GAO, COFC, and state equivalents test the documentation directly * Bias in scoring — if model favors certain proposal styles, it disadvantages small/disadvantaged businesses **Stakes:** * A successful bid protest can vacate an award * Federal FAR Part 33 and state analogs require fair, impartial evaluation — AI bias is now an examined question * Small Business Administration set-aside programs require documented compliance **LayerLens evaluation criteria:** * **Criterion-coverage rule** — Every solicitation criterion is scored; missing criteria fails the trace * **Score-justification citation** — Each score cites specific proposal text * **Bias monitoring** — Group fairness across business size, location, ownership classifications * **Reproducibility rule** — Same proposal scored twice produces same score (within threshold) ## 5. Public-safety analytics **Setup:** Crime-pattern analysis, traffic-incident hotspots, code-enforcement triage. AI surfaces patterns; human officials decide actions. **Why it's hard:** * Predictive-policing concerns — historical data encodes historical bias; deploying it forward perpetuates disparate impact * Constitutional rights (Fourth Amendment, Equal Protection) constrain what's permissible * Many states/cities have outright banned predictive policing or specific facial-recognition uses **Stakes:** * DOJ Civil Rights Division consent decrees in cities have specifically named algorithmic policing * Class-action exposure under §1983 for algorithmic bias * Public trust collapses fast if a community perceives the AI as targeting them **LayerLens evaluation criteria:** * **Disparate-impact grader** — Required for any AI output influencing enforcement allocation * **Constitutional-rights gate** — Hard rule: outputs must be advisory to human decisions, never automated enforcement * **Source-data disclosure** — Every output cites the data sources used * **Drift-monitoring** — Bias scores tracked over time; retraining triggered on threshold breach * **Redaction at output** — PII never appears in operational outputs without authorized purpose ## See also * [Government compliance](/more-in-this-section-2/compliance-3) * [Government eval patterns](/more-in-this-section-2/eval-patterns-3) * [Concept: Graders](/8.-evaluate-score-the-outputs/judges-1) # Healthcare compliance Healthcare compliance — HIPAA, BAA, audit retention, PHI handling, FDA SaMD considerations. ## Regulatory frame Clinical AI sits at the intersection of three regulatory regimes. Each has its own evaluation evidence requirement. | Regime | Applies when | Evaluation-evidence shape | | ------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | | **HIPAA / BAA** | Anything touching Protected Health Information (PHI) | Tenant-isolation, audit log, PHI handling controls | | **FDA SaMD** | The AI makes or substantially supports a clinical decision (Class II+ Software as a Medical Device) | Clinical-validation evidence, change-control records | | **State Medical Practice Acts** | The AI's output could be construed as practicing medicine | Disclaimers, scope-of-use constraints, escalation paths | LayerLens supports the evidence side of all three. It does not certify your product — that's between you and your regulator. It produces the artifacts that make certification tractable. ## HIPAA / BAA LayerLens supports a Business Associate Agreement. **Required for production:** 1. Executed BAA with LayerLens 2. PHI redaction at trace ingestion (in your application code, not LayerLens's) 3. SSO with role-based access (no shared logins) 4. Extended audit retention for the period your record-retention policy requires **Discuss with** [**sales@layerlens.ai**](mailto:sales@layerlens.ai) **and** [**legal@layerlens.ai**](mailto:legal@layerlens.ai) to start the BAA workflow. See **DPA process** (contact sales) for the legal mechanics. ## PHI handling PHI must never enter LayerLens trace bodies in plaintext unless your BAA covers it AND your security team has approved the data flow. **Recommended pre-trace redaction:** * Names → `` token * DOB → age-band only * MRN / SSN / addresses → category tokens * Free-text clinical notes — entity-level redaction (Presidio, AWS Comprehend Medical, or equivalent) before the trace leaves your boundary The redaction happens **client-side**, before LayerLens sees the data. LayerLens's [PHI-redaction rule pattern](/more-in-this-section-2/eval-patterns-4) then verifies redaction was effective by scoring traces for any leftover PII. ## FDA Software as a Medical Device (SaMD) If your product makes or substantially supports a clinical decision, FDA may classify it as SaMD. The classification (Class I, II, or III) drives the evidence burden. **LayerLens's role in the SaMD evidence pack:** * **Pre-market validation** — agentic evaluations against curated clinical trace sets become the validation evidence * **Change control** — every release pin cites the evaluation ID that gated it; before-and-after evaluations document the impact of any change * **Post-market surveillance** — continuous trace evaluations on production traffic become the post-market quality monitoring artifact Talk to your FDA regulatory consultant about how the evaluation evidence LayerLens produces maps to your specific submission shape (510(k), De Novo, PMA). ## Audit retention Audit retention is configurable. Common configurations: | Use | Retention | | ----------------------------------------------------- | --------------------------------------------- | | HIPAA-compliant operational logs | 6 years (HIPAA minimum) | | FDA-regulated clinical-AI change logs | Lifecycle of the product + post-market period | | Quality audit trail (eval IDs cited in release notes) | Indefinite | Configure during onboarding; retention period is a contractual term. ## State medical practice acts State boards regulate "practice of medicine." AI that crosses certain thresholds (diagnosis, prescription, treatment recommendation) may be regulated as such. **Mitigations LayerLens's evaluation framework supports:** * **Disclaimer-presence rule** on patient-facing outputs (state-specific) * **No-diagnosis rule** that a triage assistant must not assert a specific diagnosis * **Escalation rule** to a licensed clinician for any output above configured risk These are documentation patterns, not legal opinions. Consult counsel for your specific deployment. ## Recommended setup 1. Executed BAA 2. PHI redaction at trace ingestion (your code) 3. SSO with role-based access 4. Trace evaluations with evaluation IDs cited in release notes 5. Continuous trace evaluation on production traffic for post-market quality monitoring 6. Audit retention configured to your record-retention policy ## See also * [Enterprise](/4.2-industry-use-cases/healthcare) * **DPA process** (contact sales) * [Security and compliance](/4.2-industry-use-cases/healthcare) * [Concept: Multi-tenancy](/7.-observe-see-whats-happening/multi-tenancy) * [Industry pattern: healthcare contraindicated cocktail](/4.2-industry-use-cases/pattern) # Healthcare cookbook and samples Healthcare cookbook starter samples. * [Clinical Q\&A safety grader](/more-in-this-section-2/industry-healthcare-clinical-qa) ## Reference * [Use case: RAG evaluation](/4.1-general-use-cases/rag-evaluation) * [Concept: Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation) # Recipe: healthcare clinical Q\&A safety Recipe — healthcare clinical Q\&A. Pointer to the SDK industry sample. Canonical sample: [`samples/industry/healthcare_clinical.py`](https://github.com/layerlens/stratix-python/blob/main/samples/industry/healthcare_clinical.py) — clinical decision support evaluation: medical accuracy, drug-interaction detection, guideline adherence. ## See also * [Industry pattern: clinical decision support](/4.2-industry-use-cases/pattern) * [Industry → Healthcare](/4.2-industry-use-cases/healthcare) * [Bootstrap a grader before GEPA](/more-in-this-section-6/bootstrap-judges) # Healthcare evaluation patterns Healthcare evaluation patterns — common grader and rule combinations. A field guide to which LayerLens primitive to reach for in clinical-AI scenarios. The pattern for healthcare differs from other industries because of the asymmetric cost of errors: **a deterministic rule that catches one bad output is worth more than a grader that grades fifty good ones.** ## Rule of thumb: 70 / 20 / 10 For a typical healthcare evaluation: * **\~70% deterministic rules** — the bulk of safety enforcement * **\~20% natural-language assertions** — fuzzy correctness (cited correctly? appropriate care level?) * **\~10% LLM graders** — residual subjective dimensions (tone, plain-language quality) Inverting this ratio (grader-heavy) is the most common mistake. ## Pattern table | Dimension | Approach | Notes | | ----------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------ | | Clinical-claim faithfulness | LLM grader, GEPA-optimized | Tune against ≥50 clinician-labeled examples; hold out ≥20 for validation | | Citation existence | Deterministic rule | Cited section IDs must resolve to real chunks | | Dose-recommendation surfacing | Deterministic rule + numeric grader | Extract every dose mention; require exact match to source | | Drug-interaction coverage | Deterministic rule | Every drug pair in input must trigger a check | | Emergency escalation | Deterministic rule (regex / classifier) | Hard floor below LLM judgment | | Reasoning soundness | LLM grader, GEPA-optimized | Penalize correct-answer-by-flawed-reasoning | | PHI redaction | Deterministic rule + regex | Block unredacted PHI in any logged output | | Care-level appropriateness | LLM grader | Triage scenarios | | Empathy / patient-facing tone | LLM grader | Use sparingly | | Reading level | Deterministic grader (Flesch-Kincaid or similar) | Required for patient-facing | | Required-field presence | Custom code grader | Diagnoses, medications, allergies, vitals | ## Span-level rules Healthcare agents often need rules at the span level — examples: * **Retrieval span:** "the retrieval span must not return content from outside the approved guideline corpus." * **LLM span:** "the prescribing span must include a citation to a prescribing reference for any dose mentioned." * **Tool span:** "the EHR write-back span may only fire for the patient ID in `inputs`." ## Trace-set curation A healthcare trace set should over-sample edge cases: * Allergy combinations * Pediatric / geriatric weight-based dosing * Pregnancy / lactation contraindications * Renal / hepatic impairment dosing * Crisis scenarios (suicidal ideation, anaphylaxis, MI symptoms) * Multilingual presentations * Low-health-literacy phrasings A 50-trace curated set tilted toward edge cases catches more failures than a 500-trace random sample. ## GEPA labeling for clinical graders Clinicians label inconsistently across specialties; calibrate before optimizing. 1. Have ≥3 clinicians label the same 30 examples 2. Compute inter-rater agreement (Cohen's kappa or similar) 3. Discuss disagreements; converge on a rubric 4. Have those clinicians label 100+ examples with the converged rubric 5. GEPA-optimize the grader against the consensus labels Skipping the calibration step leaves you tuning a grader to noise. ## Continuous evaluation cadence Production healthcare AI should run trace evaluations: * **Per-PR** on a curated 50-trace gating set * **Daily** on a sampled production slice * **Weekly** full review with clinical leadership of any drift ## See also * [Scenarios](/more-in-this-section-2/scenarios-4) * [Compliance](/more-in-this-section-2/compliance-4) * [Cookbook: clinical Q\&A](/more-in-this-section-2/industry-healthcare-clinical-qa) * [Tutorial 5: GEPA optimize](/9.-improve-tune-the-system/05-gepa-optimize) # Healthcare scenarios Healthcare evaluation scenarios — clinical Q\&A, triage, summarization, medication safety, prior-authorization, and more. Healthcare AI has the highest stakes on the platform. The scenarios below are anchored on real workflows and the specific failure modes LayerLens is shaped to catch. ## Clinical decision-support Q\&A **Setup:** AI retrieves guideline content (UpToDate, internal protocols, PubMed abstracts) and answers a clinician's question. Used at point-of-care to surface the right evidence quickly. **Why evaluation is hard:** clinicians have low tolerance for fabricated citations and zero tolerance for incorrect medication dosing. The system must cite — not just answer. **Eval criteria:** * **Citation existence rule** — every cited guideline section must exist in the retrieved corpus. Fabricated section IDs = CRITICAL fail. * **Faithfulness grader** (GEPA-tuned against clinician labels) — every clinical claim in the answer must be directly supported by the retrieved chunks. * **Dose-mention rule** — surface every dosage figure for human review; never present a dose without a citation to a real prescribing reference. * **Reasoning soundness grader** — penalize correct answers that arrived via flawed reasoning. **Trace shape:** ``` trace ├── span: retrieval (guideline_corpus) │ └── outputs: chunks[] ├── span: llm (answer-synthesis) │ ├── inputs: question, chunks │ └── outputs: answer, citations └── span: post-process (citation-validation) ``` Score both retrieval (precision\@k against ground-truth chunks) and synthesis (faithfulness against retrieved chunks) separately. **Cookbook:** [healthcare clinical Q\&A safety grader](/more-in-this-section-2/industry-healthcare-clinical-qa). ## Patient triage chatbot **Setup:** Patient describes symptoms; chatbot triages and recommends care level (emergency / urgent / routine / self-care). **Why evaluation is hard:** missing an emergency is catastrophic; over-triaging routine cases overwhelms clinical staff. Tone matters — patients in distress need clear, empathetic guidance. **Eval criteria:** * **Emergency escalation rule** — if the trace contains symptom phrases matching emergency patterns (chest pain + shortness of breath, stroke FAST symptoms, anaphylaxis indicators, etc.), the response MUST contain "call 911" or "go to ER." CRITICAL fail otherwise. * **No-diagnosis rule** — the chatbot may triage but must not assert specific diagnoses (no "you have X"). * **Care-level appropriateness grader** (GEPA-tuned) — across edge cases, the recommendation matches a clinician's verdict. * **Empathy / tone grader** — appropriately warm without minimizing or alarming. **Trace shape:** ``` trace ├── span: symptom-extraction ├── span: emergency-pattern-match (deterministic) ├── span: llm (triage-decision) └── span: response-formatting ``` The deterministic emergency-pattern-match span is critical: it gives you a hard floor below the LLM's judgment. ## Medical document summarization **Setup:** AI summarizes a chart, discharge notes, or prior-authorization request for downstream review. **Why evaluation is hard:** subtle omissions can mislead the next clinician. Required structured fields (diagnosis codes, medications, allergies) must be preserved. **Eval criteria:** * **Faithfulness grader** — every claim in the summary appears in the source document. * **Required-field grader** (deterministic) — diagnosis codes, medications, allergies, vitals — required-field presence. * **Length grader** — summaries within configured bounds. * **Omission-risk grader** — secondary check: is anything *important* in the source missing from the summary? ## Medication safety: prescribing assistance **Setup:** AI assists clinicians with prescribing — drug-drug interactions, dosing for renal impairment, allergy cross-reactivity. **Why evaluation is hard:** medication errors are the highest-risk class of clinical-AI mistake. Off-by-one decimal places matter. **Eval criteria:** * **Interaction-coverage rule** — every drug-pair in the input must be checked against an authoritative interaction database. Missing checks = CRITICAL. * **Dosing-precision rule** — numeric extractions of doses must match the source exactly (no rounding, no unit substitution). * **Allergy-cross-check rule** — patient's documented allergies must be reflected in any prescribing recommendation. * **Prescribing-explanation grader** — explanation cites the rule that triggered any warning. ## Prior authorization **Setup:** AI drafts a prior-authorization request based on patient chart + insurer formulary. **Why evaluation is hard:** wrong drafts get rejected, costing clinician time and patient delay. Insurer-specific formulary rules vary. **Eval criteria:** * **Formulary-rule rule** — every requirement from the insurer's formulary must be addressed in the draft. * **Faithfulness grader** — every clinical claim cites the chart accurately. * **Completeness grader** (deterministic) — required form fields all populated. ## Patient-facing Q\&A (post-discharge) **Setup:** Patient asks questions after discharge — "what should I do if I miss a dose," "when should I call the office." **Why evaluation is hard:** patient-facing tone differs from clinician-facing; complex instructions need plain-language clarity. **Eval criteria:** * **Reading-level grader** — outputs at or below configured grade level. * **Plain-language grader** — avoids jargon; uses concrete examples. * **Escalation rule** — when symptoms suggest urgent follow-up, response includes the office contact path. ## See also * [Industry pattern: clinical decision support](/4.2-industry-use-cases/pattern) — these scenarios composed into one runnable evaluation * [Evaluation patterns](/more-in-this-section-2/eval-patterns-4) * [Compliance](/more-in-this-section-2/compliance-4) * [Cookbook: clinical Q\&A safety grader](/more-in-this-section-2/industry-healthcare-clinical-qa) * [Concept: Agentic evaluation](/8.-evaluate-score-the-outputs/agentic-evaluation) * [Use case: RAG evaluation](/4.1-general-use-cases/rag-evaluation) # Insurance compliance Insurance compliance — state DOI rate filings, NAIC AI bulletin, GLBA, ECOA/Fair Housing, HIPAA (health lines), audit retention. Insurance regulation is **state-led with federal overlays** — the regulatory frame for AI in insurance is the most fragmented of any major industry. NAIC coordinates model bulletins; each state DOI adopts, adapts, or supplements them. ## Regulatory frame | Regime | Applies when | Evaluation-evidence shape | | ------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------ | | **State DOI rate filings** | Any rating, underwriting, or claims-decision AI | Filed-factor citation per decision, retention with the rate filing | | **NAIC Model Bulletin on AI** | Adopted by \~25 states (and growing) as of 2026 | Governance documentation; bias monitoring; inventory of consequential AI | | **NY DFS Insurance Circular Letter No. 7 (2024)** | Insurers operating in NY using AI/ECDIS in underwriting | Quantitative disparate-impact analysis; fairness testing across protected classes | | **CO SB21-169** | Insurers using external consumer data and algorithms in CO | Annual testing for unfair discrimination; testing methodology disclosure to division | | **GLBA / Safeguards Rule** | All non-public personal information processing | Data protection program; incident reporting; vendor diligence | | **HIPAA** | Health-line carriers (group health, supplemental) | PHI handling, BAA in place, audit logs | | **ECOA / Fair Housing Act** | Credit-tied insurance products and homeowners | Adverse-action notices, disparate-impact testing | | **State unfair-claims-practice acts** | All claims handling | Audit retention; documented basis for denial | ## State DOI rate filings Most consequential surface for insurance AI: * Premium varying more than \~1.25× across otherwise similar applicants without filed factor → DOI flag * Rate filings must include the model methodology and the variables used * Some states (NY, CO, CA) require ongoing post-deployment monitoring evidence **LayerLens evaluation evidence supports rate-filing compliance:** * Filed-factor citation per premium — every recommendation cites a variable in the filing * Disparate-impact ratio computed per cohort — frozen at filing, tracked in production * Audit log retained for the same period as the rate filing (typically 5 years; some states longer) ## NAIC Model Bulletin on the Use of AI Systems by Insurers Adopted in some form by Alabama, Connecticut, Illinois, Iowa, Kentucky, Louisiana, Maryland, Massachusetts, Michigan, Nevada, New Hampshire, New Jersey, North Dakota, Pennsylvania, Rhode Island, Vermont, Virginia, Washington, West Virginia, others as of 2026. Common elements: * Written AI Systems Program ("AIS Program") covering governance, risk management, internal controls * Inventory of "consequential" AI systems * Pre- and post-deployment testing for accuracy, bias, drift * Ongoing monitoring with documented thresholds * Third-party AI vendor diligence **LayerLens evaluation evidence aligns with AIS Program requirements:** * Pre- and post-deployment scenario suite with frozen baselines * Drift monitoring on grader scores and rule pass rates * Vendor diligence package — LayerLens tenant isolation, security review on request ## NY DFS Insurance Circular Letter No. 7 (2024) NY DFS has the strictest current operationalization. Insurers using "external consumer data and information sources" (ECDIS) and AI systems in underwriting and pricing must: * Conduct quantitative testing for unfair or unlawful discrimination * Document data sources, data lineage, and the proxy-discrimination analysis * Maintain ongoing testing, not just a one-time validation **LayerLens evaluation evidence shape:** * Group-fairness graders run on every underwriting evaluation; results retained * Drift report on disparate impact ratio over time * Source-of-data labels on every input feature in trace metadata ## CO SB21-169 Insurance regulation Colorado was first to enact algorithmic-fairness legislation for insurance. Effective regulations require: * Insurers to test their algorithms and external data for unfair discrimination * Disclosure of testing methodology to the Colorado Division of Insurance * Annual reporting **LayerLens evaluation evidence shape:** * Annual fairness audit pulled from frozen scenario suite + production sampled traces * Rule-engine config version locked at filing time ## GLBA Safeguards Rule Modernized GLBA Safeguards Rule (effective 2023) requires: * Designated qualified individual responsible for the security program * Risk assessments documented in writing * Access controls; encryption of NPI in transit and at rest * Service-provider oversight * Incident response with notification to FTC for breaches > 500 consumers **LayerLens supports:** * Encryption at rest, TLS in transit * Audit trail of who accessed what * DPA available with security commitments and breach notification clauses ## HIPAA (health insurance lines) For group health, supplemental health, dental, vision lines processing PHI: * BAA required between carrier and LayerLens * Tenant isolation enforced * BYOK custom models for PHI processing recommended ## ECOA, Fair Housing Act For credit-scored insurance and homeowners products: * Adverse-action notice must include specific reasons; AI-generated reasons must be human-readable * Disparate-impact testing required * Records retained for at least 25 months (ECOA) or 24 months (FHA) — longer per state ## Unfair-claims-practice acts Every state has one. Common requirements: * Prompt acknowledgment of claims * Documented basis for any denial * Good-faith claim handling * Records retained typically 3-7 years post-claim closure ## Recommended setup 1. Executed DPA + BAA where health lines apply 2. SSO with role-based access; underwriter / adjuster / SIU role separation 3. State-aware deployment — rate filing-specific evaluation suites per state 4. Citation-existence hard gate on every customer-facing output 5. Audit retention matching the longest applicable state requirement (default to 7 years) 6. Annual fairness audit with documented methodology aligned to NY DFS / CO SB21-169 standards 7. AI inventory maintained per NAIC Model Bulletin ## See also * [Insurance scenarios](/more-in-this-section-2/scenarios-5) * [Insurance eval patterns](/more-in-this-section-2/eval-patterns-5) * [Concept: Multi-tenancy](/7.-observe-see-whats-happening/multi-tenancy) * [Enterprise](/4.2-industry-use-cases/insurance) * [BYOK custom models](/5.-select-pick-the-model/byok-custom-models) # Insurance cookbook and samples Insurance starter samples. * [Claims triage scoring](/more-in-this-section-2/industry-insurance-claims) # Recipe: insurance claims & underwriting Recipe — insurance claims and underwriting. Pointer to the SDK industry samples. Canonical samples: * [`samples/industry/insurance_claims.py`](https://github.com/layerlens/stratix-python/blob/main/samples/industry/insurance_claims.py) — coverage determination accuracy and settlement fairness. * [`samples/industry/insurance_underwriting.py`](https://github.com/layerlens/stratix-python/blob/main/samples/industry/insurance_underwriting.py) — risk-assessment accuracy and fair-lending compliance. ## See also * [Industry pattern: claims-triage agent](/4.2-industry-use-cases/pattern-3) * [Industry → Insurance](/4.2-industry-use-cases/insurance) # Insurance evaluation patterns Insurance evaluation patterns — primitive ratio, span rules, GEPA labeling, cadence. The recommended evaluation-primitive ratio for insurance AI is **65/25/10** — deterministic rules, code assertions, and graders respectively. Insurance has more rule-bound regulatory surface than most industries (state DOI rate filings, statute deadlines, sub-limit math) so rules and assertions dominate. ## Primitive ratio | Primitive | Share | Why | | ------------------- | ----- | ---------------------------------------------------------------------------------- | | Deterministic rules | 65% | State filings, document-version match, dollar-amount math, SoL deadlines | | Code assertions | 25% | Policy-section citation existence, premium-band feasibility, SIU reason-code count | | LLM graders | 10% | Explainability, hedge-language quality, liability-finding rationale | Reserve graders for "does this read like a regulator-defensible explanation" tasks. Use rules everywhere a number, date, or document version can be checked. ## Pattern table | Scenario | Primary primitive | Specific check | | --------------------------------------- | ----------------- | ---------------------------------------------------- | | Claims triage — severity bucket | Rule | Severity ≥ "major" auto-routes to senior adjuster | | Claims triage — coverage citation | Code assertion | Cited policy section ID exists in policy file | | Claims triage — photo presence | Rule | Required-photo flag set when policy demands it | | Underwriting — fairness | Code grader | Disparate impact ratio ≥ 0.8 (four-fifths) | | Underwriting — premium justification | Rule | Every premium decision cites a rate filing factor | | Underwriting — state rule applicability | Rule | State-specific exclusions match policyholder state | | Underwriting — explainability | Grader | Reasoning is regulator-followable | | Policy Q\&A — document grounding | Grader | Faithfulness to customer's policy version | | Policy Q\&A — version match | Rule | Cited doc version = active version | | Policy Q\&A — sub-limit math | Rule | Dollar amounts in output match policy file | | Policy Q\&A — hedge language | Rule | Required disclaimer string present | | Fraud — top-decile precision | Code assertion | High-score band matches confirmed-fraud rate | | Fraud — referral fairness | Code grader | Group-rate ratio within tolerance | | Subrogation — SoL deadline | Rule | Every subrogation file has SoL date and jurisdiction | ## Span rules Insurance traces should expose at minimum these spans: * `policy.lookup` — which policy version was retrieved * `coverage.eval` — which exclusions and endorsements applied * `rate.filing.cite` — which filed factor justified the premium * `siu.scoring` — fraud score and contributing factors * `subrogation.identify` — liable-party identification and SoL computation ## GEPA labeling For each insurance grader, label ≥ 50 paired examples drawn from real (de-identified) claim files: * **Claims triage:** 30 routine + 15 escalations + 5 fraud-flagged * **Underwriting:** 25 standard + 15 substandard + 10 declined * **Policy Q\&A:** stratified across coverage types and policy form versions * **Fraud SIU:** match labeled-fraud set with non-fraud controls (1:3 ratio is typical) * **Subrogation:** labeled by senior subrogation attorney Run [grader optimization](/more-in-this-section-9/judge-optimizations-1) with `budget="medium"` until agreement-with-humans ≥ 92%. Drop iteration if the optimized grader regresses on a held-out 20% slice. ## Cadence | Cadence | Activity | | ------------- | ----------------------------------------------------------------------------- | | **Per-PR** | Run the scenario test set as a CI gate; deterministic rules + assertions only | | **Daily** | Sample 1-5% of production traces; run all graders + rules | | **Weekly** | Drift report on score distributions; underwriting fairness audit | | **Monthly** | GEPA re-optimization of graders if drift detected | | **Quarterly** | Bench against a frozen scenario suite to catch silent regressions | | **Annually** | DOI examination prep — pull last 12 months of evaluations and rule outcomes | ## See also * [Insurance scenarios](/more-in-this-section-2/scenarios-5) * [Insurance compliance](/more-in-this-section-2/compliance-5) * [Recipe: GEPA basics](/9.-improve-tune-the-system/gepa-basics) * [Trace evaluations (SDK)](/more-in-this-section-9/trace-evaluations-1) # Insurance scenarios Insurance industry scenarios — claims triage, underwriting, policy Q\&A, fraud detection, subrogation. Industry patterns, not customer case studies. Five canonical scenarios where insurance carriers and InsurTech teams use LayerLens evaluation to gate AI features into production. Patterns, not specific customers. ## 1. Claims triage and processing **Setup:** First-notice-of-loss assistant ingests claim narrative, photos, prior policy data; produces severity bucket, coverage-applicability flag, and recommended next action (assign adjuster, pay-in-place, deny-with-reason). **Why it's hard:** * Severity must be calibrated — under-triage delays critical claims; over-triage clogs senior adjusters with low-value work * Coverage-applicability requires reading a policy document the model has likely never seen * Photos add modality risk (occluded damage, poor lighting, staged-loss indicators) **Stakes:** * Industry estimates put insurance fraud at \~$80B/year across P\&C lines * Average mis-estimation cost on commercial property claims: $50K-$500K per incident * State DOI complaint rates rise sharply when triage routes legitimate claims to denial paths **LayerLens evaluation criteria:** | Aspect | What you measure | Pattern | | ----------------------- | --------------------------------------------------------------------- | ------------------------------------------------------- | | Severity calibration | Buckets vs. ground-truth final paid amount | Deterministic rule on bucket-vs-actual delta | | Coverage citation | Cited policy section exists and supports decision | Citation-existence + grader for citation-supports-claim | | Escalation rule | Severity ≥ "major" auto-routes to senior adjuster | Hard rule — zero violations tolerated | | Photo evidence handling | Claim doesn't proceed past triage without photos when policy requires | Deterministic rule | | Disparate impact | Approval rate parity across demographic proxies (zip code, age band) | Group fairness grader | ## 2. Underwriting risk assessment **Setup:** AI co-pilot helps underwriter price a new policy. Inputs include applicant data, third-party risk signals, prior claims history. Output: recommended premium, risk class, exclusions to attach. **Why it's hard:** * Proxy discrimination — zip codes, occupation codes, even credit-tier features can encode protected class * State-by-state rate filing rules diverge * Recommendation must be explainable to a regulator post-hoc **Stakes:** * Most state DOIs flag underwriting decisions when premium varies by more than \~1.25× across otherwise similar applicants without documented basis * DOJ and state AG investigations have followed proxy-discrimination findings in algorithmic underwriting **LayerLens evaluation criteria:** * **Disparate impact ratio** — Fairness grader; ratio < 0.8 fails (four-fifths rule analog) * **Premium-band justification** — Each premium decision must cite the rate filing factor that drove it; citation-existence rule * **State-rule applicability** — Hard rule that state-specific exclusions match the policyholder state * **Explainability** — Grader scores whether a non-technical regulator could follow the reasoning ## 3. Policy Q\&A and customer self-service **Setup:** Customer-facing chatbot answers questions about an active policy: "Am I covered for water damage from the supply line in my dishwasher?" Response must reference the actual policy document version on file, not generic policy language. **Why it's hard:** * Stale-policy interpretation — model trained on yesterday's policy form misreads today's renewal * Coverage edge cases (deductibles, sub-limits, exclusion endorsements) are legally precise and don't tolerate paraphrase **Stakes:** * Carriers face state DOI complaints and class-action exposure when chatbot output contradicts the actual policy * A miscommunicated coverage decision can trigger bad-faith claims handling allegations **LayerLens evaluation criteria:** * **Policy-document grounding** — Retrieval-faithfulness grader against the customer's specific policy version * **Document-version match** — Hard rule: cited document version equals customer's active version * **Sub-limit accuracy** — Deterministic rule: any dollar amount cited matches the policy file * **Hedge language enforcement** — Output must include "this is informational, refer to your policy / contact a licensed agent" caveat for non-trivial coverage questions ## 4. Fraud detection and special-investigations referral **Setup:** Model scores claims for fraud-investigation referral. Output: risk score 0-100, top three contributing factors, recommended action (close, investigate, refer to SIU). **Why it's hard:** * False positives drag legitimate claimants into invasive investigations and create complaint exposure * False negatives cost the carrier directly * Demographic disparity in referral rates is a regulator hot button **Stakes:** * Industry-wide P\&C fraud cost \~$80B annually; SIU efficiency drives loss-ratio meaningfully * Disparate referral rates have been the subject of state DOI consent orders **LayerLens evaluation criteria:** * **Precision at top decile** — Deterministic; tracks whether the high-score band matches confirmed-fraud rate * **Recall on labeled known-fraud set** — Deterministic * **Group-fairness grader** — Referral rate ratio across demographic proxies * **Reason-code coverage** — Hard rule: every referral must cite ≥ 3 contributing factors * **Audit-trail completeness** — Every score logged with model version, feature inputs, threshold decision ## 5. Subrogation and recovery **Setup:** AI assistant reads a claim file and identifies subrogation potential — the third party potentially liable for the loss the carrier just paid. Output includes liable-party identification, dollar-recovery estimate, statute-of-limitations deadline. **Why it's hard:** * Liability identification requires reading police reports, third-party policies, and jurisdictional law * Statute-of-limitations errors directly bar recovery * False-positive subrogation pursuits annoy customers and damage agency relationships **Stakes:** * Subrogation recovery typically returns 1-3% of P\&C loss expense — material to combined ratio * Missed-deadline claims are unrecoverable and an E\&O exposure for the carrier **LayerLens evaluation criteria:** * **Liability-party identification accuracy** — Grader scores against ground-truth liability findings * **Statute-of-limitations rule** — Hard deterministic rule: every subrogation file must include the SoL date and jurisdiction * **Recovery-estimate calibration** — Deterministic; estimated vs. actual recovery on closed files * **Document-citation completeness** — Every liability claim must cite the supporting document section ## See also * [Insurance eval patterns](/more-in-this-section-2/eval-patterns-5) * [Insurance compliance](/more-in-this-section-2/compliance-5) * [Concept: Graders](/8.-evaluate-score-the-outputs/judges-1) * [Trace evaluations (SDK)](/more-in-this-section-9/trace-evaluations-1) # Legal Legal AI evaluation — contract review, legal research, document analysis. ## Top scenarios
Contract reviewIdentify risky clauses; cite to sections.Read
Legal research Q&AAnswer with case citations.Read
Document analysisDiscovery / e-discovery analysis.Read
## Why LayerLens Faithfulness graders + deterministic citation-existence rules combine to catch fabricated case citations — the most common legal-AI failure mode. ## Industry pattern → [Legal: AI research citation verification pattern](/4.2-industry-use-cases/pattern-2) ## Where to next * [Scenarios](/more-in-this-section-2/scenarios-6) | [Eval patterns](/more-in-this-section-2/eval-patterns-6) | [Compliance](/more-in-this-section-2/compliance-6) | [Cookbook](/more-in-this-section-2/cookbook-and-samples-6) # Legal compliance Legal compliance — privilege, work product, court rules, bar discipline, malpractice exposure, retention. Legal AI's compliance frame is **bar discipline + court rules + privilege protection** — not regulatory enforcement in the same sense as financial services or healthcare. But the consequences of misconduct are public, professionally damaging, and increasingly financial. ## Regulatory frame | Regime | Applies when | Evaluation-evidence shape | | ------------------------------------- | ----------------------------------------------- | --------------------------------------------------------------- | | **Court rules of procedure** | Filing AI-assisted briefs in any jurisdiction | Citation-existence audit trail, attorney certification | | **Bar Rules of Professional Conduct** | All licensed-attorney practice; varies by state | Documentation of attorney supervision over AI output | | **Attorney-client privilege** | Any AI processing of privileged communications | Privilege-detection accuracy, tenant isolation | | **Work-product doctrine** | AI processing litigation work product | Same as privilege | | **GDPR / CCPA / state privacy** | Discovery / e-discovery handling personal data | PII redaction, data subject rights handling | | **Court sanctions** | Filing fabricated citations or unverified work | Pre-filing evaluation evidence; *Mata v. Avianca* line of cases | | **Malpractice insurance** | All practice; carrier-specific requirements | Documentation of AI-supervision protocols | ## Court rules and the *Mata* doctrine *Mata v. Avianca, Inc.*, No. 22-cv-1461 (S.D.N.Y. June 22, 2023) and successor cases establish: * Attorneys remain personally responsible for verification of every citation in a filing * AI-generated material does not lower this bar * Some courts (S.D.N.Y., N.D. Tex., others) now require disclosure of AI assistance in filings * Sanctions cited in *Mata* line: $5K + reputational **LayerLens evaluation evidence supports court compliance:** * Citation-existence audit trail — every filed citation tied to an evaluation that verified its existence * Attorney-certification workflow — evaluation gate is hard; only after the attorney explicitly acknowledges can the output be acted on * Audit retention — the evaluation evidence is retained indefinitely ## Bar Rules of Professional Conduct State-by-state variance, but common themes: * **Competence** (Rule 1.1) — attorneys using AI must understand its limitations * **Supervision** (Rule 5.3) — AI as non-lawyer assistance must be supervised * **Confidentiality** (Rule 1.6) — privileged material must not flow to third parties without consent * **Communication** (Rule 1.4) — clients should be informed about AI assistance per state rules ## Attorney-client privilege Privileged communications must not enter trace bodies in plaintext unless: * Executed terms with LayerLens protecting privilege (talk to legal) * Tenant isolation enforced — privileged data confined to the firm's tenant **Recommended trace-handling pattern:** * Strip privileged content before posting to traces * For privileged matter analysis, use BYOK custom models running in the firm's own infrastructure (no external SDK calls) * All trace evaluations on privileged material must occur within the firm's tenant boundary ## Work-product doctrine Same as privilege: never expose litigation work product to systems outside the firm's tenant. Audit-log who-accessed-what. ## GDPR / CCPA / state privacy in discovery Discovery review at scale processes large volumes of personal data. Compliance considerations: * PII / personal-data detection grader at ≥99% recall * Per-jurisdiction handling — EU subject data may not flow to US-region tenants without proper transfer mechanism * Data Subject Right responses — LayerLens supports DSR via ## Malpractice exposure Multi-million-dollar settlements documented for AI-assisted misconduct. Mitigations: * Pre-filing evaluation evidence package per matter * Attorney sign-off recorded against each evaluation ID * Audit retention matching the longest applicable record-retention period ## Recommended setup 1. Executed DPA and tenant isolation contractual terms 2. SSO with role-based access; client-matter-based segmentation 3. BYOK custom models for privileged-matter analysis (firm's own inference) 4. Citation-existence hard gate on every filed-output path (zero unverified citations reach a brief) 5. Audit retention matching state bar record-retention requirements 6. Per-matter audit log — every AI interaction tied to a matter file ## See also * [Industry pattern: AI research citation verification](/4.2-industry-use-cases/pattern-2) * [Enterprise](/more-in-this-section-2/legal) * [BYOK custom models](/5.-select-pick-the-model/byok-custom-models) * [Concept: Multi-tenancy](/7.-observe-see-whats-happening/multi-tenancy) # Legal cookbook and samples Legal starter samples. * [Contract review faithfulness](/more-in-this-section-2/industry-legal-contract) ## Reference * [Use case: RAG evaluation](/4.1-general-use-cases/rag-evaluation) --- [Next Page](https://docs.layerlens.ai/llms-full.txt/1)