> For the complete documentation index, see [llms.txt](https://docs.layerlens.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layerlens.ai/more-in-this-section-2/eval-patterns-2.md). # Financial-services evaluation patterns The asymmetric-cost shape of financial-services AI drives the primitive ratio. Detection accuracy is necessary but not sufficient — fair-treatment and disclosure-presence dimensions are where regulatory exposure lives. ## Rule of thumb: 60 / 25 / 15 For a typical financial-services evaluation: * **\~60% deterministic rules / scorers** — sanctions coverage, disclaimer presence, position limits, PII handling, no-specific-recommendation patterns * **\~25% natural-language assertions and code-based fairness scorers** — disparity computation, suitability matching, document-extraction accuracy * **\~15% LLM judges** — explanation quality (SAR narration, customer-facing rationale), suitability judgment in ambiguous cases Inverting this ratio (judge-heavy) is the most common mistake. ## Pattern table | Dimension | Approach | Notes | | -------------------------------- | ------------------------------------------- | -------------------------------------------------------------------------- | | Detection precision/recall | Custom code grader | Against labeled fraud / sanctions / SAR datasets | | False-positive disparity | Custom code grader | Per-segment (ZIP, surname-derived demographic, age band) — 1.25× threshold | | Sanctions coverage | Hard rule | Recall must be 100%; missing checks = CRITICAL | | Suitability matching | LLM judge, GEPA-tuned per risk-profile band | ≥50 labels per band | | Required-disclaimer presence | Substring scorer | FINRA / SOX / TILA disclosures | | No-specific-securities rule | Regex / classifier | Customer-facing AI never recommends individual securities | | Document-extraction accuracy | Custom code grader | Field-by-field exact match | | Cross-document consistency | LLM judge | Values match across application + supporting docs | | Numeric / dollar-amount accuracy | Custom code grader | Critical for income, balances, transaction amounts | | PII redaction | Regex rule | No SSN / account number echo in any logged span | | Adverse-media relevance | LLM judge | Per match — keep precision high to limit reviewer overload | | Latency under SLA | Latency scorer | Real-time fraud decisions need <300ms p95 | ## Span-level rules Span-level rules matter for compound workflows: * **Feature-extraction span:** sensitive fields (SSN, full account number) must be hashed before scoring. * **Model-scoring span:** every score above the SAR threshold must include the contributing features. * **Explanation span:** every explanation must cite specific transaction or customer features (no opaque rationale). ## GEPA labeling for financial-services judges Suitability and SAR-narration judges benefit substantially from GEPA — but the labeling needs careful calibration. 1. Label by **multi-reviewer agreement** — at least one compliance officer + one product reviewer per example 2. Stratify labeling by risk-profile band (suitability) or by transaction shape (SAR) 3. Hold out 20% as validation 4. Re-optimize quarterly as regulator interpretive guidance shifts ## Trace-set curation Over-sample edge cases: * High-net-worth conservative profiles paired with growth-product recommendations * Sanctions-adjacent names (close to OFAC list but not on it) * Self-employment income with depreciation add-backs * Identical-amount transactions across multiple devices (account takeover signal) * Cross-border transactions where both endpoints might be sanctioned * Names that are PEP-adjacent in non-Latin scripts ## Continuous evaluation cadence * **Per-PR** on the regression suite (small, fast) * **Hourly** on a 1% sample of production fraud decisions during business hours * **Daily** on the full SAR-eligible decision set * **Quarterly** disparity audit with compliance signoff ## See also * [Scenarios](/more-in-this-section-2/scenarios-2.md) | [Compliance](/more-in-this-section-2/compliance-2.md) * [Industry pattern: fair-lending platform pattern](/4.2-industry-use-cases/pattern-1.md) * [SDK sample: financial\_fraud.py](https://github.com/layerlens/stratix-python/blob/main/samples/industry/financial_fraud.py) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.layerlens.ai/more-in-this-section-2/eval-patterns-2.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.