For the complete documentation index, see llms.txt. This page is also available as Markdown.

Financial-services evaluation patterns

Financial-services evaluation patterns — primitive ratios, dimensions, fair-lending considerations, GEPA labeling, continuous cadence.

The asymmetric-cost shape of financial-services AI drives the primitive ratio. Detection accuracy is necessary but not sufficient — fair-treatment and disclosure-presence dimensions are where regulatory exposure lives.

Rule of thumb: 60 / 25 / 15

For a typical financial-services evaluation:

  • ~60% deterministic rules / scorers — sanctions coverage, disclaimer presence, position limits, PII handling, no-specific-recommendation patterns

  • ~25% natural-language assertions and code-based fairness scorers — disparity computation, suitability matching, document-extraction accuracy

  • ~15% LLM judges — explanation quality (SAR narration, customer-facing rationale), suitability judgment in ambiguous cases

Inverting this ratio (judge-heavy) is the most common mistake.

Pattern table

Dimension
Approach
Notes

Detection precision/recall

Custom code grader

Against labeled fraud / sanctions / SAR datasets

False-positive disparity

Custom code grader

Per-segment (ZIP, surname-derived demographic, age band) — 1.25× threshold

Sanctions coverage

Hard rule

Recall must be 100%; missing checks = CRITICAL

Suitability matching

LLM judge, GEPA-tuned per risk-profile band

≥50 labels per band

Required-disclaimer presence

Substring scorer

FINRA / SOX / TILA disclosures

No-specific-securities rule

Regex / classifier

Customer-facing AI never recommends individual securities

Document-extraction accuracy

Custom code grader

Field-by-field exact match

Cross-document consistency

LLM judge

Values match across application + supporting docs

Numeric / dollar-amount accuracy

Custom code grader

Critical for income, balances, transaction amounts

PII redaction

Regex rule

No SSN / account number echo in any logged span

Adverse-media relevance

LLM judge

Per match — keep precision high to limit reviewer overload

Latency under SLA

Latency scorer

Real-time fraud decisions need <300ms p95

Span-level rules

Span-level rules matter for compound workflows:

  • Feature-extraction span: sensitive fields (SSN, full account number) must be hashed before scoring.

  • Model-scoring span: every score above the SAR threshold must include the contributing features.

  • Explanation span: every explanation must cite specific transaction or customer features (no opaque rationale).

GEPA labeling for financial-services judges

Suitability and SAR-narration judges benefit substantially from GEPA — but the labeling needs careful calibration.

  1. Label by multi-reviewer agreement — at least one compliance officer + one product reviewer per example

  2. Stratify labeling by risk-profile band (suitability) or by transaction shape (SAR)

  3. Hold out 20% as validation

  4. Re-optimize quarterly as regulator interpretive guidance shifts

Trace-set curation

Over-sample edge cases:

  • High-net-worth conservative profiles paired with growth-product recommendations

  • Sanctions-adjacent names (close to OFAC list but not on it)

  • Self-employment income with depreciation add-backs

  • Identical-amount transactions across multiple devices (account takeover signal)

  • Cross-border transactions where both endpoints might be sanctioned

  • Names that are PEP-adjacent in non-Latin scripts

Continuous evaluation cadence

  • Per-PR on the regression suite (small, fast)

  • Hourly on a 1% sample of production fraud decisions during business hours

  • Daily on the full SAR-eligible decision set

  • Quarterly disparity audit with compliance signoff

See also

Last updated

Was this helpful?