Financial-services evaluation patterns

Financial-services evaluation patterns — primitive ratios, dimensions, fair-lending considerations, GEPA labeling, continuous cadence.

The asymmetric-cost shape of financial-services AI drives the primitive ratio. Detection accuracy is necessary but not sufficient — fair-treatment and disclosure-presence dimensions are where regulatory exposure lives.

Rule of thumb: 60 / 25 / 15

For a typical financial-services evaluation:

• ~60% deterministic rules / scorers — sanctions coverage, disclaimer presence, position limits, PII handling, no-specific-recommendation patterns
• ~25% natural-language assertions and code-based fairness scorers — disparity computation, suitability matching, document-extraction accuracy
• ~15% LLM judges — explanation quality (SAR narration, customer-facing rationale), suitability judgment in ambiguous cases

Inverting this ratio (judge-heavy) is the most common mistake.

Pattern table

Dimension	Approach	Notes
Detection precision/recall	Custom code grader	Against labeled fraud / sanctions / SAR datasets
False-positive disparity	Custom code grader	Per-segment (ZIP, surname-derived demographic, age band) — 1.25× threshold
Sanctions coverage	Hard rule	Recall must be 100%; missing checks = CRITICAL
Suitability matching	LLM judge, GEPA-tuned per risk-profile band	≥50 labels per band
Required-disclaimer presence	Substring scorer	FINRA / SOX / TILA disclosures
No-specific-securities rule	Regex / classifier	Customer-facing AI never recommends individual securities
Document-extraction accuracy	Custom code grader	Field-by-field exact match
Cross-document consistency	LLM judge	Values match across application + supporting docs
Numeric / dollar-amount accuracy	Custom code grader	Critical for income, balances, transaction amounts
PII redaction	Regex rule	No SSN / account number echo in any logged span
Adverse-media relevance	LLM judge	Per match — keep precision high to limit reviewer overload
Latency under SLA	Latency scorer	Real-time fraud decisions need <300ms p95

Span-level rules

Span-level rules matter for compound workflows:

• Feature-extraction span: sensitive fields (SSN, full account number) must be hashed before scoring.
• Model-scoring span: every score above the SAR threshold must include the contributing features.
• Explanation span: every explanation must cite specific transaction or customer features (no opaque rationale).

GEPA labeling for financial-services judges

Suitability and SAR-narration judges benefit substantially from GEPA — but the labeling needs careful calibration.

1. Label by multi-reviewer agreement — at least one compliance officer + one product reviewer per example
2. Stratify labeling by risk-profile band (suitability) or by transaction shape (SAR)
3. Hold out 20% as validation
4. Re-optimize quarterly as regulator interpretive guidance shifts

Trace-set curation

Over-sample edge cases:

• High-net-worth conservative profiles paired with growth-product recommendations
• Sanctions-adjacent names (close to OFAC list but not on it)
• Self-employment income with depreciation add-backs
• Identical-amount transactions across multiple devices (account takeover signal)
• Cross-border transactions where both endpoints might be sanctioned
• Names that are PEP-adjacent in non-Latin scripts

Continuous evaluation cadence

• Per-PR on the regression suite (small, fast)
• Hourly on a 1% sample of production fraud decisions during business hours
• Daily on the full SAR-eligible decision set
• Quarterly disparity audit with compliance signoff

See also

• Scenarios | Compliance
• Industry pattern: fair-lending platform pattern
• SDK sample: financial_fraud.py