Financial services scenarios
Financial services AI evaluation scenarios — fraud detection, advisor chatbots, KYC, document processing, trading compliance.
Financial-services AI carries asymmetric risk: false negatives cost direct losses (fraud, mis-pricing); false positives drive customer friction and fair-lending exposure. Every scenario below pairs accuracy with fair-treatment evaluation.
Fraud detection
Setup: AI analyzes transaction patterns, account behavior, and device signals to detect potential fraud in real-time.
Why evaluation is hard: detection rate is the obvious metric, but false-positive disparity by demographic drives fair-lending exposure that's invisible to aggregate accuracy.
Stakes: $4.2B annual US card fraud (Federal Reserve / Nilson Report); per-declined-legitimate-transaction friction ~$25 (card-network industry benchmarks); ECOA fair-lending civil penalties on systematic bias.
Eval criteria:
Custom code grader — detection precision/recall on labeled fraud dataset
Custom code grader — false-positive rate disparity across protected-class proxies (1.25× ratio threshold)
Faithfulness judge — explanation quality for SAR (Suspicious Activity Report) filing
Latency scorer — decision time under SLA
Trace shape:
trace
├── span: feature-extraction (transaction, account, device)
├── span: model-scoring (risk score)
├── span: explanation-generation (for SAR-eligible decisions)
└── span: outcome (approve / decline / flag-for-review)Advisory / robo-advisor chatbot
Setup: AI provides personalized investment guidance, portfolio recommendations, market analysis to retail investors.
Why evaluation is hard: suitability violations (recommending unsuitable products to a customer's risk profile) are FINRA enforcement targets — recent sweeps include $1M-$50M fines. Required disclosures must appear consistently.
Stakes: SEC / FINRA enforcement bands ($1M-$50M); customer losses + civil litigation; failure to disclose risk-of-loss; bias toward proprietary products.
Eval criteria:
Faithfulness judge (GEPA-tuned ≥50 examples) — performance and product claims grounded in real fund data
Suitability judge (GEPA-tuned ≥50 examples per risk-profile band) — recommendation matches customer's stated risk tolerance
Disclaimer-presence scorer — required FINRA disclosures appear in every recommendation
No-specific-recommendation rule — customer-facing AI never recommends specific securities (deterministic regex)
KYC / AML document review
Setup: AI verifies identity documents (passports, driver's licenses, utility bills) and screens against sanctions / PEP / adverse-media lists.
Why evaluation is hard: sanctions-screening recall must be ~100%. Document authenticity detection must catch digitally altered documents.
Stakes: FinCEN penalties ($1M-$1B per BSA/AML enforcement action); correspondent banking relationship loss; onboarding fraudulent accounts.
Eval criteria:
Sanctions-coverage rule — every input name screened against the active sanctions list (recall 100%)
Document-authenticity scorer — image-tamper detection on submitted IDs and bills
OCR-accuracy scorer — field extraction matches certified-coder ground truth
Adverse-media judge — relevance scoring for media hits
PEP-identification scorer — politically-exposed-person matches against the active PEP database
Loan / mortgage document processing
Setup: AI extracts structured data from loan applications, tax returns, bank statements, pay stubs for underwriting.
Why evaluation is hard: Schedule C self-employment income parsing is a known weak point — depreciation add-backs, multiple schedules, net-vs-gross.
Stakes: $50K-$500K per bad loan; TILA / ECOA violations; manual rework cost; audit failures.
Eval criteria:
Field-level extraction scorer — names, amounts, dates, account numbers exact-match against ground truth
Cross-document consistency judge — values match across application + W-2 + bank statement
Schedule C / self-employment numeric scorer — income computation matches certified-coder ground truth
PII-redaction rule — no SSN / account-number echo in any logged span
Trading compliance / SOX-aligned advisory
Setup: AI assists trading workflows — recommendations, audit narration, compliance checks.
Why evaluation is hard: SOX trading compliance requires immutable narration of decision rationale. Suitability for institutional vs. retail counterparties differs.
Stakes: SOX violations; SEC/FINRA enforcement; fiduciary breach litigation.
Eval criteria:
Audit-narration faithfulness judge — every claim cites the underlying market data or policy
Suitability judge (per counterparty type) — recommendation matches counterparty's authorized scope
Position-limit rule — recommendation doesn't breach hard position limits
Required-disclaimer scorer — SOX/FINRA disclosures present
See also
Industry pattern: fair-lending platform pattern — these scenarios composed into one runnable evaluation
Last updated
Was this helpful?