Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Financial services scenarios

Financial services AI evaluation scenarios — fraud detection, advisor chatbots, KYC, document processing, trading compliance.

Financial-services AI carries asymmetric risk: false negatives cost direct losses (fraud, mis-pricing); false positives drive customer friction and fair-lending exposure. Every scenario below pairs accuracy with fair-treatment evaluation.

Fraud detection

Setup: AI analyzes transaction patterns, account behavior, and device signals to detect potential fraud in real-time.

Why evaluation is hard: detection rate is the obvious metric, but false-positive disparity by demographic drives fair-lending exposure that's invisible to aggregate accuracy.

Stakes: $4.2B annual US card fraud (Federal Reserve / Nilson Report); per-declined-legitimate-transaction friction ~$25 (card-network industry benchmarks); ECOA fair-lending civil penalties on systematic bias.

Eval criteria:

  • Custom code grader — detection precision/recall on labeled fraud dataset

  • Custom code grader — false-positive rate disparity across protected-class proxies (1.25× ratio threshold)

  • Faithfulness judge — explanation quality for SAR (Suspicious Activity Report) filing

  • Latency scorer — decision time under SLA

Trace shape:

trace
├── span: feature-extraction (transaction, account, device)
├── span: model-scoring (risk score)
├── span: explanation-generation (for SAR-eligible decisions)
└── span: outcome (approve / decline / flag-for-review)

Advisory / robo-advisor chatbot

Setup: AI provides personalized investment guidance, portfolio recommendations, market analysis to retail investors.

Why evaluation is hard: suitability violations (recommending unsuitable products to a customer's risk profile) are FINRA enforcement targets — recent sweeps include $1M-$50M fines. Required disclosures must appear consistently.

Stakes: SEC / FINRA enforcement bands ($1M-$50M); customer losses + civil litigation; failure to disclose risk-of-loss; bias toward proprietary products.

Eval criteria:

  • Faithfulness judge (GEPA-tuned ≥50 examples) — performance and product claims grounded in real fund data

  • Suitability judge (GEPA-tuned ≥50 examples per risk-profile band) — recommendation matches customer's stated risk tolerance

  • Disclaimer-presence scorer — required FINRA disclosures appear in every recommendation

  • No-specific-recommendation rule — customer-facing AI never recommends specific securities (deterministic regex)

KYC / AML document review

Setup: AI verifies identity documents (passports, driver's licenses, utility bills) and screens against sanctions / PEP / adverse-media lists.

Why evaluation is hard: sanctions-screening recall must be ~100%. Document authenticity detection must catch digitally altered documents.

Stakes: FinCEN penalties ($1M-$1B per BSA/AML enforcement action); correspondent banking relationship loss; onboarding fraudulent accounts.

Eval criteria:

  • Sanctions-coverage rule — every input name screened against the active sanctions list (recall 100%)

  • Document-authenticity scorer — image-tamper detection on submitted IDs and bills

  • OCR-accuracy scorer — field extraction matches certified-coder ground truth

  • Adverse-media judge — relevance scoring for media hits

  • PEP-identification scorer — politically-exposed-person matches against the active PEP database

Loan / mortgage document processing

Setup: AI extracts structured data from loan applications, tax returns, bank statements, pay stubs for underwriting.

Why evaluation is hard: Schedule C self-employment income parsing is a known weak point — depreciation add-backs, multiple schedules, net-vs-gross.

Stakes: $50K-$500K per bad loan; TILA / ECOA violations; manual rework cost; audit failures.

Eval criteria:

  • Field-level extraction scorer — names, amounts, dates, account numbers exact-match against ground truth

  • Cross-document consistency judge — values match across application + W-2 + bank statement

  • Schedule C / self-employment numeric scorer — income computation matches certified-coder ground truth

  • PII-redaction rule — no SSN / account-number echo in any logged span

Trading compliance / SOX-aligned advisory

Setup: AI assists trading workflows — recommendations, audit narration, compliance checks.

Why evaluation is hard: SOX trading compliance requires immutable narration of decision rationale. Suitability for institutional vs. retail counterparties differs.

Stakes: SOX violations; SEC/FINRA enforcement; fiduciary breach litigation.

Eval criteria:

  • Audit-narration faithfulness judge — every claim cites the underlying market data or policy

  • Suitability judge (per counterparty type) — recommendation matches counterparty's authorized scope

  • Position-limit rule — recommendation doesn't breach hard position limits

  • Required-disclaimer scorer — SOX/FINRA disclosures present

See also

PreviousFinancial-services evaluation patternsNextGovernment and public sector compliance

Last updated

Was this helpful?