Insurance evaluation patterns

Insurance evaluation patterns — primitive ratio, span rules, GEPA labeling, cadence.

The recommended evaluation-primitive ratio for insurance AI is 65/25/10 — deterministic rules, code assertions, and judges respectively. Insurance has more rule-bound regulatory surface than most industries (state DOI rate filings, statute deadlines, sub-limit math) so rules and assertions dominate.

Primitive ratio

Primitive	Share	Why
Deterministic rules	65%	State filings, document-version match, dollar-amount math, SoL deadlines
Code assertions	25%	Policy-section citation existence, premium-band feasibility, SIU reason-code count
LLM judges	10%	Explainability, hedge-language quality, liability-finding rationale

Reserve judges for "does this read like a regulator-defensible explanation" tasks. Use rules everywhere a number, date, or document version can be checked.

Pattern table

Scenario	Primary primitive	Specific check
Claims triage — severity bucket	Rule	Severity ≥ "major" auto-routes to senior adjuster
Claims triage — coverage citation	Code assertion	Cited policy section ID exists in policy file
Claims triage — photo presence	Rule	Required-photo flag set when policy demands it
Underwriting — fairness	Code scorer	Disparate impact ratio ≥ 0.8 (four-fifths)
Underwriting — premium justification	Rule	Every premium decision cites a rate filing factor
Underwriting — state rule applicability	Rule	State-specific exclusions match policyholder state
Underwriting — explainability	Judge	Reasoning is regulator-followable
Policy Q&A — document grounding	Judge	Faithfulness to customer's policy version
Policy Q&A — version match	Rule	Cited doc version = active version
Policy Q&A — sub-limit math	Rule	Dollar amounts in output match policy file
Policy Q&A — hedge language	Rule	Required disclaimer string present
Fraud — top-decile precision	Code assertion	High-score band matches confirmed-fraud rate
Fraud — referral fairness	Code scorer	Group-rate ratio within tolerance
Subrogation — SoL deadline	Rule	Every subrogation file has SoL date and jurisdiction

Span rules

Insurance traces should expose at minimum these spans:

• policy.lookup — which policy version was retrieved
• coverage.eval — which exclusions and endorsements applied
• rate.filing.cite — which filed factor justified the premium
• siu.scoring — fraud score and contributing factors
• subrogation.identify — liable-party identification and SoL computation

Span-level scorers run on each span independently. Failure of any one fails the trace.

GEPA labeling

For each insurance judge, label ≥ 50 paired examples drawn from real (de-identified) claim files:

• Claims triage: 30 routine + 15 escalations + 5 fraud-flagged
• Underwriting: 25 standard + 15 substandard + 10 declined
• Policy Q&A: stratified across coverage types and policy form versions
• Fraud SIU: match labeled-fraud set with non-fraud controls (1:3 ratio is typical)
• Subrogation: labeled by senior subrogation attorney

Run judge optimization with budget="medium" until agreement-with-humans ≥ 92%. Drop iteration if the optimized judge regresses on a held-out 20% slice.

Cadence

Cadence	Activity
Per-PR	Run the scenario test set as a CI gate; deterministic rules + assertions only
Daily	Sample 1-5% of production traces; run all judges + rules
Weekly	Drift report on score distributions; underwriting fairness audit
Monthly	GEPA re-optimization of judges if drift detected
Quarterly	Bench against a frozen scenario suite to catch silent regressions
Annually	DOI examination prep — pull last 12 months of evaluations and rule outcomes

See also

• Insurance scenarios
• Insurance compliance
• Recipe: GEPA basics
• Trace evaluations (SDK)