Telecommunications evaluation patterns

Telecommunications evaluation patterns — primitive ratio, span rules, GEPA labeling, cadence.

Recommended primitive ratio for telecom AI: 65/25/10 — deterministic rules, code assertions, judges. Carrier policy, regulatory rules, and lawful-intercept scope are tightly bounded; judges focus on customer-facing copy quality.

Primitive ratio

Primitive	Share	Why
Deterministic rules	65%	Authentication, NORS triggers, advisory-only on network changes, scope-limitation on lawful intercept
Code assertions	25%	Top-decile precision on fraud, fairness, freshness thresholds
LLM judges	10%	Plan-promise faithfulness, customer-impact statements, multilingual parity

Pattern table

Scenario	Primary primitive	Specific check
Customer service — plan promise	Rule + Judge	Promise matches policy
Customer service — outage freshness	Rule	Cited data within threshold
Customer service — authentication	Rule	Verified identity for changes
NOC — mitigation faithfulness	Judge	Output grounded in runbook
NOC — NORS trigger	Rule	911-affecting outages flag NORS
NOC — advisory-only	Rule	AI never auto-applies changes
Plan recommendations — promotion realizability	Rule	Achievable as offered
Plan recommendations — negative-option guard	Rule	Affirmative consent required
Plan recommendations — fairness	Code scorer	Group-level offered-price parity
Fraud — top-decile precision	Code assertion	Match confirmed fraud
Fraud — fairness	Code scorer	Group-level lockout parity
Fraud — recovery path	Rule	Lockout has documented recovery
Lawful intercept — order type	Rule	Subpoena/2703(d)/warrant/Title III correctly classed
Lawful intercept — scope	Rule	Response bounded by order
Lawful intercept — attorney gate	Rule	No production without sign-off

Span rules

• account.lookup — customer state, plan, identity-verification
• policy.cite — policy section reference
• outage.lookup — outage data freshness and source
• network.advisory — flag confirming advisory-only
• fraud.score — score and contributing signals
• intercept.order — order classification and scope

GEPA labeling

For each telecom judge, label ≥ 50 paired examples:

• Plan-promise faithfulness: CX leads + policy team label
• Customer-impact statements: Comms team + legal label
• Multilingual parity: Native-speaker QA per language

Run judge optimization with budget="medium". Hold out 20%.

Cadence

Cadence	Activity
Per-deploy	CI gate runs scenario suite (rules + assertions)
Continuous	Outage-data freshness; NORS-trigger watch
Daily	Sample production traces; fraud P/R report
Weekly	Fairness audit; plan-promise audit; SIM-swap rate watch
Monthly	GEPA re-optimization on judges
Quarterly	FCC reporting prep; STIR/SHAKEN compliance audit
Annually	CALEA capability audit; full bias audit

See also

• Telecom scenarios
• Telecom compliance
• Recipe: GEPA basics