Pattern: customer-service pricing accuracy

Telecom — customer-service AI pricing-accuracy pattern. JSON-schema scorer, numeric exact-match, per-PR CI gating, continuous sampled trace eval.

A telecom AI customer-service agent handles billing questions, plan changes, and pricing inquiries. The agent reads the active rate card and applies promotional pricing rules. When it gets the math right, churn drops and customer-care hours fall. When it gets it wrong — even occasionally — the bill is wrong and revenue leaks.

This pattern shows how to evaluate pricing accuracy continuously, in CI and in production.

What's at stake

Risk dimension	Magnitude	Framework
Annual revenue leakage from underbilling at 6% error rate	~$2M for a mid-market carrier	Internal billing-reconciliation
FCC complaint exposure from overcharging	Per-violation civil-penalty bands	FCC enforcement bureau guidance
Per-subscriber churn cost from a billing-dispute departure	$300–$600	Telecom retention benchmarks
Audit / SOX exposure on unreviewed pricing-engine outputs	Material-weakness risk	Public-company controls frameworks

The evaluation pattern

Pre- and post-deployment, in CI: a fast (≤5 min) regression evaluation runs on every PR touching the customer-service prompt or pricing logic.

1. JSON-schema scorer — every pricing change in the agent's output must include a rate_card_row_id and a citation timestamp. Missing field = fail.
2. Numeric scorer — extracted prices match the rate-card row exactly (no rounding, no unit substitution).
3. Compare-models view — when proposing a prompt variant, compare the variant against the current main-branch baseline on the same regression set. Pick the variant with the lowest error rate.
4. Tolerance gate — fail the build if pricing-error rate exceeds 0.5%.

Post-deployment, continuous: trace evaluation on a 5% sample of live customer interactions, hourly cadence.

• Same JSON-schema and numeric scorers
• Threshold alert routed to a Slack channel below 99.5% pass rate
• Per-trace alert on any CRITICAL error (extracted price differs from rate card by >$5)

Configuration in code

# Python (SDK) — CI-side gating evaluation
from layerlens import Stratix
client = Stratix()

schema_scorer = client.scorers.create(
 name="pricing-citation",
 type="json_schema_validate",
 config={"schema": {
 "type": "object",
 "required": ["rate_card_row_id", "citation_ts"],
 "properties": {
 "rate_card_row_id": {"type": "string"},
 "citation_ts": {"type": "string", "format": "date-time"},
 },
 }},
)

numeric_scorer = client.scorers.create_code(
 name="price-exact-match",
 code="result = abs(output['price'] - expected['rate_card_price']) < 0.01",
)

evaluation = client.evaluations.create(
 name="cs-pricing-gate",
 model_id=os.environ["MODEL_ID"],
 dataset_id="pricing-regression-v1",
 scorers=[schema_scorer.id, numeric_scorer.id],
)
result = client.evaluations.wait_for_completion(evaluation.id)
sys.exit(0 if result.pass_rate >= 0.995 else 1)

// TypeScript (REST) — same gate from a Node CI runner
const r = await fetch("https://stratix.layerlens.ai/api/v1/evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 name: "cs-pricing-gate",
 model_id: process.env.MODEL_ID,
 dataset_id: "pricing-regression-v1",
 scorers: [schemaScorerId, numericScorerId],
 }),
});
const evaluation = await r.json();
// Poll evaluation.id until status === "completed"

What you get

• Pricing regressions caught at PR — never reach production.
• Continuous evaluation surfaces drift when the rate card updates without the prompt being refreshed.
• Revenue reconciliation moves from discovery surface to verification surface.
• Per-call request IDs in Stratix evaluation results pair with internal call logs for fast incident triage.

Stratix capabilities used

• SDK + CLI evaluations wired into GitHub Actions
• Compare models — prompt-variant A/B
• JSON-schema and numeric scorers
• Trace evaluations — continuous sampled
• Notifications — Slack routing on threshold

Replicate this

Get started: Tutorial 3: Wire CI/CD quality gates walks the end-to-end build (~30 min).

• Use case: AI quality gates in CI/CD
• Industry → Telecommunications
• Workflow: Govern
• Cookbook: cost-aware evaluation