Cost-control levers
Cost-control levers in Stratix — prompt caching pass-through, batch judges, sampling, BYOK.
Last updated
Was this helpful?
⌘Ctrlk
Cost-control levers in Stratix — prompt caching pass-through, batch judges, sampling, BYOK.
ECU consumption follows from the workload. Below are the levers that move the bill without compromising signal.
When a BYOK custom model is selected, model-inference tokens bill against your provider account, not Stratix ECU. Stratix ECU only bills the platform engine work (judges, scorers, GEPA, trace eval orchestration). For high-volume workloads, BYOK is the largest single lever.
When the judge model supports prompt caching (Anthropic, OpenAI), Stratix passes through caching annotations on the rubric portion of the prompt. Repeated judge calls against the same rubric reuse the cached prefix.
Effect: judge calls beyond the first (per cache window) drop ~70-90% in token cost. The savings show up as lower ECU consumption automatically; no config required.
Verification: check the per-call cost in the trace evaluation results — the second-and-onward calls should be markedly cheaper.
For trace evaluations spanning many traces, Stratix batches judge calls when the underlying provider supports it (OpenAI Batch API, Anthropic Batch API where available).
Effect: ~50% cost reduction on batch-eligible workloads, at the cost of higher latency (batches resolve over hours rather than seconds).
Use: continuous-evaluation cadences — overnight scoring of the prior 24h of traces.
For high-volume production traces, sampling is the right shape:
5% sampling gives statistically meaningful continuous-eval signal at 5% of the cost. Stratify the sample (by feature, by user tier, by tag) to avoid bias.
Deterministic scorers cost ~zero. Judges are 1000-10000× more expensive per call.
Pattern: use scorers and rules to filter; reserve judges for residual subjective bar.
Example: a faithfulness scorer extracts every claim and checks each against the source via regex/embedding. The judge only runs on the small subset where the scorer flagged ambiguity.
Most rubrics work on a balanced judging model. Reserve the strongest (most expensive) judging model for genuinely subtle dimensions. After GEPA-optimizing the rubric, often a smaller model meets the same agreement rate.
For idempotent evaluations (same inputs, same outputs), Stratix caches results by content hash. Re-running an eval against an unchanged dataset is free after the first run.
Use: CI gates that run the same eval on every PR — the unchanged dataset rows aren't re-scored.
The Premium dashboard shows worst-case ECU consumption before any compute-intensive run. Approve before it starts. Real consumption is usually lower.
In order:
Check the Consumption tab in Settings → Billing — what's burning ECU?
Switch the judging model down a tier on judges that don't need the strongest one
Reduce sampling rate on continuous evaluations
Move to BYOK custom models for the highest-volume model inferences
Negotiate Volume ECU on Enterprise tier if monthly run-rate justifies it
Last updated
Was this helpful?
Was this helpful?
trace_eval = client.trace_evaluations.create(
trace_set={"sample_rate": 0.05, "tags": {"env": "production"}},
judges=[...]
)