Daily smoke eval

Recipe — daily smoke evaluation against a representative trace set, scheduled by your own cron.

Run a small (≤50 trace) evaluation daily. Goal: detect "did anything break overnight" — model provider drift, prompt regression, infra issues.

The SDK doesn't ship a scheduler. Wire the SDK call into your existing scheduler (cron, GitHub Actions schedule, Lambda EventBridge, k8s CronJob).

SDK call (run from your scheduler)

from layerlens import Stratix
client = Stratix()

# Pull recent representative traces
traces = client.traces.get_many(page=1, page_size=50, sort_by="created_at", sort_order="desc")

# Evaluate each with your smoke-test judge
import asyncio
from layerlens import AsyncStratix
async def smoke():
 aclient = AsyncStratix()
 creates = [aclient.trace_evaluations.create(trace_id=t.id, judge_id=SMOKE_JUDGE_ID) for t in traces.traces]
 evals = await asyncio.gather(*creates)
 return await asyncio.gather(*[aclient.trace_evaluations.wait_for_completion(e.id) for e in evals])

results = asyncio.run(smoke())
failed = [r for r in results if not r.passed]
if failed:
 raise SystemExit(f"Smoke eval failed: {len(failed)} of {len(results)}")

Wire to a scheduler

GitHub Actions:

on:
 schedule:
 - cron: "0 7 * * *" # 07:00 UTC daily
jobs:
 smoke:
 runs-on: ubuntu-latest
 steps:
 - run: python scripts/smoke_eval.py
 env:
 LAYERLENS_STRATIX_API_KEY: ${{ secrets.LAYERLENS_STRATIX_API_KEY }}

Or Linux cron, k8s CronJob, etc.

See also

• Concept: Continuous evaluation
• Cookbook: containerized CI