Recipe: multi-grader consensus
Recipe — run multiple graders against the same trace and aggregate via consensus.
When no single grader captures a multi-faceted dimension, run several graders over the same trace and aggregate (majority vote, weighted average). Reduces single-grader noise meaningfully — 3 graders with majority vote roughly halves the false-positive rate compared to one.
from layerlens import Client
client = Client()
JUDGE_IDS = ["judge_helpfulness", "judge_safety", "judge_factual"]
def consensus_for_trace(trace_id: str):
"""Run all judges against one trace, return majority vote."""
verdicts = []
for jid in JUDGE_IDS:
evaluation = client.trace_evaluations.create(trace_id=trace_id, judge_id=jid)
result = client.trace_evaluations.wait_for_completion(evaluation.id)
verdicts.append(result.passed)
# Majority vote
return verdicts.count(True) > len(verdicts) // 2For concurrent execution across many graders and traces, use AsyncClient:
import asyncio
from layerlens import AsyncClient
client = AsyncClient()
async def consensus_async(trace_id: str):
creates = [client.trace_evaluations.create(trace_id=trace_id, judge_id=jid) for jid in JUDGE_IDS]
evals = await asyncio.gather(*creates)
waits = [client.trace_evaluations.wait_for_completion(e.id) for e in evals]
results = await asyncio.gather(*waits)
return [r.passed for r in results]See also
PreviousRecipe: Run a grader in CI without auto-merging on pass
NextRecipe: Score multilingual outputs
Last updated
Was this helpful?