Recipe: multi-judge consensus

Recipe — run multiple judges against the same trace and aggregate via consensus.

When no single judge captures a multi-faceted dimension, run several judges over the same trace and aggregate (majority vote, weighted average). Reduces single-judge noise meaningfully — 3 judges with majority vote roughly halves the false-positive rate compared to one.

from layerlens import Stratix
client = Stratix()

JUDGE_IDS = ["judge_helpfulness", "judge_safety", "judge_factual"]

def consensus_for_trace(trace_id: str):
 """Run all judges against one trace, return majority vote."""
 verdicts = []
 for jid in JUDGE_IDS:
 evaluation = client.trace_evaluations.create(trace_id=trace_id, judge_id=jid)
 result = client.trace_evaluations.wait_for_completion(evaluation.id)
 verdicts.append(result.passed)

 # Majority vote
 return verdicts.count(True) > len(verdicts) // 2

For concurrent execution across many judges and traces, use AsyncStratix:

import asyncio
from layerlens import AsyncStratix
client = AsyncStratix()

async def consensus_async(trace_id: str):
 creates = [client.trace_evaluations.create(trace_id=trace_id, judge_id=jid) for jid in JUDGE_IDS]
 evals = await asyncio.gather(*creates)
 waits = [client.trace_evaluations.wait_for_completion(e.id) for e in evals]
 results = await asyncio.gather(*waits)
 return [r.passed for r in results]

See also

• Concept: Judges
• SDK reference: trace_evaluations