For the complete documentation index, see llms.txt. This page is also available as Markdown.

Recipe: multi-grader consensus

Recipe — run multiple graders against the same trace and aggregate via consensus.

When no single grader captures a multi-faceted dimension, run several graders over the same trace and aggregate (majority vote, weighted average). Reduces single-grader noise meaningfully — 3 graders with majority vote roughly halves the false-positive rate compared to one.

from layerlens import Client
client = Client()

JUDGE_IDS = ["judge_helpfulness", "judge_safety", "judge_factual"]

def consensus_for_trace(trace_id: str):
 """Run all judges against one trace, return majority vote."""
 verdicts = []
 for jid in JUDGE_IDS:
 evaluation = client.trace_evaluations.create(trace_id=trace_id, judge_id=jid)
 result = client.trace_evaluations.wait_for_completion(evaluation.id)
 verdicts.append(result.passed)

 # Majority vote
 return verdicts.count(True) > len(verdicts) // 2

For concurrent execution across many graders and traces, use AsyncClient:

import asyncio
from layerlens import AsyncClient
client = AsyncClient()

async def consensus_async(trace_id: str):
 creates = [client.trace_evaluations.create(trace_id=trace_id, judge_id=jid) for jid in JUDGE_IDS]
 evals = await asyncio.gather(*creates)
 waits = [client.trace_evaluations.wait_for_completion(e.id) for e in evals]
 results = await asyncio.gather(*waits)
 return [r.passed for r in results]

See also

Last updated

Was this helpful?