Recipe: validate JSON output against a schema

Recipe — validate JSON outputs against a strict schema using a custom judge or pre-evaluation validation.

When to use: the model must return valid schema-conformant JSON every time.

The Stratix SDK doesn't ship a dedicated "JSON-schema scorer" type. Two viable patterns:

Pattern A — pre-evaluation validation (cheapest)

Validate JSON shape client-side before posting traces. Catches schema errors before consuming evaluation budget.

import json, jsonschema
from layerlens import Stratix

schema = {
 "type": "object",
 "properties": {
 "answer": {"type": "string"},
 "confidence": {"type": "number", "minimum": 0, "maximum": 1},
 },
 "required": ["answer", "confidence"],
 "additionalProperties": False,
}

client = Stratix()

def trace_with_schema_check(model_output: dict, **trace_kwargs):
 try:
 jsonschema.validate(instance=model_output, schema=schema)
 except jsonschema.ValidationError as e:
 # Tag failed traces explicitly so you can filter them downstream
 trace_kwargs.setdefault("tags", {})["schema_valid"] = False
 trace_kwargs["tags"]["schema_error"] = str(e)
 else:
 trace_kwargs.setdefault("tags", {})["schema_valid"] = True
 return client.traces.upload(...) # your trace upload path

Pattern B — custom judge that scores schema conformance

Build a scorer/judge whose prompt encodes the schema rule:

scorer = client.scorers.create(
 name="response-schema-conformance",
 description="Score whether output is valid JSON matching the response schema",
 model_id=model.id,
 prompt="""You are validating model output against a JSON schema.

Schema:
{
 "type": "object",
 "required": ["answer", "confidence"],
 "properties": {
 "answer": {"type": "string"},
 "confidence": {"type": "number", "minimum": 0, "maximum": 1}
 }
}

Score 0 if the output is not valid JSON or does not conform.
Score 1 if it conforms exactly.
""",
)

Pattern A is faster and free. Pattern B is what to use when the deviation must be quantified across many traces.

See also

• Concept: Scorers
• Stratix Premium — Scorers
• SDK reference: scorers