Skip to content

Evaluation Primitives

Local evaluation primitives that replaced the previously-required pydantic_evals types. These are plain dataclasses — cheap to construct, serializable, and decoupled from upstream framework changes.

EvaluationReason

ragpill.eval_types.EvaluationReason dataclass 

EvaluationReason(value, reason=None)

Result of running an evaluator with optional explanation.

Attributes:

Name Type Description
value bool | int | float | str

The evaluator's verdict. Typically a bool; evaluators may return int, float, or str for non-binary assertions.
reason str | None

Optional human-readable explanation of the verdict.

EvaluationResult

ragpill.eval_types.EvaluationResult dataclass 

EvaluationResult(name, value, reason, source)

Details of an individual evaluation result.

Attributes:

Name Type Description
name str

The evaluator's name as it appears in the aggregated results.
value bool | int | float | str

The verdict value (bool/int/float/str).
reason str | None

Optional explanation of the verdict.
source EvaluatorSource

Provenance describing which evaluator produced this result.

EvaluatorSource

ragpill.eval_types.EvaluatorSource dataclass 

EvaluatorSource(name, arguments=dict())

Provenance information for an evaluation result.

Attributes:

Name Type Description
name str

Identifier of the evaluator that produced the result (typically the class name).
arguments dict[str, Any]

Arbitrary metadata describing the evaluator instance.

EvaluatorContext

ragpill.eval_types.EvaluatorContext dataclass 

EvaluatorContext(*, name, inputs, metadata, expected_output, output, duration, attributes=dict(), metrics=dict(), trace=None, run_span_id=None)

Bases: Generic[InputsT, OutputT, MetadataT]

Context passed to evaluators during evaluation.

Attributes:

Name Type Description
name str | None

Optional case name.
inputs InputsT

The task inputs for this case.
metadata MetadataT | None

Case-level metadata (typically TestCaseMetadata), or None.
expected_output OutputT | None

The expected output for this case, if any.
output OutputT

The actual task output.
duration float

Wall-clock seconds the task took to run.
attributes dict[str, Any]

Arbitrary attributes attached to the run.
metrics dict[str, int | float]

Arbitrary numeric metrics attached to the run.
trace Trace | None

MLflow Trace captured during the run. Populated by the execute layer and consumed by span-based evaluators. None when tracing was disabled (Phase 1+ only).
run_span_id str | None

Span ID of the per-run parent span inside trace. Used by span-based evaluators to restrict attention to the current run's subtree when multiple runs share a trace. Empty string or None when tracing was disabled.

Case

ragpill.eval_types.Case dataclass 

Case(inputs, name=None, metadata=None, expected_output=None, evaluators=list())

Bases: Generic[InputsT, OutputT, MetadataT]

A single test case in a dataset.

Attributes:

Name Type Description
inputs InputsT

The inputs to pass to the task under test.
name str | None

Optional display name for the case.
metadata MetadataT | None

Case-level metadata (typically TestCaseMetadata).
expected_output OutputT | None

Optional expected output for comparison.
evaluators list[Any]

Case-level evaluators (each a ragpill BaseEvaluator).

Dataset

ragpill.eval_types.Dataset dataclass 

Dataset(cases=list(), evaluators=list())

Bases: Generic[InputsT, OutputT, MetadataT]

A collection of test cases with optional global evaluators.

Attributes:

Name Type Description
cases list[Case[InputsT, OutputT, MetadataT]]

The test cases in this dataset.
evaluators list[Any]

Dataset-level evaluators applied to every case.

See Also