Result Types

Data classes returned by evaluation runs.

EvaluationOutput

ragpill.types.EvaluationOutput dataclass

EvaluationOutput(runs, cases, case_results, dataset_run=None)

Top-level output returned by evaluate_testset_with_mlflow.

Provides three views of the evaluation data: - .runs: One row per (run x evaluator) — the most granular view. - .cases: One row per (case x evaluator) — aggregated across runs. - .summary: One row per case — overall pass/fail with pass_rate. - .case_results: The structured CaseResult objects for programmatic access.

Attributes:

Name	Type	Description
runs	DataFrame	DataFrame with one row per (run x evaluator).
cases	DataFrame	DataFrame with one row per (case x evaluator), aggregated.
case_results	list[CaseResult]	List of CaseResult objects.

summary property

summary

One row per case with overall pass/fail and pass_rate.

from_json classmethod

from_json(s)

Deserialize an :class:EvaluationOutput produced by :meth:to_json.

Source code in src/ragpill/types.py

@classmethod
def from_json(cls, s: str) -> EvaluationOutput:
    """Deserialize an :class:`EvaluationOutput` produced by :meth:`to_json`."""
    return _evaluation_output_from_dict(json.loads(s))

per_attribute_accuracy

per_attribute_accuracy(attribute)

Mean evaluator result grouped by value of a single attribute key.

Iterates self.case_results and looks up cr.metadata.attributes[attribute] for each case. Cases missing the key are skipped (not counted as failures). For each surviving case, every assertion across every run contributes its value (True -> 1.0, False -> 0.0, numeric passthrough). Returns an empty dict when no case carries the key or no usable assertions exist. Attribute values are stringified for dict-key safety (covers unhashable values such as lists).

Source code in src/ragpill/types.py

def per_attribute_accuracy(self, attribute: str) -> dict[str, float]:
    """Mean evaluator result grouped by value of a single attribute key.

    Iterates ``self.case_results`` and looks up
    ``cr.metadata.attributes[attribute]`` for each case. Cases missing
    the key are skipped (not counted as failures). For each surviving
    case, every assertion across every run contributes its value
    (``True`` -> 1.0, ``False`` -> 0.0, numeric passthrough). Returns
    an empty dict when no case carries the key or no usable
    assertions exist. Attribute values are stringified for dict-key
    safety (covers unhashable values such as lists).
    """
    buckets: dict[str, list[float]] = {}
    for cr in self.case_results:
        attrs = cr.metadata.attributes
        if attribute not in attrs:
            continue
        value_key = str(attrs[attribute])
        bucket = buckets.setdefault(value_key, [])
        for rr in cr.run_results:
            for r in rr.assertions.values():
                v = r.value
                if isinstance(v, bool):
                    bucket.append(1.0 if v else 0.0)
                elif isinstance(v, (int, float)):
                    bucket.append(float(v))
    return {k: sum(vs) / len(vs) for k, vs in buckets.items() if vs}

per_attribute_accuracy_all

per_attribute_accuracy_all()

Auto-discovered per-attribute breakdown across the dataset.

Returns a {attribute_key: {value: mean_score}} mapping for every attribute key that appears on at least one case and yields at least one usable score. Useful for surfacing every attribute breakdown in a single call (e.g. the triage report).

Source code in src/ragpill/types.py

def per_attribute_accuracy_all(self) -> dict[str, dict[str, float]]:
    """Auto-discovered per-attribute breakdown across the dataset.

    Returns a ``{attribute_key: {value: mean_score}}`` mapping for every
    attribute key that appears on at least one case and yields at least
    one usable score. Useful for surfacing every attribute breakdown
    in a single call (e.g. the triage report).
    """
    keys: set[str] = set()
    for cr in self.case_results:
        keys.update(cr.metadata.attributes.keys())
    out: dict[str, dict[str, float]] = {}
    for k in sorted(keys):
        scores = self.per_attribute_accuracy(k)
        if scores:
            out[k] = scores
    return out

per_tag_accuracy

per_tag_accuracy()

Mean evaluator_result grouped by tag, across all runs and evaluators.

Tags appear on both case metadata (TestCaseMetadata.tags) and evaluator metadata (BaseEvaluator.tags); these are union-merged during evaluation so each row in self.runs carries the full tag set. Rows with NaN evaluator_result (evaluator failures) are excluded.

For boolean evaluators (the common case) the returned value is the pass rate. For numeric evaluators the mean of the raw values is returned alongside, since both share the evaluator_result column.

Returns:

Type	Description
dict[str, float]	Mapping from tag -> mean evaluator result in [0.0, 1.0]. Empty
dict[str, float]	dict when self.runs is empty, has no evaluator_result
dict[str, float]	column, or has no tagged rows.

Source code in src/ragpill/types.py

def per_tag_accuracy(self) -> dict[str, float]:
    """Mean ``evaluator_result`` grouped by tag, across all runs and evaluators.

    Tags appear on both case metadata (``TestCaseMetadata.tags``) and
    evaluator metadata (``BaseEvaluator.tags``); these are union-merged
    during evaluation so each row in ``self.runs`` carries the full tag
    set. Rows with NaN ``evaluator_result`` (evaluator failures) are
    excluded.

    For boolean evaluators (the common case) the returned value is the
    pass rate. For numeric evaluators the mean of the raw values is
    returned alongside, since both share the ``evaluator_result``
    column.

    Returns:
        Mapping from tag -> mean evaluator result in [0.0, 1.0]. Empty
        dict when ``self.runs`` is empty, has no ``evaluator_result``
        column, or has no tagged rows.
    """
    if self.runs.empty or "evaluator_result" not in self.runs.columns:
        return {}
    runs: Any = self.runs
    df_valid = runs[runs["evaluator_result"].notna()]
    if df_valid.empty:
        return {}
    grouped = df_valid.explode("tags").groupby("tags")["evaluator_result"].mean()
    return {str(tag): float(acc) for tag, acc in grouped.items() if pd.notna(tag)}  # pyright: ignore[reportUnknownMemberType]

to_json

to_json()

Serialize this :class:EvaluationOutput to a JSON string.

DataFrames are encoded via pandas.to_json(orient="split") and traces via Trace.to_json(). from_json is the inverse.

Source code in src/ragpill/types.py

def to_json(self) -> str:
    """Serialize this :class:`EvaluationOutput` to a JSON string.

    DataFrames are encoded via ``pandas.to_json(orient="split")`` and
    traces via ``Trace.to_json()``. ``from_json`` is the inverse.
    """
    return json.dumps(_evaluation_output_to_dict(self))

to_llm_text

to_llm_text(*, max_chars=32000, include_passing=True, include_spans=True, redact=True, redact_patterns=None)

Render a triage-focused markdown view suitable for LLM input.

See :func:ragpill.report.triage.render_evaluation_output_as_triage.

Source code in src/ragpill/types.py

def to_llm_text(
    self,
    *,
    max_chars: int = 32_000,
    include_passing: bool = True,
    include_spans: bool = True,
    redact: bool = True,
    redact_patterns: list[str] | None = None,
) -> str:
    """Render a triage-focused markdown view suitable for LLM input.

    See :func:`ragpill.report.triage.render_evaluation_output_as_triage`.
    """
    from ragpill.report.triage import render_evaluation_output_as_triage

    return render_evaluation_output_as_triage(
        self,
        max_chars=max_chars,
        include_passing=include_passing,
        include_spans=include_spans,
        redact=redact,
        redact_patterns=redact_patterns,
    )

CaseResult

ragpill.types.CaseResult dataclass

CaseResult(case_name, inputs, metadata, base_input_key, trace_id, run_results, aggregated)

Result for a single test case across all its runs.

Attributes:

Name	Type	Description
case_name	str	Display name or string representation of the case inputs.
inputs	Any	The original test case inputs.
metadata	TestCaseMetadata	The TestCaseMetadata for this case.
base_input_key	str	Hash of the inputs (without run index suffix).
trace_id	str	MLflow trace ID for the parent span.
run_results	list[RunResult]	One RunResult per repeat execution.
aggregated	AggregatedResult	Aggregated verdict across all runs.

RunResult

ragpill.types.RunResult dataclass

RunResult(run_index, input_key, run_span_id, output, duration, assertions, evaluator_failures=list(), error=None)

Result of a single task execution (one run of one test case).

Attributes:

Name	Type	Description
run_index	int	Zero-based index of this run within the case's repeat sequence.
input_key	str	Unique key for this run, formatted as {base_hash}_{run_index}.
run_span_id	str	MLflow span ID captured during Phase 1, used to set ContextVar in Phase 2.
output	Any	The task's return value, or None if the task raised an exception.
duration	float	Wall-clock seconds the task took to execute.
assertions	dict[str, EvaluationResult]	Evaluator name -> EvaluationResult mapping for this run.
evaluator_failures	list[EvaluatorFailureInfo]	Evaluators that raised exceptions (not pass/fail, but code errors).
error	Exception | None	The exception raised by the task, or None if it succeeded.

all_passed property

all_passed

True if the task succeeded and every assertion passed.

AggregatedResult

ragpill.types.AggregatedResult dataclass

AggregatedResult(passed, pass_rate, threshold, summary, per_evaluator_pass_rates)

Aggregated pass/fail verdict across multiple runs of the same test case.

Attributes:

Name	Type	Description
passed	bool	Whether pass_rate >= threshold.
pass_rate	float	Fraction of runs where all_passed was True (0.0 to 1.0).
threshold	float	The minimum pass_rate required to pass.
summary	str	Human-readable summary string (e.g. "2/3 runs passed").
per_evaluator_pass_rates	dict[str, float]	Per-evaluator pass rates across runs.

EvaluatorFailureInfo

ragpill.types.EvaluatorFailureInfo dataclass

EvaluatorFailureInfo(name, error_message, error_stacktrace)

Information about an evaluator that raised an exception during evaluation.

See Also

• MLflow Helper - Functions that produce these result types
• Repeated Runs Guide - Reading and interpreting results