Reporting (LLM-Readable Views)¶

Markdown renderers that convert ragpill outputs into a form an LLM (or a human in a chat session) can read directly. See the LLM Reports Guide for when to use which view.

render_evaluation_output_as_triage¶

ragpill.report.triage.render_evaluation_output_as_triage ¶

render_evaluation_output_as_triage(eo, *, max_chars=32000, include_passing=True, include_spans=True, redact=True, redact_patterns=None)

Render an :class:EvaluationOutput as a triage-focused markdown document.

Parameters:

Name	Type	Description	Default
`eo`	`EvaluationOutput`	The evaluation output to render.	required
`max_chars`	`int`	Total character budget. When exceeded the renderer drops sections in this order: passing-case section, run-detail past index 5, then trailing failing cases.	`32000`
`include_passing`	`bool`	Include the collapsed passing-case section.	`True`
`include_spans`	`bool`	Include "Relevant spans" subsections for failing runs. When `False` no trace data is touched.	`True`
`redact`	`bool`	Apply redaction to span inputs/outputs (see :func:`ragpill.report._trace.render_spans`).	`True`
`redact_patterns`	`Iterable[str] \| None`	Override the default redaction regex list.	`None`

Returns:

Type	Description
`str`	A markdown string at most `max_chars` characters long.

Source code in src/ragpill/report/triage.py

def render_evaluation_output_as_triage(
    eo: EvaluationOutput,
    *,
    max_chars: int = 32_000,
    include_passing: bool = True,
    include_spans: bool = True,
    redact: bool = True,
    redact_patterns: Iterable[str] | None = None,
) -> str:
    """Render an :class:`EvaluationOutput` as a triage-focused markdown document.

    Args:
        eo: The evaluation output to render.
        max_chars: Total character budget. When exceeded the renderer drops
            sections in this order: passing-case section, run-detail past
            index 5, then trailing failing cases.
        include_passing: Include the collapsed passing-case section.
        include_spans: Include "Relevant spans" subsections for failing runs.
            When ``False`` no trace data is touched.
        redact: Apply redaction to span inputs/outputs (see
            :func:`ragpill.report._trace.render_spans`).
        redact_patterns: Override the default redaction regex list.

    Returns:
        A markdown string at most ``max_chars`` characters long.
    """
    failing, passing = _split_cases(eo.case_results)
    failing.sort(key=lambda c: (c.aggregated.pass_rate, c.base_input_key))

    case_traces = _build_case_trace_index(eo.dataset_run)

    # Render in stages so we can shed sections under budget pressure.
    header = _render_header(eo)
    failing_blocks = [
        _render_failing_case(
            cr,
            case_traces.get(cr.base_input_key),
            include_spans=include_spans,
            collapse_runs=False,
            redact=redact,
            redact_patterns=redact_patterns,
        )
        for cr in failing
    ]
    passing_block = _render_passing_section(passing) if include_passing and passing else ""

    document = _assemble(header, failing_blocks, passing_block)
    if len(document) <= max_chars:
        return document

    # Step 1: drop passing-case section.
    if passing_block:
        document = _assemble(header, failing_blocks, "")
        if len(document) <= max_chars:
            return document

    # Step 2: collapse run detail on failing cases past index 5.
    collapsed = list(failing_blocks)
    for i in range(5, len(failing)):
        collapsed[i] = _render_failing_case(
            failing[i],
            case_traces.get(failing[i].base_input_key),
            include_spans=False,
            collapse_runs=True,
            redact=redact,
            redact_patterns=redact_patterns,
        )
    document = _assemble(header, collapsed, "")
    if len(document) <= max_chars:
        return document

    # Step 3: drop trailing failing cases.
    kept: list[str] = []
    running = len(header) + 2  # header trailing newlines
    failing_header = "\n## Failing cases\n\n"
    running += len(failing_header)
    for i, block in enumerate(collapsed):
        if running + len(block) + 50 > max_chars:
            kept.append(f"\n_… ({len(collapsed) - i} additional failing cases not shown)_\n")
            break
        kept.append(block)
        running += len(block)
    return truncate(_assemble(header, kept, "", _no_split_failing=True), max_chars)

render_dataset_run_as_exploration¶

ragpill.report.exploration.render_dataset_run_as_exploration ¶

render_dataset_run_as_exploration(dr, *, max_chars=16000, include_spans=True, redact=True, redact_patterns=None)

Render a :class:DatasetRunOutput as an exploration-focused markdown document.