LLM Judge

Low-level LLM-as-a-judge helpers used by LLMJudge. These wrap a pydantic_ai.Agent and return a structured GradingOutput.

GradingOutput

ragpill.llm_judge.GradingOutput

Bases: BaseModel

Structured output from the LLM judge.

Attributes:

Name	Type	Description
reason	str	Human-readable explanation of the verdict.
pass_	bool	Whether the output passes the rubric. Aliased to "pass" for JSON compatibility.
score	float	Continuous score in [0.0, 1.0].

judge_output

ragpill.llm_judge.judge_output async

judge_output(output, rubric, model, model_settings=None)

Judge a task output against a rubric.

Parameters:

Name	Type	Description	Default
output	Any	The task output to grade.	required
rubric	str	Rubric describing what "passing" means for this output.	required
model	Model | KnownModelName | str	The pydantic-ai model to use for grading.	required
model_settings	ModelSettings | None	Optional pydantic-ai ModelSettings.	None

Returns:

Type	Description
GradingOutput	A GradingOutput with reason, pass_, and score.

Example

from pydantic_ai.models.test import TestModel
from ragpill.llm_judge import judge_output

result = await judge_output("Hello world", "contains a greeting", TestModel())
print(result.pass_, result.reason)

Source code in src/ragpill/llm_judge.py

async def judge_output(
    output: Any,
    rubric: str,
    model: models.Model | models.KnownModelName | str,
    model_settings: ModelSettings | None = None,
) -> GradingOutput:
    """Judge a task output against a rubric.

    Args:
        output: The task output to grade.
        rubric: Rubric describing what "passing" means for this output.
        model: The pydantic-ai model to use for grading.
        model_settings: Optional pydantic-ai ``ModelSettings``.

    Returns:
        A ``GradingOutput`` with ``reason``, ``pass_``, and ``score``.

    Example:
        ```python
        from pydantic_ai.models.test import TestModel
        from ragpill.llm_judge import judge_output

        result = await judge_output("Hello world", "contains a greeting", TestModel())
        print(result.pass_, result.reason)
        ```
    """
    user_prompt = _build_prompt(output=output, rubric=rubric)
    return (await _judge_output_agent.run(user_prompt, model=model, model_settings=model_settings)).output

judge_input_output

ragpill.llm_judge.judge_input_output async

judge_input_output(inputs, output, rubric, model, model_settings=None)

Judge a task output against a rubric given the inputs.

Parameters:

Name	Type	Description	Default
inputs	Any	The task inputs included alongside the output in the prompt.	required
output	Any	The task output to grade.	required
rubric	str	Rubric describing what "passing" means for this output.	required
model	Model | KnownModelName | str	The pydantic-ai model to use for grading.	required
model_settings	ModelSettings | None	Optional pydantic-ai ModelSettings.	None

Returns:

Type	Description
GradingOutput	A GradingOutput with reason, pass_, and score.

Example

from pydantic_ai.models.test import TestModel
from ragpill.llm_judge import judge_input_output

result = await judge_input_output(
    inputs="Pirate",
    output="Avast ye!",
    rubric="does speak like a pirate",
    model=TestModel(),
)

Source code in src/ragpill/llm_judge.py

async def judge_input_output(
    inputs: Any,
    output: Any,
    rubric: str,
    model: models.Model | models.KnownModelName | str,
    model_settings: ModelSettings | None = None,
) -> GradingOutput:
    """Judge a task output against a rubric given the inputs.

    Args:
        inputs: The task inputs included alongside the output in the prompt.
        output: The task output to grade.
        rubric: Rubric describing what "passing" means for this output.
        model: The pydantic-ai model to use for grading.
        model_settings: Optional pydantic-ai ``ModelSettings``.

    Returns:
        A ``GradingOutput`` with ``reason``, ``pass_``, and ``score``.

    Example:
        ```python
        from pydantic_ai.models.test import TestModel
        from ragpill.llm_judge import judge_input_output

        result = await judge_input_output(
            inputs="Pirate",
            output="Avast ye!",
            rubric="does speak like a pirate",
            model=TestModel(),
        )
        ```
    """
    user_prompt = _build_prompt(inputs=inputs, output=output, rubric=rubric)
    return (await _judge_input_output_agent.run(user_prompt, model=model, model_settings=model_settings)).output

See Also

• LLMJudge — the high-level evaluator that wraps these functions.