Environment Variables

For development, we recommend using a '.env' file in the root of your project to manage environment variables for ragpill. This allows you to easily configure settings for components like LLMJudge and MLFlow without hardcoding sensitive information in your code.

This file is automatically loaded by pydantic-settings in the LLMJudgeSettings and MLFlowSettings classes.

Settings Classes

MLFlowSettings

ragpill.settings.MLFlowSettings

Bases: BaseSettings

MLflow connection and evaluation settings.

Controls where evaluation results are logged and the default repeat/threshold behaviour for multi-run evaluations. All fields can be set via environment variables with the MLFLOW_ prefix (e.g. MLFLOW_RAGPILL_TRACKING_URI).

Example

from ragpill.settings import MLFlowSettings

settings = MLFlowSettings(
    ragpill_tracking_uri="http://mlflow.internal:5000",
    ragpill_experiment_name="my_evaluation",
)

LLMJudgeSettings

ragpill.settings.LLMJudgeSettings

Bases: BaseSettings

Configuration for the LLMJudge evaluator's backing LLM.

Controls which model, temperature, and API endpoint the LLMJudge evaluator uses. All fields can be set via environment variables with the RAGPILL_LLMJUDGE_ prefix.

This class supports a singleton pattern via get_llm_judge_settings and configure_llm_judge. The singleton caches a pydantic_ai.models.Model instance so that custom httpx / SSL configuration (e.g. corporate CA bundles) only needs to be set up once.

Example

from ragpill.settings import LLMJudgeSettings, configure_llm_judge

settings = LLMJudgeSettings(
    model_name="gpt-4o",
    base_url="https://my-proxy.example.com/v1",
    api_key="sk-...",
)

# Or configure the singleton with custom SSL handling
configure_llm_judge(
    api_key="sk-...",
    base_url="https://my-proxy/v1",
    model_name="gpt-4o",
    ssl_ca_cert="/path/to/custom-ca-bundle.pem",
)

llm_model property

llm_model

Lazily build and cache a pydantic_ai.models.Model from the current settings.

The model is created on first access and reused on subsequent calls. Use :meth:set_model to inject a fully custom model instance instead.

set_model

set_model(model)

Override the cached model with a fully custom instance.

Useful when you need full control over the httpx client or provider (e.g. custom middleware, mTLS, retry policies).

Source code in src/ragpill/settings.py

def set_model(self, model: models.Model) -> None:
    """Override the cached model with a fully custom instance.

    Useful when you need full control over the ``httpx`` client or provider
    (e.g. custom middleware, mTLS, retry policies).
    """
    self._cached_model = model

get_llm_judge_settings

ragpill.settings.get_llm_judge_settings

get_llm_judge_settings()

Return the global :class:LLMJudgeSettings singleton.

Creates a default instance (reading from env vars) on first call. Use :func:configure_llm_judge to set up custom values before first use.

Source code in src/ragpill/settings.py

def get_llm_judge_settings() -> LLMJudgeSettings:
    """Return the global :class:`LLMJudgeSettings` singleton.

    Creates a default instance (reading from env vars) on first call.
    Use :func:`configure_llm_judge` to set up custom values before first use.
    """
    global _llm_judge_settings
    if _llm_judge_settings is None:
        _llm_judge_settings = LLMJudgeSettings()  # pyright: ignore[reportCallIssue]
    return _llm_judge_settings

configure_llm_judge

ragpill.settings.configure_llm_judge

configure_llm_judge(settings=None, **kwargs)

Create or replace the global :class:LLMJudgeSettings singleton.

Pass either a pre-built settings instance or keyword arguments that will be forwarded to the LLMJudgeSettings constructor.

Returns the newly configured singleton.

Source code in src/ragpill/settings.py

def configure_llm_judge(
    settings: LLMJudgeSettings | None = None,
    **kwargs: Any,
) -> LLMJudgeSettings:
    """Create or replace the global :class:`LLMJudgeSettings` singleton.

    Pass either a pre-built ``settings`` instance or keyword arguments that will
    be forwarded to the ``LLMJudgeSettings`` constructor.

    Returns the newly configured singleton.
    """
    global _llm_judge_settings
    if settings is not None:
        _llm_judge_settings = settings
    else:
        _llm_judge_settings = LLMJudgeSettings(**kwargs)
    return _llm_judge_settings

Example .env File

Here's a complete example .env file:

# MLFlow Configuration
EVAL_MLFLOW_TRACKING_URI=http://localhost:5000
EVAL_MLFLOW_EXPERIMENT_NAME=my_project_evaluation
EVAL_MLFLOW_RUN_DESCRIPTION="Testing new prompts"
EVAL_MLFLOW_TRACKING_USERNAME=your-username
EVAL_MLFLOW_TRACKING_PASSWORD=your-password

# LLMJudge Configuration
RAGPILL_LLMJUDGE_MODEL_NAME=gpt-4o
RAGPILL_LLMJUDGE_TEMPERATURE=0.0
RAGPILL_LLMJUDGE_BASE_URL=https://my_domain.com/v1
RAGPILL_LLMJUDGE_API_KEY=your-actual-api-key-here