When migrating Large Language Model (LLM) applications and autonomous agents from prototype to production, traditional software testing paradigms fail. Because LLM outputs are non-deterministic, static code assertions can’t tell you if a response generated an unhelpful tone, leaked private data, or hallucinated details.
To bridge this operational gap, the modern AI engineering stack relies on LLM-as-a-Judge frameworks. By using an independent, high-capability language model (like GPT-4o or Claude 3.5 Sonnet) to evaluate application inputs and outputs, you get continuous, scalable monitoring that mirrors human reviewer preferences.
MLflow provides native, robust primitives to write custom judges, score metrics using natural language criteria, and run automated evaluations checking your application’s outputs against strict structural expectations.
🏗️ The LLM Evaluation Architecture in MLflow
MLflow orchestrates evaluation workflows by compiling model data, evaluation parameters, and metric judges into an isolated execution pipeline.
- Evaluation Dataset Ingestion: A static or streaming dataframe consisting of prompt variables (
inputs) and optional ground truth benchmarks (expectations). - The Predictive Function (
predict_fn): The live hook calling your target system (e.g., your LangGraph agent or raw OpenAI wrapper pipeline). - The Scorer Framework: A sequence layout containing built-in or custom statistical judges.
- The Evaluation Loop (
mlflow.genai.evaluate): Orchestrates the data flow, securely passes execution traces to the judge models, tracks latency and token expenses, and populates a side-by-side visualization dashboard.
📐 1. Creating Custom Metrics Using Scores (LLM-as-a-Judge)
Based on MLflow’s evaluation design rules, you can compile completely custom judges using make_judge or natural language Guidelines.
Suppose you want to enforce a distinct metric evaluating a translation or communication pipeline’s Cultural Sensitivity & Idiomatic Accuracy across a continuous scoring hierarchy (e.g., scoring from 1 to 5). You construct the judge dynamically using the Python SDK:
Python
from typing import Literal
import mlflow
from mlflow.genai.judges import make_judge
# 1. Ensure your core MLflow experiment tracker is set up
mlflow.set_experiment("llm-as-a-judge-metrics")
# 2. Define a custom score-based metric judge using natural language instructions
cultural_sensitivity_judge = make_judge(
name="cultural_sensitivity",
instructions=(
"Assess how faithfully the translation in {{ outputs }} captures the "
"cultural nuances, context, and idiomatic expressions of the query in {{ inputs }}. "
"Rate the translation quality strictly based on these four values:\n"
"- excellent: Completely preserves idioms, cultural relevance, and appropriate tone.\n"
"- good: Culturally accurate but phrasing feels slightly literal.\n"
"- fair: Translation is understandable but completely misses local idioms.\n"
"- poor: Culturally inappropriate, offensive, or structurally wrong."
),
# Force the judge model to respond using strict Pydantic Literal categories
feedback_value_type=Literal["excellent", "good", "fair", "poor"],
model="openai:/gpt-4o-mini" # The model tasked with executing the judgment
)
🛠️ 2. Evaluating Outputs via Prompts and Expectations
In a production scenario, you want to test how your model performs across an evaluation dataset. A robust pipeline passes target inputs to your app, collects the outputs, and hands them to a Correctness judge that evaluates compliance against ground-truth business contracts (expectations).
Let’s look at the complete end-to-end Python script implementing this:
Python
import pandas as pd
import mlflow
from mlflow.genai.scorers import Correctness, Guidelines
# 1. Formulate your test suite data (including prompt inputs and ground truth expectations)
eval_dataset = [
{
"inputs": {"question": "What is the return policy for international shipments?"},
"expectations": {
"expected_facts": [
"International returns are accepted within 30 days.",
"Customers must cover international return shipping fees.",
"Original duties and taxes are non-refundable."
]
}
},
{
"inputs": {"question": "Can I return a clearance product item?"},
"expectations": {
"expected_facts": [
"Clearance items are final sale.",
"No refunds or exchanges allowed on clearance merchandise."
]
}
}
]
# Convert the layout array into a standard Pandas DataFrame
eval_df = pd.DataFrame(eval_dataset)
# 2. Define your application's operational target function
def customer_support_bot(inputs_df):
results = []
for _, row in inputs_df.iterrows():
question = row["inputs"]["question"]
# Simulating your active production model pipeline response:
if "international" in question.lower():
results.append("We accept returns within 30 days! However, you must pay for your own international return shipping.")
else:
results.append("Sorry, clearance items are final sale and we don't do refunds.")
return results
# 3. Instantiate built-in and custom evaluation judges
# The Correctness judge compares 'outputs' against 'expected_facts' automatically
correctness_scorer = Correctness(model="openai:/gpt-4o-mini")
# Combine with an explicit style compliance Guidelines judge
compliance_scorer = Guidelines(
name="brand_voice",
guidelines=[
"The response must maintain a polite, professional, and helpful customer support tone.",
"The response must not contain internal code block variables or technical jargon."
],
model="openai:/gpt-4o-mini"
)
# 4. Trigger the full automated evaluation pipeline run
print("🚀 Launching automated MLflow evaluation pipeline...")
evaluation_results = mlflow.genai.evaluate(
data=eval_df,
predict_fn=customer_support_bot,
scorers=[correctness_scorer, compliance_scorer, cultural_sensitivity_judge]
)
# 5. Inspect global execution metrics summary
print("\n📊 --- GLOBAL RUN AGGREGATE METRICS --- 📊")
print(evaluation_results.metrics)
# 6. Print individual row-by-row verdicts and rationales
print("\n📝 --- DETAIL VERDICTS BREAKDOWN --- 📝")
results_df = evaluation_results.result_df
for idx, row in results_df.iterrows():
print(f"\n[Prompt Input]: {row['inputs/question']}")
print(f"[Bot Output]: {row['outputs']}")
print(f"[Correctness Score]: {row['correctness/value']}")
print(f"[Judge Rationale]: {row['correctness/rationale']}")
print("-" * 50)
📈 Interpreting Evaluation Results in the Dashboard
When the script runs, MLflow securely stores the outputs inside your local or remote backend tracking server storage.
value: Returns the specific chosen string metric category (e.g.,yes/noorexcellent/poor).rationale: The absolute superpower of MLflow judges. The judging model leaves an explicit, text-based justification explaining why it docked points (e.g., “The response accurately captured that international returns are 30 days, but completely omitted the expected fact that duties/taxes are non-refundable.”).
By opening the native MLflow Web UI (mlflow ui), you can visualize your experiment runs side-by-side. If you adjust your core system prompt template or swap models from GPT-3.5 to GPT-4o, you can immediately run evaluation iterations and compare charts to ensure your accuracy scores trend upward before deploying your agents to production.