Table of contents
Why do we need? What problem does it solve? Why should we even care?
What kind of techniques do we have? (Methodology)
Introduction
With the advent of AI, the rapid evolution of software engineering has been quite evident. Unlike traditional software, where we write unit tests for deterministic outputs, LLMs are auto-regressive models (Ask the same question twice, and the model might give different answers, both of which could be perfectly valid). It becomes very difficult to chalk out a fix testing methodology to evaluate how the application.
Why do we need? What problem does it solve? Why should we even care?
You ship an LLM feature
It passes your manual tests. A few days later, during sanity testing, the QA team reports that the response is hallucinating wildly! You tweak the prompt again, and it works fine. Did you fix it, or did you get lucky?
This isn’t a judgment problem; it’s a measurement problem.
Welcome to the world of LLM engineering
Before we deep dive into the solutions, you need to understand something
1. Non-deterministic Problem
You write a function
def add(a:int, b: int)->int:
"""Add two numbers"""
return a + b
print(add(2,3))Call it twice with the same input, and you’ll get the same output every time. But that’s not the case with LLMs
from openai import OpenAI
client = OpenAI()
response = client.responses.create(
model="gpt-5.5",
input="Add two numbers 2 and 3."
)
print(response.output_text)The same prompt can produce a different response on every run (Maybe slightly or completely different). This isn’t a bug; it’s called temperature.
Your entire mental model for testing is based on the assumption of deterministic outputs. In the case of LLMs, passing a test case is not an end in itself; instead, you’re sampling from a probability distribution. One sample tells you almost nothing.
2. Fuzzy Correctness Problem
If your eval dashboard looks like this, you’re doing it wrong:
Relevance Score: 3.2/5
Coherence: 3.8/5
Completeness: 2.9/5
Accuracy: 4.1/5
Helpfulness: 3.5/5
What does a 3.2 even mean? What do you do with that number? Is 3.5 better than 3.2? Should you celebrate or panic?
Nobody knows. And that’s the problem.
LLM responses are multi-dimensional and subjective. There’s a range of acceptable answers and a range of bad ones. The line between them is a judgment call.
What does ‘good’ mean for your use case? You need a clear answer to that before you can evaluate anything. Most people skip this step.
3. Silent Regression Problem
In traditional software, CI(Continuous Integration) catches issues before they reach users. In LLM engineering, there’s NO such equivalent. So people rely on their intuition (Which might have its own inherent biases based on past interaction with LLMs). You find out about regression when a user complains!
These three problems make evaluating LLM's responses a bit challenging; we need some supposedly unconventional ways to validate the output
Share this post & get rewards for the referrals.
What kind of techniques do we have?
When evaluating LLMs, we have several approaches available. Let’s explore the trade-offs among the main categories, keeping in mind the need to balance scalability, cost, and reliability
1. Automatic Evaluations
It is one of the increasingly popular approaches, where we use another LLM as a judge. We use a more capable model like GPT or Claude to rate our application’s output on criteria like helpfulness, accuracy, or relevance. This approach can capture nuance that simpler metrics miss.
But here’s the catch: judges can be wrong.
They have their own biases and blind spots. An LLM judge is an approximation of human judgment, not a replacement for it.
2. Benchmark-based Evaluations
ML research communities have developed a standardized benchmark for evaluating LLM models.
MMLU (Massive Multitask Language Understanding)
Tests knowledge across many subjects, such as math, science, law, and medicine. Think of it as a general knowledge exam for LLMs.
HellaSwag
Tests common sense reasoning. The model is given the start of a scenario and must predict what happens next. (Many earlier models struggled with this.)
HumanSwag
Test code generation. The model receives a function signature and must implement it correctly. It’s measured using pass@k: how often the model gets the right answer within k attempts.
However, benchmarks have limitations. A model that scores highly on academic benchmarks might still perform poorly on our customer support application.
3. Human Evaluations
Human in the loop of evals remains the gold standard for assessing the nuanced aspects of the LLM response. Although you can’t use a human for every eval, it is not feasible to scale.
But one shouldn’t remove human evaluation completely either. It’s what keeps your system grounded. So, use human evaluation strategically:
To build and validate your golden dataset.
To periodically check the accuracy of your automated evals.
To debug when something breaks and you don’t know why.
This way, you balance cost with reliability.
Decision framework (Bonus)
There’s no single tool that solves LLM evaluation.
In practice, we've seen that what differentiates good engineers from great ones is following a structured process that both keeps them from getting stuck and ensures they deliver a reliable system (Less on guts and more metric-driven). Here’s a practical approach to building an eval process.
What does ‘Good’ mean for your use case: Defining what does good enough means should be the very first step before building the pipeline.
This will be a product decision, NOT a technical one.
Creating a golden set: Everything in your eval gets measured against one thing: your golden set. and it has to be written by an expert (Not your team member, not your manager, but an actual domain expert).
Start with ~50 examples. We don’t need thousands of examples to start getting value.
Choosing an evaluation approach: Totally depends on the availability of resources.
If you have limited resources, start with automatic evals.
If quality is paramount and the budget allows, incorporating humans in the loop is certainly not a bad idea.
Best practice is to go in hybrid mode, where we use automatic eval for broad coverage and human evals for final validation.
Set up an iteration: Once the eval approach is finalized, just run it and identify where the model is failing, make relevant tweaks in the prompt or code, and re-evaluate.
Track performance over time: Keeping a record of eval scores across changes helps us understand whether the changes are helping in preventing regression
Versioning: Track which model version, which prompt version, and which eval dataset version produced each result. This makes debugging much more efficient.
The key is to start simple and iterate quickly. The earlier you get the signs of hallucinations and simultaneously resolve them, the less likely it is to break in production.
Conclusion
Evaluation isn’t something you do at the end.
Traditional software engineering learned this lesson the hard way: CI, automated tests, production monitoring, these aren’t optional. They’re what make fast, reliable development possible.
LLM systems need the same approach. The only difference is in non-deterministic outputs that are hard to measure. By making evaluation a core part of the development process, we transform what would otherwise be guesswork into engineering.
If you got this far, someone probably came to mind. Maybe a colleague who is struggling with eval pipelines, forward this to them. Don’t write a long note. Just say “thought this was worth 10 minutes, curious what you think.”