Part 2 of a series on testing AI systems in production In Part 1, we explored why testing AI systems is fundamentally different from traditional software. We talked about non-determinism, prompt sensitivity, and why unit tests aren’t enough. Now let’s move from theory to practice. How do you actually build a system to test AI reliably? This post walks through a practical approach to building an AI evaluation pipeline—from dataset creation to CI/CD integration. At a high level, an evaluation pipeline looks like this: Dataset → System → Evaluation → Metrics → Analysis More concretely: You define a dataset of test cases Run them through your AI system Evaluate outputs using defined metrics Store and analyze results over time This becomes your source of truth for system quality. Your evaluation pipeline is only as good as your dataset. Production logs (most valuable) Synthetic examples (for coverage) Edge cases and failure scenarios { "input": "What is the refund policy?", "expected": "Answer should mention 30-day refund window", "context": "Optional (for RAG systems)", "metadata": { "type": "faq", "difficulty": "easy" } } Represents real user behavior Includes edge cases Covers known failure modes Insight: Most teams underestimate this step. Dataset quality matters more than model choice in many cases. Unlike traditional systems, correctness isn’t always binary. You’ll need a mix of evaluation strategies. 1. Exact match (for structured tasks) Useful for classification or JSON outputs 2. Semantic similarity Measures meaning, not exact wording 3. LLM-as-a-judge Uses a model to evaluate output quality 4. Task success (for agents) Did the system complete the objective? Exact match → precise but brittle Semantic → flexible but fuzzy LLM judge → scalable but imperfect The key is combining multiple signals. At this stage, you execute your system against the dataset. A simple evaluation loop might look like this: results = [] for sample in dataset: output = system.run(sample["input"]) score = evaluator( output=output, expected=sample.get("expected"), context=sample.get("context") ) results.append({ "input": sample["input"], "output": output, "score": score }) Keep it simple at first. Complexity can come later. Raw scores are not enough. You need visibility. Inputs Outputs Scores Metadata Failure tagging Error categories (hallucination, formatting, etc.) Trace logs (especially for agents) This is what allows you to answer: Why did the system fail? Without this layer, debugging becomes guesswork. An evaluation pipeline is not a one-time exercise. You should be able to answer: Did the latest change improve performance? Did hallucination rates increase? Did a prompt tweak break edge cases? Accuracy Hallucination rate Task success rate Version your datasets and compare results across runs. This is where evaluation becomes part of engineering discipline. Run evaluations when: Prompts change Models are updated Retrieval logic is modified Code Change → Run Evals → Compare Metrics → Pass/Fail You can define thresholds like: Fail if accuracy drops below X% Fail if hallucination rate increases This prevents silent regressions. Putting it all together: Dataset ↓ Run System ↓ Evaluate Outputs ↓ Store Results ↓ Compare with Previous Runs ↓ Trigger Alerts / Decisions This is your AI quality control loop. Let’s say you’re testing a support chatbot. Manual testing Inconsistent results Hard to track improvements ~200 real queries as dataset Automated evaluation on every update Clear metrics (correctness, grounding) Faster iteration Reduced hallucinations Better confidence in releases Even with a pipeline, teams run into issues: Overfitting to the evaluation dataset Blind trust in LLM-as-a-judge Not updating datasets with real usage Lack of dataset versioning Avoid treating evals as static—they should evolve with your system. In the next part of this series, I’ll go deeper into: Evaluating RAG systems (retrieval + generation) Measuring context relevance and faithfulness Common failure patterns in retrieval pipelines AI systems don’t fail loudly—they drift. An evaluation pipeline gives you a way to detect, measure, and control that drift. It’s not just about testing once. Is my AI still working as expected?