AI-EVAL · Path 4: Build and operate
Evaluation, Observability and Guardrails
Without a golden dataset and release gates, every AI release is an experiment on your users.
- Duration
- 2 days
- Group
- up to 12 people
- Format
- In-house · Remote · In the lab
- Language
- German or English
- Price
- €15,800 flat, plus VAT
Who this track is for
Developers, platform engineers and technical owners taking an AI system towards production or already operating one. Ideal for teams whose current response to prompt changes is trying things and hoping.
Who it is not for
Not for teams without a running or concretely planned AI system, they lack the object to measure and start with AI-RAG. For the security view on threats, Secure GenAI and Agent Threat Modeling (AI-SEC) is the right track.
Starting situation
The system is live, but nobody can prove whether the last model update made answers better or worse. Complaints arrive from users instead of from monitoring, token costs grow unnoticed and every prompt change ships unchecked. What is missing is not an opinion but a measurement system.
This exists after the track
- A golden dataset for your own system: curated cases with expected results, edge cases and deliberately hard queries
- Implemented quality metrics for groundedness, relevance and completeness, including a calibrated LLM-as-judge with human spot-check validation
- Tracing across the full chain from request through retrieval and model calls to the answer, with cost and latency per step
- A cost and latency budget per request type with alert thresholds
- Defined release gates: which measurements a deployment must pass and when a rollback is triggered
Prerequisites
Programming skills in Python and a real AI system as the measurement object, or the one we provide in the training. Experience with CI/CD pipelines helps.
Preparation before the track
In advance you collect 20 real requests to your system, at least 5 of them with poor or disputed answers, anonymized in the provided format. These cases become the core of your golden dataset on Day 1. Teams without their own system receive a prepared RAG system on the lab environment.
What's included
- 2 training days with Dino Bordonaro, on site, remote or in our lab
- A personal lab environment with a prepared reference system and evaluation stack, plus 14 days of continued access
- A code repository with metric implementations, judge prompts and pipeline templates
- Templates for golden dataset, budget definition and release gate checklist
- Certificates of attendance and training documentation for your compliance archive
- A 60-minute remote office hour 4 weeks after the training to review your own gates
Agenda
Day 1
The blind flight diagnosis
Live experiment on the reference system: a seemingly harmless prompt fix measurably degrades half the answers. Question to the group: how many of your recent changes went live without measurement?
Building the golden dataset
Workshop on the cases you brought: formulating expected results, adding edge cases and trap questions, checking representativeness. The result is a versioned dataset of 30-50 cases per team.
Implementing metrics: groundedness, relevance, completeness
Hands-on: automated scoring with LLM-as-judge, calibration against human spot-check judgments, handling judge errors. By the end, every metric runs against your own dataset.
The first evaluation run
A full run of your dataset against the system, then outlier analysis: is the failure in retrieval, in the prompt or in the judge? The error classification determines the right fix.
Day result: a reliable baseline
Each team presents its baseline: dataset size, metric values, classified outliers. Test question: could you approve or reject a change based on these numbers?
Day 2
Tracing: making the chain visible
Hands-on: instrumenting the pipeline with traces per request, from retrieval hits through model calls to the answer. Exercise: diagnose a real bad case from the trace alone.
Cost and latency budget
Analyzing the trace data: what does each request type cost in tokens and milliseconds? Each team defines budgets and alert thresholds and wires them into monitoring.
Guardrails before and after the model
Implementation: input filters against prompt injection and data exfiltration, output checks for grounding and format violations. Measurement exercise: what do the guardrails catch and what do they cost in latency?
Release gates and regression detection
Workshop: the evaluation run as a mandatory stage in the deployment pipeline, thresholds per metric, rollback criteria. Test on your own system: a deliberately introduced regression must be stopped by the gate.
Day result: the gate holds
Sign-off per team: the prepared regression is detected and blocked by your own release gate, the clean version passes. The documented gate ruleset is the verifiable closing artifact.
Exercises and lab share
Around 70 percent of the time is implementation work on the lab environment: building the dataset, coding metrics, analyzing traces, testing gates. The final stress test is a real injected regression that your own gate has to catch.
Platforms
The evaluation stack is built on open-source components and runs on our lab environment, either against the system you bring or against our reference RAG. The patterns transfer to Azure and on-premises environments, platform services in the respective available version.
Transfer evidence
The transfer proof is the passed gate test on Day 2: regression caught, clean version released, ruleset documented. Attendance and results are documented in an audit-proof way.
Artifacts you take home
- A golden dataset for your own system, versioned and extensible
- An implemented metric suite with calibrated judge prompts
- A tracing setup with cost and latency analysis per request type
- A budget definition with alert thresholds as an operations document
- A release gate ruleset with rollback criteria for your own pipeline
Optional extensions
- Secure GenAI and Agent Threat Modeling (AI-SEC) for the security view beyond quality metrics
- LLMOps for connected, disconnected and air-gapped environments (AI-OPS) for the full operations framework
- Agentic Systems Engineering (AI-AGENT) if your gates are to check agents instead of assistants next
Boundaries
This track enables your team to build and run its own measurement system. It is neither an audit of your production system nor a managed monitoring service, for those we talk about a project.
Frequently asked questions
Our system is not in production yet. Is this track premature?
No, rather the opposite: teams that build the golden dataset and gates before the first release measure from day one. The only requirement is a running system as the measurement object, our reference RAG on the lab environment covers that if needed.
Aren't LLM-as-judge scores unreliable themselves?
Uncalibrated, yes, which is why calibration is a core part: judge verdicts are checked against human spot samples, deviation patterns are documented and the prompts sharpened. You also learn when a judge is the wrong choice and rule-based checks carry the load.
Does remote work as well as the lab?
Yes, the entire exercise environment runs on our lab anyway and is operated remotely. On site or in the lab you gain the whiteboard discussions, remote you save travel costs, the artifacts are identical.