AI agents now answer customers, look up orders, write database queries, and take actions inside business systems. They also fail in new ways: they pick the wrong tool, pass the wrong order number, loop, or get talked into actions they should refuse. This instructor-led, hands-on course teaches technical staff how to show an agent works before it ships and how to keep it working afterward. Trainees read agent traces, run error analysis to build a failure taxonomy, and turn that taxonomy into a test suite. The suite combines code-based checks, LLM judges validated against human labels, and structured human review. They evaluate every part of an agent: router decisions, tool calls and arguments, skills such as retrieval and SQL, the full trajectory, and the final outcome. They also measure reliability across repeated trials, cost, and latency. The course covers agent security using the OWASP Top 10 for Agentic Applications, prompt-injection red-teaming, guardrails, OpenTelemetry tracing, online monitoring, release gates, and governance documentation. Labs use provider-built practice agents for fictional Massachusetts businesses in logistics, manufacturing, insurance, and life sciences. They run in a sandbox with fictional data only, on Arize Phoenix, Langfuse, or LangSmith with the employer's approved model. Trainees leave with a working evaluation suite and a written agent evaluation and operations plan. For developers, data staff, IT, and AI product owners who build, buy, or support agents.
What you will be able to do
By the end of the course, trainees will be able to:
- Run error analysis on agent traces and turn observed failures into a prioritized failure taxonomy and evaluation plan.
- Build code-based graders, LLM judges validated against human labels, and a human review rubric, and match each grader to a failure mode.
- Evaluate router decisions, tool calls and arguments, skills, trajectories, and final outcomes of a multi-step agent against a golden dataset.
- Measure reliability (pass@k and pass^k), cost per successful task, and latency across repeated trials, and compare two agent versions behind a release gate.
- Red-team an agent against the OWASP Top 10 for Agentic Applications, with prompt-injection cases, and report guardrail miss and false-alarm rates.
- Design production tracing, online evaluation, monitoring, and incident review for an agent, with privacy controls that meet Massachusetts data rules.
Who this course is for
Software developers and engineers; Data analysts and data scientists; IT and systems administrators; AI product owners and product managers; QA and test engineers; Technical support and solutions staff
Before you start
Comfortable reading Python and JSON and running a notebook; has called an LLM API or configured an AI assistant or agent at work. FAST-03 or equivalent prompting experience recommended. No machine-learning training background required.
Hands-on activities
- Lab 1 Error analysis: open-code and group 40 fictional traces from a Merrimack Freight & Logistics shipment-status agent into a failure taxonomy.
- Lab 2 Graders and a validated judge: golden dataset, code graders, and an LLM judge for a Pioneer Valley Mutual coverage-question agent; paired labeling with Cohen's kappa.
- Lab 3 Component and trajectory scorecard: router, tool-argument, SQL result-match, trajectory-match, convergence, and outcome grading on a Bayside Precision Parts quoting agent.
- Lab 4 Reliability, cost, and red-team: two agent versions run five trials each (pass^k, cost per success), plus prompt-injection cases hidden in supplier emails and guardrail measurement for a Lowell BioSupply order-desk agent.
- Lab 5 Capstone: instrument a practice agent, review its traces, and present an Agent Evaluation and Operations Plan with online evals, alerts, and a release gate.
- Trace clinic: each session opens with one real-world-style failure trace that the cohort diagnoses together.
How your learning is measured
- Attendance at all five sessions (start and end check).
- Five lab deliverables (W1 to W5) scored complete / incomplete against written criteria.
- Pre- and post-course knowledge check drawn from the 30-item quiz bank; gain reported per trainee; 80% on the post-check to meet.
- Capstone Agent Evaluation and Operations Plan scored Meets on every rubric criterion (one resubmission allowed).
- Post-course confidence survey and a 60-day follow-up asking whether the trainee applied an eval or monitoring practice at work.
