# Session 2 Handout: Checking Well: Evaluating and Verifying AI Output

<p class="meta">FAST-16 Working Well with AI · 7formations · Instructor: Wadood Chaudhary · All examples fictional · Suggested reading: Svekis, AI Literacy for All Employees (2026), chapter 7</p>

## Why this session matters

AI output is fluent, fast, and confident whether it is right or wrong. The person who uses it is responsible for it. A short, repeatable check catches most problems before they reach a customer or a manager.

## Key concepts

### 1. Five types of output error

| Type | What it looks like | Merrimack Freight example (fictional) |
| --- | --- | --- |
| Factual error | A wrong number, name, date, or rule | "Route 9: 120 deliveries" when the data says 102 |
| Incomplete information | A key point left out | No mention of the two-day snow closure |
| Context misread | Wrong purpose, audience, or format | A customer newsletter instead of an internal memo |
| Logical inconsistency | The output contradicts itself | "Improved to 91%" in one place, "88%" in another |
| Bias or imbalance | One-sided or unfair generalization | Blames "older drivers" with no evidence |

### 2. The five-step evaluation

1. **Accuracy:** check facts and figures against the source.
2. **Completeness:** what is missing?
3. **Relevance:** does it answer what was asked, for the right reader?
4. **Bias:** is anyone treated or described unfairly? Any missing perspective?
5. **Judgment:** knowing the business and the stakes, would you put your name on it?

### 3. Verification techniques

- **Trace to the source** and recalculate key figures.
- **Lateral reading (SIFT):** Stop; Investigate the source; Find better coverage; Trace claims to the original. Leave the page to see what independent and official sources say.
- **Compare** two or three outputs; differences flag uncertainty (agreement does not prove accuracy).
- **Ask an expert** for specialized, high-stakes topics such as wage rules or safety.

### 4. Invented citations

Assistants can produce realistic-looking sources that do not exist. A source counts only when you have opened it and it says what the AI claims. Do not ask the same AI whether its source is real. The book suggests asking AI for its sources; that is a good start only if you then check them.

## Worked example

A report cites "Northeast Freight Review, May 2025, p. 14" for a 94% industry on-time average. A search finds no such publication. The fix: remove the figure or replace it with a verified source, and tell the report's owner what you found. In Mata v. Avianca (2023), lawyers were sanctioned for filing AI-invented case citations they did not check.

## Try this before session 3

- Run the five-step evaluation on one AI output you see this week.
- Read Svekis chapter 7.