When I first heard the CPO of OpenAI talk about how writing model evals is going to be a core product function, I had no idea what he was talking about. If that’s you, below is a short explanation of evals, when to use them, and a template for you to use to write your first evals.
BUT FIRST! Make sure to subscribe to my substack so you can get all my hacks to accelerate your career.
http://alecfullmer.substack.com
AI model evals (short for “evaluations”) are a way to test how well an AI is doing its job. Instead of just trusting that the AI works, you give it a set of real examples, ask it to perform a specific task (like summarizing a review or answering a question), and then score the results based on things like accuracy, clarity, or usefulness.
Use evals to quickly and objectively assess whether an AI-powered feature is going to be valuable and reliable before investing time and resources into building it. By testing model outputs against real-world examples and scoring them on clear criteria like accuracy, clarity, and relevance, you can compare different approaches, track improvements, and decide whether to move forward, iterate, or cut the idea entirely. Evals help teams align on what "good" looks like, reduce the risk of shipping low-quality AI, and ensure they’re building features that actually solve user problems.
| Metric | Description | Scale |
|---|---|---|
| Accuracy | Is the output factually correct? | 1 (poor) – 5 (perfect) |
| Clarity | Is the output easy to understand? | 1 (unclear) – 5 (very clear) |
| Relevance | Does the output address the most important ideas? | 1 (off-topic) – 5 (fully focused) |
| # | Input (User Review) | Model Output | Accuracy (1-5) | Clarity (1-5) | Relevance (1-5) | Comments |
|---|---|---|---|---|---|---|
| 1 | “The video takes forever to load, but the picture quality is great.” | “Great picture quality, long load time.” | 5 | 5 | 5 | Clear and accurate. |
| 2 | “I love the show selection, but the app crashes a lot.” | “The app is great and never crashes.” | 1 | 5 | 2 | Factually wrong – misses the crash issue. |
| 3 | “Too many ads during movies. It ruins the experience.” | “There are a few ads, but it’s okay overall.” | 2 | 4 | 2 | Minimizes user complaint and misrepresents tone. |
| 4 | “Simple UI, but hard to find the continue watching section.” | “Users find the app easy to use, but navigation could be improved.” | 4 | 4 | 5 | Reasonable abstraction, could be clearer. |
| 5 | “Streaming works perfectly on my phone, but not on my TV.” | “The app works well on phones but not TVs.” | 5 | 5 | 5 | Excellent summary. |