Measuring Quality and Safety in LLM Applications
Introduction Before adding controls to an LLM application, you need to know what is happening. Quality and safety measurement gives you that visibility. This course focuses on metrics and monitoring rather than runtime blocking. It uses chat datasets, WhyLogs, LangKit, custom UDFs, model-based scoring, and active monitoring patterns to inspect hallucinations, data leakage, toxicity, refusals, and prompt injection. The useful lesson is practical: do not treat “the model seems fine” as evidence. Log the interactions, compute signals, inspect critical examples, and evaluate filtered subsets. ...