Evaluating an LLM before launch: 'looks good' is not a pass

LLM output is non-deterministic, and 'tried it a few times, seems fine' is nowhere near enough to launch. Without systematic evaluation, problems cluster under real traffic, edge inputs, and adversarial prompts.

A usable evaluation covers at least: accuracy and relevance; refusal and guardrails for sensitive and high-risk questions; robustness to adversarial input (jailbreaks, injection); and consistency across groups and languages.

The pragmatic approach is to build a reproducible eval set and scoring rubric, fold it into the release process, and regression-test after any model or prompt change. Evaluation isn't a one-off — it's ongoing quality and compliance evidence.