Test actions as well as answers
An AI assistant can state a correct rule and still act against it. The test cases cover public services, finance and tourism.
Consider a fictional payment test. The assistant must prepare a draft and wait for approval. If it sends the payment without approval, the test records a critical failure. The evaluator examines the answer and the action.
Check whether the cases fit the local service
A local service can use different documents, languages, connections and referral procedures from those in a model’s earlier tests. A place name in a prompt does not demonstrate local coverage. The HELM authors examine “what’s missing or underrepresented” in model evaluation. [helm]
All LAC Eval prompts are in English. The set has nine place labels and six risk categories. It does not establish performance in Haitian Creole, Jamaican language varieties, Spanish, Portuguese, Dutch or other languages. Each language needs reviewed cases and competent assessors. A translated interface does not provide that evidence.
Test permitted help and prohibited requests
Eighteen cases ask for permitted help. Eighteen ask for prohibited or unsupported help. A service that refuses every request would fail permitted tasks.
Evaluators rate task completion, use of evidence, compliance with action limits, and recovery or referral. A critical failure has a separate record and blocks a release recommendation. A high average cannot cancel an unauthorised transfer or disclosure. AgentHarm supports the assessment of actions through software tools, as well as written refusals. [agentharm]
Prepare one service for a test
Choose one service. List the sources it can use. Specify which actions require a named person’s approval. Define the response to a tool failure. Use fictional records before you introduce customer information.
A model comparison also needs separate final test cases, fixed settings, independent ratings and local review. LAC Eval provides the public cases and test plan for this preparation. It does not certify that a service is safe.