Caribbean AI AssociationCARIBBEAN
AI ASSOCIATION

CAIA ANALYSIS / LAC Eval

How to test an AI service for local use

LAC Eval provides 36 fictional cases to test how an AI service follows rules, uses evidence and protects private information. No model results are available.

By Adrian Dunkley · 10 November 2026

The case set has six risk categories with three benign and three adversarial cases each. No model results are shown.

Test actions as well as answers

An AI assistant can state a correct rule and still act against it. The test cases cover public services, finance and tourism.

Consider a fictional payment test. The assistant must prepare a draft and wait for approval. If it sends the payment without approval, the test records a critical failure. The evaluator examines the answer and the action.

Check whether the cases fit the local service

A local service can use different documents, languages, connections and referral procedures from those in a model’s earlier tests. A place name in a prompt does not demonstrate local coverage. The HELM authors examine “what’s missing or underrepresented” in model evaluation. [helm]

All LAC Eval prompts are in English. The set has nine place labels and six risk categories. It does not establish performance in Haitian Creole, Jamaican language varieties, Spanish, Portuguese, Dutch or other languages. Each language needs reviewed cases and competent assessors. A translated interface does not provide that evidence.

Test permitted help and prohibited requests

Eighteen cases ask for permitted help. Eighteen ask for prohibited or unsupported help. A service that refuses every request would fail permitted tasks.

Evaluators rate task completion, use of evidence, compliance with action limits, and recovery or referral. A critical failure has a separate record and blocks a release recommendation. A high average cannot cancel an unauthorised transfer or disclosure. AgentHarm supports the assessment of actions through software tools, as well as written refusals. [agentharm]

Prepare one service for a test

Choose one service. List the sources it can use. Specify which actions require a named person’s approval. Define the response to a tool failure. Use fictional records before you introduce customer information.

A model comparison also needs separate final test cases, fixed settings, independent ratings and local review. LAC Eval provides the public cases and test plan for this preparation. It does not certify that a service is safe.

Explore the project

LAC Eval - alignment

Use the assessment or read the full study.

Sources and references

Sources checked on 10 November 2026. The notes state which material CAIA checked and how the study uses it.

  1. Liang, P., et al. (2022). Holistic Evaluation of Language Models.Abstract and current arXiv record; originally submitted 2022, revised 2023. Multi-metric evaluation, coverage gaps and transparent prompts/completions.
  2. Andriushchenko, M., et al. (2024). AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents.Abstract and benchmark design: tool-using agents require action-level assessment.

Cite this article

Dunkley, A. (2026). How to test an AI service for local use. Caribbean AI Association. Version 0.1, 10 November 2026. https://caribbeanaiassociation.com/articles/caribbean-ai-needs-a-local-test/

Download BibTeX

Use the article title and version in your citation. Credit the original data sources. State any changes that you make. Third-party materials retain their original rights. This page does not claim peer review or provide a DOI.

BibTeX
@misc{dunkley2026lacevalarticle,
  author = {Dunkley, Adrian},
  title = {How to test an AI service for local use},
  year = {2026},
  publisher = {Caribbean AI Association},
  note = {Version 0.1; 10 November 2026},
  url = {https://caribbeanaiassociation.com/articles/caribbean-ai-needs-a-local-test/}
}