How to test an AI skill before your team relies on it.

Test an AI skill against the job it is meant to do. Define what a useful output contains, give it representative inputs and inspect both the result and any actions it took. Include cases where the right response is to expose a gap or ask for information.

A successful demonstration is a starting point. A team also needs to know how the workflow behaves when the inputs are incomplete, the source material disagrees or the person running it is new to the job.

Define success in terms the reviewer can observe.

“Write a good project update” is difficult to judge consistently. “Keep approved dates separate from proposed dates and cite the record behind each commitment” gives the reviewer something specific to check.

For existing work, ask the practitioner what makes an output usable and what would cause them to reject it. For a new capability, have its accountable owner set an initial standard. A new function can have acceptance criteria even when there is no past expert output to compare against.

Check the output's structure, support for its statements and usefulness to the responsible person. A complete document can still contain an unsupported assertion; a factually sound summary can still omit the decision the meeting needs.

Illustration: test a weekly delivery brief.

The SOP-to-workflow example describes an invented engineering firm preparing a weekly brief. The skill should distinguish approved project commitments from proposals and return a draft for a project lead. The cases below are a proposed test set, not completed runs.

Choose cases from the failures that would matter in your own workflow. A brief used in an internal meeting and an action that changes a financial record need different acceptance criteria and review arrangements.

Match the check to the question.

Some checks are mechanical: required fields, files that open, date formats or totals that reconcile. Other checks need subject knowledge: whether a risk is material, whether a summary leaves out an important qualification, or whether a proposed next step is sensible.

Anthropic's evaluation guidance distinguishes code-based checks, model-based judgments and human review. For business workflows, we recommend simple checks for explicit requirements and a knowledgeable reviewer where interpretation matters. An AI reviewer can help apply written criteria; compare its judgments with a human reviewer's and adjust the criteria before relying on its scores. Evaluation guidance.

Inspect the produced file or destination as well as the conversation. Open the document and check that it contains the requested material. If the workflow changed a record or sent something, inspect what actually happened.

Keep a record that makes a miss explainable.

For each run, record the case, input and skill versions, application, model where available, output and the reviewer's judgment with its reason. Note relevant settings and connections. Use invented or permitted test material and handle it under your business's information rules.

Maintain expected outcomes with each test case. A reference outcome or golden record might identify the approved date, the unresolved proposal and the missing owner. It records what must be correct, without requiring identical prose. Supply the test inputs to the workflow, but keep held-out answers unavailable through its instructions, files and tools. A separate folder is not an access control.

Start with a small table and saved outputs. Repeat important cases to look for inconsistent behavior, and keep some fresh cases outside the examples used to improve the instructions. Choose the amount of testing to fit the consequences of a failure.

Test what the workflow must not do.

Add cases where source material tries to change the task or send information elsewhere. In an isolated test using invented data, inspect attempted tool calls, actual changes and output destinations as well as the final answer. A statement that the workflow refused a request is insufficient if an earlier action already sent the records. (OWASP outcome-based testing)

Test denied permissions, missing approvals and attempts to reach unapproved destinations. Record what the application or server blocked, what ran and what could not be assessed. Include legitimate tasks to check that the controls still permit useful work. These tests complement output-quality cases; neither set proves immunity to prompt injection. See AI skill security for package inspection, MCP access and approval design.

Retest the whole set after changing the skill.

A repair for one exception can disrupt a previously useful response. Keep earlier cases and run them again alongside the new one. Review failures in the instructions, source material and application setup separately; rewriting the prompt cannot supply a missing file or restore a disconnected system.

Decide what “ready” means before reviewing the scores. For the invented delivery brief, the project lead could reject any run that replaces an approved date with an unapproved proposal. Passing this check would establish only what happened in those tests; every future draft would still receive review.

At Fully Powered AI, evals, validation and verification are part of turning a capability into work a team can use. We agree the standard, show misses and refine the workflow alongside the people responsible for it.

Keep the same checks when you schedule a recurring routine, and review the record of each run.