An agent project needs a complete task your company can evaluate. Start with one bounded job rather than a general promise of autonomy.
Specify the result
Describe the trigger, permitted inputs and successful finish. Include a case that cannot be completed. Name the person who receives incomplete work.
Bound the tools
Separate reading, proposing and writing. Application services enforce authorization and business rules. A model request cannot grant itself more rights.
Buy a testable delivery
Ask for representative runs, failed cases and an operating handover. Measure accepted results, cost and correction effort. Anthropic's agent guidance describes architectural patterns; the supplier should justify its choice against your task. Expand the mandate only after the initial boundaries are demonstrated.
Write a concrete job contract
Suppose the agent prepares a support handoff. Its inputs are the authorized ticket and applicable procedures. Its output is a summary, references and an identified destination. Sending a customer message or changing account state remains outside that initial job.
This contract lets you test success without interpreting a broad promise of autonomy. Include incomplete information and unavailable tools. A useful failure result explains what was attempted and which question needs an authorized person.
Inspect tool boundaries
| Tool capability | Acceptance question |
|---|---|
| Search | Are customer and document rights enforced? |
| Read a record | Is the target authorized for this user? |
| Prepare a change | Are values and evidence reviewable? |
| Execute a change | Are current conditions rechecked? |
| Retry | Can prior effects be identified safely? |
| Stop | Does an incomplete job remain visible? |
Keep credentials in controlled services. Tool outputs should return only the information needed for the task. Retrieved documents cannot grant new rights merely by containing instructions.
Evaluate the whole run
Inspect actual tool targets and results alongside the final answer. An agent can write a correct-looking response after accessing the wrong record or taking an unwanted intermediate action. Both outcomes belong in evaluation.
Hold out some cases from preparation. Include known failures and review them after changes to instructions, tools or model configuration. Ask the supplier to explain which failures are recoverable automatically and which require human intervention.
Plan ownership after the pilot
Assign people to source maintenance, tool contracts and interrupted runs. Cap duration and consumption with controls your operators can verify. A cost estimate should include retries and review effort per accepted job.
Have a second operator reproduce a run using the handover. Then investigate a known failed case. Missing context discovered here should be corrected before the engagement ends.
These deliverables make competing proposals comparable. Your organization can judge a complete authorized task, its operating cost and its recovery behavior, rather than relying on the appearance of a successful demonstration.
Related reading : AI agents with tool credentials: keep secrets outside generated context. Discuss your project scope with Sky Vaults.