Before an AI prototype becomes production software

1. Define the whole workflow.

Suppose a system reads incoming supplier documents and prepares records for an operations tool. The model extracts a supplier name, reference number, and requested date. A person checks the proposed record before it is written to the destination.

The useful output is an approved, traceable record. Specify the accepted document types, required fields, permitted transformations, and review responsibilities. Keep the source document available to the reviewer. Missing information should remain visibly missing.

Start with a predictable workflow when the steps are known. Anthropic’s guidance on building agents distinguishes predefined workflows from systems that choose their own steps and recommends adding complexity only when it helps the task.

2. Test representative cases.

Build a set of examples with expected results, including difficult and unsuccessful inputs. Separate the cases used to develop the system from those used to check a change. Model, prompt, parser, and integration changes can all affect the outcome.

Evaluate the extracted fields and the resulting workflow state. A well-formed response can still contain the wrong supplier or leave a record in the wrong place. Anthropic’s evaluation guidance discusses why clear tasks, appropriate grading, and inspection of system behavior matter.

Choose acceptance thresholds according to the consequence of each error. Track incorrect accepted records separately from cases sent for review. Also measure review effort, response time, and cost per completed item; a single average accuracy number cannot explain the whole workflow.

3. Control access and actions.

Enforce access in the application and tools. Give the extraction step only the documents it needs. Let the write operation accept validated, approved records with the appropriate authorization.

Document text is input data, even when it contains instructions. It should not be able to grant permissions, select an unrelated destination, or authorize a write. Include misleading instructions and attempts to access other records in the test cases.

4. Plan for partial failure.

A timeout does not tell you whether a remote write happened. Use a stable operation identifier where the destination supports one, or reconcile against the destination before retrying. If the outcome cannot be established, show it as unresolved and provide a review path.

Define bounded retries, failure queues or review states as the workflow requires. Record enough information to diagnose an operation without copying sensitive document contents into routine logs. Agree who handles exceptions and how source documents, results, and logs are retained.

5. Choose a bounded first release.

Begin with a defined document type and a manageable set of users. Compare proposed records with the current process before enabling writes, then keep human approval while collecting evidence about actual errors and review effort.

Agree on when to pause the workflow, how to return to manual handling, and who owns changes. Wider automation should follow evidence that the workflow meets its requirements, including the failure cases—not just a successful demonstration.