The evidence to request before moving from a demo to production

The decision to move a privacy workflow into production should rest on evidence about the intended environment. A successful demo establishes something narrower: a selected path worked under the conditions shown.

Before expansion, ask what changes when real users, real integrations, failures and continuing operations enter the picture. The answer should be a concrete acceptance record, with clear limits and owners.

SWARN is in development. The evidence described here is what a future production decision should request; it is not a claim that these capabilities or assessments already exist.

Establish the capability being accepted

A hypothetical business has seen a demonstration of an AI-assisted redaction workflow. The first task is to name the proposed production scope: accepted inputs, languages, entity categories, recipients, systems and expected volume.

Identify mocked components and manual steps in the demonstration. A manual reviewer may be a legitimate part of the production design, but the staffing and procedure must then be real.

Ask for a dated capability record showing what is available, what was tested and what remains planned. A roadmap item cannot fill a missing production requirement unless the deployment is deliberately limited until that item is delivered.

Request evidence for the data path

The pack should show where original content goes, which services receive it, which copies are created and how access and retention are controlled. Include OCR, embeddings, model calls, telemetry, queues and support access where relevant.

For local-processing claims, ask which components remain local and how the configured path was checked. For remote services, identify the actual provider and relevant contractual and technical arrangements.

NIST’s Generative AI Profile addresses privacy risks and third-party dependencies. It supports asking about the complete system rather than accepting an assurance limited to the visible model interface.

The evidence should also explain deletion and export boundaries. An active-store deletion result does not establish treatment of backups or data held by an external service.

Examine the tested limits

For the redaction workflow, request the label guide, evaluation set description, configuration and results by important category. Ask to see consequential misses and unnecessary transformations, not only successful examples.

Presidio’s evaluation guidance supports use-case-specific testing. A published benchmark or a vendor’s score on another dataset does not establish performance on the business’s intended documents.

Include the final output stage. Correct detection is insufficient if the exported file retains the original content or if a later step reintroduces identifying information.

If the workflow uses retrieval or tools, request evidence for tenant and resource permissions. OWASP’s authorisation guidance recommends checks on every request. Test denied paths and background operations as well as the authorised demo account.

Check whether the service can be operated

Someone must own failed jobs, credential changes, model updates, user questions and exceptions. The production pack should identify those responsibilities and provide usable procedures.

Ask what happens when a privacy check is unavailable, an integration times out or a customer instruction cannot be matched. The answer should include state visibility, escalation and recovery evidence.

A queue labelled “manual review” is not an operating capability until reviewers have access, guidance and capacity. A retry button is not a recovery plan until duplicate and uncertain outcomes have been considered.

Inspect diagnostic handling too. OpenTelemetry’s sensitive-data guidance supports minimising sensitive telemetry. Production support should not depend on unrestricted prompt or document logging.

Record the acceptance decision and its limits

The business should be able to inspect a compact pack containing:

• The intended scope and dated capability record.
• The complete data path and responsible service providers.
• Access-control and isolation evidence.
• Evaluation results, representative errors and unsupported inputs.
• Failure, recovery and deletion procedures.
• Operational owners and support arrangements.
• Remaining issues with explicit deployment consequences.

Not every unresolved issue requires the same response. Some may justify a restricted pilot; others may prevent the proposed use altogether. The decision should state why the chosen boundary is acceptable and when it must be reviewed.

Avoid substituting a certificate, contract or attractive interface for the entire assessment. Each can provide relevant evidence within its scope, but none automatically answers every technical, operational and legal question.

The transition to production is complete when the proposed workflow has an accountable operating model supported by inspectable evidence. If that evidence supports only a narrower deployment, keep the deployment narrow until the missing conditions are met.