False positives have a cost: measure the review work they create

An unnecessary redaction can make a support message hard to understand, send a harmless document to manual review or stop an operator from completing a routine task. Those costs deserve measurement.

They do not make missed personal data less important. The evaluation should show both kinds of error and the work needed to manage them, so a lower review queue is not mistaken for a safer system.

Define the operational consequence

A hypothetical privacy filter checks supplier correspondence before it enters a shared analysis workspace. It flags possible names, contact details and private addresses.

Some flags are correct. Others may capture product names or ordinary phrases. The operational effect depends on the workflow: a false detection could be silently removed, presented for confirmation or used to block the entire document.

Record that effect alongside the detection label. Two false positives can have very different consequences if one removes an irrelevant greeting and the other hides the product reference needed to resolve a complaint.

Presidio’s evaluation guidance supports measuring detection performance for the intended use case. Extend that evaluation with the downstream work your design actually creates.

Measure documents and decisions as well as spans

Span-level precision tells you how many detections were correct. It does not directly tell you how many documents a reviewer must open or how much effort each requires.

Track the number of documents routed to review, the number that need correction and the types of decisions reviewers make. Where timing is measured, define when the task starts and ends, and distinguish active review from time waiting in a queue.

Use a controlled pilot to measure these quantities. Do not estimate review cost from an invented average or borrow a benchmark from a different organisation.

Keep the denominator visible. “Reviews per hundred processed documents” answers a different question from “incorrect detections per hundred predicted spans”. Both can be useful, provided the report does not interchange them.

The standard precision definition uses predicted positives as its denominator. A low precision result is not itself a measured false-positive rate across all non-sensitive text.

Review the difficult cases

For the supplier example, ask reviewers to classify why a flag was unnecessary. Was the entity policy unclear? Did a recogniser mistake a product for a person? Was the detected span too broad? Did the routing rule escalate an otherwise harmless uncertainty?

These categories point to different fixes. Improving the annotation guide may resolve policy disagreement. Narrowing a transformation may preserve useful context. Changing a threshold may trade fewer flags for more misses.

Do not label every reviewer override a detector error. The reviewer may have additional context, or the policy may have changed. Record enough reasoning to distinguish those cases.

A small adjudicated sample can help check whether reviewers interpret the rules consistently. Disagreement is useful evidence about the workflow, not something to erase before calculating a score.

Compare changes against the same cases

During development, run the old and new configurations on the same development cases. Compare the errors that disappeared with the errors that appeared. Once the choices are fixed, confirm the comparison on separate held-out material that has not guided the tuning. Scikit-learn’s guidance on data leakage explains why test data should not influence model choices.

A reduction in review volume can be misleading if the new configuration simply stops detecting a difficult personal-data category. Show the corresponding recall and consequential misses.

Also inspect workload distribution. A modest total queue may still be unmanageable if one language or document type requires specialist review and arrives in bursts.

Do not optimise a weighted score until the team understands what its weights represent. Converting all errors into one number can conceal decisions that deserve explicit approval.

Set a practical acceptance boundary

The pilot report should describe what reviewers can handle, which cases must pause and what happens when the queue exceeds capacity. A manual-review fallback is only credible if the people, permissions and procedure exist.

Use this checklist:

• Detection errors are linked to their actual downstream effect.
• Review volume and correction effort have defined denominators.
• Timing data comes from an observed, described pilot.
• Reviewer disagreement is recorded and resolved where necessary.
• Every tuning change is checked for newly missed personal data.
• High-consequence categories remain visible outside aggregate scores.
• Queue capacity and escalation rules match the proposed deployment.

The decision may be to narrow the pilot, improve the entity policy or change the interface before changing the detector. Measuring review work makes those choices possible without pretending that fewer alerts automatically mean better privacy protection.