Devanagari text needs its own evaluation cases

A detector can identify the right name and still redact the wrong characters if another component changes the text between detection and rendering. For Devanagari workflows, evaluation should follow the text through every transformation, including extraction, normalisation and the final visible output.

This is a proposed testing approach, not a claim that Devanagari always causes offset errors. The actual behaviour depends on the libraries, representation and document path. The useful question is whether the system preserves the relationship between a detected span and the original content it intends to protect.

Separate three capabilities

First, can the system obtain readable text from the input? A digital text document and a photographed form have different extraction paths.

Second, can the detector identify the required entities in that text? A Hindi named-entity model may recognise some names or locations, but that is not the same as recognising every category in a privacy policy.

Third, can the transformation remove or replace the correct material in the delivered output? This may involve text spans, page coordinates or both.

HiNER is a Hindi named-entity recognition dataset. It provides a primary research reference for Hindi NER evaluation. Its scope does not establish performance on your scanned forms, Romanised Hindi messages or complete personal-data taxonomy.

Use cases that expose boundaries

Imagine a hypothetical clinic testing redaction on appointment slips. Its approved evaluation fixtures use invented people and non-functional contact details. Some slips are digitally generated; others are images created for the test.

The test collection should vary multiword names, titles beside names, addresses across lines and punctuation adjacent to an identifier. Include cases where the same word appears in different roles. Mark the exact expected span rather than accepting any overlap with the general area.

The team should also include negative examples: headings, ordinary words and non-personal place references that its policy allows to remain. Otherwise the detector could appear successful simply by removing too much text.

For scanned cases, vary the conditions the actual intake process permits. Test different text sizes, image orientation and document quality. Do not invent an accuracy claim from a few clean screenshots.

Preserve the path back to the original

Text processing may change spacing, punctuation or character representation. If a component normalises input, record what transformation it applies and how a result maps back to the original text.

One defensible design is to keep a protected original, a processing representation and an explicit mapping between them. Another may avoid transformations that alter positions. The right choice depends on the stack, but the acceptance test should be the same: does the final output protect the intended content without damaging unrelated text?

Check offsets after every stage that can change representation. A detector span evaluated against a different version of the string is not a reliable redaction instruction.

This is particularly important when extraction returns both text and coordinates. A correct text match does not prove that the corresponding page area has been covered.

Evaluate OCR as its own stage

Tesseract’s documentation describes how skew, noise, borders and page segmentation can affect recognition quality. These factors deserve test cases when OCR is part of the workflow.

If the OCR engine misses a word entirely, a downstream text detector cannot find that missing word in its input. Keep an extraction error category separate from a recognition error category. Otherwise the team may keep tuning the detector for a failure it cannot correct.

Presidio’s image-redaction documentation describes an OCR-based path from image text to detected entities and image redaction. As checked on 22 October 2026, the documentation marks the package as beta and unsuitable for production; that limitation belongs in the evaluation decision. It is not evidence that a specific document set will be safely redacted.

Inspect the delivered document

For the clinic example, review the final image and any text layer that accompanies it. Check headers, repeated names, page edges and other locations included in the workflow. Where files contain metadata or attachments, decide separately whether those are in scope and how they will be inspected.

Use this release checklist:

• The expected entity spans were independently labelled.
• Extraction errors and detector errors were recorded separately.
• Text transformations preserve a tested mapping to the original.
• Final output inspection covers visible content and relevant embedded content.
• The report names the scripts, document types and quality conditions evaluated.

A successful test supports a bounded statement about those conditions. It does not justify “Devanagari supported” across every image, font, language mixture and identifier. Preserve the cases that challenged the system so later updates can be checked against the same evidence.