Testing Hinglish names and addresses in a privacy detector

“Hindi supported” is not a sufficient answer when customers write Hindi in Roman letters, switch to English halfway through a sentence and shorten familiar place names. A language label hides the variation that the detector will actually encounter.

Hinglish testing should begin with the messages the workflow is designed to handle. Separate the language of the words, the script used to write them and the kinds of personal information being sought. Each can change independently.

Research offers a reason to take these distinctions seriously. GLUECoS studies code-switching across tasks including named-entity recognition. Its results concern the datasets and models studied; they do not establish a privacy detector’s performance on your support queue.

Specify what a useful detection means

A hypothetical delivery business wants to remove names and precise delivery locations before its support messages enter an external summarisation service. The proposed detector must handle Romanised Hindi, English and mixed sentences.

A useful test specification would state whether a named locality, a building description and a person’s name should be marked separately. It would explain how to label a phrase that becomes identifying only when combined with the rest of the message. It would also define the permitted output: complete removal, a category placeholder or routing for review.

These choices belong in an annotation guide. Without them, two reviewers may disagree for sensible reasons, and a single score will conceal the ambiguity.

Avoid turning every unfamiliar word into a name. A respectful review process needs examples of ordinary vocabulary, commercial names and public landmarks that should remain when the policy permits them.

Vary the language deliberately

Build families of test cases around a single underlying meaning. A family might include an English sentence, a Romanised Hindi version and a mixed version. Then vary spelling, word order, spacing and abbreviations within the family.

Do not present these variants as an exhaustive dictionary of how people write. They are probes for particular failure modes. Add new families when authorised review reveals a different pattern.

A useful coverage table can have columns for language mixture, entity type, spelling variation, message length and channel. Short chat messages may provide little context. Longer messages can introduce several people and places. Both deserve representation if both enter the service.

Research on Hindi-English code-mixed named-entity recognition examines the challenges of informal mixed-language text. Use it to justify evaluation coverage, without borrowing a published model score as a product claim.

Keep related variants together when splitting

Suppose one fixture says “Meera ko parcel bhejna” and another changes only the person’s name. Putting one in development and one in the final test can make the test easier than unfamiliar input.

Group related templates and paraphrase families before dividing the dataset. Reserve some families for evaluation, including some vocabulary and sentence structures not used to tune recognisers. Document the split so that later changes do not quietly move difficult cases into development.

Presidio’s evaluation material discusses evaluating detection systems and the risks of synthetic data that is too closely related across splits. A generated dataset is useful only when its construction and limits remain visible.

Configure the whole recognition path

Presidio’s language documentation explains that additional language support involves NLP configuration and suitable recognisers, including language-specific context. An Indian identifier recogniser and a Hindi language model answer different questions. Neither alone demonstrates Hinglish name recognition.

Record the exact recognisers, model versions, supported language settings and thresholds used in a test. If routing first classifies the language, evaluate routing mistakes too. A mixed message forced into the wrong route can fail before the name detector sees it.

Review the final transformed message, not only the detected spans. Removing part of a name while leaving an identifying address can produce an apparently tidy output that misses the workflow’s objective.

Make the result actionable

For the delivery example, the pilot review should answer five questions:

• Which entity categories have enough labelled examples to support a decision?
• Which spelling or language mixtures produce recurring misses?
• Which ordinary phrases are being removed unnecessarily?
• What review path handles uncertain or unfamiliar text?
• What change will trigger a fresh evaluation before release?

Report results by meaningful groups alongside the overall score. A strong average should not hide a weak category with serious consequences. If the test set is small, say so and keep the deployment boundary narrow.

The practical deliverable is a reproducible collection of cases, decisions and observed errors. That gives the next engineer a way to improve coverage without turning “Hinglish support” into an unsupported promise.