How to anonymize production documents for AI agent evals
Document AI agents (claims triage, dispute handling, KYC review, medical record summarization) fail on the things synthetic test data rarely has: odd layouts, poor scans, multi-page packets and documents that refer to each other. The most useful eval set is built from real production case files. The problem is the personal data in them.
What an eval-ready anonymized copy needs
- The same structure as the original. Fonts, layout, scan quality and file format stay the same, so OCR, parsers and the agent see what they see in production. Black-box redaction removes the very fields many tasks depend on.
- Plausible, format-valid values. Card numbers that pass checksums, MRZ lines with valid check digits, dates of birth that fit the case. Agents that validate inputs should behave as they do on real data.
- Consistency across the case. If a claimant appears on a claim form, an EOB and a letter, the same replacement must appear in all three, or a cross-document task no longer has a correct answer.
- A mapping for expected answers. When identities change, the expected outputs in your eval set must change too. You need to see which original became which replacement.
- Human review. Automated detection can miss values. Someone should check every page before the copy leaves the team.
Doing it with CasePatch
- Upload the case (PDF, PNG or JPG) at the CasePatch home page. Use test or specimen documents in the hosted beta; for production files, run CasePatch in your own environment.
- Review each change. Every detected value is listed by category next to its synthetic replacement. Override any value or keep the original; overrides are re-checked so they do not reintroduce real data.
- Use the identity mapping ("who became who") to update expected answers.
- Export. PDFs stay PDFs, and the download includes a record of reviewer overrides.
Diagnoses, medications, procedures and lab results are detected but kept by default, because many eval tasks depend on them. Dates other than dates of birth, such as dates of service, are kept too.
Limits to keep in mind
CasePatch output is not a formal de-identification determination and CasePatch does not certify compliance with HIPAA, PCI DSS or GDPR. Review every page. See what CasePatch detects and pricing: 100 pages a month are free with an account.