Auditing Private Evolution: When DP Bounds Meet Real Attacks
A new arXiv preprint (Canaries in the Bank) does something the differential privacy community rarely bothers with: it measures what actually leaks from a production-shaped federated synthetic-data pipeline, then compa…
Auditing Private Evolution: When DP Bounds Meet Real Attacks
A new arXiv preprint (Canaries in the Bank) does something the differential privacy community rarely bothers with: it measures what actually leaks from a production-shaped federated synthetic-data pipeline, then compares that number to the worst-case guarantee on paper. The gap is large, and it matters for anyone shipping synthetic data under DP contracts.
The audit setup
Private Evolution (PE) is a federated synthetic-data mechanism where clients score a shared candidate bank of samples and the server evolves the bank toward higher-utility text. The audit swaps roughly 1% of that bank with probes derived from a known canary — a user-level insertion that lets the auditor recover a lower bound on user-level leakage without touching the server code. Two corpora, Yelp reviews and Sentiment140 tweets, ran under eight attack variants. Natural-text canaries stayed far under the theoretical DP bound, as expected — most private text looks like other private text. High-entropy nonce-based canaries pushed leakage closest to the mechanism’s ceiling, which is the number a red team would actually design.
That distinction — natural inputs behave, adversarial inputs push the envelope — mirrors what NIST’s guidance on privacy attacks against machine learning has been arguing since NIST AI 100-2 landed: reported ε rarely equals achieved ε.
What breaks in production
The audit is protocol-aware, meaning it exploits PE’s own scoring loop rather than treating the mechanism as a black box. That is the honest way to measure — ENISA’s data protection engineering guidance makes the same argument in less pointed language. A team relying on a paper-declared ε of 8 might be operating at an empirical lower bound closer to 2 on natural-language workloads, but an attacker with control over even a sliver of the bank can force the number up toward the stated ceiling.
Sample audit output looks roughly like this:
2026-09-15T04:22Z pe-audit run_id=yelp-v3
probes_injected=612 (1.02% of bank)
attack=nonce_high_entropy emp_epsilon_lb=6.71 theoretical=8.00
attack=nonce_low_entropy emp_epsilon_lb=4.18
attack=paraphrase_natural emp_epsilon_lb=0.94
attack=membership_baseline emp_epsilon_lb=0.31Teams building synthetic pipelines under GDPR Art. 5(1)(f) need both numbers on the deployment ticket: theoretical ε for compliance paperwork, empirical ε for the risk register. Reporting only one is malpractice. The ACM audit-DP literature has been converging on the same conclusion for two years now.
Where CAI Technology sits
We ship privacy-audited synthetic corpora for regulated clients, and our position is that ε on the contract is a starting point, not an SLA. Every deployment gets a canary bank sized to the client’s threat model and a re-run every quarter, with the attack surface documented alongside the model card. If your vendor cannot show you an empirical audit trace, ask for one — and if you want to see ours, the Janus pillar page has the current methodology on file.