NATASHA detector
natasha PII detection benchmark
Measured on 41 of 41 retained datasets under the same frozen scoring and masking protocol.
24.15%fully hidden annotations
28.03%detected / overlapped
1.07%extra masking
130,799CPU characters / second
Strengths
Highest complete-masking rates by normalized data type.
People's names49.43%32,495 / 65,736
Organizations40.19%7,746 / 19,274
Addresses & locations34.75%13,763 / 39,602
Watch list
Lowest observed complete-masking rates; inspect before deployment.
Dates & times0.04%2 / 5,502
Phone numbers & email0.07%15 / 21,981
Network identifiers0.08%20 / 25,631
Language coverage
The overall leader can differ from the best choice for one language slice.
| Language | Datasets | Fully hidden ↑ | Detected ↑ | Extra masking ↓ |
|---|---|---|---|---|
| English | 17 | 9.22% | 9.78% | 0.94% |
| Russian | 19 | 35.54% | 41.40% | 1.39% |
| Multilingual | 5 | 5.59% | 6.77% | 0.27% |
Configuration, source and version
- Family
- NATASHA
- Revision
- 1.6.0
- Eligible datasets
- 41 / 41
- Character F1
- 0.316
- Untouched diagnostic
- 71.97% · 163,696 / 227,466
- Benchmark version
- 1.0.3 · 2026-09-09
- CPU seconds / 10k chars
- 0.076
- GPU seconds / 10k chars
- Not measured
Results by dataset
Complete masking is primary; detection shows whether any character overlapped.
| Dataset | Gold annotations | Fully hidden ↑ | Detected ↑ | Extra masking ↓ |
|---|---|---|---|---|
| arthur-passwords | 280 | 0.00% | 0.00% | 1.03% |
| corrupt-secrets-issues | 288 | 0.00% | 2.43% | 0.81% |
| creddata | 776 | 0.00% | 0.52% | 0.69% |
| leaky-repo | 95 | 0.00% | 1.05% | 0.61% |
| secrets-rules | 746 | 0.00% | 0.00% | 1.87% |
| synth-env-configs | 392 | 0.00% | 0.26% | 1.44% |
| synth-secrets-en | 581 | 0.00% | 0.00% | 0.35% |
| synth-secrets-ru | 581 | 0.17% | 0.17% | 0.63% |
| secrets-issues | 288 | 0.69% | 1.04% | 0.62% |
| multiconer-ru | 1,208 | 0.75% | 0.83% | 0.02% |
| nym-multi | 8,169 | 2.83% | 3.32% | 0.00% |
| ameau01 | 2,242 | 3.93% | 3.97% | 0.44% |
| leak-museum | 101 | 3.96% | 3.96% | 0.55% |
| redact-multi | 22,864 | 4.14% | 5.30% | — |
| rubai-ru | 3,458 | 4.34% | 53.70% | — |
| gretel-multi | 4,484 | 4.77% | 10.64% | 0.27% |
| nemotron-pii | 9,391 | 7.32% | 7.70% | 0.00% |
| synth-ru-tickets | 32,099 | 8.09% | 10.57% | — |
| dialogpii-multi | 10,345 | 8.43% | 8.98% | — |
| privy | 1,899 | 8.85% | 9.37% | 3.93% |
| corrupt-hivetrace | 1,667 | 9.06% | 16.80% | 2.11% |
| nym-en | 3,584 | 9.26% | 10.30% | 0.18% |
| kiji-multi | 7,565 | 9.58% | 9.62% | — |
| dialogpii-en | 3,087 | 9.59% | 10.01% | 0.60% |
| synth-wiki-tables | 13,094 | 9.84% | 12.13% | — |
| tonicai | 2,417 | 11.25% | 12.99% | <0.01% |
| synth-jira-comments | 2,705 | 12.13% | 13.20% | — |
| kiji-en | 7,623 | 14.04% | 14.19% | — |
| hivetrace | 1,667 | 14.10% | 20.94% | 0.19% |
| tab-echr | 3,830 | 14.33% | 15.59% | — |
| corrupt-redmadrobot | 5,531 | 18.79% | 21.86% | 1.32% |
| alexen2 | 1,261 | 21.25% | 42.11% | 13.33% |
| russian-pii-66k | 4,805 | 23.56% | 30.11% | — |
| nym-ru | 9,497 | 27.46% | 33.62% | — |
| redmadrobot | 5,516 | 30.42% | 33.72% | 1.17% |
| redact-ru | 9,230 | 42.15% | 49.09% | 5.71% |
| jayguard | 1,195 | 44.35% | 61.09% | 2.36% |
| scanpatch | 8,708 | 49.31% | 54.87% | 1.55% |
| alrosait | 1,862 | 50.05% | 79.11% | 2.46% |
| nerel | 24,369 | 81.37% | 87.28% | — |
| factrueval | 7,966 | 94.39% | 95.61% | — |