SPACY detector
spacy-alrosait PII detection benchmark
Measured on 39 of 41 retained datasets under the same frozen scoring and masking protocol.
14.35%fully hidden annotations
16.85%detected / overlapped
0.12%extra masking
53,307CPU characters / second
Strengths
Highest complete-masking rates by normalized data type.
People's names34.03%20,743 / 60,951
Addresses & locations26.03%9,653 / 37,090
Organizations3.06%582 / 18,997
Watch list
Lowest observed complete-masking rates; inspect before deployment.
Dates & times0.00%0 / 4,852
Logins & usernames0.00%0 / 11,382
Passwords, keys & tokens0.03%2 / 6,870
Language coverage
The overall leader can differ from the best choice for one language slice.
| Language | Datasets | Fully hidden ↑ | Detected ↑ | Extra masking ↓ |
|---|---|---|---|---|
| English | 17 | 0.01% | 0.02% | <0.01% |
| Russian | 17 | 24.62% | 28.88% | 0.40% |
| Multilingual | 5 | 0.25% | 0.35% | 0.00% |
Configuration, source and version
- Family
- SPACY
- Revision
- 8e0d9e49ae62dcb0772f8d8599ed3ffe28d43d7d
- Eligible datasets
- 39 / 41
- Character F1
- 0.238
- Untouched diagnostic
- 83.15% · 180,353 / 216,896
- Benchmark version
- 1.0.3 · 2026-09-09
- CPU seconds / 10k chars
- 0.188
- GPU seconds / 10k chars
- Not measured
source overlap: scanpatch, alrosait
Results by dataset
Complete masking is primary; detection shows whether any character overlapped.
| Dataset | Gold annotations | Fully hidden ↑ | Detected ↑ | Extra masking ↓ |
|---|---|---|---|---|
| ameau01 | 2,242 | 0.00% | 0.00% | 0.00% |
| arthur-passwords | 280 | 0.00% | 0.00% | 0.04% |
| corrupt-secrets-issues | 288 | 0.00% | 0.00% | 0.02% |
| creddata | 776 | 0.00% | 0.00% | 0.00% |
| gretel-multi | 4,484 | 0.00% | 0.00% | 0.00% |
| kiji-en | 7,623 | 0.00% | 0.00% | — |
| leak-museum | 101 | 0.00% | 0.00% | 0.00% |
| leaky-repo | 95 | 0.00% | 0.00% | 0.00% |
| nemotron-pii | 9,391 | 0.00% | 0.01% | 0.00% |
| nym-en | 3,584 | 0.00% | 0.00% | 0.00% |
| privy | 1,899 | 0.00% | 0.00% | 0.00% |
| secrets-issues | 288 | 0.00% | 0.00% | 0.00% |
| secrets-rules | 746 | 0.00% | 0.00% | 0.00% |
| synth-env-configs | 392 | 0.00% | 0.00% | 0.00% |
| synth-secrets-en | 581 | 0.00% | 0.00% | 0.00% |
| synth-secrets-ru | 581 | 0.00% | 0.00% | 0.00% |
| tab-echr | 3,830 | 0.00% | 0.03% | — |
| redact-multi | 22,864 | 0.01% | 0.10% | — |
| tonicai | 2,417 | 0.04% | 0.04% | 0.00% |
| dialogpii-multi | 10,345 | 0.06% | 0.14% | — |
| kiji-multi | 7,565 | 0.11% | 0.11% | — |
| dialogpii-en | 3,087 | 0.13% | 0.13% | 0.00% |
| multiconer-ru | 1,208 | 0.17% | 0.66% | 0.09% |
| nym-multi | 8,169 | 1.44% | 1.73% | 0.00% |
| rubai-ru | 3,458 | 2.66% | 50.43% | — |
| corrupt-hivetrace | 1,667 | 6.78% | 12.42% | 0.22% |
| synth-ru-tickets | 32,099 | 6.88% | 8.80% | — |
| corrupt-redmadrobot | 5,531 | 10.05% | 11.19% | 0.18% |
| synth-wiki-tables | 13,094 | 11.75% | 12.33% | — |
| synth-jira-comments | 2,705 | 14.12% | 14.16% | — |
| alexen2 | 1,261 | 18.32% | 38.22% | 11.78% |
| hivetrace | 1,667 | 21.78% | 22.56% | 0.16% |
| redmadrobot | 5,516 | 24.38% | 26.54% | 0.30% |
| nym-ru | 9,497 | 31.10% | 34.04% | — |
| russian-pii-66k | 4,805 | 36.13% | 37.02% | — |
| redact-ru | 9,230 | 42.06% | 45.30% | 4.35% |
| nerel | 24,369 | 44.89% | 51.44% | — |
| jayguard | 1,195 | 48.20% | 59.75% | 1.43% |
| factrueval | 7,966 | 50.95% | 52.54% | — |