Methodology
A transparent scoring contract. Every number has a denominator.
A benchmark you can inspect.
Frozen inputs. Declared masking rules. Exact counts.
A descriptive comparison under one protocol.
01 · The inputs
41 retained dataset cuts cover Russian, English and multilingual PII and secrets. Rows and normalized gold annotations retain their source identity.
02 · The mask
Predicted offsets are normalized to word boundaries. Detection requires any character overlap; full hiding requires every annotated character to be covered.
03 · The comparison
Known training-source overlaps are excluded. Complete coverage is the default. Pairwise comparisons use the intersection of eligible datasets.
04 · The limits
Unannotated rows are not verified clean. Compositions were inspected on this benchmark. Observed percentages are not production leak probabilities.
How to read the metrics
Untouched
Sensitive items with no detector overlap. Lower is better. This is not a production leak rate.
Fully hidden
Sensitive items whose whole normalized value is covered by the mask. This is the main ranking metric.
Sensitive items with any detector overlap. Part of the value may still remain visible.
Extra masking
Text masked in rows with no labels. These rows were not manually verified clean, so this is not a false-positive rate.
Character F1
Character-level balance of precision and recall. It measures masked characters, not entity types.
CPU throughput
Seconds per 10,000 characters on the fixed CPU workload. This measures batch throughput, not request latency.
GPU throughput
Seconds per 10,000 characters on the fixed GPU workload. This measures batch throughput, not request latency.
Reproduce and extend
The public verification regenerates publication artifacts; it does not rerun inference.
uv sync --frozen uv run python scripts/verify.py uv run python scripts/render.py uv run python scripts/verify.py
496f1ccd1bf354c972e001c4ed78aed22fcae1e7df6c9fc7e65859f585e27198