← All days

Day 7 · The full picture: a voice-privacy benchmark

Goal in one line: turn everything from Days 1 to 6 into one honest evaluation: how private is each anonymization system, what does it cost in content, naturalness and prosody, and how sure are we about each number.

What you will build: a mini leaderboard generator (make_leaderboard) that recomputes the attacker EERs (from shipped ECAPA embeddings) and the CER (from shipped transcripts) for the 8 systems that have shipped files, and the pitch correlation ρF0 on the 20 demo clips, with speaker-bootstrap confidence intervals, writes outputs/day7/leaderboard_recomputed.csv, and draws the privacy-utility Pareto plot.

Time: ~3.5 h

By the end you can

• explain the VoicePrivacy protocol: enrollment vs trial, and the three attacker knowledge levels (ignorant, lazy-informed, semi-informed);
• recompute the EERs of the 8 leaderboard systems that have shipped ECAPA embeddings and say why the numbers differ by attacker;
• put a 95 % confidence interval on an EER or a CER with a speaker bootstrap, and a paired CI on a difference between two systems;
• read a privacy-utility scatter plot and a forest plot and say which systems are actually distinguishable;
• spot the shortcuts an attacker can take (gender, recording channel) and the "two EERs trap";
• design a small blind listening test for naturalness and speaker similarity;
• defend every number in the project's one-page summary in front of an advisor.

How to read this notebook (for the team)

You see	What it is	Do you need it?
Purple heading	a new section and its explanation	yes: read the bold lines and the "Why it matters" line
Visualization Version	pictures (one is animated) that explain the section	yes: the first image of each section is the fastest way in
Additional reading	what the next code cell does and what its output means, plus extra notes	optional: read just the In short lines if you are busy
code cell + output	the real computation on the 559-clip test set, the 20 demo clips and 8 systems	run it, look at the result

Fast path (≈ 45 min): purple text → first Visualization image → "In short" lines. Full path (≈ 3.5 h): read everything and run every cell top to bottom.

The numbers in the explanations come from the run saved in this file. Seeds are fixed, so a re-run on the same setup should give the same numbers. Timings (seconds) change from machine to machine and are not quoted.

Important, what is fresh and what is shipped. This notebook has no model download step, so nothing fails on a machine without internet. But it does not run the big models either:

• The EERs of the 8 systems are computed from shipped ECAPA-TDNN embeddings (data/embeddings/ and data/embeddings_semi_informed/), not from a fresh ECAPA run. The Day 4 numbers for original and mcadams come from the same files.
• The CERs of the 7 systems with transcripts are computed from the shipped ASR transcripts (data/meta/transcripts.csv): the judge was not run again here.
• UTMOS is read from leaderboard.csv. The optional recompute needs a ~400 MB download and is switched off (RUN_UTMOS = False). The recompute code is the course's and was not run here.
• Fresh here: the speaker bootstraps, the CER bootstrap, ρF0 and the duration ratio (pitch tracking on the shipped audio of the 20 demo clips), the paired tests, and the listening-test sheets.

Every code cell ran here, top to bottom, with no errors.

The big idea

An anonymization system is judged by two kinds of people. An attacker tries to find out who spoke. A user (or a downstream model) wants to keep what was said, how natural it sounds, and how it was said (prosody). A system is good only if it wins against the attacker and keeps the content. Both halves are measured with the models you built this week.

                         anonymization systems
         original  mcadams  knnvc_en  knnvc_km  asr_tts  sslsas  private_knnvc
             |        |        |         |         |        |         |
             v        v        v         v         v        v         v
   +---------------------------------------------------------------------+
   |  PRIVACY   attacker ECAPA (Day 4)   -> EER  (higher = more private) |
   |            x 3 attacker levels: ignorant / lazy-informed / semi-inf.|
   |  CONTENT   ASR judge (Day 3)        -> CER  (lower = better)        |
   |  QUALITY   UTMOS predictor          -> MOS 1..5 (higher = better)   |
   |  PROSODY   pitch tracker (Day 1)    -> rho_F0 (how much F0 leaks)   |
   +---------------------------------------------------------------------+
             |                                   |
             v                                   v
     each number +- 95 % CI           privacy vs utility plot, Pareto front
     (speaker bootstrap)              paired tests between two systems

The VoicePrivacy protocol (used by the VoicePrivacy Challenge since 2020 (background fact from the course text, not checked here)) fixes the rules. Each test speaker has a few enrollment utterances (the attacker's reference: "this is what speaker 17 sounds like") and several trial utterances (what the attacker intercepts). The system anonymizes the trials. The attacker compares each trial with each enrolled speaker and says "same" or "different". A good anonymizer makes this comparison as hard as guessing: EER near 50 %.

The twist is what the attacker knows. An attacker who also has the anonymization system can anonymize the enrollment speech too, or even retrain the speaker model on anonymized speech. In our table the EER usually gets lower as the attacker gets smarter, but not always (for original, the semi-informed EER is higher than the lazy one; for three systems the lazy EER is higher than the ignorant one, see section 1). So a benchmark must report which attacker produced each EER.

The privacy-utility trade-off. Destroying the voice completely is easy (replace it with white noise: EER about 50 %, CER about 100 %; a course example, not run here). Keeping the content completely is easy (do nothing: CER 7.6 %, EER 2.4 % for original, from shipped transcripts and embeddings). The interesting systems sit in between, and no single number ranks them. You need both axes and a confidence interval on each.

In our research: this notebook is the project's evaluation. The leaderboard in data/meta/leaderboard.csv was produced, the course says, by exactly the pipeline you rebuild here. After today you can explain every column, reproduce the EER and CER columns from shipped embeddings and transcripts (UTMOS is read from leaderboard.csv), and say which differences between systems are real.

Why it matters: one number can hide a lot. The next sections show four ways a single number misleads: the attacker level, the trial list, the interval, and the missing utility.

Visualization Version

Eight systems, four judges, and four real numbers for four of the systems.

1. The attacker ladder

The attacker always has the same tool: an ECAPA-TDNN speaker encoder (Day 4) that maps an utterance to a 192-d unit vector, and a cosine score between a trial and a speaker model (the mean of the enrollment vectors). What changes is what the attacker feeds it.

attacker	enrollment side	trial side	speaker encoder	what it models
ignorant	original speech	anonymized	trained on normal speech	an attacker who does not know anonymization was applied
lazy-informed	anonymized with the same system	anonymized	trained on normal speech	an attacker who owns the system and runs it on the enrollment speech too, so both sides live in the same "anonymized voice space"
semi-informed	anonymized	anonymized	fine-tuned on anonymized speech	an attacker who also retrains the encoder to undo the system's regularities (the VoicePrivacy 2022/2024 standard (background fact from the course text, not checked here))

Worked example (real numbers, same-gender trials). SSL-SAS maps every speaker to one fixed pseudo-voice (the shipped description in leaderboard.csv says "one fixed pseudo-speaker per speaker"). The ignorant attacker compares that pseudo-voice with the real voice and fails: EER 50.7 %.

The lazy attacker enrolls the pseudo-voice itself: 12.8 %. The semi-informed attacker does a little better still: 9.8 % (9.77).

The course explains this by the fixed pseudo-voice: all of one speaker's utterances sound the same, so they are easy to link (course explanation, not tested here). The ignorant number measured the attacker's confusion, not privacy.

Why EER drops as the attacker learns (course explanation, not tested here). The anonymizer applies a deterministic (or nearly deterministic) map $f$. The ignorant attacker compares $e(x)$ with $e(f(x'))$: a mismatch between two spaces. The lazy attacker compares $e(f(x))$ with $e(f(x'))$: whatever speaker information survives $f$ is now on both sides. The semi-informed attacker replaces $e$ by $e'$ trained to be sensitive to exactly that surviving information.

$$\mathrm{EER} = \text{the error rate at the threshold where } \mathrm{FAR}(\theta) = \mathrm{FRR}(\theta)$$

It is not a law. In our table the ignorant EER is the highest of its row for 4 of the 7 anonymizers (McAdams, kNN-VC km same-gender, SSL-SAS, Private kNN-VC). For the other three (kNN-VC en, kNN-VC km, ASR→TTS) the lazy EER is higher. An ignorant EER alone is therefore often, but not always, an upper bound.

Why it matters: a paper that reports only the ignorant EER may report a number that the owner of the system can beat. The course's advice is to quote the semi-informed EER as the headline number.

Below we recompute all three from the shipped embeddings.

• data/embeddings/<system>.npz holds the ordinary ECAPA embedding of every evaluation clip after that system.
• data/embeddings_semi_informed/<system>.npz holds the embedding from an ECAPA fine-tuned on that system's anonymized speech.

Because the anonymized .npz files include the 129 enrollment utterances (checked: each anonymized file holds 579 clips, which include all 559 test clips), the lazy-informed attacker can be recomputed as well. All of this uses shipped embeddings, no fresh ECAPA run.

Visualization Version

The picture shows the three attackers, how four systems fall along the ladder, and one bootstrap histogram. The animation grows that histogram.

Animated: the speaker bootstrap draws 1 000 resamples of the 43 speakers; the real EERs of kNN-VC km (ignorant) pile up and the 95 % interval settles at [41.4, 46.5]

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short load the libraries and the metadata, list the 8 systems, and print the size of the test set.

• from src.utils import ...: the Day 1 helpers (eer, cer, median_f0, load, ...).
• utts = 559 test clips (129 enrollment + 430 trial, 43 speakers), trials = every (enrolled speaker, trial clip) pair, board = the shipped leaderboard, paired_ref = the shipped paired comparisons.
• SYSTEMS lists 8 systems: the reference original and 7 anonymizers (the code comment says "the six systems you listened to on Day 6, plus the reference and one same-gender variant").

What the output means

In short {'trial': 430, 'enroll': 129} | speakers: 43; trials: 18490 | same-gender: 10090 | targets: 430; leaderboard systems: 20; 8 shipped embedding files.

• 43 speakers × 3 enrollment clips = 129; 43 × 10 trial clips = 430; 43 × 430 = 18 490 trials. The 430 targets are the trials where the enrolled speaker is the speaker of the clip.
• The leaderboard has 20 systems; we recompute 8 of them (the ones with shipped embeddings).

%matplotlib inline
import os, sys, json, time, warnings
os.environ.setdefault("OMP_NUM_THREADS", "4")   # before numpy/torch: keep CPU use predictable
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from pathlib import Path
warnings.filterwarnings("ignore")
np.random.seed(0)
sys.path.insert(0, "..")
from src.utils import play, show_wave, show_spec, median_f0, eer, cer, load as load_audio

SR = 16000
DATA = Path("../data")
OUT = Path("../outputs/day7"); OUT.mkdir(parents=True, exist_ok=True)
T0 = time.time()

# --- metadata: who spoke, what role, which trials the attacker scores ---------------------------
utts   = pd.read_csv(DATA / "meta/test_utts.csv")          # 559 utts: 129 enroll + 430 trial, 43 speakers
trials = pd.read_csv(DATA / "meta/trials_test.csv")        # 18 490 (enroll_speaker, trial_utt) pairs
board  = pd.read_csv(DATA / "meta/leaderboard.csv").set_index("system")
paired_ref = json.load(open(DATA / "meta/paired_comparisons.json"))

# the six systems you listened to on Day 6, plus the reference and one same-gender variant
SYSTEMS = ["original", "mcadams", "knnvc_english_pool", "knnvc_khmer_pool",
           "knnvc_khmer_pool_samegender", "asr_tts", "sslsas", "private_knnvc_english_durations"]
SHORT = {"original": "original", "mcadams": "McAdams", "knnvc_english_pool": "kNN-VC en",
         "knnvc_khmer_pool": "kNN-VC km", "knnvc_khmer_pool_samegender": "kNN-VC km same-g",
         "asr_tts": "ASR->TTS", "sslsas": "SSL-SAS", "private_knnvc_english_durations": "Private kNN-VC"}

print(utts.role.value_counts().to_dict(), "| speakers:", utts.speaker.nunique())
print("trials:", len(trials), "| same-gender:", int(trials.same_gender.sum()),
      "| targets:", int((trials.label == "target").sum()))
print("leaderboard systems:", len(board), "| shipped embeddings:", sorted(p.stem for p in (DATA/"embeddings").glob("*.npz")))

{'trial': 430, 'enroll': 129} | speakers: 43
trials: 18490 | same-gender: 10090 | targets: 430
leaderboard systems: 20 | shipped embeddings: ['asr_tts', 'knnvc_english_pool', 'knnvc_khmer_pool', 'knnvc_khmer_pool_samegender', 'mcadams', 'original', 'private_knnvc_english_durations', 'sslsas']

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short define how to load embeddings, build speaker models and score trials; compute the EER of 8 systems under 3 attackers on the 10 090 same-gender trials; print it next to the leaderboard.

• speaker_models: each speaker model is the mean of 3 enrollment vectors, scaled to length 1. score_trials: the cosine of each trial.
• ignorant: enrollment from original, trials from the system. lazy: both sides from the system. semi_informed: both sides from the fine-tuned encoder's files (only if that file exists).
• eer(...) is the Day 1 helper. The table shows our value and the shipped board_* value.
• These are shipped ECAPA embeddings; the cell never runs ECAPA.

What the output means

In short a table with 8 rows × 3 attackers. For example original 2.36 / 2.36 / 3.02, mcadams 29.58 / 25.31 / 14.18, sslsas 50.70 / 12.80 / 9.77. asr_tts has NaN for semi-informed.

• The recomputed values are within 0.05 points of the shipped columns (for example mcadams ignorant 29.58 vs 29.63; original ignorant 2.36 vs 2.40). knnvc_khmer_pool_samegender and asr_tts (lazy) match exactly.
• asr_tts has no semi-informed value because no fine-tuned encoder was shipped for it; the leaderboard has the same gap.
• Same numbers as Day 4 (shipped embeddings): original 2.36 / 2.36 / 3.02 and mcadams 29.58 / 25.31 / 14.18 are the Day 4 section 7 values.

def load_emb(folder, system):
    # .npz with two arrays: utt_id (str) and emb (n, 192), already L2-normalized -> dict utt_id -> vector
    z = np.load(DATA / folder / f"{system}.npz")
    return dict(zip(z["utt_id"].tolist(), z["emb"].astype(np.float32)))

enroll_utts = utts[utts.role == "enroll"].groupby("speaker").utt_id.apply(list).to_dict()   # 3 per speaker

def speaker_models(E):
    # one speaker model = mean of its enrollment vectors, renormalized to unit length
    M = {}
    for spk, ids in enroll_utts.items():
        v = np.mean([E[u] for u in ids], axis=0)
        M[spk] = v / np.linalg.norm(v)
    return M

def score_trials(E_enroll, E_trial, tr=trials):
    # cosine score of every trial: higher = attacker thinks "same speaker"
    M = speaker_models(E_enroll)
    enr = np.stack([M[s] for s in tr.enroll_speaker])
    tri = np.stack([E_trial[u] for u in tr.trial_utt])
    return np.einsum("ij,ij->i", enr, tri)

E_orig = load_emb("embeddings", "original")
labels_all = (trials.label == "target").astype(int).values
sg = trials.same_gender.values            # the honest trial list: 10 090 same-gender trials

# scores[attacker][system] -> one score per row of `trials` (all 18 490 rows; we mask later)
scores = {"ignorant": {}, "lazy": {}, "semi_informed": {}}
for s in SYSTEMS:
    E = load_emb("embeddings", s)
    scores["ignorant"][s] = score_trials(E_orig, E)        # original enrollment vs anonymized trials
    scores["lazy"][s]     = score_trials(E, E)             # both sides anonymized, same encoder
    if (DATA / "embeddings_semi_informed" / f"{s}.npz").exists():
        S = load_emb("embeddings_semi_informed", s)
        scores["semi_informed"][s] = score_trials(S, S)    # both sides anonymized, fine-tuned encoder

rows = []
for s in SYSTEMS:
    r = {"system": s}
    for a in scores:
        r[a] = eer(scores[a][s][sg], labels_all[sg])[0] if s in scores[a] else np.nan
    r["board_ignorant"] = board.loc[s, "eer_ignorant"]; r["board_lazy"] = board.loc[s, "eer_lazy"]; r["board_semi"] = board.loc[s, "eer_semi_informed"]
    rows.append(r)
eer_point = pd.DataFrame(rows).set_index("system").round(2)
print("EER (%) on same-gender trials, recomputed vs leaderboard.csv (board_*):")
eer_point

EER (%) on same-gender trials, recomputed vs leaderboard.csv (board_*):

	ignorant	lazy	semi_informed	board_ignorant	board_lazy	board_semi
system
original	2.36	2.36	3.02	2.40	2.40	3.02
mcadams	29.58	25.31	14.18	29.63	25.27	14.19
knnvc_english_pool	40.66	43.69	37.93	40.62	43.65	37.96
knnvc_khmer_pool	43.96	46.74	43.49	43.98	46.74	43.49
knnvc_khmer_pool_samegender	40.70	33.95	31.40	40.70	33.95	31.40
asr_tts	45.33	49.99	NaN	45.31	49.99	NaN
sslsas	50.70	12.80	9.77	50.70	12.82	9.77
private_knnvc_english_durations	48.13	36.78	36.51	48.13	36.82	36.51

Additional reading: you can read all of it or just read the "In short"

Reading the table

In short the recomputed EERs agree with the shipped leaderboard within 0.05 points, and with Day 4.

The recomputed columns match the leaderboard to within 0.05 points (the exact maximum is printed in a later cell: 0.05 for the ignorant column, 0.04 lazy, 0.03 semi-informed). The course says the leaderboard was written from these exact embeddings; the last digit still differs. Day 4 gives a likely reason: the leaderboard interpolates the FAR/FRR crossing, while utils.eer takes the nearest observed score (not tested here).

The semi-informed column is empty for asr_tts because no fine-tuned encoder was shipped for that system.

Same attacker, Day 4 ECAPA numbers. Day 4 printed original 1.84 / 2.36 / 3.02 and mcadams 23.95 / 29.58 / 31.16 for three trial lists (all / same-gender / same-gender and same-corpus), also from shipped embeddings. The middle values are the ignorant column here (2.36 and 29.58); section 2 reproduces all of them.

Additional reading: you can read all of it or just read the "In short"

Confidence intervals: the speaker bootstrap

In short resample the 43 speakers, not the trials; each EER gets a 95 % interval from 1 000 resamples.

An EER is a number computed from 10 090 trials, but those trials come from only 43 speakers. Trials sharing a speaker are not independent. The course says that a "resample the trials" bootstrap would give an interval that is too narrow, so the right unit to resample is the speaker.

We checked this once (kNN-VC km, ignorant attacker, 300 trial resamples, seed 0) (computed separately, not printed by the cells): the trial bootstrap was not narrower than the one-sided speaker bootstrap (5.5 vs 5.1 points wide). So treat "too narrow" as a general warning, not as something shown for this one system. We use the speaker bootstrap, as the leaderboard does.

Procedure (1 000 repetitions, seed 0): draw 43 speakers with replacement. Every trial "belongs" to the speaker it tests, the enrolled speaker (target trials are that speaker's own utterances, non-target trials are impostors against that speaker's model). A speaker drawn $k$ times gives all of its trials weight $w = k$ (0 = left out). Compute the weighted EER. The 2.5 % and 97.5 % percentiles of the 1 000 EERs are the 95 % CI.

A stricter two-sided variant also multiplies by the draw count of the trial speaker, $w = k_i k_j$. For kNN-VC km (ignorant) it gave an interval 8.1 points wide against 5.1 for the one-sided version, about 60 % wider (the course said about 40 %) (computed separately, not printed by the cells). The leaderboard used the one-sided version, and so do we.

Weighted EER is the same formula with weighted counts:

$$\mathrm{FAR}(\theta)=\frac{\sum_{\text{non-target}} w\,[s>\theta]}{\sum_{\text{non-target}} w},\qquad \mathrm{FRR}(\theta)=\frac{\sum_{\text{target}} w\,[s\le\theta]}{\sum_{\text{target}} w}$$

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short define the weighted EER and the speaker bootstrap, draw the 1 000 × 43 speaker counts once (DRAWS), and compute the EER and its 95 % interval for every (attacker, system) pair.

• speaker_draws: 1 000 rows, each row = how many times each of the 43 speakers was drawn (seed 0). DRAWS is shared by every system, so later paired comparisons are free.
• eer_boot: for each row, weight each trial by the draw count of its enrolled speaker, compute the weighted EER, then take the 2.5th and 97.5th percentiles.
• The table also shows the shipped CI of the leaderboard (ignorant and lazy only).

What the output means

In short 23 bootstraps × 1000 reps (24 pairs minus the missing asr_tts semi-informed; EERs from shipped embeddings). Example: original 2.4 [1.4, 3.5], knnvc_khmer_pool ignorant 44.0 [41.4, 46.5], sslsas ignorant 50.7 [46.1, 55.5].

• The interval is about ±1 point for original and about ±2 to ±5 for most other rows. The widest is knnvc_khmer_pool_samegender ignorant: [36.0, 46.8], about 11 points wide.
• Our intervals are close to the shipped ones but not equal, for example mcadams ignorant [26.3, 33.0] here vs [26.51, 32.99] shipped (we use 1 000 reps; the course says the shipped ones came from a longer run, which we did not check).
• The time is printed by the cell and changes from machine to machine.

def eer_weighted(scores, labels, w):
    # same sliding-threshold EER as src.utils.eer, but each trial carries a weight (0 = dropped)
    order = np.argsort(-scores)
    l, w = labels[order].astype(float), w[order].astype(float)
    tp, fp = np.cumsum(w * l), np.cumsum(w * (1 - l))
    far = fp / max(fp[-1], 1e-9)
    frr = 1 - tp / max(tp[-1], 1e-9)
    i = np.argmin(np.abs(far - frr))
    return 100 * (far[i] + frr[i]) / 2

SPEAKERS = sorted(utts.speaker.unique())
spk_idx = {s: i for i, s in enumerate(SPEAKERS)}
enr_i = trials.enroll_speaker.map(spk_idx).values
tri_i = trials.trial_speaker.map(spk_idx).values
N_BOOT = 1000

def speaker_draws(n_boot=N_BOOT, seed=0):
    # one row per bootstrap rep: how many times each of the 43 speakers was drawn
    rng = np.random.default_rng(seed)
    return np.stack([np.bincount(rng.integers(0, len(SPEAKERS), len(SPEAKERS)), minlength=len(SPEAKERS)) for _ in range(n_boot)])

DRAWS = speaker_draws()          # shared by every system -> paired comparisons come for free later

def eer_boot(sc, mask=sg, draws=DRAWS):
    # returns (point EER, CI low, CI high, all bootstrap EERs)
    s, l = sc[mask], labels_all[mask]
    e = enr_i[mask]
    boots = np.array([eer_weighted(s, l, k[e]) for k in draws])     # weight = how often the enrolled speaker was drawn
    return eer(s, l)[0], np.percentile(boots, 2.5), np.percentile(boots, 97.5), boots

t0 = time.time()
eer_tab, eer_boots = [], {}
for a in scores:
    for s in SYSTEMS:
        if s not in scores[a]:
            continue
        pt, lo, hi, b = eer_boot(scores[a][s])
        eer_boots[(a, s)] = b
        eer_tab.append({"system": s, "attacker": a, "eer": pt, "ci_low": lo, "ci_high": hi})
eer_tab = pd.DataFrame(eer_tab)
print(f"{len(eer_tab)} bootstraps x {N_BOOT} reps in {time.time()-t0:.1f} s")

# side by side with the shipped CI (the leaderboard ships CIs for ignorant and lazy only)
cmp = eer_tab.pivot(index="system", columns="attacker", values=["eer", "ci_low", "ci_high"]).round(1)
cmp[("board", "ignorant")] = [f"{board.loc[s,'eer_ignorant']} [{board.loc[s,'eer_ignorant_ci_low']}, {board.loc[s,'eer_ignorant_ci_high']}]" for s in cmp.index]
cmp[("board", "lazy")] = [f"{board.loc[s,'eer_lazy']} [{board.loc[s,'eer_lazy_ci_low']}, {board.loc[s,'eer_lazy_ci_high']}]" for s in cmp.index]
cmp.loc[SYSTEMS]

23 bootstraps x 1000 reps in 7.9 s

	eer			ci_low			ci_high			board
attacker	ignorant	lazy	semi_informed	ignorant	lazy	semi_informed	ignorant	lazy	semi_informed	ignorant	lazy
system
original	2.4	2.4	3.0	1.4	1.4	1.7	3.5	3.5	3.9	2.4 [1.4, 3.49]	2.4 [1.4, 3.49]
mcadams	29.6	25.3	14.2	26.3	21.6	10.2	33.0	28.3	16.6	29.63 [26.51, 32.99]	25.27 [21.63, 28.06]
knnvc_english_pool	40.7	43.7	37.9	38.4	41.4	35.4	43.5	46.0	40.9	40.62 [37.91, 44.04]	43.65 [41.0, 46.51]
knnvc_khmer_pool	44.0	46.7	43.5	41.4	44.3	40.3	46.5	49.1	46.0	43.98 [40.7, 47.67]	46.74 [44.14, 49.35]
knnvc_khmer_pool_samegender	40.7	34.0	31.4	36.0	31.2	28.7	46.8	36.3	33.7	40.7 [37.67, 44.88]	33.95 [31.06, 36.74]
asr_tts	45.3	50.0	NaN	42.4	47.5	NaN	47.7	52.5	NaN	45.31 [41.14, 49.72]	49.99 [47.36, 52.35]
sslsas	50.7	12.8	9.8	46.1	11.1	7.7	55.5	14.7	12.0	50.7 [44.96, 55.35]	12.82 [11.05, 14.87]
private_knnvc_english_durations	48.1	36.8	36.5	44.2	34.1	33.5	51.8	39.5	39.5	48.13 [44.19, 51.63]	36.82 [33.91, 39.77]

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short draw the EER of the 8 systems as bars, three bars (one per attacker) per system, with the 95 % interval as whiskers.

• Bar height = EER (from the shipped ECAPA embeddings); whisker = the 95 % speaker-bootstrap interval; the dashed line = 50 % (coin flip).
• The asr_tts semi-informed bar is missing (NaN).

What the output means

In short one bar chart, 8 systems × 3 attackers.

• Read it with the picture above: the same numbers, now with whiskers.

fig, ax = plt.subplots(figsize=(12, 4))
attackers = ["ignorant", "lazy", "semi_informed"]
x = np.arange(len(SYSTEMS)); wbar = 0.26
for k, a in enumerate(attackers):
    sub = eer_tab[eer_tab.attacker == a].set_index("system").reindex(SYSTEMS)
    ax.bar(x + (k - 1) * wbar, sub.eer, wbar, label=a,
           yerr=[sub.eer - sub.ci_low, sub.ci_high - sub.eer], capsize=3)
ax.axhline(50, ls="--", c="gray", lw=1); ax.text(len(SYSTEMS) - 0.5, 51, "50 % = coin flip", ha="right", fontsize=9)
ax.set(xticks=x, xticklabels=[SHORT[s] for s in SYSTEMS], ylabel="EER (%)  (higher = more private)",
       title="Attacker EER per system, same-gender trials, 95 % speaker-bootstrap CI")
ax.legend(); plt.tight_layout(); plt.show()

Additional Reading

How to read this plot

In short bars (EERs from shipped embeddings): higher = more private; whiskers: the 95 % interval. SSL-SAS: the ignorant bar sits on the 50 % line, the other two fall to 10-13 %. kNN-VC km stays at 43-47 % for all three attackers.

Three bars per system, one per attacker; the whisker is the 95 % CI.

original: all three bars sit near 2-3 % (2.4, 2.4, 3.0). The attacker is nearly perfect. That is the floor.

SSL-SAS: the left (ignorant) bar is at 50.7 %, on the coin-flip line. The middle and right bars fall to 12.8 and 9.8 %. The height of the first bar was never privacy.

kNN-VC km: all three bars stay around 43-47 % (44.0, 46.7, 43.5), so a smarter attacker gains almost nothing. That is what a robust system looks like in this table. McAdams sits in the middle: 29.6 → 25.3 → 14.2 %, the attacker learns a lot.

Use the whiskers. kNN-VC en (40.7, interval [38.4, 43.5]) and kNN-VC km (44.0, [41.4, 46.5]) overlap for the ignorant attacker. If the intervals overlap, do not call a winner from this plot; wait for the paired test in section 5.

The Private kNN-VC bars look good (48.1 % ignorant, 36.5 % semi-informed), but hold that thought until you see its CER in section 3.

2. Shortcuts the attacker can use

An attacker says "same speaker" or "different speaker". Anything that separates speakers helps, even if it is not the voice. Two shortcuts live in our data. (Every EER in this section comes from the shipped ECAPA embeddings, not a fresh ECAPA run.)

Gender. An anonymizer that keeps gender (the course names McAdams, same-gender kNN-VC and every system that keeps F0) lets the attacker reject many non-target trials for free: a male trial against a female enrollment is obviously "different". In our trial list 8 400 of the 18 060 non-target trials (46.5 %) are cross-gender (computed separately, not printed by the cells). Mixing them in usually lowers the EER with no real speaker information. The honest trial list contains same-gender trials only (10 090 of the 18 490), which is what the leaderboard reports.

Corpus / channel. Our 43 speakers come from two recordings: 32 phone speakers (iAny) and 11 studio speakers (OpenSLR 42, all female (computed separately, not printed by the cells)). A phone trial against a studio enrollment can be rejected from the channel alone (bandwidth, noise, microphone). The 3 740 same-gender cross-corpus trials (all non-target (computed separately, not printed by the cells)) are "easy rejects". Dropping them tells you how the attacker does when the channel gives no hint.

Worked example. For McAdams the ignorant EER is 23.95 % on all trials (the cell rounds it to 24.0) but 29.58 % (29.6) on same-gender trials: 5.63 points of apparent "attacker skill" were gender (computed separately, not printed by the cells). Day 4 printed the same values (original 1.84 / 2.36 / 3.02, McAdams 23.95 / 29.58 / 31.16) from the same shipped embeddings; the cell here shows them with one decimal.

Why it matters: when two papers report EERs on different trial lists, the numbers are not comparable. Always say which trials.

Visualization Version

The three trial lists, the real EER on each list for five systems, and the size of the gender step.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short build the corpus flag for every trial, define 5 trial lists (all, same-gender, same-gender and same-corpus, phone only, studio only), and compute the ignorant-attacker EER of each system on each list.

• same_corpus is true when the enrolled speaker and the trial speaker come from the same recording corpus.
• The EER uses the ignorant scores of section 1; the table is rounded to one decimal.
• The first line prints how many trials each list has.

What the output means

In short EERs from shipped embeddings: list sizes 18490 / 10090 / 6350 / 5140 / 1210. McAdams: 24.0 / 29.6 / 31.2 / 30.0 / 31.8. SSL-SAS: 41.9 / 50.7 / 52.8 / 52.2 / 48.2.

• 18 490 → 10 090 (same-gender) → 6 350 (also same-corpus). 5 140 + 1 210 = 6 350: phone-only and studio-only.
• original on the first three lists: 1.8 / 2.4 / 3.0, the Day 4 values. mcadams: 24.0 / 29.6 / 31.2, the Day 4 values (23.95 / 29.58 / 31.16).
• The numbers are rounded, so differences you compute from the table can be off by 0.1.

spk_corpus = utts.drop_duplicates("speaker").set_index("speaker").corpus
same_corpus = (trials.enroll_speaker.map(spk_corpus) == trials.trial_speaker.map(spk_corpus)).values
masks = {"all trials": np.ones(len(trials), bool),
         "same-gender (honest)": sg,
         "same-gender & same-corpus": sg & same_corpus,
         "same-gender & phone only": sg & same_corpus & (trials.enroll_speaker.map(spk_corpus) == "phone (iAny)").values,
         "same-gender & studio only": sg & same_corpus & (trials.enroll_speaker.map(spk_corpus) == "studio (OpenSLR 42)").values}
rows = []
for s in SYSTEMS:
    r = {"system": SHORT[s]}
    for name, m in masks.items():
        r[name] = eer(scores["ignorant"][s][m], labels_all[m])[0]
    rows.append(r)
short_tab = pd.DataFrame(rows).set_index("system").round(1)
print("ignorant-attacker EER (%) by trial list;  n trials:", {k: int(v.sum()) for k, v in masks.items()})
short_tab

ignorant-attacker EER (%) by trial list;  n trials: {'all trials': 18490, 'same-gender (honest)': 10090, 'same-gender & same-corpus': 6350, 'same-gender & phone only': 5140, 'same-gender & studio only': 1210}

	all trials	same-gender (honest)	same-gender & same-corpus	same-gender & phone only	same-gender & studio only
system
original	1.8	2.4	3.0	3.1	1.6
McAdams	24.0	29.6	31.2	30.0	31.8
kNN-VC en	40.2	40.7	42.1	41.2	45.5
kNN-VC km	44.2	44.0	45.1	45.9	46.2
kNN-VC km same-g	34.6	40.7	42.6	39.3	39.2
ASR->TTS	48.6	45.3	48.4	47.8	48.9
SSL-SAS	41.9	50.7	52.8	52.2	48.2
Private kNN-VC	44.2	48.1	50.9	47.9	47.4

Additional reading: you can read all of it or just read the "In short"

Reading the table

In short (EERs from shipped embeddings) the gender shortcut is large for McAdams, kNN-VC km same-gender, SSL-SAS and Private kNN-VC, about zero for three systems, and negative for ASR→TTS; the corpus shortcut is smaller (0.66 to 3.05 points).

Gender step (first column vs second). Same-gender minus all-trials EER, from unrounded values (computed separately, not printed by the cells): McAdams +5.6, kNN-VC km same-gender +6.1, SSL-SAS +8.8, Private kNN-VC +3.9, original +0.5, kNN-VC en +0.5, kNN-VC km -0.2 and ASR→TTS -3.3. So the gender shortcut is large for four systems, near zero for three, and ASR→TTS goes the other way (48.6 on all trials, 45.3 on same-gender). The course reason is that ASR→TTS uses one synthetic voice for everybody, so there is no gender to exploit and the extra trials only add noise (course explanation, not tested here).

Corpus step (second column vs third). Removing cross-corpus trials raises the EER by 0.66 to 3.05 points for every system (computed separately, not printed by the cells). That is smaller than the largest gender steps. The direction is as expected: the removed trials were easy non-targets.

Phone vs studio columns. These are small subsets: studio has 11 speakers and 110 target trials (computed separately, not printed by the cells) (1 210 trials in all). The two columns differ by up to 4.3 points, in both directions (SSL-SAS 52.2 vs 48.2, kNN-VC en 41.2 vs 45.5). Do not read a 3-point difference between phone and studio as a finding.

The leaderboard reports the second column (same-gender). That is the number to quote.

3. Utility: did the content, prosody and quality survive?

Privacy alone is meaningless (white noise is perfectly private). Four utility numbers:

• CER from the Khmer ASR judge (Day 3). transcripts.csv has the reference and the judge's hypothesis for the 430 trial utterances after each system. The hypotheses are shipped; the judge was not run again here. We use the corpus-level CER (total edits / total reference characters), which is what the leaderboard reports. The course says the mean of per-utterance CERs would weight short sentences too much; in our data the two differ by less than 1 point (original 7.59 vs mean-per-utterance 7.41; McAdams 38.36 vs 39.02) (computed separately, not printed by the cells). CI by speaker bootstrap again: each speaker contributes (edits, chars), and we resample speakers.
• ρF0: Spearman correlation between the median pitch of each original clip and of its anonymized version. ρ ≈ 1 means pitch (and so much of the prosody, and the gender) is untouched; ρ ≈ 0 means the system discarded it. We recompute it fresh on the 20 demo clips (a small sample); the leaderboard value used all 430 trials.
• UTMOS: a neural predictor of the mean opinion score (1 = terrible, 5 = natural), trained on VoiceMOS challenge data (English and Chinese (background fact from the course text, not checked here)), so for Khmer it is a hint, not a measurement. We cannot recompute it here (model download); it is read from the leaderboard. The course's optional recompute code (a ~400 MB download) was not run here.
• duration ratio: anonymized length / original length, a cheap proxy for "did the system re-time the speech" (ASR→TTS and Private kNN-VC generate their own timing).

$$\mathrm{CER} = \frac{\sum_u \mathrm{edits}(ref_u, hyp_u)}{\sum_u |ref_u|}$$

Why it matters: a system that hides the speaker but destroys the words is useless. These numbers say what each system costs.

Visualization Version

Real CER with interval, phone vs studio CER, and pitch correlation (leaderboard vs our 20 clips).

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short read the 430 transcripts, compute the corpus-level CER of each system with a speaker bootstrap (same 1 000 draws as before), and also the CER for phone and studio clips; compare with the leaderboard.

• per_utt_edits: number of character edits = CER × reference length (spaces are removed, as in src.utils.cer).
• cer_boot: sum edits and characters per speaker, then resample speakers.
• 7 systems have transcripts (knnvc_khmer_pool_samegender has none).
• Hypotheses are shipped, not a fresh ASR run.

What the output means

In short original 7.59 [5.94, 9.44]; mcadams 38.36; knnvc_english_pool 27.56; knnvc_khmer_pool 24.43; asr_tts 31.25; sslsas 29.62; private_knnvc_english_durations 113.64 [105.32, 123.56].

• The overall CER agrees with the shipped cer column within 0.04 points (for example original 7.59 vs 7.58, Private kNN-VC 113.64 vs 113.60); phone and studio CER agree with the shipped columns within 0.06 (table in the next cell).
• Phone / studio for original: 6.89 / 9.09. For mcadams: 43.77 / 26.79.
• The intervals are narrow for original (about ±2) and asr_tts (about ±2), wide for private_knnvc_english_durations (about -8 / +10) (computed separately, not printed by the cells).

tx = pd.read_csv(DATA / "meta/transcripts.csv").fillna("")
tx["speaker"] = tx.utt_id.map(utts.set_index("utt_id").speaker)
CER_SYSTEMS = [s for s in SYSTEMS if f"hyp_{s}" in tx.columns]

def per_utt_edits(ref, hyp):
    # number of character edits = CER x reference length (src.utils.cer strips spaces, so do we)
    ref = ref.replace(" ", "")
    return cer(ref, hyp) * len(ref), len(ref)

def cer_boot(system, corpus=None, draws=DRAWS):
    sub = tx if corpus is None else tx[tx.corpus == corpus]
    ed, n = zip(*[per_utt_edits(r, h) for r, h in zip(sub.reference, sub[f"hyp_{system}"])])
    df = pd.DataFrame({"spk": sub.speaker.map(spk_idx).values, "ed": ed, "n": n}).groupby("spk").sum()
    e_s = np.zeros(len(SPEAKERS)); n_s = np.zeros(len(SPEAKERS))
    e_s[df.index] = df.ed; n_s[df.index] = df.n
    point = 100 * e_s.sum() / n_s.sum()
    boots = 100 * (draws @ e_s) / np.maximum(draws @ n_s, 1)   # same speaker draws as the EER bootstrap
    return point, np.percentile(boots, 2.5), np.percentile(boots, 97.5), boots

t0 = time.time()
cer_rows, cer_boots = [], {}
for s in CER_SYSTEMS:
    pt, lo, hi, b = cer_boot(s); cer_boots[s] = b
    cer_rows.append({"system": s, "cer": pt, "ci_low": lo, "ci_high": hi,
                     "cer_phone": cer_boot(s, "phone (iAny)")[0], "cer_studio": cer_boot(s, "studio (OpenSLR 42)")[0],
                     "board_cer": board.loc[s, "cer"], "board_phone": board.loc[s, "cer_phone"], "board_studio": board.loc[s, "cer_studio"]})
cer_tab = pd.DataFrame(cer_rows).set_index("system").round(2)
print(f"CER with {N_BOOT}-rep speaker bootstrap in {time.time()-t0:.1f} s")
cer_tab

CER with 1000-rep speaker bootstrap in 0.4 s

	cer	ci_low	ci_high	cer_phone	cer_studio	board_cer	board_phone	board_studio
system
original	7.59	5.94	9.44	6.89	9.09	7.58	6.88	9.09
mcadams	38.36	33.13	44.74	43.77	26.79	38.35	43.77	26.77
knnvc_english_pool	27.56	22.53	33.90	32.33	17.36	27.55	32.31	17.36
knnvc_khmer_pool	24.43	18.94	31.35	27.90	17.01	24.42	27.89	16.99
asr_tts	31.25	29.44	33.17	30.02	33.89	31.25	30.02	33.87
sslsas	29.62	22.84	38.06	35.50	17.05	29.62	35.50	17.05
private_knnvc_english_durations	113.64	105.32	123.56	121.51	96.80	113.60	121.45	96.80

Additional reading: you can read all of it or just read the "In short"

Reading the CER table

In short phone CER is higher than studio CER for 5 of 7 systems (1.6 to 2.1 times for four of them), but not for original and ASR→TTS; Private kNN-VC has CER 113.6 %.

The overall CER agrees with the leaderboard within 0.04 points; phone and studio CER within 0.06 (see the table above).

Phone vs studio. For five of the seven systems the phone CER is higher than the studio CER. After anonymization it is 1.6 to 2.1 times the studio CER for McAdams (1.63), kNN-VC en (1.86), kNN-VC km (1.64) and SSL-SAS (2.08), and 1.26 times for Private kNN-VC (computed separately, not printed by the cells). Two systems go the other way: original (6.89 phone vs 9.09 studio) and ASR→TTS (30.02 vs 33.89). The course reason for the anonymized systems is that the anonymizer and the noisy channel compound (course explanation, not tested here).

Private kNN-VC has a CER above 100 %. The judge makes more edits than there are reference characters, so its output is far from the reference. The course says the output is not Khmer any more because the English phone recognizer inside it has no Khmer phones to work with (course explanation, not tested here); we did not read the hypotheses here. Listen to it on Day 6 again if you doubt the number.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short track the pitch of the 20 demo clips (originals and 6 anonymized versions), then compute ρF0 and the duration ratio per system.

• median_f0(x): the median pitch of one clip. spearmanr compares the original and anonymized median pitch over the 20 clips (a clip with no voiced frames gives NaN and is dropped).
• duration_ratio = anonymized length / original length, averaged over the clips.
• This is a fresh computation on the shipped audio (data/audio/demo, data/audio/anonymized/<system>/). Only 6 anonymizers have audio.

What the output means

In short 140 clips tracked (20 × (1 original + 6 systems)); median F0 of the originals 217 Hz. ρF0 (20 clips): McAdams 0.920, kNN-VC en -0.132, kNN-VC km -0.486, ASR→TTS 0.323, SSL-SAS 0.923, Private kNN-VC 0.596.

• Duration ratio: McAdams 1.000, kNN-VC 0.996, SSL-SAS 0.997, ASR→TTS 1.152, Private kNN-VC 0.788.
• Median F0 of the anonymized clips: 213.8 (McAdams), 158.5 and 156.3 (the two kNN-VC), 136.9 (ASR→TTS), 220.1 (SSL-SAS), 143.9 (Private kNN-VC) Hz.
• The board_rho_f0 column is the shipped value for 430 clips.

# rho_F0 and duration ratio on the 20 demo clips (the only shipped audio per system)
from scipy.stats import spearmanr
demo = pd.read_csv(DATA / "meta/demo_clips.csv")
AUDIO_SYSTEMS = [s for s in SYSTEMS if (DATA / "audio/anonymized" / s).is_dir()]

t0 = time.time()
f0_orig, dur_orig = {}, {}
for u in demo.utt_id:
    x = load_audio(DATA / "audio/demo" / f"{u}.wav")
    f0_orig[u] = median_f0(x); dur_orig[u] = len(x) / SR

pros_rows = []
for s in AUDIO_SYSTEMS:
    f0s, ratios = [], []
    for u in demo.utt_id:
        y = load_audio(DATA / "audio/anonymized" / s / f"{u}.wav")
        f0s.append(median_f0(y)); ratios.append(len(y) / SR / dur_orig[u])
    rho = spearmanr([f0_orig[u] for u in demo.utt_id], f0s, nan_policy="omit").correlation   # a clip with no voiced frames -> NaN, dropped
    pros_rows.append({"system": s, "rho_f0_demo20": rho, "board_rho_f0": board.loc[s, "rho_f0"],
                      "duration_ratio": np.mean(ratios), "median_f0_anon_Hz": np.nanmedian(f0s)})
pros_tab = pd.DataFrame(pros_rows).set_index("system").round(3)
print(f"pitch tracking on {len(demo)*(1+len(AUDIO_SYSTEMS))} clips in {time.time()-t0:.1f} s; median F0 of originals: {np.nanmedian(list(f0_orig.values())):.0f} Hz")
pros_tab

pitch tracking on 140 clips in 4.7 s; median F0 of originals: 217 Hz

	rho_f0_demo20	board_rho_f0	duration_ratio	median_f0_anon_Hz
system
mcadams	0.920	0.732	1.000	213.825
knnvc_english_pool	-0.132	0.673	0.996	158.475
knnvc_khmer_pool	-0.486	0.614	0.996	156.265
asr_tts	0.323	0.317	1.152	136.878
sslsas	0.923	0.699	0.997	220.065
private_knnvc_english_durations	0.596	0.180	0.788	143.862

Additional reading: you can read all of it or just read the "In short"

Reading ρF0 and the duration ratio

In short on the leaderboard, five systems keep pitch (0.61-0.73) and two do not (0.18, 0.32); on our 20 clips the values are noisy and the kNN-VC ones turn negative.

On the leaderboard (430 clips) McAdams, both kNN-VC variants and SSL-SAS keep a strong pitch correlation (0.61 to 0.73): a high-pitched speaker is still high-pitched after anonymization. The course links this to the gender shortcut of section 2 (course explanation, not tested here). ASR→TTS and Private kNN-VC are at 0.32 and 0.18: one synthetic voice (or one duration model) flattens everyone's pitch, so prosody is gone, and with it a lot of the meaning in a language like Khmer where sentence intonation carries questions and emphasis (background fact from the course text, not checked here).

On our 20 demo clips the picture is only half reproduced.

• McAdams and SSL-SAS come out at ρ ≈ 0.92. The course says they do not touch F0 (course explanation, not tested here); the median F0 of the anonymized clips (213.8 and 220.1 Hz vs 217 Hz) fits that.
• ASR→TTS matches the leaderboard (0.323 vs 0.317).
• Private kNN-VC comes out at 0.60 instead of 0.18, on 19 clips (one clip has no voiced frames) (computed separately, not printed by the cells).
• The two kNN-VC systems come out negative (-0.13 and -0.49).

This is not a bug, it is a lesson about n = 20. 16 of the 20 demo clips are female (median F0 202-245 Hz) and 4 are male (118-140 Hz) (computed separately, not printed by the cells). A correlation over 20 clips is dominated by that male/female gap and by a few points, so it is noisy; the 0.92 of McAdams and SSL-SAS is probably this gap, not within-gender pitch tracking (our guess, not tested).

kNN-VC draws a random target voice per utterance, so the anonymized pitch is a random draw from the pool (median 158 and 156 Hz here vs 217 Hz for the originals). With 430 clips and 43 speakers the leaderboard sees the source-pitch leak (0.61 to 0.67) through that randomness; with 20 clips the randomness wins (course explanation, not tested here).

Rule: a Spearman ρ on 20 points is noisy (the standard error at ρ = 0 is about 0.23-0.24, roughly 1/√(n−1) for n = 19-20; a standard approximation, not tested here). Quote the leaderboard value and treat the demo-clip value as a sanity check.

The duration ratio shows which systems re-time speech: McAdams, kNN-VC and SSL-SAS keep the frame grid (ratio 1.00 / 0.996 / 0.997), ASR→TTS is 15 % longer (1.152), Private kNN-VC 21 % shorter (0.788).

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short read the UTMOS values of the leaderboard (the recompute is switched off) and print them for the 6 audio systems and the original.

• RUN_UTMOS = False: the optional recompute (a ~400 MB model download from torch.hub) is skipped, so nothing is downloaded and the cell cannot fail.
• utmos_tab takes the utmos column of leaderboard.csv.

What the output means

In short UTMOS (shipped values, not recomputed): McAdams 1.64, kNN-VC en 2.52, kNN-VC km 2.39, ASR→TTS 2.49, SSL-SAS 2.93, Private kNN-VC 1.23, original 2.40.

• If RUN_UTMOS = True, the course's code is meant to add a utmos_demo20 column from the 20 demo clips. That code was not run here, so we do not know that it works.

# UTMOS: predicted MOS from the leaderboard. Optional recompute (needs internet + torch hub, ~1 min).
RUN_UTMOS = False   # set True to try; the notebook does not depend on it
utmos_tab = board.loc[AUDIO_SYSTEMS + ["original"], ["utmos"]].copy()
if RUN_UTMOS:
    try:
        import torch
        torch.manual_seed(0)
        # model: tarepan/SpeechMOS utmos22_strong (~400 MB download) -> ~/.cache/torch/hub   # needs internet (first run only)
        predictor = torch.hub.load("tarepan/SpeechMOS:v1.2.0", "utmos22_strong", trust_repo=True)
        rec = {}
        for s in AUDIO_SYSTEMS + ["original"]:
            folder = DATA / "audio/demo" if s == "original" else DATA / "audio/anonymized" / s
            rec[s] = np.mean([float(predictor(torch.tensor(load_audio(folder / f"{u}.wav"))[None], SR)) for u in demo.utt_id])
        utmos_tab["utmos_demo20"] = pd.Series(rec)
    except Exception as ex:
        print("UTMOS model not available (offline or not installed) -> using leaderboard values only.", type(ex).__name__)
utmos_tab.round(2)

	utmos
system
mcadams	1.64
knnvc_english_pool	2.52
knnvc_khmer_pool	2.39
asr_tts	2.49
sslsas	2.93
private_knnvc_english_durations	1.23
original	2.40

Additional reading: you can read all of it or just read the "In short"

Reading UTMOS

In short UTMOS is a prediction read from the leaderboard: SSL-SAS 2.93 is highest, Private kNN-VC 1.23 is lowest.

UTMOS is read from the leaderboard (shipped leaderboard.csv value, not recomputed here): 2.4 for the original phone/studio mix, 2.9 for SSL-SAS, 1.6 for McAdams, 1.2 for Private kNN-VC. The course gives reasons: the predictor dislikes phone recordings, a neural vocoder makes clean audio, the formant warp leaves a nasal quality (course explanation, not tested here).

A score below the original does not have to mean "noise"; it means the predictor, trained on English and Chinese, rates the voice as less like clean human speech (interpretation; course explanation, not tested here). This is why section 7 sets up a human listening test.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short put CER, interval, phone/studio CER, ρF0 (leaderboard and 20 clips), duration ratio and UTMOS in one table.

• Rows = the 8 systems. kNN-VC km same-g has no transcripts and no audio, so most of its cells are empty.
• CER and the intervals are recomputed; rho_F0 (board) and UTMOS (board) are shipped.

What the output means

In short one utility table; for example Private kNN-VC: CER 113.64 [105.3, 123.6], ρF0 board 0.18, UTMOS 1.23.

• original has no rho_F0 (20 demo) (it is the reference), and no duration ratio.

util = pd.DataFrame(index=SYSTEMS)
util["CER %"] = cer_tab.cer.reindex(SYSTEMS)
util["CER CI"] = [f"[{cer_tab.loc[s,'ci_low']:.1f}, {cer_tab.loc[s,'ci_high']:.1f}]" if s in cer_tab.index else "" for s in SYSTEMS]
util["CER phone"] = cer_tab.cer_phone.reindex(SYSTEMS); util["CER studio"] = cer_tab.cer_studio.reindex(SYSTEMS)
util["rho_F0 (board)"] = board.rho_f0.reindex(SYSTEMS); util["rho_F0 (20 demo)"] = pros_tab.rho_f0_demo20.reindex(SYSTEMS)
util["dur ratio"] = pros_tab.duration_ratio.reindex(SYSTEMS); util["UTMOS (board)"] = board.utmos.reindex(SYSTEMS)
util.index = [SHORT[s] for s in SYSTEMS]
util.round(2)

	CER %	CER CI	CER phone	CER studio	rho_F0 (board)	rho_F0 (20 demo)	dur ratio	UTMOS (board)
original	7.59	[5.9, 9.4]	6.89	9.09	1.00	NaN	NaN	2.40
McAdams	38.36	[33.1, 44.7]	43.77	26.79	0.73	0.92	1.00	1.64
kNN-VC en	27.56	[22.5, 33.9]	32.33	17.36	0.67	-0.13	1.00	2.52
kNN-VC km	24.43	[18.9, 31.4]	27.90	17.01	0.61	-0.49	1.00	2.39
kNN-VC km same-g	NaN		NaN	NaN	0.63	NaN	NaN	2.34
ASR->TTS	31.25	[29.4, 33.2]	30.02	33.89	0.32	0.32	1.15	2.49
SSL-SAS	29.62	[22.8, 38.1]	35.50	17.05	0.70	0.92	1.00	2.93
Private kNN-VC	113.64	[105.3, 123.6]	121.51	96.80	0.18	0.60	0.79	1.23

4. The privacy-utility plot

Put privacy on the x axis (EER, higher = more private) and content loss on the y axis (CER, lower = better). Every system is a point with error bars in both directions. A system is Pareto-optimal if no other system is at least as good on both axes and strictly better on one. The Pareto front is the staircase through those points; anything above-left of the staircase is dominated and should not be chosen, whatever its other merits.

Worked example (ignorant attacker, this run; EERs from shipped ECAPA embeddings, CERs from shipped ASR transcripts, no fresh model run). kNN-VC km (EER 43.96, CER 24.43) dominates kNN-VC en (EER 40.66, CER 27.56): more private and better content. McAdams (29.58, 38.36) is dominated by four of the six other systems with a CER (ASR→TTS, kNN-VC en, kNN-VC km, SSL-SAS) (computed separately, not printed by the cells). Private kNN-VC (48.13, 113.64) looks very private, but SSL-SAS (50.70, 29.62) is more private and far more intelligible, so Private kNN-VC is dominated too.

Why it matters: a leaderboard sorted by one column hides the trade. The front is the honest summary of "what you can have".

Visualization Version

7 systems under three attackers (6 in the right panel: ASR→TTS has no semi-informed number). Orange = on the front, blue = dominated; the triangle is Private kNN-VC, CER 113.6, off scale.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short define the Pareto front and the plotting function, then draw the privacy-utility plot for the ignorant attacker (left) and the semi-informed attacker (right) and print both fronts.

• pareto_front: keep a point if no other point has EER ≥ and CER ≤ with one strict.
• privacy_utility_table(attacker) joins our EER and CER (with intervals). Systems without a CER or without an EER (e.g. knnvc_khmer_pool_samegender, asr_tts for semi-informed) are dropped.
• Error bars show the 95 % intervals in both directions.

What the output means

In short (EERs from shipped embeddings, CERs from shipped transcripts) Pareto front, ignorant attacker: original, kNN-VC km, SSL-SAS. Pareto front, semi-informed attacker: original, kNN-VC km.

• SSL-SAS drops off the front when the attacker becomes semi-informed (its EER falls from 50.7 to 9.8).
• The lazy-informed front (not drawn by the course cell) is original, kNN-VC km and ASR→TTS (computed separately, not printed by the cells); it is the middle panel of the picture above.

def pareto_front(df, x="eer", y="cer"):
    # keep points not dominated: no other point with x >= and y <= and at least one strict
    keep = []
    for i, r in df.iterrows():
        dominated = ((df[x] >= r[x]) & (df[y] <= r[y]) & ((df[x] > r[x]) | (df[y] < r[y]))).any()
        if not dominated:
            keep.append(i)
    return df.loc[keep].sort_values(x)

def plot_pareto(df, attacker="ignorant", ax=None, title=None):
    # df: index system; columns eer, eer_lo, eer_hi, cer, cer_lo, cer_hi
    if ax is None:
        _, ax = plt.subplots(figsize=(8, 6))
    ax.errorbar(df.eer, df.cer, xerr=[df.eer - df.eer_lo, df.eer_hi - df.eer],
                yerr=[df.cer - df.cer_lo, df.cer_hi - df.cer], fmt="o", capsize=3, color="C0", ecolor="gray")
    for s, r in df.iterrows():
        ax.annotate(SHORT.get(s, s), (r.eer, r.cer), xytext=(5, 5), textcoords="offset points", fontsize=9)
    pf = pareto_front(df)
    ax.step(pf.eer, pf.cer, where="post", color="C3", lw=1.5, alpha=0.8, label="Pareto front")
    ax.axvline(50, ls="--", c="gray", lw=1)
    ax.set(xlabel=f"EER % ({attacker} attacker)  ->  more private", ylabel="CER %  (lower = better content)",
           title=title or f"Privacy vs utility, {attacker} attacker, 95 % CIs")
    ax.legend(loc="upper left"); ax.grid(alpha=0.3)
    return pf

def privacy_utility_table(attacker):
    e = eer_tab[eer_tab.attacker == attacker].set_index("system")
    df = pd.DataFrame({"eer": e.eer, "eer_lo": e.ci_low, "eer_hi": e.ci_high,
                       "cer": cer_tab.cer, "cer_lo": cer_tab.ci_low, "cer_hi": cer_tab.ci_high}).dropna()
    return df

fig, axes = plt.subplots(1, 2, figsize=(14, 6))
pf_ign = plot_pareto(privacy_utility_table("ignorant"), "ignorant", axes[0])
pf_semi = plot_pareto(privacy_utility_table("semi_informed"), "semi_informed", axes[1])
plt.tight_layout(); plt.show()
print("Pareto front, ignorant attacker:     ", [SHORT[s] for s in pf_ign.index])
print("Pareto front, semi-informed attacker:", [SHORT[s] for s in pf_semi.index])

Pareto front, ignorant attacker:      ['original', 'kNN-VC km', 'SSL-SAS']
Pareto front, semi-informed attacker: ['original', 'kNN-VC km']

Additional Reading

How to read this plot

In short (EERs from shipped embeddings, CERs from shipped transcripts) bottom-right is the dream (private and intelligible). Under the ignorant attacker the front is original → kNN-VC km → SSL-SAS; under the semi-informed attacker it is only original → kNN-VC km.

Bottom-right is the dream (private and intelligible); top-left is bad. The original sits bottom-left (readable, no privacy); white noise would sit top-right. The red staircase is the Pareto front.

Left panel (ignorant attacker). The front is original (2.4, 7.6) → kNN-VC km (44.0, 24.4) → SSL-SAS (50.7, 29.6). kNN-VC km is the knee: the step to SSL-SAS buys 6.7 EER points for about 5 CER points (6.73 and 5.19 (computed separately, not printed by the cells)).

ASR→TTS (45.3, 31.3) sits just inside the front, dominated by SSL-SAS. Private kNN-VC (48.1, 113.6) is the useless corner at the top, also dominated. McAdams and kNN-VC en sit inside, on the left.

Right panel (semi-informed attacker). The picture changes. SSL-SAS moves from x = 50.7 to x = 9.8 and drops out of the front; McAdams falls to 14.2. kNN-VC km barely moves (43.5) and the front is now just original → kNN-VC km; even Private kNN-VC (36.5) is to its left. ASR→TTS is missing because no semi-informed encoder was shipped for it.

The course calls this "the single most important comparison of the week": the Pareto front under a weak attacker is misleading.

The error bars. On the x axis the half-widths are about ±1 to ±5 points (median ±2.9), on the y axis about ±2 to ±10 points (median ±5.9) (left panel) (computed separately, not printed by the cells). Neighbouring points along the front overlap, so only the paired test in the next section can separate them.

A drawing problem in the course plot: in the left panel the labels of kNN-VC en, kNN-VC km, ASR->TTS and SSL-SAS overlap each other and the error bars. This is how the course cell draws it (the code is unchanged). Use the picture above for the positions.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short plot the ignorant-attacker EER against the leaderboard ρF0, with the EER interval as a horizontal whisker.

• x = our ignorant EER (with 95 % interval); y = rho_f0 of leaderboard.csv (430 clips, shipped, not recomputed).
• Each point is labelled with the system name.

What the output means

In short one scatter plot, 8 points.

• Note that only the ignorant EER is on the x axis. The semi-informed EERs are in the text below.

# prosody leak: EER vs rho_F0 (leaderboard values, all 430 trials)
fig, ax = plt.subplots(figsize=(8, 5))
e = eer_tab[eer_tab.attacker == "ignorant"].set_index("system").reindex(SYSTEMS)
ax.errorbar(e.eer, board.rho_f0.reindex(SYSTEMS), xerr=[e.eer - e.ci_low, e.ci_high - e.eer], fmt="o", capsize=3, ecolor="gray")
for s in SYSTEMS:
    ax.annotate(SHORT[s], (e.loc[s, "eer"], board.loc[s, "rho_f0"]), xytext=(5, 5), textcoords="offset points", fontsize=9)
ax.set(xlabel="EER % (ignorant attacker)  ->  more private", ylabel="rho_F0 (Spearman, original vs anonymized median pitch)",
       title="Prosody leak: pitch correlation vs privacy", ylim=(0, 1.05)); ax.grid(alpha=0.3)
plt.tight_layout(); plt.show()

Additional Reading

How to read this plot

In short two groups: five anonymizers keep pitch (ρ 0.61-0.73), two do not (0.18-0.32). The pitch-keeping group contains both the least private (McAdams, SSL-SAS under informed attackers) and a more robust system (kNN-VC km).

Top-right would be "private and still sounds like your own intonation". Bottom-right is "private because it threw the prosody away". The real systems form two groups.

Along the top (ρ ≈ 0.6 to 0.73): McAdams, both kNN-VC variants (and the same-gender one at 0.63) and SSL-SAS keep the speaker's pitch. They anonymize timbre, not prosody. At the bottom (ρ 0.18 to 0.32): ASR→TTS and Private kNN-VC regenerate speech with their own prosody: private, but the speaker's emphasis and question intonation are gone, and for Private kNN-VC so is the content.

Is the top-right corner empty? No. kNN-VC km sits at EER 44.0 and ρ 0.61, and it keeps 43.5 % against the semi-informed attacker.

What no system in the plot has is a ρ of about 0.7 or more together with a high semi-informed EER: the two highest among the anonymizers, McAdams (0.73) and SSL-SAS (0.70; 0.699 unrounded), have semi-informed EERs of 14.2 and 9.8.

Over the 16 leaderboard systems that have a semi-informed EER (including original), the correlation between ρF0 and that EER is -0.61 (Pearson; -0.53 without original) (computed separately, not printed by the cells) (shipped leaderboard.csv value, not recomputed here). In this table, systems that keep more pitch tend to have a lower semi-informed EER, with kNN-VC km as the big exception. This is a pattern over 16 points, not a cause. The course names a prosody-preserving system that still defeats a semi-informed attacker as the research opportunity.

A drawing problem in the course plot: the labels kNN-VC en, kNN-VC km same-g and kNN-VC km overlap (their points are close: ρ 0.67, 0.63, 0.61). This is how the course cell draws it.

Additional reading: you can read all of it or just read the "In short"

What each system trades

In short (shipped embeddings and transcripts) each system sits at a different point: McAdams cheap but learnable, kNN-VC km the best trade in this table, SSL-SAS a trap, Private kNN-VC unusable.

What each real system trades (ignorant / semi-informed EER from this run, CER, then ρF0 and UTMOS from leaderboard.csv (shipped leaderboard.csv value, not recomputed here)):

• McAdams (29.6 / 14.2, CER 38.4, ρ 0.73, UTMOS 1.6): a signal-processing warp of the formants (the code docstring says it warps the LPC pole angles; α is drawn per utterance, 0.55 to 0.88 on the demo clips (computed separately, not printed by the cells)). Cheap and language-independent (course explanation, not tested here), keeps pitch and timing (duration ratio 1.000), but the semi-informed attacker roughly halves its EER, and the ASR judge suffers. The VoicePrivacy baseline, and our floor.
• kNN-VC, English pool (40.7 / 37.9, CER 27.6, ρ 0.67): the shipped description says "random target from 40 English LibriSpeech voices". The ignorant-to-semi drop is small (2.7 points). The course says matching Khmer frames to English frames costs content (course explanation, not tested here).
• kNN-VC, Khmer pool (44.0 / 43.5, CER 24.4, ρ 0.61): "random target from 17 Khmer voices". Better on both axes than the English pool in this table; the paired test in section 5 says how sure we are. The same-gender variant (40.7 / 31.4) keeps gender, and its EER is lower under the smarter attackers (the course explains this by the kept gender (course explanation, not tested here)).
• ASR→TTS (45.3 / n.a., CER 31.3, ρ 0.32): "Qwen3-ASR-Khmer transcript read by MMS-TTS-khm (one synthetic voice)". Private by construction (every speaker becomes the TTS voice; lazy EER 50.0), but the CER is the ASR's errors plus the judge's (course explanation, not tested here), prosody is gone, and no semi-informed number exists yet.
• SSL-SAS (50.7 / 9.8, CER 29.6, ρ 0.70, UTMOS 2.9): "HuBERT content + F0 + one fixed pseudo-speaker per speaker". The best predicted naturalness in this table and a perfect-looking ignorant EER, but against an attacker who can enroll the pseudo-voice the EER falls to 9.8-12.8 %, much lower than the 50.7 % the ignorant attacker shows. The textbook two-EERs trap (section 6).
• Private kNN-VC (48.1 / 36.5, CER 113.6, ρ 0.18, UTMOS 1.2): "English phone recognizer + duration model". Private, but its CER is 113.6 %, far from the reference (the course says its output is not Khmer; not tested here). The shipped Khmer-duration variant (CER 49.1, ρ 0.32, from leaderboard.csv) suggests that the duration model, not the matching, was the main problem; that is the course's reading and was not tested here (course explanation, not tested here).

5. Paired comparisons: is B really better than A?

"kNN-VC en EER 40.7 vs kNN-VC km 44.0" (ignorant attacker) looks like a 3.3-point win (EERs from shipped ECAPA embeddings, as everywhere in this notebook). But both numbers have an interval of about ±2.5 points (en: [38.4, 43.5], km: [41.4, 46.5]), and the intervals overlap. Does that mean no difference? Not necessarily: the two systems were evaluated on the same 43 speakers, and a hard speaker is hard for both. The uncertainty in the difference can be smaller than the two separate uncertainties suggest.

The paired speaker bootstrap handles this: for each bootstrap rep, draw one set of speakers and evaluate both systems on that same set, then take the difference. We already stored the per-rep EERs in eer_boots using the shared DRAWS, so the paired CI is just the percentile of the per-rep differences.

$$\Delta_b = \mathrm{EER}_B(\text{draw}_b) - \mathrm{EER}_A(\text{draw}_b),\qquad \mathrm{CI}_{95} = [\,\Delta_{(2.5\%)},\ \Delta_{(97.5\%)}\,]$$

If the CI of $\Delta$ contains 0, the data do not show a difference at this sample size. It does not prove the systems are equal.

Pre-registration. We pick the comparisons before looking at the differences. With 8 systems there are 28 possible pairs and 3 attackers: 84 differences. If you test all of them at the 5 % level, about 4 (84 × 0.05 = 4.2) would miss zero by chance alone, if the tests were independent (an approximation): the multiple-comparisons problem. The project pre-registered six comparisons in paired_comparisons.json (checked: the file has six); we recompute the three for which we have embeddings. If you later want to test a pair you noticed in a plot, say so, and treat it as exploratory.

Why it matters: "the intervals overlap" is not a test. A paired test on the same speaker draws is.

Visualization Version

All 11 paired differences: this run (thick) and the shipped 2 000-rep interval (thin orange).

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short compute the paired difference, its 95 % interval and a verdict for three pre-registered pairs, and list the shipped value next to it.

• PAIRS: (English pool, Khmer pool), (McAdams, Khmer pool), (Khmer pool, Khmer pool same-gender).
• For each attacker with embeddings in both systems, paired_delta takes boots_b - boots_a of the same draw and its 2.5th / 97.5th percentiles. CER uses the CER bootstrap.
• shipped = the observed difference and 95 % interval of the project's 2 000-rep run.
• evidence = "CI excludes 0" if the interval does not contain 0.

What the output means

In short 11 rows (EERs from shipped embeddings, CERs from shipped transcripts). Khmer pool minus English pool: ignorant +3.30 [0.02, 6.28], lazy +3.05 [-0.26, 6.58], semi-informed +5.55 [1.62, 8.78], CER -3.13 [-6.20, -0.13]. Khmer pool minus McAdams: +14.38 / +21.43 / +29.30, CER -13.93. Same-gender minus Khmer pool: -3.27 [-8.59, 3.95] (ignorant), -12.79 (lazy), -12.09 (semi).

• The signs agree with the shipped values in all 11 rows. Where a CI edge is close to 0, the verdict can differ: the first row's lower edge is 0.02 here (CI excludes 0) but -0.47 in the shipped run (CI includes 0).
• There is no CER row for the third pair (knnvc_khmer_pool_samegender has no transcripts column).
• The pre-registered file also holds three more pairs (ASR→TTS, Private kNN-VC Khmer durations, SSL-SAS Khmer pool) that we cannot recompute here.

PAIRS = [("knnvc_english_pool", "knnvc_khmer_pool"),              # does a Khmer pool beat an English pool?
         ("mcadams", "knnvc_khmer_pool"),                         # does learned VC beat the signal-processing baseline?
         ("knnvc_khmer_pool", "knnvc_khmer_pool_samegender")]     # does keeping gender cost privacy?

def paired_delta(boots_a, boots_b, point_a, point_b):
    d = boots_b - boots_a
    return point_b - point_a, np.percentile(d, 2.5), np.percentile(d, 97.5)

rows = []
for a, b in PAIRS:
    key = f"{b} minus {a}"
    ref = paired_ref.get(key, {})
    for att, refname in [("ignorant", "delta_eer_ignorant"), ("lazy", "delta_eer_lazy"), ("semi_informed", "delta_eer_semi")]:
        if (att, a) in eer_boots and (att, b) in eer_boots:
            pa = eer_tab[(eer_tab.system == a) & (eer_tab.attacker == att)].eer.item()
            pb = eer_tab[(eer_tab.system == b) & (eer_tab.attacker == att)].eer.item()
            obs, lo, hi = paired_delta(eer_boots[(att, a)], eer_boots[(att, b)], pa, pb)
            r = ref.get(refname, {})
            rows.append({"comparison": key, "metric": f"EER {att}", "delta": obs, "ci_low": lo, "ci_high": hi,
                         "shipped": f"{r.get('observed','')} {r.get('ci95','')}"})
    if a in cer_boots and b in cer_boots:
        obs, lo, hi = paired_delta(cer_boots[a], cer_boots[b], cer_tab.loc[a, "cer"], cer_tab.loc[b, "cer"])
        r = ref.get("delta_cer", {})
        rows.append({"comparison": key, "metric": "CER", "delta": obs, "ci_low": lo, "ci_high": hi,
                     "shipped": f"{r.get('observed','')} {r.get('ci95','')}"})
paired_tab = pd.DataFrame(rows)
paired_tab["evidence"] = np.where((paired_tab.ci_low > 0) | (paired_tab.ci_high < 0), "CI excludes 0", "no evidence")
paired_tab.round(2)

	comparison	metric	delta	ci_low	ci_high	shipped	evidence
0	knnvc_khmer_pool minus knnvc_english_pool	EER ignorant	3.30	0.02	6.28	3.35 [-0.47, 6.96]	CI excludes 0
1	knnvc_khmer_pool minus knnvc_english_pool	EER lazy	3.05	-0.26	6.58	3.09 [-0.83, 6.99]	no evidence
2	knnvc_khmer_pool minus knnvc_english_pool	EER semi_informed	5.55	1.62	8.78	5.53 [1.32, 9.05]	CI excludes 0
3	knnvc_khmer_pool minus knnvc_english_pool	CER	-3.13	-6.20	-0.13	-3.14 [-6.25, -0.05]	CI excludes 0
4	knnvc_khmer_pool minus mcadams	EER ignorant	14.38	11.20	17.63	14.35 [9.97, 18.77]	CI excludes 0
5	knnvc_khmer_pool minus mcadams	EER lazy	21.43	18.13	25.43	21.48 [17.69, 26.06]	CI excludes 0
6	knnvc_khmer_pool minus mcadams	EER semi_informed	29.30	26.52	32.79	29.3 [25.48, 33.86]	CI excludes 0
7	knnvc_khmer_pool minus mcadams	CER	-13.93	-20.64	-7.68	-13.94 [-20.69, -7.3]	CI excludes 0
8	knnvc_khmer_pool_samegender minus knnvc_khmer_...	EER ignorant	-3.27	-8.59	3.95	-3.28 [-6.39, 0.86]	no evidence
9	knnvc_khmer_pool_samegender minus knnvc_khmer_...	EER lazy	-12.79	-16.14	-9.55	-12.79 [-16.04, -9.7]	CI excludes 0
10	knnvc_khmer_pool_samegender minus knnvc_khmer_...	EER semi_informed	-12.09	-15.75	-8.17	-12.09 [-15.58, -8.35]	CI excludes 0

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short draw the 11 differences as a forest plot: dot = observed difference, line = 95 % interval, red if the interval excludes 0.

• Row label = "second - first | metric". The dashed vertical line is zero.
• Positive EER difference = the second system is more private; negative CER difference = the second system keeps content better.

What the output means

In short one forest plot with 11 rows; 9 red dots and 2 grey ones (the lazy row of the first pair and the ignorant row of the third pair).

• The red / grey colour comes from the evidence column of the previous cell.

fig, ax = plt.subplots(figsize=(10, 6))
# label = "second - first": the key is "<second> minus <first>"
ylab = [f"{SHORT[c.split(' minus ')[0]]} - {SHORT[c.split(' minus ')[1]]}  |  {m}" for c, m in zip(paired_tab.comparison, paired_tab.metric)]
y = np.arange(len(paired_tab))[::-1]
colors = ["C3" if e == "CI excludes 0" else "gray" for e in paired_tab.evidence]
ax.errorbar(paired_tab.delta, y, xerr=[paired_tab.delta - paired_tab.ci_low, paired_tab.ci_high - paired_tab.delta],
            fmt="none", capsize=4, ecolor="gray")
ax.scatter(paired_tab.delta, y, c=colors, s=60, zorder=3)
ax.axvline(0, ls="--", c="k", lw=1)
ax.set(yticks=y, yticklabels=ylab, xlabel="difference (second - first), percentage points, 95 % paired speaker-bootstrap CI",
       title="Forest plot of pre-registered comparisons (red = CI excludes 0)")
ax.grid(axis="x", alpha=0.3); plt.tight_layout(); plt.show()

Additional Reading

How to read this plot

In short (EERs from shipped embeddings, CERs from shipped transcripts) a line that crosses the dashed zero = no evidence of a difference. kNN-VC km vs McAdams: every row excludes 0. kNN-VC km vs en: CER and semi-informed EER exclude 0, the ignorant EER is at the edge.

Each row is one difference (second system minus first); the dot is the observed difference, the line is the 95 % CI. A line that crosses the dashed zero = no evidence of a difference.

• kNN-VC km - kNN-VC en. The ignorant-EER difference (+3.3) has a CI of [0.02, 6.28] in this run: it just excludes 0. The shipped 2 000-rep CI is [-0.47, 6.96] and includes it. With an edge at 0.02 the verdict can flip between runs, so read it as not established.

  The semi-informed difference (+5.5, shipped CI [1.32, 9.05]) and the CER difference (-3.1, shipped [-6.25, -0.05]) exclude 0, though the CER edge is close to 0. The lazy row includes 0 here (-0.26) and in the shipped run (-0.83).

  A careful sentence is: "the Khmer pool improves CER by about 3 points and semi-informed EER by about 5 points; the ignorant-EER gain is not established".

• kNN-VC km - McAdams. Every CI is far from 0 in this run and in the shipped run (EER +14 / +21 / +29, CER -14; lower edges 11.2, 18.1, 26.5 and an upper CER edge of -7.7 here). This one you may call a winner without hedging.

• kNN-VC km same-gender - kNN-VC km. Ignorant-EER difference -3.3 with a CI that includes 0 ([-8.59, 3.95]); lazy and semi-informed differences about -12.8 and -12.1 with CIs that clearly exclude 0. Keeping gender costs about 12 points of privacy against informed attackers. The CER difference is not in this run; the shipped file says +0.14 [-4.42, 4.92] (shipped paired_comparisons.json value, not recomputed here), which includes 0.

The shipped column shows the project's 2 000-rep numbers (shipped paired_comparisons.json value, not recomputed here). The signs agree in every row. Eight of the 11 rows agree within 1 point at both edges; the largest edge difference is 3.09 points (same-gender minus Khmer pool, ignorant, upper edge) (computed separately, not printed by the cells). Two of the three ignorant rows are not clear (the first at the edge, the third includes 0), while the lazy and semi-informed rows are clearer. We did not check why.

6. Linkability and the "two EERs trap"

Privacy has two faces. Linkability asks: can the attacker tell that two anonymized utterances come from the same person (even without knowing who)? Identification asks: can the attacker link an anonymized utterance to a known original voice? The ignorant attacker measures identification against original voices; the lazy and semi-informed attackers measure linkability inside the anonymized world, which is what matters once the attacker has the system.

A system with one fixed pseudo-voice per speaker can make the ignorant EER look like a coin flip while linkability stays easy (its lazy EER is small). That is the trap: a paper reports the first number as "privacy". Our leaderboard holds a textbook case: SSL-SAS, 50.7 % ignorant but 12.8 % lazy and 9.8 % semi-informed.

Why it matters: an EER without its attacker is not a privacy claim.

Visualization Version

Seven systems under three attackers, and the drop from ignorant to semi-informed. All values come from the shipped leaderboard.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short take 7 systems from the shipped leaderboard (3 of them not recomputed here) and draw one line per system through its three attacker EERs.

• trap = the eer_ignorant, eer_lazy, eer_semi_informed columns of leaderboard.csv for McAdams, kNN-VC km, SSL-SAS, SSL-SAS Khmer pool, SSL-SAS English pool variant, Private kNN-VC and OpenVoice, plus the drop ignorant → semi.
• These are shipped values, not recomputed; the other cells of the notebook recompute only the 8 systems with embeddings.

What the output means

In short one line chart with 7 lines.

• The numbers are in the next cell.

trap = board.loc[["mcadams", "knnvc_khmer_pool", "sslsas", "sslsas_khmer_pool", "sslsas_english_pool_variant", "private_knnvc_english_durations", "openvoice"],
                 ["eer_ignorant", "eer_lazy", "eer_semi_informed"]].copy()
trap["drop ignorant -> semi"] = (trap.eer_ignorant - trap.eer_semi_informed).round(1)
fig, ax = plt.subplots(figsize=(10, 5))
for s, r in trap.iterrows():
    ax.plot([0, 1, 2], [r.eer_ignorant, r.eer_lazy, r.eer_semi_informed], "-o", label=s)
    ax.annotate(s, (2, r.eer_semi_informed), xytext=(6, 0), textcoords="offset points", fontsize=8, va="center")
ax.axhline(50, ls="--", c="gray", lw=1)
ax.set(xticks=[0, 1, 2], xticklabels=["ignorant", "lazy-informed", "semi-informed"], ylabel="EER (%)", xlim=(-0.2, 2.9),
       title="The same system under three attackers (leaderboard values)"); ax.grid(alpha=0.3)
plt.tight_layout(); plt.show()

Additional Reading

How to read this plot

In short each line is one system; the three x positions are the three attackers. A flat line (kNN-VC km) = privacy independent of attacker knowledge. A line that falls off a cliff (SSL-SAS) is the trap.

Each line is one system; the three x positions are the three attackers, left to right the attacker learns more. All values are from leaderboard.csv (shipped leaderboard.csv value, not recomputed here).

A flat line is a system whose privacy does not depend on the attacker's knowledge: kNN-VC km, 44.0 → 46.7 → 43.5.

A line that falls off a cliff is the trap: SSL-SAS 50.7 → 12.8 → 9.8, and its English-pool variant 45.3 → 4.7 → 4.0. Against an attacker who owns the system the EER falls to 4.0-12.8 %, much lower than the ignorant EER (45.3-50.7 %). McAdams falls more gently (29.6 → 25.3 → 14.2) and so does OpenVoice (29.0 → 30.2 → 20.0). Private kNN-VC loses 11.3 points between ignorant and lazy (48.1 → 36.8) and then holds (36.5).

The course plot labels the lines with the system names of the CSV (for example knnvc_khmer_pool = kNN-VC km) and has no legend. Its colours differ from the picture above, so use the labels.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short print the same 7 systems as numbers, with the drop from the ignorant to the semi-informed EER.

• trap.round(1): one decimal.
• The cell prints the table already used by the line chart.

What the output means

In short drop ignorant → semi: SSL-SAS English-pool variant 41.3, SSL-SAS 40.9, SSL-SAS Khmer pool 37.2, McAdams 15.4, Private kNN-VC 11.6, OpenVoice 9.0, kNN-VC km 0.5.

• The three SSL-SAS systems lose 37 to 41 points. kNN-VC km loses 0.5 (43.98 → 43.49 in the shipped file).

trap.round(1)

	eer_ignorant	eer_lazy	eer_semi_informed	drop ignorant -> semi
system
mcadams	29.6	25.3	14.2	15.4
knnvc_khmer_pool	44.0	46.7	43.5	0.5
sslsas	50.7	12.8	9.8	40.9
sslsas_khmer_pool	43.5	7.4	6.3	37.2
sslsas_english_pool_variant	45.3	4.7	4.0	41.3
private_knnvc_english_durations	48.1	36.8	36.5	11.6
openvoice	29.0	30.2	20.0	9.0

Additional reading: you can read all of it or just read the "In short"

How to report

In short print all three EERs per system; never fill a missing semi-informed value with the ignorant one.

Rule for your reports (the course's advice): always print the whole row. If you have to choose one number, choose the semi-informed EER. If the semi-informed attacker was not run, say so explicitly, as the leaderboard does with a blank cell for asr_tts, cosyvoice, seedvc and mcadams_tuned_khmer (four blank cells in leaderboard.csv (computed separately, not printed by the cells)), and never fill the gap with the ignorant number.

7. Listening test design

UTMOS is a prediction. The only ground truth for "does it sound natural" and "does it still sound like her" is a human. For Khmer no MOS predictor was trained on Khmer listeners (background fact from the course text, not checked here). The course says a small test with 3 to 5 listeners and the 20 demo clips is enough to catch a bad predictor (its opinion).

Two formats:

• MOS (naturalness): the listener hears one clip, blind, and rates 1 (very unnatural) to 5 (completely natural). Average per system. Needs a hidden reference (the original) and, ideally, a hidden anchor (a clearly degraded clip) so listeners use the whole scale.
• ABX / AB preference (speaker similarity or naturalness): the listener hears A and B (the same sentence through two systems, blind, random order) and picks the one that sounds more natural, or the one that sounds more like reference X (the original). Preference tests are easier for untrained listeners and less sensitive to scale use.

Rules that keep the test honest: randomize the order of clips and of A/B, never show the system name, keep every listener's sheet different, do not let listeners hear the same sentence twice in a row, and decide the analysis (mean MOS per system with a bootstrap CI over listeners × sentences, sign test for AB) before collecting ratings.

Why it matters: this is the check on UTMOS, and the cell below builds both sheets so you can run it.

Visualization Version

The two sheets, a smoke test with SIMULATED random ratings (not real listeners), and the rules.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short build a MOS sheet and an AB sheet for listener L01, with answer keys; define the scorers; and test the scorer with simulated random ratings.

• make_listening_sheets writes 4 CSV files into outputs/day7/: the MOS sheet and its key, the AB sheet and its key. Everything is seeded, so the files are the same on every run.
• MOS sheet: 20 demo clips × (6 anonymized systems + the original) = 140 items, shuffled. AB sheet: 20 trials, each with a reference X and two clips from two different systems, random side.
• score_mos: mean rating per system and a bootstrap interval over items. score_ab: win rate per system.
• The smoke test fills the sheet with random integers 1 to 5. These are not listeners.

What the output means

In short MOS sheet: 140 items; AB sheet: 20 trials; written to ../outputs/day7; the first 3 AB trials; a table of simulated MOS per system; one audio player.

• The simulated MOS is between 2.80 (McAdams) and 3.35 (kNN-VC en), with n = 20 per system and 95 % interval half-widths between 0.50 and 0.65 (computed separately, not printed by the cells). Random ratings have a true mean of 3, so these are noise.
• The audio player plays clip_A of trial T000.
• Re-running rewrites the four CSV files with the same content (checked by comparing the files before and after).

def make_listening_sheets(listener_id, seed, n_ab=20, out=OUT):
    rng = np.random.default_rng(seed)
    # MOS sheet: every demo clip x every system + the original, shuffled, blind ids
    items = [(u, "original", DATA / "audio/demo" / f"{u}.wav") for u in demo.utt_id]
    items += [(u, s, DATA / "audio/anonymized" / s / f"{u}.wav") for s in AUDIO_SYSTEMS for u in demo.utt_id]
    order = rng.permutation(len(items))
    mos = pd.DataFrame([{"listener": listener_id, "item": f"M{i:03d}", "path": str(items[j][2]), "rating_1_to_5": ""} for i, j in enumerate(order)])
    mos_key = pd.DataFrame([{"item": f"M{i:03d}", "utt_id": items[j][0], "system": items[j][1]} for i, j in enumerate(order)])
    # AB sheet: same sentence, two random distinct systems, random A/B side
    ab, ab_key = [], []
    for i in range(n_ab):
        u = demo.utt_id.iloc[i % len(demo)]
        s1, s2 = rng.choice(AUDIO_SYSTEMS, 2, replace=False)
        ab.append({"listener": listener_id, "trial": f"T{i:03d}", "reference_X": str(DATA / "audio/demo" / f"{u}.wav"),
                   "clip_A": str(DATA / "audio/anonymized" / s1 / f"{u}.wav"), "clip_B": str(DATA / "audio/anonymized" / s2 / f"{u}.wav"),
                   "more_natural_A_or_B": "", "more_like_X_A_or_B": ""})
        ab_key.append({"trial": f"T{i:03d}", "utt_id": u, "system_A": s1, "system_B": s2})
    ab, ab_key = pd.DataFrame(ab), pd.DataFrame(ab_key)
    mos.to_csv(out / f"listening_mos_{listener_id}.csv", index=False); mos_key.to_csv(out / f"listening_mos_{listener_id}_KEY.csv", index=False)
    ab.to_csv(out / f"listening_ab_{listener_id}.csv", index=False); ab_key.to_csv(out / f"listening_ab_{listener_id}_KEY.csv", index=False)
    return mos, mos_key, ab, ab_key

def score_mos(sheet, key):
    # mean rating per system with a bootstrap CI over items (one listener) -> extend to listeners x items with more sheets
    df = sheet.merge(key, on="item"); df["rating_1_to_5"] = pd.to_numeric(df.rating_1_to_5, errors="coerce")
    rng = np.random.default_rng(0)
    rows = []
    for s, g in df.dropna().groupby("system"):
        r = g.rating_1_to_5.values
        b = [rng.choice(r, len(r)).mean() for _ in range(500)]
        rows.append({"system": s, "n": len(r), "MOS": r.mean(), "ci_low": np.percentile(b, 2.5), "ci_high": np.percentile(b, 97.5)})
    return pd.DataFrame(rows).set_index("system").round(2)

def score_ab(sheet, key, column="more_natural_A_or_B"):
    # win rate per system over the trials it took part in; a sign test needs ~20 trials per pair to be useful
    df = sheet.merge(key, on="trial"); df = df[df[column].isin(["A", "B"])]
    wins = {}
    for _, r in df.iterrows():
        w, l = (r.system_A, r.system_B) if r[column] == "A" else (r.system_B, r.system_A)
        wins.setdefault(w, [0, 0]); wins.setdefault(l, [0, 0]); wins[w][0] += 1; wins[w][1] += 1; wins[l][1] += 1
    return pd.DataFrame({s: {"wins": w, "trials": n, "win_rate": w / n} for s, (w, n) in wins.items()}).T.sort_values("win_rate", ascending=False)

mos, mos_key, ab, ab_key = make_listening_sheets("L01", seed=1)
print("MOS sheet:", len(mos), "items;  AB sheet:", len(ab), "trials;  written to", OUT)
display(ab.head(3)[["trial", "clip_A", "clip_B"]])

# smoke test of the scorers with simulated ratings (replace with a real filled sheet)
fake = mos.copy(); fake["rating_1_to_5"] = np.random.default_rng(0).integers(1, 6, len(fake))
fake_ab = ab.copy(); fake_ab["more_natural_A_or_B"] = np.random.default_rng(0).choice(["A", "B"], len(ab))
print("simulated MOS (random ratings, so every system ~3):"); display(score_mos(fake, mos_key))
play(load_audio(ab.clip_A.iloc[0]))

MOS sheet: 140 items;  AB sheet: 20 trials;  written to ../outputs/day7

	trial	clip_A	clip_B
0	T000	../data/audio/anonymized/knnvc_khmer_pool/slr4...	../data/audio/anonymized/mcadams/slr42_7548_34...
1	T001	../data/audio/anonymized/asr_tts/slr42_7548_04...	../data/audio/anonymized/knnvc_english_pool/sl...
2	T002	../data/audio/anonymized/mcadams/slr42_3154_41...	../data/audio/anonymized/asr_tts/slr42_3154_41...

simulated MOS (random ratings, so every system ~3):

	n	MOS	ci_low	ci_high
system
asr_tts	20	3.20	2.70	3.80
knnvc_english_pool	20	3.35	2.70	3.95
knnvc_khmer_pool	20	2.90	2.37	3.45
mcadams	20	2.80	2.15	3.40
original	20	3.15	2.55	3.73
private_knnvc_english_durations	20	3.20	2.65	3.75
sslsas	20	2.95	2.30	3.50

Additional reading: you can read all of it or just read the "In short"

Reading the smoke test

In short random ratings give intervals of about ±0.6 for 20 items; one listener cannot separate systems that are half a point apart.

Listen to trial T000's clip A above, then open listening_ab_L01.csv next to it.

Note what the simulated scores show: with random ratings every system lands between 2.80 and 3.35, with an interval of about ±0.6 from only 20 items per system. With ratings this noisy, a difference of about half a MOS point is smaller than the interval. The course says that with 5 listeners the interval shrinks by roughly √5 (a standard result for independent ratings, not tested here), and that MOS papers report 10 to 20 listeners (background fact from the course text, not checked here). That is why UTMOS, imperfect as it is, is still worth having for ranking many systems cheaply before a human test on the top few.

Build: mini leaderboard generator

Everything above, packed into one function. make_leaderboard(systems) recomputes the three attacker EERs (where the embeddings exist) and the overall CER, each with a speaker-bootstrap CI. It also computes the phone and studio CER and ρF0 on the demo clips (no CI for these three), copies UTMOS from the shipped leaderboard, writes outputs/day7/leaderboard_recomputed.csv and a Markdown table, and returns both. The EERs come from shipped ECAPA embeddings and the CER from shipped transcripts. The next cell compares six columns with the shipped leaderboard (the three EERs, the CER, and the two ignorant-attacker CI edges). plot_pareto was defined in section 4.

Why it matters: one function, one command, the whole benchmark again.

Visualization Version

What the function takes in and writes out, and how far its results are from the shipped leaderboard.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short define make_leaderboard, run it for the 8 systems, write the CSV and Markdown files, and show the table.

• For each system: EER and interval for each attacker with embeddings; CER and interval if a transcripts column exists; phone and studio CER; ρF0 and duration ratio on the demo clips if audio exists; UTMOS and ρF0 from the leaderboard.
• It writes outputs/day7/leaderboard_recomputed.csv (rounded to 3 decimals) and leaderboard_recomputed.md. These two files are rewritten with identical content on every run (checked by comparing the files before and after).
• UTMOS is shipped, as in section 3.

What the output means

In short an 8-row table such as kNN-VC km: EER 44.0 [41.4, 46.5] / 46.7 [44.3, 49.1] / 43.5 [40.3, 46.0], CER 24.4 [18.9, 31.4], phone/studio 27.9 / 17.0, ρF0 (20) -0.49, UTMOS* 2.39.

• kNN-VC km same-g has n.a. for CER and ρF0 (no transcripts, no audio). ASR->TTS has n.a. for the semi-informed EER.
• original has no ρF0 on the demo clips (n.a.): it is the reference.
• The line make_leaderboard: ... s shows the time; it changes from machine to machine.

def make_leaderboard(systems, n_boot=N_BOOT, seed=0, out=OUT):
    draws = speaker_draws(n_boot, seed)
    rows = []
    for s in systems:
        r = {"system": s}
        for att in ["ignorant", "lazy", "semi_informed"]:
            if s in scores[att]:
                pt, lo, hi, _ = eer_boot(scores[att][s], draws=draws)
                r[f"eer_{att}"], r[f"eer_{att}_ci_low"], r[f"eer_{att}_ci_high"] = pt, lo, hi
        if f"hyp_{s}" in tx.columns:
            pt, lo, hi, _ = cer_boot(s, draws=draws)
            r["cer"], r["cer_ci_low"], r["cer_ci_high"] = pt, lo, hi
            r["cer_phone"] = cer_boot(s, "phone (iAny)", draws)[0]; r["cer_studio"] = cer_boot(s, "studio (OpenSLR 42)", draws)[0]
        if s in pros_tab.index:
            r["rho_f0_demo20"] = pros_tab.loc[s, "rho_f0_demo20"]; r["duration_ratio"] = pros_tab.loc[s, "duration_ratio"]
        r["utmos_board"] = board.loc[s, "utmos"]; r["rho_f0_board"] = board.loc[s, "rho_f0"]
        r["n_speakers"] = len(SPEAKERS); r["n_same_gender_trials"] = int(sg.sum()); r["n_boot"] = n_boot
        rows.append(r)
    lb = pd.DataFrame(rows).set_index("system")
    lb.round(3).to_csv(out / "leaderboard_recomputed.csv")
    show = pd.DataFrame(index=lb.index)
    for att, short in [("ignorant", "ign"), ("lazy", "lazy"), ("semi_informed", "semi")]:
        show[f"EER {short}"] = [f"{r[f'eer_{att}']:.1f} [{r[f'eer_{att}_ci_low']:.1f}, {r[f'eer_{att}_ci_high']:.1f}]" if pd.notna(r.get(f"eer_{att}")) else "n.a." for _, r in lb.iterrows()]
    show["CER"] = [f"{r.cer:.1f} [{r.cer_ci_low:.1f}, {r.cer_ci_high:.1f}]" if pd.notna(r.get("cer")) else "n.a." for _, r in lb.iterrows()]
    show["CER phone/studio"] = [f"{r.cer_phone:.1f} / {r.cer_studio:.1f}" if pd.notna(r.get("cer")) else "n.a." for _, r in lb.iterrows()]
    show["rho_F0 (20)"] = [f"{r.rho_f0_demo20:.2f}" if pd.notna(r.get("rho_f0_demo20")) else "n.a." for _, r in lb.iterrows()]
    show["UTMOS*"] = lb.utmos_board.round(2)
    show.index = [SHORT.get(s, s) for s in show.index]
    (out / "leaderboard_recomputed.md").write_text("# Recomputed leaderboard (Day 7)\n\n" + show.to_markdown() + "\n\n*UTMOS read from the shipped leaderboard (needs the predictor model).\n")
    return lb, show

t0 = time.time()
lb, lb_md = make_leaderboard(SYSTEMS)
print(f"make_leaderboard: {time.time()-t0:.1f} s -> {OUT/'leaderboard_recomputed.csv'} and .md")
lb_md

make_leaderboard: 9.9 s -> ../outputs/day7/leaderboard_recomputed.csv and .md

	EER ign	EER lazy	EER semi	CER	CER phone/studio	rho_F0 (20)	UTMOS*
original	2.4 [1.4, 3.5]	2.4 [1.4, 3.5]	3.0 [1.7, 3.9]	7.6 [5.9, 9.4]	6.9 / 9.1	n.a.	2.40
McAdams	29.6 [26.3, 33.0]	25.3 [21.6, 28.3]	14.2 [10.2, 16.6]	38.4 [33.1, 44.7]	43.8 / 26.8	0.92	1.64
kNN-VC en	40.7 [38.4, 43.5]	43.7 [41.4, 46.0]	37.9 [35.4, 40.9]	27.6 [22.5, 33.9]	32.3 / 17.4	-0.13	2.52
kNN-VC km	44.0 [41.4, 46.5]	46.7 [44.3, 49.1]	43.5 [40.3, 46.0]	24.4 [18.9, 31.4]	27.9 / 17.0	-0.49	2.39
kNN-VC km same-g	40.7 [36.0, 46.8]	34.0 [31.2, 36.3]	31.4 [28.7, 33.7]	n.a.	n.a.	n.a.	2.34
ASR->TTS	45.3 [42.4, 47.7]	50.0 [47.5, 52.5]	n.a.	31.3 [29.4, 33.2]	30.0 / 33.9	0.32	2.49
SSL-SAS	50.7 [46.1, 55.5]	12.8 [11.1, 14.7]	9.8 [7.7, 12.0]	29.6 [22.8, 38.1]	35.5 / 17.0	0.92	2.93
Private kNN-VC	48.1 [44.2, 51.8]	36.8 [34.1, 39.5]	36.5 [33.5, 39.5]	113.6 [105.3, 123.6]	121.5 / 96.8	0.60	1.23

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short compare the recomputed leaderboard with the shipped one: the largest absolute difference per column; and draw plot_pareto for the recomputed ignorant table.

• Columns checked: eer_ignorant, eer_lazy, eer_semi_informed, cer, and the lower and upper interval edge of the ignorant EER.
• The cell prints its own explanation of the CI gap ("different seed, 1000 vs 2000 reps, percentile noise"); we did not test it.

What the output means

In short max |recomputed - shipped|: eer_ignorant 0.05, eer_lazy 0.04, eer_semi_informed 0.03, cer 0.04, ign_ci_low 1.62, ign_ci_high 2.04 percentage points.

• The point estimates agree within 0.05 points; the printed sentence says "agree to rounding", but the shipped values have two decimals, so 0.05 is a small real difference.
• The interval edges differ by up to about 2 points.
• The plot is the same Pareto plot as in section 4, ignorant attacker only; the timing line at the end changes between machines.

# agreement with the shipped leaderboard: max absolute difference per column
chk = pd.DataFrame({
    "eer_ignorant": (lb.eer_ignorant - board.eer_ignorant.reindex(lb.index)).abs(),
    "eer_lazy": (lb.eer_lazy - board.eer_lazy.reindex(lb.index)).abs(),
    "eer_semi_informed": (lb.eer_semi_informed - board.eer_semi_informed.reindex(lb.index)).abs(),
    "cer": (lb.cer - board.cer.reindex(lb.index)).abs(),
    "ign_ci_low": (lb.eer_ignorant_ci_low - board.eer_ignorant_ci_low.reindex(lb.index)).abs(),
    "ign_ci_high": (lb.eer_ignorant_ci_high - board.eer_ignorant_ci_high.reindex(lb.index)).abs(),
})
print("max |recomputed - shipped| per column (percentage points):")
print(chk.max().round(2).to_string())
print("\nPoint estimates agree to rounding; CI edges differ by up to ~2 points (different seed, 1000 vs 2000 reps, percentile noise).")
fig, ax = plt.subplots(figsize=(8, 6))
plot_pareto(privacy_utility_table("ignorant"), "ignorant", ax, title="plot_pareto(): recomputed leaderboard, ignorant attacker")
plt.tight_layout(); plt.show()
print(f"total runtime so far: {(time.time()-T0)/60:.1f} min")

max |recomputed - shipped| per column (percentage points):
eer_ignorant         0.05
eer_lazy             0.04
eer_semi_informed    0.03
cer                  0.04
ign_ci_low           1.62
ign_ci_high          2.04

Point estimates agree to rounding; CI edges differ by up to ~2 points (different seed, 1000 vs 2000 reps, percentile noise).

total runtime so far: 0.4 min

Additional Reading

How to read this plot

In short the same privacy-utility plot as in section 4 (ignorant attacker), now drawn from lb, the table made by make_leaderboard.

Same layout as the left panel of section 4: x = ignorant EER, y = CER, red = Pareto front, the dashed line = 50 %. The front is original → kNN-VC km → SSL-SAS, as before. The point positions are the same because make_leaderboard uses the same scores; this plot is a check that the function reproduces the table.

8. The one-page project summary

Pipeline. Khmer test set: 43 speakers (32 phone from iAny, 11 studio from OpenSLR 42), 129 enrollment + 430 trial utterances, 10 090 same-gender trials. Each system anonymizes the 430 trials (and, for the informed attackers, the 129 enrollments). Privacy: ECAPA-TDNN cosine scoring → EER under three attackers; semi-informed uses an ECAPA fine-tuned on that system's output (background fact from the course text, not checked here). Content: Khmer ASR judge → corpus CER. Quality: UTMOS. Prosody: Spearman ρ between original and anonymized median F0. All CIs: speaker bootstrap; the course says 2 000 reps for the leaderboard intervals (the shipped paired file says n_boot 2000); we use 1 000.

Why it matters: this is the page you would show an advisor. Every number below is either recomputed in this run from shipped files, or a shipped value from leaderboard.csv or paired_comparisons.json, and we say which.

Visualization Version

Four headline systems under three attackers (this run), and what the paired tests allow us to say.

Additional reading: you can read all of it or just read the "In short"

The numbers, the differences, the one-sentence result

In short kNN-VC km is the best trade in this table; SSL-SAS is the trap; the table below is the shipped leaderboard.

Numbers to know (leaderboard.csv, same-gender trials; EER ignorant / lazy / semi-informed, CER, UTMOS, ρF0). All values below are the shipped leaderboard values (shipped leaderboard.csv value, not recomputed here). Eight rows are systems we recompute: our values agree with them within 0.05 points for EERs and 0.04 for CER (where we have transcripts). Four rows (SSL-SAS Khmer pool, Private kNN-VC Khmer durations, VoicePrivacy 2024 B5, OpenVoice) are not recomputed here; we only checked that the table agrees with the shipped file.

system	EER ign	EER lazy	EER semi	CER %	UTMOS	ρF0
original	2.4	2.4	3.0	7.6	2.40	1.00
McAdams (VoicePrivacy baseline)	29.6	25.3	14.2	38.4	1.64	0.73
kNN-VC, English pool	40.6	43.7	38.0	27.6	2.52	0.67
kNN-VC, Khmer pool	44.0	46.7	43.5	24.4	2.39	0.61
kNN-VC, Khmer pool, same gender	40.7	34.0	31.4	24.6	2.34	0.63
ASR→TTS (Qwen3-ASR-Khmer + MMS-TTS)	45.3	50.0	–	31.3	2.49	0.32
SSL-SAS (as released)	50.7	12.8	9.8	29.6	2.93	0.70
SSL-SAS, Khmer pool	43.5	7.4	6.3	27.0	2.49	0.73
Private kNN-VC (English durations)	48.1	36.8	36.5	113.6	1.23	0.18
Private kNN-VC (Khmer durations)	48.1	36.7	37.4	49.1	1.91	0.32
VoicePrivacy 2024 B5	46.5	48.8	46.8	102.7	1.83	0.61
OpenVoice	29.0	30.2	20.0	34.4	2.12	0.75

Note the one difference to our run: the shipped kNN-VC English-pool values are 40.6 and 38.0; our recompute gives 40.66 and 37.93 (40.7 and 37.9 in the cell).

Pre-registered differences (shipped 2 000-rep paired bootstrap (shipped paired_comparisons.json value, not recomputed here)): Khmer pool − English pool: CER −3.1 [−6.3, −0.1] (the CER upper edge is only −0.05, close to 0), semi EER +5.5 [1.3, 9.1]; kNN-VC km − McAdams: semi EER +29.3 [25.5, 33.9], CER −13.9 [−20.7, −7.3]; same-gender − random-gender Khmer pool: semi EER −12.1 [−15.6, −8.4]. Section 5 recomputes these with 1 000 reps and gets the same values within about 1 point at the edges.

What each system is (from the shipped descriptions in leaderboard.csv, the code in src/anonymize.py and the course text).

• McAdams: LPC pole-angle warp (α ~ U(0.5, 0.9) per utterance), no training.
• kNN-VC: the course says WavLM layer-6 features of the source are replaced by their nearest neighbours in a target speaker's pool and decoded by a HiFi-GAN vocoder (background fact from the course text, not checked here); pools of 40 English or 17 Khmer voices.
• ASR→TTS: transcribe, then synthesize with one MMS voice.
• SSL-SAS: HuBERT content + F0 + a pseudo-speaker embedding, decoded by a HiFi-GAN vocoder (background fact from the course text, not checked here).
• Private kNN-VC: kNN-VC with a phone recognizer and a duration model so timing does not leak (background fact from the course text, not checked here).
• VPC B5: ASR bottleneck + vector quantization + HiFi-GAN.

The one-sentence result (the course's conclusion). On Khmer, kNN-VC with a Khmer target pool is the best trade in this table: EER about 44 % under every attacker (44.0, 46.7, 43.5), CER 24 % (vs 7.6 % on the original audio). The system with the best-looking ignorant EER, SSL-SAS (50.7 %), falls to 12.8 % (lazy) and 9.8 % (semi-informed) once the attacker can enroll anonymized speech.

Additional reading: you can read all of it or just read the "In short"

Still open (research directions)

In short five open questions: pool design, prosody, the semi-informed gap, a better judge, human listening.

1. Khmer pool design: 17 voices is small; does privacy or CER change with pool size, pool gender balance, or pool recording channel (phone vs studio)? The same-gender result says gender matching costs about 12 EER points against informed attackers; what about channel matching?
2. Prosody anonymization: the three kNN-VC systems keep ρF0 of 0.61 to 0.67. A pitch-shift or F0-contour transform on top of kNN-VC that keeps Khmer intonation but changes the speaker's range is untested.
3. The semi-informed gap: kNN-VC km loses only 0.5 points to the semi-informed attacker (43.96 → 43.49), Private kNN-VC loses 11.6 (48.13 → 36.51); both recomputed from shipped embeddings. Why? Is the duration model leaking rhythm?
4. A better Khmer ASR judge: the judge has 7.6 % CER on the original (not anonymized) audio, and the phone/studio gap compounds after anonymization (section 3). A judge fine-tuned on anonymized Khmer (as VoicePrivacy does for its ASR (background fact from the course text, not checked here)) would separate "content destroyed" from "judge confused".
5. Human listening: no Khmer MOS data exists (background fact from the course text, not checked here); the 20-clip sheets from section 7 with 5 Khmer listeners would calibrate UTMOS for this language.

Additional reading: you can read all of it or just read the "In short"

Ten advisor questions (with model answers)

In short ten advisor questions with short answers; each number is either recomputed from shipped files or a shipped value (see the first line).

Where the numbers come from. EER and CER values below are recomputed in this notebook from the shipped embeddings and transcripts; ρF0 and UTMOS are leaderboard.csv values; the paired intervals are from the shipped paired_comparisons.json.

1. Why is your headline EER the semi-informed one and not the 50.7 % of SSL-SAS? Because the ignorant attacker measures its own confusion; an attacker owning the system gets 9.8 % on SSL-SAS. We report all three and rank by semi-informed.
2. Why same-gender trials only? Cross-gender non-targets are free rejects for systems that keep gender. Including them changes the ignorant EER by between −8.8 and +3.3 points depending on the system (McAdams −5.6, SSL-SAS −8.8, but ASR→TTS +3.3) (computed separately, not printed by the cells), without any speaker information. (The same-gender EER is 3.3 points lower to 8.8 points higher than the all-trials EER.)
3. How did you get the confidence intervals? Speaker bootstrap: resample the 43 speakers with replacement, weight each trial by how often its enrolled speaker was drawn, 2.5 / 97.5 percentiles (the course says 2 000 reps for the leaderboard; 1 000 here). A trial-level bootstrap treats trials as independent although they share 43 speakers; the course says it is too narrow, and in our one check it was not narrower than the one-sided speaker bootstrap (section 1), so we simply use the speaker level.
4. Is the Khmer pool really better than the English pool? Paired bootstrap on the same speaker draws: CER −3.1 and semi-informed EER +5.5 exclude 0 (shipped CIs [−6.3, −0.1] and [1.3, 9.1]; the CER upper edge is only −0.05, close to 0); the ignorant-EER gain (+3.3) is borderline: [0.02, 6.28] here, [−0.5, 7.0] in the shipped run. We pre-registered this comparison.
5. Why corpus-level CER and not the mean per utterance? It is the standard in ASR evaluation (background fact from the course text, not checked here) and matches the leaderboard. In our data the mean per utterance is within 1 point of it (section 3) (computed separately, not printed by the cells).
6. What does a CER above 100 % mean? More edits than reference characters. Private kNN-VC with its English phone recognizer does this (113.6 %); the course says its output is not Khmer (course explanation, not tested here).
7. Can you trust UTMOS on Khmer? No; it is a predictor trained on English and Chinese (background fact from the course text, not checked here). We use it to rank, and we designed a 20-clip MOS/AB test with Khmer listeners to calibrate it.
8. What is ρF0 and why does it matter? Spearman correlation of median pitch before and after. High ρ means prosody and gender survive: good for naturalness, and in the leaderboard it goes with lower semi-informed EER (correlation −0.61 over 16 systems, −0.53 without original (computed separately, not printed by the cells); a pattern, not a cause), with kNN-VC km as the exception.
9. What is the lazy-informed attacker, and why is SSL-SAS's lazy EER so low? The attacker anonymizes the enrollment speech with the same system. SSL-SAS gives each speaker one fixed pseudo-voice (shipped description), so the anonymized utterances of one speaker are easy to link although identification against the original voice fails (course explanation, not tested here).
10. What is the single number you would defend? kNN-VC km: semi-informed EER 43.5 % (interval [40.3, 46.0], about ±3 points) with CER 24.4 % (interval [18.9, 31.4]). It is on the Pareto front under all three attackers (the lazy front is original, kNN-VC km, ASR→TTS) (computed separately, not printed by the cells).