← All days

Day 4 · Who is speaking? Speaker embeddings, cosine trials, EER and DET curves: the attacker

Goal in one line: turn a clip of speech into one vector that says who is talking, decide "same person or not" with a cosine score and a threshold, and measure how often that decision is wrong (EER).

What you will build: a speaker-verification toolkit (src/verify.py, six functions: build_trials, speaker_models, score_trials, eer_ci, det_curve, min_dcf), a handcrafted speaker embedding, a tiny CNN speaker model trained from scratch, and the privacy evaluation of every anonymization system in this project, reproduced from the shipped ECAPA-TDNN embeddings and checked against data/meta/leaderboard.csv.

Time: ~3 h

• Embedding: one vector (a list of numbers) that summarizes a clip. Clips of the same speaker should give vectors that point in nearly the same direction.
• Cosine score: how much two vectors point in the same direction, from -1 to +1.
• Trial: one (enrolled speaker, test clip) pair with a known answer: target (same person) or non-target (different people).
• EER (equal error rate): the error rate at the threshold where false accepts and false rejects are equal.

By the end you can ...

• explain verification vs identification, and why a threshold is the whole game;
• build a speaker embedding by hand (MFCC + F0 statistics) and read a cosine-similarity heatmap;
• build target / non-target trials and read a score histogram;
• implement EER, FAR/FRR curves, a DET curve and minDCF from scratch;
• train a small classification-based speaker model and take embeddings from its penultimate layer;
• explain what ECAPA-TDNN adds and why a pretrained model is so much better than yours;
• run the ignorant / lazy / semi-informed attackers on the real test set and get the leaderboard numbers with 95 % confidence intervals.

How to read this notebook (for the team)

You see	What it is	Do you need it?
Purple heading	a new section and its explanation	yes: read the bold lines and the "Why it matters" line
Visualization Version	pictures (one is animated) that explain the section	yes: the first image of each section is the fastest way in
Additional reading	what the next code cell does and what its output means, plus extra notes	optional: read just the In short lines if you are busy
code cell + output	the real computation on 48 probe clips, 20 demo clips and the 559-clip test set	run it, look at the result

Fast path (≈ 40 min): purple text → first Visualization image → "In short" lines. Full path (≈ 3 h): read everything and run every cell top to bottom.

The numbers in the explanations come from the run saved in this file. Seeds are fixed, so a re-run on the same setup should give the same numbers. Timings (seconds) change from machine to machine and are not quoted.

Important, section 6 (ECAPA-TDNN): the pretrained model cannot be downloaded on the machine where this file was built (the cell prints ProxyError: 403 Forbidden). As designed in the cell, it then falls back to the shipped embeddings in data/embeddings/original.npz. So the ECAPA numbers in section 6 (and the probe check at the end of the Build section) are computed from that shipped file, not from a fresh ECAPA run. (Section 7 uses shipped .npz embeddings by design.) Every code cell ran here, top to bottom, with no errors.

1. The big idea: verification is "same voice or not", decided by a threshold

 enrollment clips ──► encoder ──► embedding e₁ ──┐
 (known speaker)                                  │   cosine
                                                  ├──► s = cos(e₁, e₂) ──► s ≥ θ ?  ──► "same speaker" / "different"
 trial clip ───────► encoder ──► embedding e₂ ──┘                            │
 (unknown speaker)                                                            └─ θ = threshold

Identification asks "which of these N people is it?" (a softmax over N names). Verification asks "is this the person they claim to be?" (a yes / no). Verification does not need the speaker in any training set: any new voice is just a new point in embedding space.

The recipe has three parts and only three:

1. Encoder: audio → one fixed-size vector (the speaker embedding). A good encoder maps all clips of one speaker close together and clips of different speakers far apart, whatever the words, the microphone or the mood.
2. Score: cosine similarity $s = \dfrac{e_1 \cdot e_2}{\|e_1\|\,\|e_2\|} \in [-1, 1]$. With L2-normalized embeddings (length 1) this is a plain dot product.
3. Threshold $\theta$: accept if $s \ge \theta$. Move $\theta$ up and you reject more impostors, but also more genuine speakers. Every error number in this notebook is a statement about this trade-off.

A trial is one (enrollment speaker, trial clip) pair with a known answer: target (same speaker) or non-target (different). A test set is a list of thousands of trials. Two kinds of mistakes: a false accept (non-target scored above $\theta$) and a false reject (target scored below $\theta$). The equal error rate (EER) is the error at the threshold where the two rates are equal.

In our research. This notebook is the attacker. The anonymization systems of this project (McAdams and kNN-VC are built on Day 6) try to hide who is speaking; a speaker-verification system is what an adversary would use to undo that. We run the attacker on anonymized trial clips and report its EER. Read it backwards from the usual meaning: a low EER means the attacker wins (the voice still links to the person), 50 % means the attacker is guessing (perfect privacy on this test), and anything above 50 % is just noise around the guessing line. The EER is the privacy score of the whole project.

Visualization Version

The picture shows the three parts, and where four real systems sit on the privacy ruler.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short load the libraries (NumPy, pandas, librosa, PyTorch, SciPy), set paths and random seeds, and import the Day 1 helpers.

• from src.utils import ...: the Day 1 helpers (load, logmel, eer, f0_track, ...). eer is used all through this notebook.
• np.random.seed(0); torch.manual_seed(0) fix the seeds so the training result in section 5 can be repeated.
• OUT = ../outputs/day4 is where the toolkit table is saved at the end.

What the output means

In short one line with the library versions: torch 2.14.1+cu130 | librosa 1.0.0 | threads 4.

• If this cell fails, nothing below runs. The exact version numbers do not matter.

%matplotlib inline
import os, sys, time, warnings, json, subprocess
os.environ.setdefault("OMP_NUM_THREADS", "4")      # keep CPU use sane before numpy/torch load
sys.path.insert(0, "..")
warnings.filterwarnings("ignore")

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import librosa
import torch
import torch.nn as nn
from pathlib import Path
from scipy.stats import norm
from IPython.display import display

from src.utils import play, show_wave, show_spec, median_f0, f0_track, logmel, load, eer

SR = 16000
DATA = Path("../data")
OUT = Path("../outputs/day4"); OUT.mkdir(parents=True, exist_ok=True)
np.random.seed(0); torch.manual_seed(0)
torch.set_num_threads(4)
pd.set_option("display.width", 160); pd.set_option("display.precision", 2)
print("torch", torch.__version__, "| librosa", librosa.__version__, "| threads", torch.get_num_threads())

torch 2.14.1+cu130 | librosa 1.0.0 | threads 4

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short load the 48 probe clips (8 speakers × 6 sentences), read all wave files into memory, and play three examples.

• data/audio/probe/ holds 8 phone speakers × 6 sentences = 48 clips. Every speaker reads the same 6 sentences. That design lets us separate "same voice" from "same words".
• The three players: two different speakers saying the same sentence (a female and a male), then the first speaker saying a different sentence.

What the output means

In short 48 clips | 8 speakers | 6 sentences; 4 female and 4 male speakers, 6 clips each; mean clip length per speaker from 2.72 s to 4.92 s.

• Short clips (2.72 s and 2.82 s) belong to two of the female speakers.
• The three audio players are embedded: press play to listen.

probe = pd.read_csv(DATA / "meta/probe_clips.csv").sort_values(["speaker", "sentence_id"]).reset_index(drop=True)
probe["path"] = probe.utt_id.map(lambda u: DATA / "audio/probe" / f"{u}.wav")
speakers = sorted(probe.speaker.unique()); sentences = sorted(probe.sentence_id.unique())
print(len(probe), "clips |", len(speakers), "speakers |", len(sentences), "sentences")
print(probe.groupby("speaker").agg(gender=("gender", "first"), clips=("utt_id", "size"), mean_dur_s=("duration", "mean")))
wavs = {u: load(p) for u, p in zip(probe.utt_id, probe.path)}   # all 48 clips in memory (~5 MB)
# Listen: two different speakers saying the SAME sentence, then the same speaker saying a different one.
r0, r1 = probe[probe.sentence_id == sentences[0]].iloc[0], probe[probe.sentence_id == sentences[0]].iloc[4]
r2 = probe[(probe.speaker == r0.speaker) & (probe.sentence_id == sentences[3])].iloc[0]
print(r0.text); display(play(wavs[r0.utt_id])); print(r0.speaker, r0.gender, "| same sentence, other speaker:", r1.speaker, r1.gender)
display(play(wavs[r1.utt_id])); print(r0.speaker, "again, different sentence:", r2.text); display(play(wavs[r2.utt_id]))

48 clips | 8 speakers | 6 sentences
                 gender  clips  mean_dur_s
speaker                                   
iany-s-046e9e11  female      6        2.82
iany-s-18151a80  female      6        4.60
iany-s-19ed77bb  female      6        4.92
iany-s-1f8f808c  female      6        2.72
iany-s-21c36386    male      6        4.12
iany-s-49b4ab7e    male      6        4.78
iany-s-600bbcad    male      6        4.43
iany-s-707a4c7a    male      6        4.80

ពួកយើងលេងបាល់ទាត់នៅទីលានក្រោយម៉ោងរៀន

iany-s-046e9e11 female | same sentence, other speaker: iany-s-21c36386 male

iany-s-046e9e11 again, different sentence: អ្នកជំនាញព្យាករណ៍ថានឹងមានភ្លៀងធ្លាក់នៅសប្ដាហ៍ក្រោយ

2. A speaker embedding by hand (no deep model)

An embedding is just "a vector that summarizes the clip". The simplest honest one: compute MFCCs (Day 1; 19 coefficients per 10 ms frame, we drop $c_0$ because it is only loudness) and keep their mean and standard deviation over time. The mean MFCCs capture the long-term spectral shape: where the formants of this mouth tend to sit. The std captures how much the speaker moves around. Add the mean and std of $\log F_0$ (pitch, Day 1) and you have a 40-d vector (19 + 19 + 2).

Worked example (checked): a male speaker with $F_0 = 110$ Hz gives $\ln 110 = 4.70$. A female at 210 Hz gives $\ln 210 = 5.35$. One number already separates most men from most women. In our 8 speakers it is not perfect: the printed values below show one female speaker at 115 Hz, lower than two of the males (125 and 127 Hz). The 38 MFCC statistics do the rest.

Two details that matter:

• Standardize each dimension (z-score) across clips before the cosine. Otherwise the dimension with the biggest raw scale decides everything. In our data the first MFCC mean has a spread of 15.2, the log F0 mean only 0.40 and the log F0 std only 0.14 (printed below): about 40 to 110 times smaller, so F0 would be nearly invisible.
• Cosine on the standardized vector: $s = \hat{z}_1 \cdot \hat{z}_2$ with $\hat{z} = z / \|z\|$.

Why it matters: this is a simple, classic baseline. It shows what a weak embedding leaks: a small trace of the words, not only the voice.

Visualization Version

The 40-d vector, its 48×48 cosine matrix, and the mean cosine of three kinds of pairs.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short turn each of the 48 clips into a 40-d vector, z-score it, make it length 1, and compute the 48 × 48 cosine matrix S_hand.

• handcrafted_embedding: 19 MFCCs (c1..c19) with the course STFT settings, plus the pitch track from pyworld (0 = unvoiced frame, ignored). Output: 19 means + 19 stds + 2 pitch numbers = 40.
• Z: each of the 40 dimensions is z-scored over the 48 clips. H: each row is scaled to length 1. S_hand = H @ H.T is the cosine of every pair of clips.
• The last two lines print the raw spread of 5 dimensions, and one pitch number per speaker.

What the output means

In short features (48, 40). Raw spread: MFCC means 15.2, 13.0, 12.9, log F0 mean 0.40, log F0 std 0.14. Per-speaker pitch: 93 to 296 Hz.

• Without z-scoring, F0 would be about 40 to 110 times smaller than MFCC 1 (0.40 and 0.14 vs 15.2).
• Pitch per speaker (Hz): females 184, 115, 296, 203; males 101, 125, 127, 93. The code label says "median", but the line computes exp(mean of log F0), which is a geometric mean, not a median.
• One female speaker (115 Hz) is lower than two males (125, 127 Hz): pitch alone does not separate all speakers.

def handcrafted_embedding(x):
    # 19 MFCCs (c1..c19; c0 dropped = loudness, not identity) with the course STFT settings
    m = librosa.feature.mfcc(y=x, sr=SR, n_mfcc=20, n_fft=512, hop_length=160, win_length=400, n_mels=80)[1:]
    f0, _ = f0_track(x, SR)                       # pyworld pitch per 10 ms frame, 0 = unvoiced
    lf0 = np.log(f0[f0 > 0]) if (f0 > 0).sum() > 5 else np.array([np.log(120.0)])
    return np.concatenate([m.mean(1), m.std(1), [lf0.mean(), lf0.std()]]).astype(np.float32)   # 19+19+2 = 40-d

t0 = time.time()
F = np.stack([handcrafted_embedding(wavs[u]) for u in probe.utt_id])        # (48, 40) raw features
print("features", F.shape, f"in {time.time()-t0:.1f} s")
Z = (F - F.mean(0)) / (F.std(0) + 1e-8)                                     # z-score each dimension over the 48 clips
H = Z / np.linalg.norm(Z, axis=1, keepdims=True)                            # unit length -> cosine = dot product
S_hand = H @ H.T                                                            # 48 x 48 cosine similarity matrix

# Notice: raw mean MFCC1 spans tens of units, log-F0 std spans ~0.1. Without z-scoring, F0 would be invisible.
print("raw scale per dim (first 3 MFCC means, then log-F0 mean/std):", np.round(F.std(0)[[0, 1, 2, 38, 39]], 3))
print("median log-F0 per speaker:", {s: round(float(np.exp(F[probe.speaker == s, 38].mean())), 0) for s in speakers})

features (48, 40) in 3.1 s
raw scale per dim (first 3 MFCC means, then log-F0 mean/std): [15.237 13.049 12.856  0.401  0.137]
median log-F0 per speaker: {'iany-s-046e9e11': 184.0, 'iany-s-18151a80': 115.0, 'iany-s-19ed77bb': 296.0, 'iany-s-1f8f808c': 203.0, 'iany-s-21c36386': 101.0, 'iany-s-49b4ab7e': 125.0, 'iany-s-600bbcad': 127.0, 'iany-s-707a4c7a': 93.0}

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short draw two cosine heatmaps and print the mean cosine of three kinds of pairs, with and without each speaker's mean.

• Left: the 48 × 48 matrix, rows sorted by speaker. Right: the same embeddings after subtracting each speaker's mean vector (the "residual"), sorted by sentence. If the residual still has structure, that structure is "what was said", not "who".
• The table at the end is the mean cosine over off-diagonal pairs of each kind.

What the output means

In short same speaker, different sentence +0.67; different speaker, same sentence -0.08; different speaker, different sentence -0.10. After removing the speaker mean: -0.19 / +0.08 / -0.02.

• The speaker part dominates the full embedding (+0.67 vs about -0.1).
• After mean removal, same-sentence pairs of different speakers are slightly above different-sentence pairs (+0.08 vs -0.02): the words leave a trace.
• The -0.19 is not a finding. When you subtract a mean of 6 clips, the residuals of the same speaker become slightly negatively related. For 6 independent clips the expected value is -1/(6-1) = -0.2 (arithmetic, not an output). The printed -0.19 is this effect.

def heatmap(S, order, labels, title, ax, vlim=1.0):
    img = ax.imshow(S[np.ix_(order, order)], cmap="magma", vmin=-vlim, vmax=vlim)
    # thin white lines between groups, group names at block centres
    lab = np.array(labels)[order]; edges = np.where(lab[1:] != lab[:-1])[0] + 0.5
    for e in edges: ax.axhline(e, color="w", lw=0.5); ax.axvline(e, color="w", lw=0.5)
    centres = [np.mean(np.where(lab == g)[0]) for g in dict.fromkeys(lab)]
    ax.set_xticks(centres); ax.set_xticklabels([g.replace("iany-s-", "") for g in dict.fromkeys(lab)], rotation=90, fontsize=7)
    ax.set_yticks(centres); ax.set_yticklabels([g.replace("iany-s-", "") for g in dict.fromkeys(lab)], fontsize=7)
    ax.set_title(title, fontsize=10); return img

fig, axes = plt.subplots(1, 2, figsize=(12, 5.5))
by_spk = np.lexsort((probe.sentence_id.to_numpy(str), probe.speaker.to_numpy(str)))   # rows grouped by speaker
by_sent = np.lexsort((probe.speaker.to_numpy(str), probe.sentence_id.to_numpy(str)))  # rows grouped by sentence
# Residual = embedding minus its speaker's mean embedding. What is left cannot be "who"; if it still has structure, it is "what".
spk_arr = probe.speaker.to_numpy(str)
R = Z - np.stack([Z[spk_arr == s].mean(0) for s in spk_arr]); R /= np.linalg.norm(R, axis=1, keepdims=True)
S_resid = R @ R.T
heatmap(S_hand, by_spk, spk_arr, "handcrafted cosine, sorted by SPEAKER", axes[0])
img = heatmap(S_resid, by_sent, probe.sentence_id.to_numpy(str), "speaker mean removed, sorted by SENTENCE (colour range ±0.5)", axes[1], vlim=0.5)
fig.colorbar(img, ax=axes[1], label="cosine similarity (right panel)", shrink=0.8); plt.show()

# Numbers behind the picture: average cosine for three kinds of pairs (off-diagonal only)
same_spk = spk_arr[:, None] == spk_arr[None, :]
same_sent = probe.sentence_id.to_numpy(str)[:, None] == probe.sentence_id.to_numpy(str)[None, :]
off = ~np.eye(48, dtype=bool)
pairs = {"same speaker, different sentence": same_spk & ~same_sent & off,
         "different speaker, SAME sentence": ~same_spk & same_sent,
         "different speaker, different sentence": ~same_spk & ~same_sent}
pair_means = pd.DataFrame({"full embedding": {k: S_hand[v].mean() for k, v in pairs.items()},
                           "speaker mean removed": {k: S_resid[v].mean() for k, v in pairs.items()}})
print(pair_means.round(3))

                                       full embedding  speaker mean removed
same speaker, different sentence                 0.67                 -0.19
different speaker, SAME sentence                -0.08                  0.08
different speaker, different sentence           -0.10                 -0.02

Additional Reading

How to read this plot

In short left: 8 bright squares = same speaker (+0.67). Right: after removing the speaker, faint structure by sentence remains (+0.08 vs -0.02). Content leaks into naive features.

Left, sorted by speaker. The bright 6 × 6 squares along the diagonal are "same speaker" pairs (mean cosine +0.67). Off the diagonal it is dark: "different speaker" (mean about -0.1). The first four speakers are female and the last four male.

Different-speaker pairs score a little higher within a gender (mean -0.01) than across genders (mean -0.17) (computed separately, not printed by the cells). This is why the top-left and bottom-right quarters look a little warmer. The reason ($\log F_0$ and the low MFCCs split gender before they split identity) is the course explanation, not tested here.

In the full embedding, "different speaker, same sentence" pairs score only slightly above "different speaker, different sentence" (-0.08 vs -0.10): the speaker part dominates, so the content leak is hidden.

Right: subtract each speaker's mean vector and look at what is left, sorted by sentence. The 8 × 8 squares along the diagonal are faint and noisy; the colour range is ±0.5 to make them visible.

The printed means are the better evidence: +0.08 for same-sentence pairs vs -0.02 for different sentences. Per sentence, the mean same-sentence cosine is about +0.12 for daily-04 and numbers-01, +0.11 for daily-10, +0.07 for health-04 and news-03, and only +0.01 for news-05 (computed separately, not printed by the cells). So the leak is not the same for every sentence.

Content leaks into naive features. Mean MFCCs over a 2 to 6 s clip (the probe clips are 2.2 to 5.9 s (computed separately, not printed by the cells)) still remember which sounds were in the sentence. It is small here because every probe sentence is a few seconds of mixed phonemes.

The course text says it gets large for short clips and for a single word; we did not test that here. A good speaker encoder has to throw the content away.

ECAPA (shipped embeddings, not a fresh run) in section 6 shows a smaller trace: +0.06 vs -0.01, a gap of about 0.075 (against about 0.10 here), but not zero.

3. Trials: target and non-target pairs, and the score histogram

A verification test is a list of trials. Each enrolled speaker gets a speaker model: the average of their enrollment embeddings (3 clips here), scaled back to length 1. Averaging cancels the per-clip noise (words, breath, a cough) and keeps the part that repeats: the voice. Every remaining clip is then scored against every speaker model:

• enrollment: 3 clips per speaker, 8 speakers → 8 speaker models;
• trial clips: the other 3 clips of each speaker = 24 clips;
• trials: 8 models × 24 clips = 192 trials, of which 24 are target (model and clip from the same person) and 168 non-target.

Score $= \cos(\text{model}, \text{clip})$. Then we draw two histograms, targets and non-targets. Why it matters: the overlap of the two histograms is the error. No threshold can separate what overlaps.

Visualization Version

The 8 × 24 trial table (orange = target) and the two real score histograms.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short define build_trials and score_trials, make the 192 trials from the handcrafted embeddings, and print the score statistics.

• build_trials: sorts each speaker's clips by sentence_id; the first 3 are enrollment, the other 3 are trial clips. Here that means enrollment = daily-04, daily-10, health-04 and trials = news-03, news-05, numbers-01 for every speaker (the sentence order of the heatmap labels above). Then it lists every (enrollment speaker, trial clip) pair.
• score_trials: the speaker model is the length-1 mean of the 3 enrollment embeddings; the score is its dot product with the trial embedding.

What the output means

In short 192 trials: 24 target, 168 non-target. Target scores: mean 0.76, min 0.32. Non-target scores: mean -0.11, max 0.41.

• The lowest target (0.32) is below the highest non-target (0.41), so the two histograms overlap a little.
• Enrollment and trial sentences are different, so a high score cannot come from identical sentences.

def build_trials(utts, n_enroll=3):
    # utts: DataFrame with utt_id, speaker, sentence_id. The first n_enroll sentences of each speaker enroll, the rest are trials.
    u = utts.sort_values(["speaker", "sentence_id"])
    rank = u.groupby("speaker").cumcount()
    u = u.assign(role=np.where(rank < n_enroll, "enroll", "trial"))
    trial_clips = u[u.role == "trial"]
    rows = [(e, t.utt_id, t.speaker, "target" if e == t.speaker else "nontarget")
            for e in sorted(u.speaker.unique()) for t in trial_clips.itertuples()]
    return u, pd.DataFrame(rows, columns=["enroll_speaker", "trial_utt", "trial_speaker", "label"])

def score_trials(trials, utts, emb):
    # emb: dict utt_id -> unit vector. Speaker model = L2-normalised mean of that speaker's enrollment embeddings.
    enr = utts[utts.role == "enroll"]
    models = {s: np.mean([emb[u] for u in g.utt_id], 0) for s, g in enr.groupby("speaker")}
    models = {s: m / np.linalg.norm(m) for s, m in models.items()}
    return np.array([models[e] @ emb[t] for e, t in zip(trials.enroll_speaker, trials.trial_utt)])

emb_hand = dict(zip(probe.utt_id, H))
utts3, trials3 = build_trials(probe, n_enroll=3)
scores3 = score_trials(trials3, utts3, emb_hand)
labels3 = (trials3.label == "target").values.astype(int)
print(len(trials3), "trials:", labels3.sum(), "target,", (1 - labels3).sum(), "non-target")
tar, non = scores3[labels3 == 1], scores3[labels3 == 0]
print(f"target scores: mean {tar.mean():.2f}  min {tar.min():.2f} | non-target: mean {non.mean():.2f}  max {non.max():.2f}")

192 trials: 24 target, 168 non-target
target scores: mean 0.76  min 0.32 | non-target: mean -0.11  max 0.41

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short compute the EER with src.utils.eer and draw the two score histograms with the threshold.

• eer(scores, labels) returns the EER in % and the threshold θ where it happens.
• Grey = non-target (168), orange = target (24), dashed line = the EER threshold.

What the output means

In short EER = 4.2 % at θ = 0.34.

• 4.2 % is 1 of 24 targets. Section 4 explains why it is exactly 1/24.

eer_hand, thr_hand = eer(scores3, labels3)
fig, ax = plt.subplots(figsize=(12, 3))
bins = np.linspace(-1, 1, 41)
ax.hist(non, bins, alpha=0.6, label=f"non-target (n={len(non)})", color="tab:gray")
ax.hist(tar, bins, alpha=0.7, label=f"target (n={len(tar)})", color="tab:orange")
ax.axvline(thr_hand, color="k", ls="--", label=f"EER threshold θ = {thr_hand:.2f}")
ax.set(title=f"handcrafted embedding: score histograms (EER = {eer_hand:.1f} %)", xlabel="cosine score", ylabel="count"); ax.legend(); plt.show()

Additional Reading

How to read this plot

In short grey = impostors, orange = genuine. The dashed line is the threshold; the overlap near 0.3 to 0.4 is where the errors are (EER 4.2 %).

Grey = non-target scores, centred around -0.1 (printed mean -0.11) and reaching up to +0.41; orange = target scores. 23 of the 24 targets are at +0.6 or higher (highest 0.93) (computed separately, not printed by the cells), with one straggler at +0.32. The dashed line (θ = 0.34) is the threshold where the two error rates are equal. Every orange bar left of the line is a false reject (a genuine speaker turned away); every grey bar right of it is a false accept (an impostor let in). The overlap region, +0.3 to +0.4 here, is narrow: that is why the EER is only 4.2 % even for a 40-d handcrafted vector on 8 phone speakers.

With only 24 targets, one target clip is 4.2 % of the false-reject rate: small test sets give coarse EERs. Keep that in mind when section 7 moves to 10 090 trials.

4. EER by hand: FAR, FRR, the crossing, DET curves, minDCF

Slide a threshold $\theta$ from $-1$ to $+1$ and count, at each position,

$$\mathrm{FAR}(\theta) = \frac{\#\{\text{non-target with } s \ge \theta\}}{\#\text{non-target}}, \qquad \mathrm{FRR}(\theta) = \frac{\#\{\text{target with } s < \theta\}}{\#\text{target}}.$$

FAR starts at 100 % (accept everyone) and falls; FRR starts at 0 % and rises. They cross once. The EER is the value at that crossing. Why the crossing? Because it is the one operating point that needs no cost model: it does not favour "being let in" over "keeping impostors out". It is a single number you can compare across systems.

What the numbers mean: 0 % = the two histograms do not overlap at all; 50 % = the score carries no speaker information, you might as well flip a coin. For the attacker, 50 % is the privacy ceiling. For scale: the ECAPA attacker gets 2.4 % on the original (not anonymized) test speech in section 7 (same-gender trials, shipped embeddings).

minDCF adds a cost model for the cases where errors are not equally bad: $\mathrm{DCF}(\theta) = C_{miss} P_{tar} \mathrm{FRR}(\theta) + C_{fa} (1 - P_{tar}) \mathrm{FAR}(\theta)$, minimised over $\theta$ and divided by the cost of the best trivial system (always accept or always reject). With $P_{tar} = 0.05$ and $C_{miss} = C_{fa} = 1$ (said to be the VoicePrivacy setting; background fact, not checked here) a false accept costs $0.95 / 0.05 = 19$ times a miss. So minDCF rewards systems whose non-target tail is short. minDCF = 1.0 means "no better than the trivial system".

Visualization Version

The 1st image is the summary. The animation shows the threshold sliding over the same 192 scores.

Animated: the threshold slides over the real 192 scores; false accepts (red) and false rejects (blue) are counted

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short write the threshold sweep by hand (far_frr_curve), the EER (eer_by_hand) and min_dcf, and compare with src.utils.eer.

• far_frr_curve: 2001 thresholds from -1 to 1; for each one, the share of non-targets with score ≥ θ (FAR) and of targets with score < θ (FRR).
• eer_by_hand: the grid point where FAR and FRR are closest; the EER is their average.
• min_dcf: p_tar * FRR + (1 - p_tar) * FAR, smallest value, divided by min(p_tar, 1 - p_tar) = 0.05.

What the output means

In short EER by hand 4.17 % at θ = 0.320; utils.eer 4.17 % at θ = 0.336; minDCF = 0.042.

• Same EER, different θ: utils.eer looks only at observed scores, the hand version at a grid. Any θ in the flat part gives the same error.
• 4.17 % = 1/24. At θ = 0.32: 7 of 168 non-targets are accepted (4.17 %) and 1 of 24 targets is rejected (4.17 %) (computed separately, not printed by the cells).
• minDCF 0.042 is also 1/24: the best point has no false accepts (θ above 0.41) and rejects the one low target; its cost is FRR + 19 × FAR = 0.042 + 0 (computed separately, not printed by the cells).

def far_frr_curve(scores, labels, n_thr=2001):
    # Sweep thresholds and count errors at each. Vectorised: (n_thr, n_trials) boolean matrix.
    thr = np.linspace(-1, 1, n_thr)
    tar, non = scores[labels == 1], scores[labels == 0]
    far = (non[None, :] >= thr[:, None]).mean(1)      # impostors accepted
    frr = (tar[None, :] < thr[:, None]).mean(1)       # genuine rejected
    return thr, far, frr

def eer_by_hand(scores, labels):
    thr, far, frr = far_frr_curve(scores, labels)
    i = np.argmin(np.abs(far - frr))                  # the crossing
    return 100 * (far[i] + frr[i]) / 2, thr[i]

def min_dcf(scores, labels, p_tar=0.05, c_miss=1.0, c_fa=1.0):
    thr, far, frr = far_frr_curve(scores, labels)
    dcf = c_miss * p_tar * frr + c_fa * (1 - p_tar) * far
    return dcf.min() / min(c_miss * p_tar, c_fa * (1 - p_tar))   # 1.0 = no better than always-accept / always-reject

e_hand, t_hand = eer_by_hand(scores3, labels3)
e_util, t_util = eer(scores3, labels3)
print(f"EER by hand  : {e_hand:.2f} % at θ = {t_hand:.3f}")
print(f"EER utils.eer: {e_util:.2f} % at θ = {t_util:.3f}   (utils sweeps the observed scores instead of a grid -> same error, θ may differ a little)")
print(f"minDCF (P_tar=0.05): {min_dcf(scores3, labels3):.3f}")

EER by hand  : 4.17 % at θ = 0.320
EER utils.eer: 4.17 % at θ = 0.336   (utils sweeps the observed scores instead of a grid -> same error, θ may differ a little)
minDCF (P_tar=0.05): 0.042

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short plot FAR and FRR against the threshold.

• Grey = FAR (impostors accepted), orange = FRR (genuine rejected). The black dot marks the EER.

What the output means

In short FAR falls from 100 % to 0 %, FRR rises from 0 % to 100 %; the crossing is at EER = 4.2 %.

• Compare with the animation above: the same two curves, drawn all at once.

thr, far, frr = far_frr_curve(scores3, labels3)
fig, ax = plt.subplots(figsize=(12, 3))
ax.plot(thr, 100 * far, label="FAR: impostors accepted", color="tab:gray")
ax.plot(thr, 100 * frr, label="FRR: genuine rejected", color="tab:orange")
ax.axvline(t_hand, color="k", ls="--", lw=0.8); ax.axhline(e_hand, color="k", ls=":", lw=0.8)
ax.annotate(f"EER = {e_hand:.1f} %", (t_hand, e_hand), xytext=(t_hand + 0.1, e_hand + 25), arrowprops=dict(arrowstyle="->"))
ax.set(title="FAR and FRR as the threshold slides", xlabel="threshold θ (cosine)", ylabel="error rate (%)", xlim=(-1, 1)); ax.legend(); plt.show()

Additional Reading

How to read this plot

In short FAR falls, FRR rises, they cross at the EER. A flat crossing like this one means the histograms barely overlap.

Grey FAR falls from 100 % at $\theta = -1$ (everyone accepted) to 0 % once the threshold is above the highest non-target score. Orange FRR rises from 0 % to 100 % once the threshold passes the highest target score. Where they cross is the EER. Here both curves are at or below 4.2 % between $\theta \approx 0.3$ and $0.4$ (FRR 0 → 4.2 %, FAR 4.2 → 1.2 %) (computed separately, not printed by the cells), and the EER is 4.2 %: the two histograms barely overlap. A wide, shallow crossing would mean that the choice of threshold matters a lot and no threshold is good. In deployment you rarely sit at the EER point: a bank wants low FAR (far right), a voice assistant wants low FRR (further left) (background fact, not checked here).

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short define det_curve (FAR and FRR at every observed score) and plot_det (probit axes), and draw the DET curve of the 192 trials.

• det_curve: sort the scores from high to low; walking down the list, the cumulative count of targets and non-targets gives FRR and FAR at every possible threshold.
• plot_det: draws norm.ppf(FAR) against norm.ppf(FRR) (inverse normal CDF) and labels the axes in %.

What the output means

In short one DET curve (handcrafted, EER 4.2 %) with the dotted FAR = FRR diagonal.

• The curve is a short horizontal step: from θ = 0.32 up to θ ≈ 0.61 exactly one target of 24 is rejected (computed separately, not printed by the cells), while FAR falls.
• Points with FAR = 0 or FRR = 0 cannot be drawn on probit axes (they would be at minus infinity), so the code leaves them out.

def det_curve(scores, labels):
    # Every observed score is a threshold -> (FAR, FRR) pairs, monotone, ready for probit axes.
    scores, labels = np.asarray(scores, float), np.asarray(labels, int)
    order = np.argsort(-scores); l = labels[order]
    tp, fp = np.cumsum(l), np.cumsum(1 - l)
    far = fp / max((1 - l).sum(), 1); frr = 1 - tp / max(l.sum(), 1)
    return far, frr

def plot_det(curves, ax, title="DET curve"):
    ticks = np.array([0.1, 0.5, 1, 2, 5, 10, 20, 40, 60]) / 100
    for name, (far, frr) in curves.items():
        m = (far > 0) & (frr > 0)
        ax.plot(norm.ppf(far[m]), norm.ppf(frr[m]), label=name)
    ax.plot(norm.ppf(ticks), norm.ppf(ticks), color="k", lw=0.5, ls=":")   # FAR = FRR diagonal: EER lives here
    ax.set_xticks(norm.ppf(ticks)); ax.set_xticklabels([f"{100*t:g}" for t in ticks])
    ax.set_yticks(norm.ppf(ticks)); ax.set_yticklabels([f"{100*t:g}" for t in ticks])
    ax.set(xlim=norm.ppf([0.001, 0.7]), ylim=norm.ppf([0.001, 0.7]), xlabel="false accept rate (%)", ylabel="false reject rate (%)", title=title)
    ax.grid(alpha=0.3); ax.legend(loc="upper right")

fig, ax = plt.subplots(figsize=(5.5, 5.5))
plot_det({f"handcrafted (EER {e_hand:.1f} %)": det_curve(scores3, labels3)}, ax, "DET curve, handcrafted embedding, 192 probe trials")
plt.show()

Additional Reading

How to read this plot

In short a DET curve is the ROC curve on probit axes. Closer to the bottom-left = better verification; for the attacker, closer to the top-right = more private.

A DET curve is the ROC curve drawn on probit axes (scipy.stats.norm.ppf): each axis is stretched so that Gaussian score distributions give a straight line (a standard result, not tested here). Left end = strict threshold (few false accepts, many false rejects), right end = loose threshold. The dotted diagonal is FAR = FRR; where the curve crosses it, you read the EER off either axis. Closer to the bottom-left = better verification. For the attacker, closer to the top-right = more private. The very coarse look here (one flat step) comes from the small test set (24 targets: each false reject is a 4.2 % step). Section 7 has 430 targets and the curves become smooth.

5. Train a tiny speaker model from scratch

Day 2's CNN classified clips (by sentence and by speaker). Today we reuse the same idea with one change: a 64-d bottleneck layer just before the softmax. Train it to name the speaker (8 classes) on 4 sentences per speaker, then throw the softmax away and use the 64-d activation as the embedding. If the model learned "who", not "what", clips of one speaker reading unseen sentences will still sit close together.

 log-mel (80 x T) ─► Conv-BN-ReLU-pool x3 ─► mean over time (frequency kept: 64 x 10 = 640) ─► Linear 640→64  ═► embedding (64-d)
                                                                                                └─► Linear 64→8 ─► softmax (training only)

What the results will show: the tiny model is perfect on the speakers it trained on, but much worse on people it has never heard (section 5 results below).

Classification-trained embeddings vs metric learning, and what ECAPA-TDNN adds (background fact from the course text, not checked here):

• A softmax classifier only needs the 8 training speakers to be separable; it does not care how far apart they are or whether unseen speakers will be. Metric-learning losses such as AAM-softmax add an angular margin $m$ so that the cosine to the correct class must beat the others by a margin. ECAPA-TDNN, x-vectors and most modern speaker encoders train with such a loss over thousands of speakers (VoxCeleb: 7 000+ speakers).
• ECAPA-TDNN (Desplanques et al., 2020), as the course text describes it: TDNN layers (1-D convolutions over time) → SE-Res2Blocks (Res2 = split the channels into groups that see wider and wider time spans; SE = squeeze-excitation, re-weight channels by a global summary) → concatenate all block outputs → attentive statistics pooling (learned attention weights per frame, then weighted mean and std) → Linear → 192-d embedding, trained with AAM-softmax (the course text gives m = 0.2, s = 30; the number of Res2 groups is given as 4).
• Our tiny net has plain mean pooling (every frame counts equally, silence included), one scale and 8 training speakers. ECAPA has attention over frames, several time scales, channel re-weighting and 7 000+ speakers of training data. Expect a large gap.

Visualization Version

The network, its real training log, and the EER of the handcrafted vector vs the tiny CNN.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short prepare log-mel features, load 20 more clips of 10 new speakers, define the network SpkNet, and train it for 15 epochs on random 1.5 s crops.

• Sentences 0 to 3 of each probe speaker train the model (32 clips); sentences 4 and 5 are held out (16 clips). The 20 demo clips are 10 speakers × 2 clips; none of them is among the 8 probe speakers (checked (computed separately, not printed by the cells)).
• CROP = 150 frames = 1.5 s: each step the net sees a different random window, a cheap data augmentation.
• SpkNet: three Conv-BN-ReLU-MaxPool blocks (16, 32, 64 channels), mean over time, Linear(640, 64) (the embedding), Linear(64, 8) (the head, thrown away later). Adam, learning rate 2e-3, batch 32.

What the output means

In short 32 training clips | 16 held-out clips. Over 15 epochs: loss 1.813 → 0.004, training accuracy 41.0 % → 100.0 %.

• Accuracy is at least 98.4 % from epoch 4 on. The loss never rises (it stays at 0.006 for epochs 13 and 14).
• This accuracy is measured on random crops of the training clips. It says the net has learned the 8 training speakers, nothing about new people.
• The time per run is printed in the cell; it changes from machine to machine.

# ---- data: log-mel per clip, normalised per clip; sentences 0..3 train, 4..5 held out
# Also load the 20 demo clips (10 speakers x 2 clips, none of them among the 8 probe speakers): the UNSEEN-SPEAKER test.
demo = pd.read_csv(DATA / "meta/demo_clips.csv").sort_values(["speaker", "utt_id"]).reset_index(drop=True)
for u in demo.utt_id: wavs[u] = load(DATA / "audio/demo" / f"{u}.wav")
mels = {u: logmel(wavs[u], SR) for u in list(probe.utt_id) + list(demo.utt_id)}  # (80, T) dB
mels = {u: ((m - m.mean()) / (m.std() + 1e-6)).astype(np.float32) for u, m in mels.items()}
spk_to_idx = {s: i for i, s in enumerate(speakers)}
train_df = probe[probe.sentence_id.isin(sentences[:4])]; held_df = probe[probe.sentence_id.isin(sentences[4:])]
print(len(train_df), "training clips (4 sentences x 8 speakers) |", len(held_df), "held-out clips (2 unseen sentences x 8 speakers)")

CROP = 150   # 1.5 s random crops: cheap data augmentation, the net never sees the same window twice
def random_crop(m):
    if m.shape[1] <= CROP: return np.pad(m, ((0, 0), (0, CROP - m.shape[1])))
    i = np.random.randint(0, m.shape[1] - CROP + 1); return m[:, i:i + CROP]

class SpkNet(nn.Module):
    def __init__(self, n_spk=8, emb_dim=64):
        super().__init__()
        def block(i, o): return nn.Sequential(nn.Conv2d(i, o, 3, padding=1), nn.BatchNorm2d(o), nn.ReLU(), nn.MaxPool2d(2))
        self.conv = nn.Sequential(block(1, 16), block(16, 32), block(32, 64))   # (1,80,T) -> (64,10,T/8)
        self.embed = nn.Linear(64 * 10, emb_dim)                                 # the layer we keep
        self.head = nn.Linear(emb_dim, n_spk)                                    # the layer we throw away
    def embedding(self, x):                      # x: (B, 1, 80, T) any T
        h = self.conv(x).mean(dim=3)             # mean over time -> (B, 64, 10)
        return self.embed(h.flatten(1))          # (B, 64)
    def forward(self, x): return self.head(torch.relu(self.embedding(x)))

net = SpkNet(); opt = torch.optim.Adam(net.parameters(), lr=2e-3); lossf = nn.CrossEntropyLoss()
Xtr = [mels[u] for u in train_df.utt_id]; ytr = torch.tensor([spk_to_idx[s] for s in train_df.speaker])
EPOCHS, CROPS_PER_CLIP, BS = 15, 8, 32
t0 = time.time()
for ep in range(EPOCHS):
    net.train(); idx = np.random.permutation(len(Xtr) * CROPS_PER_CLIP); tot = 0; correct = 0
    for b in range(0, len(idx), BS):
        ii = idx[b:b + BS] % len(Xtr)
        xb = torch.tensor(np.stack([random_crop(Xtr[i]) for i in ii]))[:, None]; yb = ytr[ii]
        out = net(xb); loss = lossf(out, yb)
        opt.zero_grad(); loss.backward(); opt.step()
        tot += loss.item() * len(ii); correct += (out.argmax(1) == yb).sum().item()
    print(f"epoch {ep+1:2d}  loss {tot/len(idx):.3f}  train acc {100*correct/len(idx):5.1f} %")
print(f"trained in {time.time()-t0:.0f} s")

32 training clips (4 sentences x 8 speakers) | 16 held-out clips (2 unseen sentences x 8 speakers)

epoch  1  loss 1.813  train acc  41.0 %

epoch  2  loss 0.973  train acc  85.2 %

epoch  3  loss 0.468  train acc  92.6 %

epoch  4  loss 0.230  train acc  99.2 %

epoch  5  loss 0.125  train acc  98.4 %

epoch  6  loss 0.062  train acc  99.2 %

epoch  7  loss 0.037  train acc 100.0 %

epoch  8  loss 0.028  train acc 100.0 %

epoch  9  loss 0.020  train acc  99.6 %

epoch 10  loss 0.015  train acc 100.0 %

epoch 11  loss 0.009  train acc 100.0 %

epoch 12  loss 0.008  train acc 100.0 %

epoch 13  loss 0.006  train acc 100.0 %

epoch 14  loss 0.006  train acc 100.0 %

epoch 15  loss 0.004  train acc 100.0 %
trained in 24 s

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short take the 64-d embeddings of all probe and demo clips, add the handcrafted embedding of the demo clips, and evaluate both embeddings on two protocols.

• Protocol A, unseen sentences: enroll on the 4 training clips of each probe speaker, test on the 16 held-out clips: 8 models × 16 clips = 128 trials, 16 target. Optimistic: the enrollment audio was in the training set.
• Protocol B, unseen speakers: the 10 demo speakers, enroll on clip 1, test on clip 2: 10 × 10 = 100 trials, 10 target. This is the real test of an embedding: does it work for people it has never heard?
• evaluate stores EER, threshold and minDCF for each embedding and protocol; table5() prints them.

What the output means

In short handcrafted: 6.25 % (A), 20.00 % (B). Tiny CNN: 0.00 % (A), 10.00 % (B). The CNN thresholds are 0.98 / 0.99, and its cosines for unseen speakers lie between 0.778 and 0.996.

• The CNN is perfect where it was trained (A) and falls to 10 % on new voices (B).
• The code comment says the CNN cosines "all sit between 0.9 and 1.0". The printed range for B is wider: 0.778 to 0.996. The space is still crowded.
• minDCF is bad for the CNN on B (0.90) although its EER is 10 %. Our reading (not tested separately): when most scores are crowded near 1, a threshold strict enough to keep false accepts rare rejects most targets.
• Protocol B has an easy part: the 10 demo speakers are 8 female and 2 male, so 32 of the 90 non-target trials are cross-gender (arithmetic from 8 + 2 speakers).
• With 10 targets, one error is a 10 % step: these numbers are indicative, not precise.

# ---- embeddings from the penultimate layer on FULL clips (any length: mean pooling handles it)
@torch.no_grad()
def embed_cnn(u):
    net.eval(); e = net.embedding(torch.tensor(mels[u])[None, None]).squeeze(0).numpy(); return e / np.linalg.norm(e)
emb_cnn = {u: embed_cnn(u) for u in list(probe.utt_id) + list(demo.utt_id)}
# Handcrafted embedding of the demo clips too (z-scored with the probe statistics so the space is the same)
Fd = np.stack([handcrafted_embedding(wavs[u]) for u in demo.utt_id]); Zd = (Fd - F.mean(0)) / (F.std(0) + 1e-8)
emb_hand.update({u: z / np.linalg.norm(z) for u, z in zip(demo.utt_id, Zd)})

# Protocol A, unseen SENTENCES: enroll = the 4 training clips per probe speaker (what the net saw), trial = the 16 held-out clips.
#   8 models x 16 clips = 128 trials, 16 target. Optimistic: the enrollment audio was in the training set.
# Protocol B, unseen SPEAKERS: the 10 demo speakers, enroll on clip 1, trial on clip 2: 10 x 10 = 100 trials, 10 target.
#   This is the real test of an embedding: does it generalise to people it has never heard?
utts4, trials4 = build_trials(probe, n_enroll=4); labels4 = (trials4.label == "target").values.astype(int)
demo_u = demo.assign(sentence_id=demo.groupby("speaker").cumcount())
utts_d, trials_d = build_trials(demo_u, n_enroll=1); labels_d = (trials_d.label == "target").values.astype(int)
PROTOCOLS = {"A: unseen sentences (128 trials)": (trials4, utts4, labels4), "B: unseen speakers (100 trials)": (trials_d, utts_d, labels_d)}
res5 = {}
def evaluate(name, emb):
    res5[name] = {}
    for pname, (tr_, ut_, lab_) in PROTOCOLS.items():
        sc = score_trials(tr_, ut_, emb); e, t = eer(sc, lab_)
        res5[name][pname] = dict(EER_pct=e, threshold=t, minDCF=min_dcf(sc, lab_), scores=sc)
def table5():
    return pd.DataFrame({(n, p): {k: v for k, v in r.items() if k != "scores"} for n, d in res5.items() for p, r in d.items()}).T
for name, emb in [("handcrafted 40-d", emb_hand), ("tiny CNN 64-d", emb_cnn)]: evaluate(name, emb)
display(table5())
# Notice the CNN thresholds: around 0.98. Its cosines all sit between 0.9 and 1.0, the space is barely spread out.
sc = res5["tiny CNN 64-d"]["B: unseen speakers (100 trials)"]["scores"]; print(f"tiny CNN, unseen speakers: cosine range {sc.min():.3f} .. {sc.max():.3f}")

		EER_pct	threshold	minDCF
handcrafted 40-d	A: unseen sentences (128 trials)	6.25	0.31	0.06
	B: unseen speakers (100 trials)	20.00	0.42	0.50
tiny CNN 64-d	A: unseen sentences (128 trials)	0.00	0.98	0.00
	B: unseen speakers (100 trials)	10.00	0.99	0.90

tiny CNN, unseen speakers: cosine range 0.778 .. 0.996

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short draw the score histograms of the two embeddings on protocol B (unseen speakers).

• Left: handcrafted; right: tiny CNN. Grey = non-target (90), orange = target (10), dashed line = EER threshold.
• The x-axes are different on purpose: the CNN histogram only spans 0.7 to 1.0.

What the output means

In short handcrafted EER 20.0 %, tiny CNN EER 10.0 %.

• On the right, the targets and many non-targets are in the same narrow band near 1.

fig, axes = plt.subplots(1, 2, figsize=(12, 3.5))
P_B = "B: unseen speakers (100 trials)"
for ax, name in zip(axes, res5):
    r = res5[name][P_B]; sc = r["scores"]; bins = np.linspace(-1, 1, 81) if "CNN" not in name else np.linspace(0.7, 1, 91)
    ax.hist(sc[labels_d == 0], bins, alpha=0.6, color="tab:gray", label="non-target (90)")
    ax.hist(sc[labels_d == 1], bins, alpha=0.7, color="tab:orange", label="target (10)")
    ax.axvline(r["threshold"], color="k", ls="--"); ax.set(title=f"{name}, unseen speakers: EER {r['EER_pct']:.1f} %", xlabel="cosine score", ylabel="count"); ax.legend()
plt.show()

Additional Reading

How to read this plot

In short same layout as section 3, on 100 unseen-speaker trials. The CNN is perfect on the speakers it trained on (A), but only 10 % EER on new ones (B), with all scores crowded near 1.

Same layout as section 3, both embeddings on the 100 unseen-speaker trials (protocol B). Mind the x-axes: the handcrafted scores span -0.37 to +0.87 (computed separately, not printed by the cells), the CNN's only 0.78 to 1.0. On the probe speakers it was trained on (protocol A) the CNN is perfect, 0 % EER: it has learned (probably memorised) 8 voices.

On 10 new voices it drops to 10 %, with targets and non-targets both crammed between cosine 0.78 and 1.0. The course text calls this the signature of a classification-trained embedding with few classes: the network only needed 8 directions to be separable, so everything else lands in the same corner of the space. We did not test that explanation here.

The handcrafted vector, which never saw any speaker, is more spread out but not better (20 % vs 10 %). With 10 targets, each false reject is a 10 % step: these numbers are indicative, not precise.

The honest comparison is section 6: a model that was trained on many other speakers (background fact from the course text, not checked here) and has never heard these 18 people.

6. Pretrained ECAPA-TDNN on the same 48 clips

Which embeddings are used here? The shipped file, not a fresh ECAPA run. The cell below first tries to download speechbrain/spkrec-ecapa-voxceleb and run it. On this machine the download fails (ProxyError: 403 Forbidden). The cell then falls back, as designed, to the shipped embeddings of the same 48 clips (and the 20 demo clips) in data/embeddings/original.npz. Everything in this section is computed from that file.

The model: ECAPA-TDNN trained with AAM-softmax on VoxCeleb 1+2 (7 205 mostly English speakers, 1 M+ clips), 192-d output, EER 0.8 % on VoxCeleb1-O (background fact from the course text, not checked here). None of our Khmer speakers is in its training set, and Khmer is not a VoxCeleb language (background fact from the course text, not checked here). The question is whether "voice identity" learned on English transfers. The course text says the shipped embeddings were produced with this model and are the ones behind the project leaderboard; we could not check that here. What we can check is what the shipped embeddings do on our clips (below).

If you run this notebook with internet, the cell needs one download (~80 MB, first run only) and the numbers may change a little.

Visualization Version

The red box says which embeddings were used. Then: EER of three embeddings, ECAPA scores, and the ECAPA cosine matrix.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short try to download and run the pretrained ECAPA-TDNN; if that fails, load the shipped embeddings; then evaluate ECAPA on both protocols and check how much of the words it still keeps.

• try: installs speechbrain if needed and loads the model. Here it raises a ProxyError, so the except: branch runs and loads data/embeddings/original.npz (the file also holds the 20 demo clips). ecapa_source becomes "shipped embeddings".
• Then evaluate(...) runs protocols A and B (section 5) with these 192-d vectors.
• The last lines compute the same "speaker mean removed" table as in section 2, for the handcrafted vector and for ECAPA.

What the output means

In short the cell prints Could not run the pretrained model (ProxyError: 403 Forbidden), then ECAPA embeddings: shipped embeddings | dim 192. ECAPA EER: 0.00 % on both protocols (thresholds 0.67 and 0.57, minDCF 0.00).

• These ECAPA numbers come from the shipped file (ECAPA (shipped embeddings, not a fresh run)).
• On unseen speakers (B): target mean 0.68, non-target mean 0.23, cosine range -0.09 to 0.81.
• Mean cosine table (ECAPA (shipped embeddings, not a fresh run), full): same speaker 0.65, different speaker same sentence 0.21, different speaker different sentence 0.19 (0.1847: the cell rounds it to 0.185, the table shows 0.19). After removing the speaker mean: -0.20 / 0.06 / -0.01.
• Again, the -0.20 is the mean-removal effect from section 2 (about -1/5).

# speechbrain/spkrec-ecapa-voxceleb (~80 MB)  # needs internet (first run only); cached in ~/.cache/sb_ecapa
ecapa_source = None
try:
    try:
        import speechbrain  # noqa
    except ImportError:
        print("installing speechbrain ...")
        subprocess.run([sys.executable, "-m", "pip", "install", "-q", "speechbrain"], check=True, timeout=600)
    try:
        from speechbrain.inference.speaker import EncoderClassifier      # speechbrain >= 1.0
    except ImportError:
        from speechbrain.pretrained import EncoderClassifier             # older speechbrain
    enc = EncoderClassifier.from_hparams(source="speechbrain/spkrec-ecapa-voxceleb",
                                         savedir=str(Path.home() / ".cache/sb_ecapa"), run_opts={"device": "cpu"})
    t0 = time.time()
    with torch.no_grad():
        emb_ecapa = {u: enc.encode_batch(torch.tensor(wavs[u])[None]).squeeze().numpy() for u in probe.utt_id}
    emb_ecapa = {u: e / np.linalg.norm(e) for u, e in emb_ecapa.items()}
    lut = None
    ecapa_source = f"downloaded model, {len(emb_ecapa)} clips embedded in {time.time()-t0:.1f} s"
except Exception as ex:
    print(f"Could not run the pretrained model ({type(ex).__name__}: {str(ex)[:120]}).")
    print("Falling back to the shipped ECAPA embeddings of the same 48 clips (data/embeddings/original.npz).")
    z = np.load(DATA / "embeddings/original.npz"); lut = dict(zip(z["utt_id"], z["emb"]))
    emb_ecapa = {u: lut[u] / np.linalg.norm(lut[u]) for u in probe.utt_id}
    ecapa_source = "shipped embeddings"
print("ECAPA embeddings:", ecapa_source, "| dim", len(next(iter(emb_ecapa.values()))))

if ecapa_source == "shipped embeddings":      # the shipped file also holds the 20 demo clips
    emb_ecapa.update({u: lut[u] / np.linalg.norm(lut[u]) for u in demo.utt_id})
else:
    with torch.no_grad():
        for u in demo.utt_id:
            e = enc.encode_batch(torch.tensor(wavs[u])[None]).squeeze().numpy(); emb_ecapa[u] = e / np.linalg.norm(e)
evaluate("ECAPA-TDNN 192-d (pretrained)", emb_ecapa)
display(table5())
sc = res5["ECAPA-TDNN 192-d (pretrained)"]["B: unseen speakers (100 trials)"]["scores"]
print(f"ECAPA, unseen speakers: target mean {sc[labels_d==1].mean():.2f}, non-target mean {sc[labels_d==0].mean():.2f}, cosine range {sc.min():.2f} .. {sc.max():.2f}")
# Same-sentence leak check for ECAPA: does content still show in the similarity matrix?
E = np.stack([emb_ecapa[u] for u in probe.utt_id]); S_ecapa = E @ E.T
R_e = E - np.stack([E[spk_arr == s].mean(0) for s in spk_arr]); R_e /= np.linalg.norm(R_e, axis=1, keepdims=True); S_eres = R_e @ R_e.T
print(pd.DataFrame({"handcrafted, full": {k: S_hand[v].mean() for k, v in pairs.items()}, "handcrafted, spk mean removed": {k: S_resid[v].mean() for k, v in pairs.items()},
                    "ECAPA, full": {k: S_ecapa[v].mean() for k, v in pairs.items()}, "ECAPA, spk mean removed": {k: S_eres[v].mean() for k, v in pairs.items()}}).round(3).T)

Could not run the pretrained model (ProxyError: 403 Forbidden).
Falling back to the shipped ECAPA embeddings of the same 48 clips (data/embeddings/original.npz).
ECAPA embeddings: shipped embeddings | dim 192

		EER_pct	threshold	minDCF
handcrafted 40-d	A: unseen sentences (128 trials)	6.25	0.31	0.06
	B: unseen speakers (100 trials)	20.00	0.42	0.50
tiny CNN 64-d	A: unseen sentences (128 trials)	0.00	0.98	0.00
	B: unseen speakers (100 trials)	10.00	0.99	0.90
ECAPA-TDNN 192-d (pretrained)	A: unseen sentences (128 trials)	0.00	0.67	0.00
	B: unseen speakers (100 trials)	0.00	0.57	0.00

ECAPA, unseen speakers: target mean 0.68, non-target mean 0.23, cosine range -0.09 .. 0.81
                               same speaker, different sentence  different speaker, SAME sentence  different speaker, different sentence
handcrafted, full                                          0.67                             -0.08                                  -0.10
handcrafted, spk mean removed                             -0.19                              0.08                                  -0.02
ECAPA, full                                                0.65                              0.21                                   0.19
ECAPA, spk mean removed                                   -0.20                              0.06                                  -0.01

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short draw the cosine matrices of the handcrafted vector and of ECAPA side by side, both sorted by speaker.

• Same heatmap helper as in section 2. Left: handcrafted (as before). Right: ECAPA (shipped embeddings).

What the output means

In short ECAPA (shipped embeddings, not a fresh run) shows 8 clean bright squares on the diagonal over a flat background.

• The colour scale is the same in both panels (-1 to +1), so you can compare them directly.

fig, axes = plt.subplots(1, 2, figsize=(12, 5.5))
heatmap(S_hand, by_spk, probe.speaker.to_numpy(str), "handcrafted, sorted by speaker", axes[0])
img = heatmap(S_ecapa, by_spk, probe.speaker.to_numpy(str), "ECAPA-TDNN, sorted by speaker", axes[1])
fig.colorbar(img, ax=axes, label="cosine similarity", shrink=0.8); plt.show()

Additional Reading

How to read this plot

In short right: ECAPA (shipped embeddings, not a fresh run) has 8 clean squares (same-speaker mean +0.65) on a flat background of about +0.2. Different speakers are mostly positive for ECAPA.

Right panel (ECAPA (shipped embeddings, not a fresh run)): the matrix has eight clean bright squares on the diagonal (same-speaker mean cosine +0.65) over a flat background around +0.2.

ECAPA's "different speaker" cosine is mostly positive (+0.21 same sentence, +0.1847 different sentence; the cell rounds it to 0.185 and the table shows 0.19), not negative as for the handcrafted vector. About 1.2 % of the different-speaker pairs in the 48 × 48 matrix are still negative (minimum -0.064) (computed separately, not printed by the cells). The course explains the positive mean by a common direction shared by all speech embeddings, which is why verification needs a learned threshold and not "cosine > 0"; we did not test the explanation.

The content leak is a little smaller than for the handcrafted vector but not zero. After removing the speaker mean, same-sentence pairs still score +0.06 vs -0.01 (gap about 0.075, against about 0.10 for the handcrafted vector), so even ECAPA keeps a trace of the words. The course note says this may matter for anonymization (an attacker could use text overlap too); we did not test that.

On unseen speakers (protocol B) ECAPA (shipped embeddings, not a fresh run) gets 0 % EER, with targets around +0.68 and non-targets around +0.23: a wide gap. The tiny CNN sat at 10 % with everything crushed between 0.78 and 1.0.

Why the gap? The course text lists four reasons: (1) 7 205 training speakers vs 8; (2) an AAM-softmax margin vs a plain softmax; (3) attentive statistics pooling and multi-scale Res2 blocks vs a mean over frames; (4) 1 M+ clips with noise and reverberation augmentation vs 32 clips (background fact from the course text, not checked here). We did not test any of them here.

Our tiny net is the right idea; the course's view is that the gap is data and loss.

Phone and studio clips: the 10 demo speakers include 7 phone and 3 studio speakers (computed separately, not printed by the cells), and ECAPA still has 0 % EER on them. With 10 targets this is coarse evidence.

7. The real evaluation: 43 speakers, 18 490 trials, every anonymization system

Now the project test set. data/meta/test_utts.csv has 559 clips from 43 speakers: 3 enrollment clips and 10 trial clips per speaker (13 × 43 = 559). Of the 43 speakers, 28 are female and 15 male, and 32 are phone speakers (iAny) and 11 are studio speakers (OpenSLR 42) (computed separately, not printed by the cells). trials_test.csv is every (enrollment speaker, trial clip) pair: 43 × 430 = 18 490 trials, 430 target. The .npz files hold the ECAPA-TDNN embedding of each clip after it was processed by each anonymization system (original = untouched speech). These are shipped embeddings (computed before this notebook; we did not recompute them).

Two shortcuts an attacker gets for free, and that we must not count as "identification":

• Gender shortcut. The course says anonymization keeps gender most of the time (a converted female voice is still female). A cross-gender non-target trial is then rejected by gender alone, which lowers the EER without the attacker knowing who it is. The leaderboard uses same-gender trials only: 10 090 trials, 430 target (28 × 280 + 15 × 150 = 7 840 + 2 250 = 10 090 trials). The attacker has to tell a woman from other women.
• Corpus shortcut. Phone (iAny) and studio (OpenSLR) clips have different channels. Cross-corpus non-targets are easy to reject for the wrong reason. We show the size of the effect; the leaderboard keeps all same-gender trials because, the course says, real attackers do get this hint.

Visualization Version

Shortcuts, the attacker ladder, and EER per system and attacker with 95 % intervals (real run).

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short load the test set and the trial list, flag same-gender and same-corpus trials, load the ECAPA embeddings of original and mcadams, and compute the EER on three trial subsets.

• load_emb(system, folder) reads data/<folder>/<system>.npz (192-d embedding per clip, scaled to length 1). speaker_models averages the 3 enrollment clips; score computes every trial.
• original: enroll and test on untouched speech. mcadams, ignorant attacker: enroll on the original voices, test on McAdams-anonymized clips.
• Subsets: all trials, same-gender only, same-gender and same-corpus.

What the output means

In short 559 utts, 43 speakers | 18490 trials, 430 target | same-gender 10090 | same-gender & same-corpus 6350. ECAPA embeddings, shipped (not a fresh run): original: EER 1.84 / 2.36 / 3.02 %. mcadams (ignorant): 23.95 / 29.58 / 31.16 %.

• Each restriction removes easy non-targets, so the EER rises: for mcadams from 23.95 % (all trials) to 29.58 % (same-gender), 5.6 points more, then to 31.16 % (also same-corpus).
• For original the same effect is small (1.84 → 2.36 → 3.02), because the attacker is already strong.
• The leaderboard number is the middle one (same-gender).

utts = pd.read_csv(DATA / "meta/test_utts.csv"); trials = pd.read_csv(DATA / "meta/trials_test.csv")
labels = (trials.label == "target").values.astype(int); same_g = trials.same_gender.to_numpy().astype(bool)
enroll_corpus = utts[utts.role == "enroll"].groupby("speaker").corpus.first()
trial_corpus = utts.set_index("utt_id").corpus
same_c = (trials.enroll_speaker.map(enroll_corpus).to_numpy(str) == trials.trial_utt.map(trial_corpus).to_numpy(str))
print(f"{len(utts)} utts, {utts.speaker.nunique()} speakers | {len(trials)} trials, {labels.sum()} target | same-gender {same_g.sum()} | same-gender & same-corpus {(same_g & same_c).sum()}")

def load_emb(system, folder="embeddings"):
    z = np.load(DATA / folder / f"{system}.npz")                # keys: utt_id (N,), emb (N, 192); row i <-> utt_id[i]
    return {u: e / np.linalg.norm(e) for u, e in zip(z["utt_id"], z["emb"])}

def speaker_models(emb):
    enr = utts[utts.role == "enroll"]
    m = {s: np.mean([emb[u] for u in g.utt_id], 0) for s, g in enr.groupby("speaker")}
    return {s: v / np.linalg.norm(v) for s, v in m.items()}

def score(models, emb):
    return np.array([models[e] @ emb[t] for e, t in zip(trials.enroll_speaker, trials.trial_utt)])

emb_orig = load_emb("original"); models_orig = speaker_models(emb_orig)
s_orig = score(models_orig, emb_orig)
for name, m in [("all trials", np.ones(len(trials), bool)), ("same-gender only", same_g), ("same-gender & same-corpus", same_g & same_c)]:
    print(f"original speech, {name:28s}: EER {eer(s_orig[m], labels[m])[0]:5.2f} %  ({m.sum()} trials)")
# The shortcuts are bigger once the voice is disguised: McAdams warping keeps gender and channel.
emb_mc = load_emb("mcadams"); s_mc = score(models_orig, emb_mc)       # ignorant attacker: enroll on ORIGINAL, test on anonymized
for name, m in [("all trials", np.ones(len(trials), bool)), ("same-gender only", same_g), ("same-gender & same-corpus", same_g & same_c)]:
    print(f"mcadams, ignorant attacker, {name:28s}: EER {eer(s_mc[m], labels[m])[0]:5.2f} %  ({m.sum()} trials)")

559 utts, 43 speakers | 18490 trials, 430 target | same-gender 10090 | same-gender & same-corpus 6350
original speech, all trials                  : EER  1.84 %  (18490 trials)
original speech, same-gender only            : EER  2.36 %  (10090 trials)
original speech, same-gender & same-corpus   : EER  3.02 %  (6350 trials)
mcadams, ignorant attacker, all trials                  : EER 23.95 %  (18490 trials)
mcadams, ignorant attacker, same-gender only            : EER 29.58 %  (10090 trials)
mcadams, ignorant attacker, same-gender & same-corpus   : EER 31.16 %  (6350 trials)

Additional reading: you can read all of it or just read the "In short"

The three attackers (ignorant, lazy, semi-informed)

In short ignorant < lazy < semi-informed in knowledge; the EER often, but not always, falls along the ladder.

The three attackers differ in what they know.

• Ignorant: enrolls on the original voices (e.g. an old recording of you) and attacks the anonymized trial clips; does not know anonymization happened.
• Lazy: knows the system exists, so enrolls on anonymized speech too (the same system applied to the enrollment clips); nothing is retrained.
• Semi-informed: additionally fine-tunes the speaker encoder on anonymized speech from other speakers, so it learns what the anonymizer leaves behind (data/embeddings_semi_informed/ holds that encoder's embeddings).

More knowledge should help the attacker, so the EER should tend to go down. It does not always, in this table. The "lazy" EER is higher than the "ignorant" one for three systems (knnvc_english_pool 43.69 vs 40.66; knnvc_khmer_pool 46.74 vs 43.96; asr_tts 49.99 vs 45.33), and for original the semi-informed EER (3.02) is higher than the lazy one (2.36) (computed separately, not printed by the cells). The big drops are for mcadams (29.58 → 14.18) and sslsas (50.70 → 9.77). The VoicePrivacy Challenge reports the semi-informed number as the headline (background fact from the course text, not checked here). A system that only beats the ignorant attacker (SSL-SAS below: 50.7 % ignorant, 9.8 % semi-informed) is not private, it is merely unfamiliar.

From here on, every EER is on the 10 090 same-gender trials.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short define eer_ci (EER with a speaker-bootstrap confidence interval), then compute the EER and its 95 % interval for 8 systems × 3 attackers.

• Speaker bootstrap (eer_ci): resample the 43 enrollment speakers with replacement (300 times), keep all their trials, recompute the EER; the 2.5th and 97.5th percentiles are the interval. Trials are not independent (10 per speaker), so a trial-level bootstrap would give too narrow an interval (a standard argument; not tested here).
• ATTACKERS says where the enrollment embeddings come from: ignorant = original speech, lazy = the same system, semi_informed = the fine-tuned encoder's files.
• For original, "enroll from the same system" and "enroll from original" are the same file, so ignorant and lazy are identical (2.36).
• If a system has no embeddings file for an attacker (asr_tts, semi-informed), the row is filled with NaN.

What the output means

In short 24 (system, attacker) EERs are printed, but 1 is NaN (asr_tts, semi-informed): 23 real EERs. Examples: original 2.36 / 2.36 / 3.02; mcadams 29.58 / 25.31 / 14.18; sslsas 50.70 / 12.80 / 9.77.

• The text printed by the cell says "24 EERs"; that count includes the NaN row.
• The reason for the missing asr_tts value, "no training data for it", comes from the course text. What we see is that no semi-informed embeddings file exists for it.
• Highest ignorant EER: sslsas 50.70 %. Lowest EER of any anonymized system: sslsas semi-informed 9.77 %.

def eer_ci(scores, labels, groups, n_boot=300, seed=0):
    # Speaker bootstrap: resample ENROLLMENT speakers with replacement, keep all their trials, recompute the EER.
    # Trials are not independent (10 per speaker), so a trial-level bootstrap would be far too optimistic.
    rng = np.random.default_rng(seed); groups = np.asarray(groups); uniq = np.unique(groups)
    idx = {g: np.where(groups == g)[0] for g in uniq}
    boots = []
    for _ in range(n_boot):
        pick = np.concatenate([idx[g] for g in rng.choice(uniq, len(uniq), replace=True)])
        boots.append(eer(scores[pick], labels[pick])[0])
    return eer(scores, labels)[0], float(np.percentile(boots, 2.5)), float(np.percentile(boots, 97.5))

SYSTEMS = ["original", "mcadams", "knnvc_english_pool", "knnvc_khmer_pool", "knnvc_khmer_pool_samegender", "asr_tts", "sslsas", "private_knnvc_english_durations"]
ATTACKERS = {"ignorant": ("embeddings", "original"),         # (folder, which system's embeddings enroll)
             "lazy": ("embeddings", "same"),
             "semi_informed": ("embeddings_semi_informed", "same")}
rows, det_store, score_store = [], {}, {}
t0 = time.time()
for sysname in SYSTEMS:
    for att, (folder, enroll_from) in ATTACKERS.items():
        try:
            emb_t = load_emb(sysname, folder)
            emb_e = emb_t if enroll_from == "same" else load_emb("original", folder)
        except FileNotFoundError:
            rows.append(dict(system=sysname, attacker=att, eer=np.nan, ci_low=np.nan, ci_high=np.nan)); continue
        s = score(speaker_models(emb_e), emb_t)
        e, lo, hi = eer_ci(s[same_g], labels[same_g], trials.enroll_speaker.to_numpy(str)[same_g])
        rows.append(dict(system=sysname, attacker=att, eer=e, ci_low=lo, ci_high=hi))
        det_store[(sysname, att)] = det_curve(s[same_g], labels[same_g]); score_store[(sysname, att)] = s
print(f"{len(rows)} (system, attacker) EERs with bootstrap CIs in {time.time()-t0:.0f} s")
res = pd.DataFrame(rows)
table = res.pivot(index="system", columns="attacker", values="eer").loc[SYSTEMS, ["ignorant", "lazy", "semi_informed"]]
display(table.round(2))

24 (system, attacker) EERs with bootstrap CIs in 5 s

attacker	ignorant	lazy	semi_informed
system
original	2.36	2.36	3.02
mcadams	29.58	25.31	14.18
knnvc_english_pool	40.66	43.69	37.93
knnvc_khmer_pool	43.96	46.74	43.49
knnvc_khmer_pool_samegender	40.70	33.95	31.40
asr_tts	45.33	49.99	NaN
sslsas	50.70	12.80	9.77
private_knnvc_english_durations	48.13	36.78	36.51

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short compare our EERs with the shipped leaderboard leaderboard.csv.

• The table puts our value next to the leaderboard value for each attacker, plus the interval of the ignorant attacker.
• It prints the largest absolute difference per attacker.

What the output means

In short largest differences 0.05 / 0.04 / 0.03 percentage points (ignorant / lazy / semi-informed): MATCH.

• The EER values agree within 0.05 points. The course explains the small gap by the way the leaderboard interpolates the FAR/FRR crossing (the cell prints this explanation).
• The intervals agree less well. original matches exactly and mcadams is within 0.2 points ([26.7, 33.0] vs [26.5, 33.0]), but for example asr_tts is [42.4, 47.6] here vs [41.1, 49.7] on the leaderboard, and knnvc_khmer_pool_samegender [36.3, 47.0] vs [37.7, 44.9]. We did not check the reason (a different number of resamples or seed is possible).

# ---- Check against the shipped leaderboard
lb = pd.read_csv(DATA / "meta/leaderboard.csv").set_index("system")
cmp = pd.DataFrame({
    "ours_ignorant": table["ignorant"], "lb_ignorant": lb.loc[SYSTEMS, "eer_ignorant"],
    "ours_lazy": table["lazy"], "lb_lazy": lb.loc[SYSTEMS, "eer_lazy"],
    "ours_semi": table["semi_informed"], "lb_semi": lb.loc[SYSTEMS, "eer_semi_informed"],
    "ours_ci_ign": res[res.attacker == "ignorant"].set_index("system").apply(lambda r: f"[{r.ci_low:.1f}, {r.ci_high:.1f}]", axis=1),
    "lb_ci_ign": lb.loc[SYSTEMS].apply(lambda r: f"[{r.eer_ignorant_ci_low:.1f}, {r.eer_ignorant_ci_high:.1f}]", axis=1)})
display(cmp.round(2))
diffs = pd.DataFrame({"ignorant": (cmp.ours_ignorant - cmp.lb_ignorant).abs(), "lazy": (cmp.ours_lazy - cmp.lb_lazy).abs(), "semi_informed": (cmp.ours_semi - cmp.lb_semi).abs()})
print("max |ours - leaderboard| per attacker (percentage points):"); print(diffs.max().round(3))
print("MATCH" if diffs.max().max() < 0.1 else "MISMATCH", "- the leaderboard EERs are reproduced to within 0.1 pp (the leaderboard interpolates the FAR/FRR crossing; utils.eer takes the nearest observed score).")

	ours_ignorant	lb_ignorant	ours_lazy	lb_lazy	ours_semi	lb_semi	ours_ci_ign	lb_ci_ign
system
original	2.36	2.40	2.36	2.40	3.02	3.02	[1.4, 3.5]	[1.4, 3.5]
mcadams	29.58	29.63	25.31	25.27	14.18	14.19	[26.7, 33.0]	[26.5, 33.0]
knnvc_english_pool	40.66	40.62	43.69	43.65	37.93	37.96	[38.6, 43.3]	[37.9, 44.0]
knnvc_khmer_pool	43.96	43.98	46.74	46.74	43.49	43.49	[41.6, 46.7]	[40.7, 47.7]
knnvc_khmer_pool_samegender	40.70	40.70	33.95	33.95	31.40	31.40	[36.3, 47.0]	[37.7, 44.9]
asr_tts	45.33	45.31	49.99	49.99	NaN	NaN	[42.4, 47.6]	[41.1, 49.7]
sslsas	50.70	50.70	12.80	12.82	9.77	9.77	[46.7, 55.6]	[45.0, 55.4]
private_knnvc_english_durations	48.13	48.13	36.78	36.82	36.51	36.51	[44.4, 51.9]	[44.2, 51.6]

max |ours - leaderboard| per attacker (percentage points):
ignorant         0.05
lazy             0.04
semi_informed    0.03
dtype: float64
MATCH - the leaderboard EERs are reproduced to within 0.1 pp (the leaderboard interpolates the FAR/FRR crossing; utils.eer takes the nearest observed score).

Additional reading: you can read all of it or just read the "In short"

What the table says

In short the table reproduces the leaderboard: original 2-3 %; mcadams 29.6 → 14.2 %; kNN-VC (Khmer pool) 43-47 %; sslsas 50.7 → 9.8 %.

The table reproduces the leaderboard. Four things to notice.

• original sits at 2 to 3 % for every attacker (2.36, 2.36, 3.02). The attacker is strong, and the test is not trivially easy: it is same-gender, with phone and studio clips of 2 to 8 s (mean 3.7 s) (computed separately, not printed by the cells).
• McAdams, the VoicePrivacy baseline, looks half-private to the ignorant attacker (29.58 %) but falls to 25.31 % (lazy) and 14.18 % once the attacker fine-tunes. The course explains this by "formant warping is a fixed, invertible-ish distortion"; we did not test that.
• kNN-VC with a random Khmer target stays at 43 to 47 % for all three attackers (43.96, 46.74, 43.49). The course explains it: the target voice is a different real person each time, so the attacker cannot learn its way back. Not tested here.
• SSL-SAS is the warning case: 50.70 % against the ignorant attacker, 9.77 % semi-informed. The course explanation: it uses one fixed pseudo-speaker per source speaker, so the mapping is consistent and learnable. Not tested here.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short plot the table as bars with 95 % intervals, and draw DET curves of three systems for the ignorant attacker.

• Bar height = EER (higher = more private); error bars = speaker-bootstrap 95 % interval; the dotted line = 50 % (guessing).
• DET: original, mcadams and knnvc_khmer_pool, same-gender trials.

What the output means

In short one bar chart (8 systems × 3 attackers) and one DET plot with three curves (EER 2.4 %, 29.6 %, 44.0 %).

• The DET legend overlaps the top of the green curve; this is how the course cell draws it.

fig, ax = plt.subplots(figsize=(12, 4))
x = np.arange(len(SYSTEMS)); w = 0.26
for k, att in enumerate(ATTACKERS):
    r = res[res.attacker == att].set_index("system").loc[SYSTEMS]
    ax.bar(x + (k - 1) * w, r.eer, w, yerr=[r.eer - r.ci_low, r.ci_high - r.eer], capsize=3, label=att)
ax.axhline(50, color="k", ls=":", lw=1); ax.text(len(SYSTEMS) - 0.5, 51, "50 % = guessing (perfect privacy)", ha="right", fontsize=9)
ax.set_xticks(x); ax.set_xticklabels(SYSTEMS, rotation=25, ha="right"); ax.set(ylabel="EER (%) on same-gender trials", title="Attacker EER per system (error bars: speaker-bootstrap 95 % CI); higher = more private"); ax.legend(title="attacker")
plt.tight_layout(); plt.show()

fig, ax = plt.subplots(figsize=(6, 6))
plot_det({f"{s} ({table.loc[s, 'ignorant']:.1f} %)": det_store[(s, "ignorant")] for s in ["original", "mcadams", "knnvc_khmer_pool"]}, ax, "DET curves, ignorant attacker, same-gender trials")
plt.show()

Additional Reading

How to read this plot

In short bars: higher = more private; the dotted line is the 50 % guessing level. Bars that shrink from ignorant to semi-informed (mcadams, sslsas, private kNN-VC) are disguises the attacker can learn.

Bar chart. Bar height = EER, higher = more private; the dotted line at 50 % is the guessing level. Error bars are the 95 % interval from resampling the 43 speakers. The half-width is about ±1 point for original and about ±2 to ±4 points for most other bars. The widest are knnvc_khmer_pool_samegender ignorant (36.3 to 47.0, about -4 / +6) and sslsas ignorant (46.7 to 55.6, about -4 / +5) (computed separately, not printed by the cells). Two systems whose bars overlap within their error bars may not be distinguishable on this test set.

Watch the colour pattern per system. Bars that shrink strongly from ignorant to semi-informed are systems whose disguise can be learned: mcadams (29.6 → 14.2), sslsas (50.7 → 9.8), private kNN-VC (48.1 → 36.5), and also knnvc_khmer_pool_samegender (40.7 → 31.4). The two kNN-VC pools knnvc_english_pool (40.7, 43.7, 37.9) and knnvc_khmer_pool (44.0, 46.7, 43.5) stay within about 6 points across attackers, so the attacker gains little. asr_tts has no semi-informed bar (no embeddings file for it), so we cannot say whether it would stay flat.

DET plot. original hugs the bottom-left corner (the attacker is near perfect, crossing the diagonal at about 2.4 %). mcadams is pushed out to the middle (about 30 %). knnvc_khmer_pool crosses the FAR = FRR diagonal near 44 %, far up in the top-right corner (the most private of the three). Closer to the top-right = more private. In the bar chart the error bar of private_knnvc_english_durations touches the "50 % = guessing" label; this is how the course cell draws it.

8. Score distributions per system

Histograms tell you how a system reaches its EER. Two possibilities give the same EER: the target scores dropped to where the non-targets are, or the non-target scores rose to where the targets are. For privacy the first is what we want (the anonymized clip no longer resembles the enrolled voice).

Visualization Version

The real score histograms of original and knnvc_khmer_pool with their means.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short draw the score histograms of original and knnvc_khmer_pool (ignorant attacker, 10 090 same-gender trials) and print the mean target and non-target scores.

• Grey = 9660 non-target trials, orange = 430 target trials, both normalized to density (each histogram has area 1).
• Dashed line = the threshold at the EER. The score is the cosine between the enrolled original voice and the trial clip.

What the output means

In short original: target mean 0.743, non-target mean 0.317, EER 2.4 %. knnvc_khmer_pool: target mean 0.272, non-target mean 0.233, EER 44.0 %.

• Threshold at the EER (from the plot legends): 0.54 for original, 0.24 for knnvc_khmer_pool.

fig, axes = plt.subplots(1, 2, figsize=(12, 3.5), sharey=True)
bins = np.linspace(-0.4, 1, 57)
for ax, sysname in zip(axes, ["original", "knnvc_khmer_pool"]):
    s = score_store[(sysname, "ignorant")][same_g]; l = labels[same_g]
    ax.hist(s[l == 0], bins, alpha=0.6, color="tab:gray", density=True, label=f"non-target (n={(l==0).sum()})")
    ax.hist(s[l == 1], bins, alpha=0.7, color="tab:orange", density=True, label=f"target (n={(l==1).sum()})")
    e, t = eer(s, l); ax.axvline(t, color="k", ls="--", label=f"θ at EER = {t:.2f}")
    ax.set(title=f"{sysname}, ignorant attacker: EER {e:.1f} %", xlabel="cosine(enrolled original voice, trial clip)"); ax.legend(fontsize=8)
    print(f"{sysname:18s} target mean {s[l==1].mean():.3f}  non-target mean {s[l==0].mean():.3f}")
axes[0].set_ylabel("density"); plt.show()

original           target mean 0.743  non-target mean 0.317
knnvc_khmer_pool   target mean 0.272  non-target mean 0.233

Additional Reading

How to read this plot

In short left: a clear gap between targets and non-targets (2.4 %). Right: the targets moved left onto the non-targets (44.0 %).

Left, original: targets (orange) cluster around 0.74, same-gender non-targets (grey) around 0.32, a wide gap with a thin overlap near 0.5 (5 of 430 targets are below 0.5 and 484 of 9 660 non-targets are above it (computed separately, not printed by the cells)), so the EER is 2.4 %. Right, knnvc_khmer_pool: the orange histogram has moved left (mean 0.272) and sits on top of the grey one (mean 0.233). The anonymized clip is only slightly more similar to its owner (mean 0.272) than to a stranger of the same gender (0.233).

What moved? The target mean fell by 0.47 (0.743 → 0.272). The non-target mean moved much less, by 0.08 (0.317 → 0.233). So in these two systems the targets moved, which is the direction we want for privacy. A dashed threshold inside the overlap, with both histograms of about the same shape, is what "EER near 44 %" looks like.

Build: a verification toolkit (src/verify.py)

Everything above, as functions you can import tomorrow. The file holds six functions: build_trials, speaker_models, score_trials, eer_ci, det_curve and min_dcf (the EER itself is imported from src/utils.py). The cell writes the file, re-imports it, and regenerates the EER table with it into outputs/day4/eer_table.csv.

Visualization Version

The six functions, and the toolkit's EERs against the leaderboard column.

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short write src/verify.py as a text file, re-import it, and list its functions.

• The source code is a Python string, written with Path('../src/verify.py').write_text(...). This overwrites the file src/verify.py of the project.
• importlib.reload loads the fresh version; dir(V) lists what it contains.

What the output means

In short src/verify.py written: ['build_trials', 'det_curve', 'eer', 'eer_ci', 'min_dcf', 'score_trials', 'speaker_models'].

• Seven names are printed: six are defined in the file, and eer is imported from src/utils.py.
• min_dcf here uses det_curve (all observed thresholds), so it can differ slightly from the grid version of section 4.

verify_src = """'''Speaker-verification toolkit (Day 4). Embeddings are dicts utt_id -> unit-norm vector.'''
from __future__ import annotations
import numpy as np
import pandas as pd
from .utils import eer


def build_trials(utts: pd.DataFrame, n_enroll: int = 3):
    '''Split each speaker's clips into the first n_enroll (sorted by sentence_id) = enroll, rest = trial,
    and list every (enroll_speaker, trial clip) pair. Returns (utts with a `role` column, trials DataFrame
    with enroll_speaker, trial_utt, trial_speaker, label in {target, nontarget}).'''
    u = utts.sort_values(["speaker", "sentence_id"]) if "sentence_id" in utts else utts.sort_values(["speaker"])
    rank = u.groupby("speaker").cumcount()
    u = u.assign(role=np.where(rank < n_enroll, "enroll", "trial"))
    tc = u[u.role == "trial"]
    rows = [(e, t.utt_id, t.speaker, "target" if e == t.speaker else "nontarget")
            for e in sorted(u.speaker.unique()) for t in tc.itertuples()]
    return u, pd.DataFrame(rows, columns=["enroll_speaker", "trial_utt", "trial_speaker", "label"])


def speaker_models(utts: pd.DataFrame, emb: dict):
    '''L2-normalised mean of each speaker's enrollment embeddings (utts needs utt_id, speaker, role).'''
    enr = utts[utts.role == "enroll"]
    m = {s: np.mean([emb[x] for x in g.utt_id], 0) for s, g in enr.groupby("speaker")}
    return {s: v / np.linalg.norm(v) for s, v in m.items()}


def score_trials(trials: pd.DataFrame, utts: pd.DataFrame, emb_trial: dict, emb_enroll: dict | None = None):
    '''Cosine score of every trial. emb_enroll lets the enrollment side use different embeddings
    (ignorant attacker: enroll on original speech, test on anonymized).'''
    models = speaker_models(utts, emb_enroll if emb_enroll is not None else emb_trial)
    return np.array([models[e] @ emb_trial[t] for e, t in zip(trials.enroll_speaker, trials.trial_utt)])


def eer_ci(scores, labels, groups, n_boot: int = 300, seed: int = 0):
    '''EER (%) with a speaker-bootstrap 95 % CI: resample the groups (enrollment speakers) with replacement.'''
    scores, labels, groups = np.asarray(scores, float), np.asarray(labels, int), np.asarray(groups)
    rng = np.random.default_rng(seed); uniq = np.unique(groups)
    idx = {g: np.where(groups == g)[0] for g in uniq}
    boots = []
    for _ in range(n_boot):
        pick = np.concatenate([idx[g] for g in rng.choice(uniq, len(uniq), replace=True)])
        boots.append(eer(scores[pick], labels[pick])[0])
    return eer(scores, labels)[0], float(np.percentile(boots, 2.5)), float(np.percentile(boots, 97.5))


def det_curve(scores, labels):
    '''(FAR, FRR) arrays over all observed thresholds. Plot norm.ppf(far) vs norm.ppf(frr) for a DET curve.'''
    scores, labels = np.asarray(scores, float), np.asarray(labels, int)
    order = np.argsort(-scores); l = labels[order]
    tp, fp = np.cumsum(l), np.cumsum(1 - l)
    return fp / max((1 - l).sum(), 1), 1 - tp / max(l.sum(), 1)


def min_dcf(scores, labels, p_tar: float = 0.05, c_miss: float = 1.0, c_fa: float = 1.0):
    '''Normalised minimum detection cost (VoicePrivacy setting by default).'''
    far, frr = det_curve(scores, labels)
    dcf = c_miss * p_tar * frr + c_fa * (1 - p_tar) * far
    return float(dcf.min() / min(c_miss * p_tar, c_fa * (1 - p_tar)))
"""

Path("../src/verify.py").write_text(verify_src)
import importlib, src.verify as V; importlib.reload(V)
print("src/verify.py written:", [n for n in dir(V) if not n.startswith("_") and callable(getattr(V, n))])

src/verify.py written: ['build_trials', 'det_curve', 'eer', 'eer_ci', 'min_dcf', 'score_trials', 'speaker_models']

Additional reading: you can read all of it or just read the "In short"

What this cell does

In short regenerate the whole EER table with the toolkit, add the leaderboard column and minDCF, save it as outputs/day4/eer_table.csv, and check the probe trials.

• Same loop as section 7, but with V.score_trials, V.eer_ci and V.min_dcf.
• The table is saved to ../outputs/day4/eer_table.csv.
• The last line runs the toolkit on the probe trials with ECAPA (the shipped embeddings of section 6).

What the output means

In short a 24-row table with eer, ci_low, ci_high, min_dcf and leaderboard; the probe check prints ECAPA EER: 0.0 %.

• The toolkit EERs equal the section-7 values. The largest difference to the leaderboard column is 0.05 points over the 23 real rows (computed separately, not printed by the cells).
• minDCF: original 0.12; mcadams 0.86 / 0.84 / 0.58; sslsas lazy and semi-informed 0.70. All rows of the kNN-VC systems, of asr_tts and of private_knnvc_english_durations, and sslsas ignorant, are 1.00 (their EERs are 31 to 51 %), i.e. no better than the trivial system.
• The probe check (0.0 %) uses the shipped ECAPA embeddings, as in section 6.

# ---- Regenerate the leaderboard table with the toolkit and save it
t0 = time.time(); rows = []
for sysname in SYSTEMS:
    for att, (folder, enroll_from) in ATTACKERS.items():
        try:
            emb_t = load_emb(sysname, folder); emb_e = emb_t if enroll_from == "same" else load_emb("original", folder)
        except FileNotFoundError:
            rows.append(dict(system=sysname, attacker=att, eer=np.nan, ci_low=np.nan, ci_high=np.nan, min_dcf=np.nan)); continue
        s = V.score_trials(trials, utts, emb_t, emb_e)
        e, lo, hi = V.eer_ci(s[same_g], labels[same_g], trials.enroll_speaker.to_numpy(str)[same_g], n_boot=300)
        rows.append(dict(system=sysname, attacker=att, eer=e, ci_low=lo, ci_high=hi, min_dcf=V.min_dcf(s[same_g], labels[same_g])))
eer_table = pd.DataFrame(rows)
eer_table["leaderboard"] = [lb.loc[r.system, {"ignorant": "eer_ignorant", "lazy": "eer_lazy", "semi_informed": "eer_semi_informed"}[r.attacker]] for r in eer_table.itertuples()]
eer_table.to_csv(OUT / "eer_table.csv", index=False)
print(f"toolkit run in {time.time()-t0:.0f} s -> {OUT/'eer_table.csv'}")
display(eer_table.round(2))
# Sanity: the toolkit on the probe trials gives the same numbers as section 5/6
u4, t4 = V.build_trials(probe, n_enroll=4)
print("probe check, ECAPA EER:", round(eer(V.score_trials(t4, u4, emb_ecapa), (t4.label == "target").values.astype(int))[0], 2), "%")

toolkit run in 5 s -> ../outputs/day4/eer_table.csv

	system	attacker	eer	ci_low	ci_high	min_dcf	leaderboard
0	original	ignorant	2.36	1.40	3.49	0.12	2.40
1	original	lazy	2.36	1.40	3.49	0.12	2.40
2	original	semi_informed	3.02	1.86	3.85	0.12	3.02
3	mcadams	ignorant	29.58	26.74	33.02	0.86	29.63
4	mcadams	lazy	25.31	21.61	28.17	0.84	25.27
5	mcadams	semi_informed	14.18	10.70	16.41	0.58	14.19
6	knnvc_english_pool	ignorant	40.66	38.60	43.26	1.00	40.62
7	knnvc_english_pool	lazy	43.69	41.61	46.03	1.00	43.65
8	knnvc_english_pool	semi_informed	37.93	35.48	41.40	1.00	37.96
9	knnvc_khmer_pool	ignorant	43.96	41.63	46.72	1.00	43.98
10	knnvc_khmer_pool	lazy	46.74	44.27	49.07	1.00	46.74
11	knnvc_khmer_pool	semi_informed	43.49	40.56	45.83	1.00	43.49
12	knnvc_khmer_pool_samegender	ignorant	40.70	36.28	46.98	1.00	40.70
13	knnvc_khmer_pool_samegender	lazy	33.95	31.14	36.31	1.00	33.95
14	knnvc_khmer_pool_samegender	semi_informed	31.40	29.17	33.73	1.00	31.40
15	asr_tts	ignorant	45.33	42.36	47.59	1.00	45.31
16	asr_tts	lazy	49.99	47.78	52.37	1.00	49.99
17	asr_tts	semi_informed	NaN	NaN	NaN	NaN	NaN
18	sslsas	ignorant	50.70	46.74	55.58	1.00	50.70
19	sslsas	lazy	12.80	10.93	14.54	0.70	12.82
20	sslsas	semi_informed	9.77	7.76	11.74	0.70	9.77
21	private_knnvc_english_durations	ignorant	48.13	44.42	51.88	1.00	48.13
22	private_knnvc_english_durations	lazy	36.78	33.95	39.77	1.00	36.82
23	private_knnvc_english_durations	semi_informed	36.51	33.59	39.77	1.00	36.51

probe check, ECAPA EER: 0.0 %