voice-likeness-eval

Before you start: read this whole install prompt, tell me in a few lines what this skill does and what installing it involves, and wait for my go-ahead before you do anything else. Prerequisites: none for the install itself: no account, no API key, no paid service, and nothing else to install first. For using the skill afterwards as described, you need a Modal account and the Modal CLI set up (`modal setup`, authenticated by Modal itself, not by an API key in a file) because the scorer runs on a GPU in Modal (an H100 in the code SKILL.md points to), your own generated takes and real voice clips for the character, and the rest of the NPCTalk repo's scorer code (`server/finetune/train.py`, class `Scorer`), which is not part of this skill and is not installed; SKILL.md itself names no price and no paid plan for Modal, and Modal GPU time is billed by Modal. Never put any key or token in a file. Install "voice-likeness-eval" for me. It is a single SKILL.md playbook for measuring how much a cloned voice sounds like the real character, and whether a change really helped: speaker-embedding similarity (ECAPA) against the centroid of held-out real clips, word error rate with Whisper, per-line comparison with paired t-tests, enough runs to beat sampling noise, checking a take-picker with an independent speaker model, and simulating picking strategies on many scored takes before deploying one. Repository: https://github.com/christos-antono/NPCTalk (the skill is the .agents/skills/voice-likeness-eval/ folder at the pinned commit: SKILL.md only; the identical .claude/skills/ copy and the root skills/ copy are duplicates, not the source of this listing, and the rest of the NPCTalk repo is not installed) Pinned version: install the code at commit 59635ceb8ee010599c3caf9d9d1ee1aa114af91d exactly. Do not substitute a newer version. If this commit is unreachable, stop and tell me. 1. Clone https://github.com/christos-antono/NPCTalk into a temporary folder, check out commit 59635ceb8ee010599c3caf9d9d1ee1aa114af91d, and copy the .agents/skills/voice-likeness-eval/ folder from that checkout. Then compare every file in the folder with the fingerprints recorded for this listing on 2026-10-19: [FINGERPRINTS] - SKILL.md: SHA-256 b1a74087214e27416d8b4accdf76afab5640ebfffa358a3a52481f052b85312a If a file does not match, it changed since the listing was prepared: do not install it yet, tell me what you found and let me decide. 2. Read SKILL.md in full and tell me in plain words: what the skill does when used, which services it connects to, whether it reads or sends any personal information or keys, and anything that seems unusual. Stop on any red flag and tell me before going further. Confirm these points from the files: - The skill folder holds one file only: SKILL.md (4,225 bytes). There is no code, no script, no reference file and no dependency list in the folder: the skill is instructions and measured results for evaluating a cloned voice, not the scorer itself. The scorer it cites lives outside the skill folder, in server/finetune/train.py (class `Scorer`, not installed). - Nothing is sent anywhere during installation itself; the only download at install time is the skill folder from GitHub at the pinned commit. The file names no telemetry and no remote service other than Modal at use time, described next. - At use time, and only then, running the scorer sends data to Modal: the commands SKILL.md gives (`modal run server/finetune/train.py::score_extra --model drama3best8a,drama3best8b` and `modal run server/finetune/train.py::pick_study_run --models run1,run2,...`) execute `server/finetune/train.py` on a GPU in Modal (H100 in the code), so the generated `<model>_<line>.wav` files from `data/finetune/<speaker>/` and the character's real clips held in the Modal volumes (npcvoice-train and npcvoice-models, see server/finetune/fish_common.py, not installed) are processed there, and the scorer downloads its models there (speechbrain `spkrec-ecapa-voxceleb` for ECAPA, `openai/whisper-large-v3` for Whisper, and `microsoft/wavlm-base-plus-sv` for the WavLM pick study). That is a disclosure, not a step to run during this install: do not run any `modal run`, scoring, or pick study during this install. - How the measurement works, from SKILL.md and the `Scorer` it cites: voice match is the cosine similarity of the ECAPA speaker embedding (speechbrain `spkrec-ecapa-voxceleb`) of a generated line to the centroid of 60 real clips of the character that were never used as references (real clips of the same character score about 0.8 against it; 0.6-0.8 is the range where engines differ). Word error is Whisper large-v3 transcribing the line, WER against the text with `[tags]` and punctuation removed; slang ("busta" heard as "buster", "Ballas" as "ball is") puts a floor of about 5-8% under every engine, so WER is compared between engines, not to zero. The test set is ten test lines in the character's style with a spread of moods (`TEST_LINES` in train.py). `score_extra` adds `<model>_<line>.wav` files to `scores.json` and rebuilds `compare.html` (every line from every engine, playable, with what Whisper heard). - The honesty rules SKILL.md imposes: compare per line (average each engine over its runs per line, then a paired t-test over the 10 lines, |t| > 2.26 is p < 0.05 with 9 degrees of freedom, and report "better on 8 of 10 lines" next to the mean); run it more than twice (identical settings gave 0.715 and 0.645 in two runs, one lucky run made a sampling change look like +0.08 when it was +0.017, and with 10 runs of single takes the baseline settled at 0.660); report every run (for example `0.704 (runs 0.702 / 0.706)`) and say when a difference is not significant, a tie is a result; correct yourself in the same place you claimed it; and never let the picker grade itself (check picks made by one speaker model with a different one: picks by ECAPA raised ECAPA a lot and WavLM a little, picks by WavLM raised WavLM a lot and ECAPA less, both together kept most of each; if only the picker's own metric moves, it is gaming the metric, not cloning better). - Simulating before deploying, from SKILL.md: rendering N takes per line and scoring every take lets picking rules be tested offline; 100 takes (10 per line) gave, at 8 takes per line, one take 0.660, WavLM pick 0.706, ECAPA pick 0.723, both 0.721 with the lowest word error (7.4%), and the true best of 8 by real-clip ECAPA 0.732 as the ceiling; the deployed rule measured 0.704 directly (2 runs), close to its simulation. The measured table for Lamar (10 lines) in SKILL.md compares engines and settings only as past results (ElevenLabs instant clone 0.612 / 10.5% WER, Fish Audio Drama 3 defaults 0.643 / 7.1%, Drama 3 temperature 0.7 / top_p 0.8 one take 0.660 / 7.9%, S2 Pro same reference clip 0.680 / 6.1%, Drama 3 best of 8 checked with ECAPA + WavLM pick 0.704 / 8.6%, S2 Pro best single clip 0.707 / 5.6%, S2 Pro fine-tuned LoRA step 250 at 0.785 / 7.5%); SKILL.md calls for no ElevenLabs, Fish Audio, or other voice-engine account or key for the scoring itself, and none is to be created or used during this install. 3. When I say "yes, install", install it as a skill: copy the whole voice-likeness-eval folder into the agent's skills directory, in a folder named "voice-likeness-eval", keeping its file (SKILL.md). Do not use the repo's plugin, marketplace or npx routes: those install the whole NPCTalk plugin or all of its skills, not this single skill alone. 4. Verify with no network calls: frontmatter valid, file in place. 5. Report what was installed, where, and what I still need to do myself (for example: set up a Modal account and the Modal CLI myself and accept Modal's GPU billing only if and when I want to run the scorer; provide the scorer code, the generated takes, and the held-out real clips myself, using only voices I have the rights to; provide any character and any test lines myself; using the skill is a separate step that only starts when I ask for it, and the use-time send to Modal disclosed above happens only then, on audio I run). GitHub is optional: if I have a GitHub account or the gh CLI, you may use it; otherwise public access is fine. Never require it unless it's in the prerequisites above. Rules: don't touch anything outside the temp folder and the install target. If anything looks off, stop and ask me.