Plazoleta / Labs

OpenDub

Experimento

Dobla vídeos de YouTube al español con voz sintética sincronizada (whisper, Claude, edge-tts).

PythonMIT
Captura de OpenDub

Más información Abrir en GitHub →

A personal tool that dubs YouTube videos into Spanish with a synthetic voice, kept in sync with the original.

URL → download (yt-dlp) → transcription (subtitles or faster-whisper)
    → separation (HTDemucs) → translation (Claude)
    → voice (Chatterbox, IndexTTS, CosyVoice or edge-tts) → mix (numpy + ffmpeg) → MP4

How each piece works: docs/architecture.md.

Installation

macOS (Apple Silicon):

brew install ffmpeg uv
uv sync

Ubuntu (optionally with an NVIDIA GPU):

sudo apt install ffmpeg
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync

Voice/music separation (--separate) needs an optional extra: it installs demucs-mlx on Apple Silicon, and demucs with PyTorch (CUDA 12.8 wheels, which RTX 50-series GPUs require) on Linux:

uv sync --extra separate

On Linux, whisper and the separator use the NVIDIA GPU automatically when one is available; on a Mac, whisper runs on the CPU and the separator on the Apple GPU.

Voice cloning (--tts clone) needs another extra, with Chatterbox Multilingual V3 (installed from a pinned GitHub commit, since it isn't on PyPI yet). It also needs the separation, so install both:

uv sync --extra separate --extra clone

The other two cloning engines, IndexTTS-2.5 (--engine indextts) and CosyVoice 3 (--engine cosyvoice), pin their own Python and PyTorch, so each lives in a separate venv and runs as a separate process. setup-engine clones a pinned commit of its repo, creates that venv and downloads the weights into ~/.local/share/opendub/<engine> (or $OPENDUB_INDEXTTS_DIR, $OPENDUB_COSYVOICE_DIR):

uv run opendub setup-engine indextts   # ~5 GB, plus ~5 GB of auxiliary models on the first run
uv run opendub setup-engine cosyvoice  # ~9 GB

IndexTTS's venv comes from its repo's own lockfile, plus flash-attn for its acceleration engine (1.3× faster, same quality; without it if that fails). CosyVoice's comes from engines/cosyvoice/pyproject.toml, whose versions were checked on an RTX 5080 (its repo pins a PyTorch without RTX 50-series support). If an engine isn't installed, choosing it says which command installs it. (setup-indextts still works too.)

There are two ways to translate:

  • Claude subscription (default, --backend cli): uses your Claude Code session. Install it and run claude once to sign in:
    curl -fsSL https://claude.ai/install.sh | bash
    
  • API (--backend api): copy .env.example to .env and fill in ANTHROPIC_API_KEY. The same translation, billed per token instead of by the subscription: the way to go on if the subscription can't be used for this.
  • Local model (--backend ollama): the same prompt on a model served by Ollama (OLLAMA_HOST, by default http://127.0.0.1:11434), so a dub needn't depend on Claude at all. The default model is Qwen3.6-35B-A3B (ollama pull qwen3.6:35b-a3b; another with --model or OPENDUB_OLLAMA_MODEL in .env). It also shortens what doesn't fit, and the model is unloaded from the GPU before the voice phase needs it. Set against Sonnet on six videos (docs/bench/README.md) it is as fast but less faithful (87% of sentences kept whole against 96%) and less precise with the length, so more sentences are shortened and late: Sonnet stays the default.

Usage

uv run opendub dub "https://www.youtube.com/watch?v=rwF-X5STYks" -o dubbed.mp4

Common dub options:

Option Default What it does
--voice es-ES-AlvaroNeural edge-tts voice (es-ES-ElviraNeural, es-MX-JorgeNeural…)
--tts edge edge: the --voice voice · clone: the original speaker's voice, cloned from the separated vocals
--engine auto Cloning engine for --tts clone: auto (CosyVoice when several people speak, Chatterbox for one voice; see below), chatterbox, indextts or cosyvoice
--min-factor 0.7 indextts, cosyvoice: lowest duration factor before shortening a sentence that doesn't fit
--keep-translation off Reuses the existing translation even if it was made for another engine: only TTS and the mix are redone
--ref-per-sentence off Experimental: clones each sentence of 3 s or more, with nobody else over it, from its own original voice (at its speaker's pace; the speaker's reference is the first fallback). Heard blind it tied with one reference per speaker
--review off Stops after the translation, to read and edit it (translation.json, or the web's editor) before the voice; the same command without it goes on from there
--style none Extra style.json (register, variant, glossary, notes; see below) over work/style.json and the video's own
--clone-ref none Fixed Spanish recording to clone from when a cloned clip fails the quality check
--bg-volume -15 Volume of the original audio, in dB
--mute-original off Drops the original audio
--separate off Mixes over the separated background (music and effects, without the original voice) instead of the turned-down original
--music-volume 0 Volume of the separated background, in dB
--separator auto mlx (Apple Silicon) or torch (CUDA or CPU); auto picks by platform
--subs manual manual: manual subtitles or whisper · auto: auto-generated ones too · none: always whisper
--source-lang YouTube's Original language (whisper detects it when unknown)
--cookies-from-browser off (OPENDUB_COOKIES_FROM_BROWSER) When YouTube asks to confirm you aren't a bot: download as your own YouTube session, with this browser's cookies (firefox, chrome, chrome:Profile 1…; see below)
--whisper-model small faster-whisper model (medium, large-v3… are slower)
--backend / --model cli / claude-sonnet-5-5 How and with which model to translate: cli, api or ollama (default model qwen3.6:35b-a3b)
--rate -10% Native edge-tts speaking rate (more negative = slower, without stretching the audio)
--chars-per-second calibrated Speaking rate used to cap each translation's length; measured with the chosen voice by default
--shorten / --no-shorten on If a sentence doesn't fit even at --max-tempo, ask Claude for a shorter version and re-synthesize it
--max-tempo / --min-tempo 1.25 / 0.9 atempo range: a clip that ends well before its sentence did in the original is slowed down, without pushing the next one (--min-tempo 1 never slows down)
--force off Redoes every phase

Each phase can also run on its own: download, transcribe, translate, tts, separate and mix (all accept the URL or the video id). separate only needs download, so it can run at any point before mix --separate. Run uv run opendub <phase> --help to see its options.

Translation style and glossary

A style.json sets how a translation reads. work/style.json applies to every video, and work/<video id>/style.json to one video, winning over the global one. Both are optional, and the web page edits them too:

{
  "register": "tu",
  "variant": "es-ES",
  "terms": [
    {"source": "itch store", "target": "tienda de itch.io"},
    {"source": "VFX", "target": ""}
  ],
  "notes": "Pixel art channel: 'frame' stays as 'frame'."
}
  • register: tu, usted or auto (keeps the original's).
  • variant: es-ES, latam or auto (follows --voice). A variant also switches an edge-tts voice of the other variant to es-ES-AlvaroNeural or es-MX-JorgeNeural.
  • terms: how each term must be rendered; an empty target keeps it as in the original. Whole words, any case.
  • notes: free instructions for this channel or video.

Each block of 30 sentences only gets the terms it says, and is cached by its own inputs. Adding a term only re-translates the blocks that say it, and a term the video never says changes nothing. Shortened sentences keep the glossary too. A sentence whose spoken text doesn't use a term its original says is listed as a glossary issue on the web page, except hand edits.

Several voices

uv run opendub setup-engine nemotron        # once: NVIDIA Nemotron 3 Diarization in its own venv
uv run opendub dub <url> --tts clone --separate --multi-speaker

--multi-speaker (the web's Several voices) tells who speaks each sentence and gives each person their own voice: the vocals are diarized (speakers with under 30 s of speech are ignored), sentences are cut where the speaker changes, each speaker is cloned from their own turns (one with under 6 s of clean speech gets the stock voice of their gender), and each sentence is translated to fit its speaker's pace, with their gender for the agreement. On the web page, each speaker can be named, heard and merged into another, and any sentence given to someone else: a retouch then speaks it with the right voice. On the web page it is ticked by default once Nemotron is installed (a single-voice video keeps one voice); on the CLI it is asked for with --multi-speaker, also by the translate and tts phases run on their own.

When YouTube blocks the download

After many downloads in a short time YouTube may answer "Sign in to confirm you're not a bot" and refuse them for a while. Waiting is enough; or the download can go as your own YouTube session, with the cookies of a browser where you are signed in:

uv run opendub dub <url> --cookies-from-browser firefox     # chrome, chrome:Profile 1, brave…

For the web page, set it once in .env (OPENDUB_COOKIES_FROM_BROWSER=firefox) and restart it. yt-dlp reads the cookies on each download; nothing is copied or saved by OpenDub. Heavy downloading on a signed-in account can get that account limited by YouTube, so this is for the odd block, not for downloading in bulk.

Exporting

uv run opendub export KS837a-TFRc -f mkv            # the latest dub; --version for a kept one
Format What you get
mkv The video with the dubbed audio (default track), the original audio and both subtitles as selectable tracks
m4a The dubbed audio alone
srt Spanish subtitles of what the voice says, timed where the mix placed each clip, in screens of up to two lines
srt-orig The original sentences at their original timestamps

To correct the translation in a subtitle editor (Subtitle Edit, Aegisub…) when the web's editor falls short, export it with who says each sentence and read it back:

uv run opendub subs-export KS837a-TFRc -f ass       # or vtt (<v Name>), srt (no speaker)
uv run opendub subs-import KS837a-TFRc fixed.ass    # texts as hand edits, names as speakers
uv run opendub retouch KS837a-TFRc                  # speak the changed sentences again

One cue per sentence at the original's times; ASS puts the speaker in each line's Actor, WebVTT in a <v Name> span (and the sentence's id as the cue's). Importing matches each cue to its sentence (by that id, else by its start time), and stops without saving anything if a cue matches no sentence or a name is no speaker. The video page has the same: Subtitles: ASS · WebVTT · SRT and Import… over the translation.

Nothing is re-encoded: an MKV of a 19-minute video takes under a second. Each export is kept next to its version (outputs/<version>.mkv…), is deleted with it, and is offered by the web page's Download menu too.

Retouching a few sentences

uv run opendub retouch KS837a-TFRc --take 12 --take 40 --separate

retouch speaks again, with the dub's own engine, the sentences whose translation changed since the latest dub (hand edits in translation.json or from the web) and each --take, then mixes a new version. Each goes through the same fitting and quality check as a full dub, judged against the whole video. The chosen takes are remembered in takes.json, so a later full dub keeps them, and the TTS phase stays cached for the new translation. Hand-edited sentences are never rewritten shorter, in a retouch or a full dub: what doesn't fit is compressed as far as the engine allows and reported.

Web frontend

A local dashboard to dub without the command line, in English or Spanish, dark or light, with a choice of accent color (Preferences):

  • Library: paste a YouTube link, pick the voice (cloned or stock), the engine (Automatic by default, or tick several to get a version with each, dubbed one after another) and whether to keep the music; a dub goes from the link to the finished video without stopping (reviewing the translation before the voice is the CLI's --review); follow the running dub phase by phase, and find every video with its versions, the latest one, its issues and its status.
  • Video: the player, a timeline with each issue where it happens (click to jump), and every dub kept: each one plays and shows its numbers (voice time, sped up, shortened, below the floor, redone, edge-tts) and its own issues, downloads as MP4, MKV (both audio tracks and subtitles), audio or SRT, and older ones can be deleted (the latest stays). The translation, segment by segment, shows each issue of the version you watch next to its sentence, filters to those or to hand edits, and saves edits. Regenerate speaks again only the edited sentences and those marked for another take (the same text, the next seed of the cloned voice), then mixes a new version: ~30–40 s on a 19-minute video instead of a whole TTS phase. Re-dub… picks the engine (with what it did last time on that video) and keeps the translation by default, so only the voice and the mix run.
  • Compare: two versions side by side, playing in sync, heard one at a time (A/B keys), with how each one fitted and the segments where they differ.
  • Engines: each engine's license, where it runs, its benchmark figures and whether it is installed (with the command that installs it).

Every dub made is kept (outputs/<date>-<engine>.mp4, with its tts.json and overflow.log), whatever the engine and however many times the video is dubbed again; long videos take ~100 MB per version.

uv sync --extra web          # plus --extra separate --extra clone for cloned voices
uv run opendub web           # http://127.0.0.1:8000, opens the browser

It only listens on 127.0.0.1 and runs one job at a time (the others wait in order); it uses the same work/ folder and cache as the CLI.

To use it from another device on your network (a tablet, the TV's browser):

uv run opendub web --lan     # prints a link with an access token for each address

Open that link once on each device: the token is kept in a cookie and taken out of the address bar. Without it, every request gets a 401 and the page only says where to find the link. The token changes on every start, so old links stop working; anyone with a current link can use OpenDub on this computer, so share it only with your own devices. There is no mode open to the network without the token.

Only on networks you trust (your home, not a café's): it is plain HTTP, so the token and everything else travel unencrypted, and anyone on the same network who sees them can use the page. To reach it from outside, use a VPN to your network (WireGuard, Tailscale) rather than opening the port.

--lan --keep-token keeps the same token across restarts, in ~/.config/opendub/lan-token (readable only by you), so the links on your devices keep working, e.g. with the systemd service below. Delete that file to revoke them: the next start makes a new one.

Starting the web page on its own

Optional, on Linux: docs/systemd/opendub-web.service is a systemd user service that starts opendub web with your session and brings it back if it crashes. Copy it, set WorkingDirectory to your checkout, and enable it:

cp docs/systemd/opendub-web.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now opendub-web

Its log is in journalctl --user -u opendub-web (with --lan, the links with the token are there too). systemctl --user stop opendub-web stops it like a Ctrl+C: a dub that is running is cut short and goes on from its cache when dubbed again. To keep it running when you log out: loginctl enable-linger.

The page is a React 19 + TypeScript app in frontend/, built with Vite into src/opendub/web/static/, which is committed: running it needs no Node, only changing it does (Node 22+):

cd frontend && npm install
npm run dev      # http://localhost:5173, against a running `opendub web`
npm run build    # rebuild src/opendub/web/static/

Cache and working folder

Everything is stored in work/<video id>/. A phase is skipped if it already ran with the same parameters, and changing a parameter only redoes that phase and the ones that depend on it (for example, changing --voice repeats TTS and mixing, but neither the translation nor the separation).

File Contents
video.mp4, audio.wav, subs.* Download
segments.json Original sentences with timestamps
translation.json Translation (you can edit it by hand and rerun tts)
takes.json The seed of each sentence given another take (retouch)
style.json, translation.style.json The video's own translation style, and the one its translation was made with
translate_blocks/ Each translated block, cached by its own inputs
tts/, tts.json One clip per sentence, with its tempo, duration and spoken text (shortened if needed)
overflow.log Sentences that didn't fit their slot, clips with abnormal duration and delayed clips
stems/vocals.wav, stems/background.wav Separated original voice and background
clone_ref.wav 10–15 s of the original speaker used as the cloning reference (--tts clone)
dub.wav, output.mp4 Dubbed track and final video
thumbnail.jpg, job.json Web frontend: thumbnail, and the last job's options and outcome

Each voice's calibrated speaking rate is stored in work/voice_rates.json; delete it to measure again.

Voice cloning

With --tts clone, dub separates the audio right after transcribing it, picks the 10–15 s stretch of the original speaker with the fewest pauses as the reference and measures the cloned voice's speaking rate before translating. The log prints the reference's mean pitch (F0) next to the engine, and tts.json keeps both. Every cloned clip is transcribed back with whisper (--whisper-model); one whose text doesn't match, or whose duration is abnormal, goes down a fallback chain that keeps the same voice as long as possible:

  1. Regenerate with another seed.
  2. Regenerate with a shorter version of the sentence, written by Claude.
  3. Clone from the fixed reference (--clone-ref), if one was given.
  4. edge-tts (--voice) as a last resort, recorded in overflow.log.

tts.json records which of these produced each clip. Chatterbox adds Resemble AI's inaudible Perth watermark to everything it generates. On a GPU with 12 GB or more, Chatterbox speaks with two processes side by side (about 1.25× faster on a long video); OPENDUB_CHATTERBOX_WORKERS=1 keeps it to one. CosyVoice does the same on a GPU with 16 GB (1.7× faster: 438 s to 255 s on an 11-minute video, with a 13.1 GB peak); OPENDUB_COSYVOICE_WORKERS=1 keeps it to one. IndexTTS (7.4 GB) always speaks one clip at a time.

Choosing the engine

The default, --engine auto, picks by the dub: CosyVoice when several people speak, Chatterbox for one voice. Heard blind (docs/audit/03-competencia.md, section 12), CosyVoice won a two-host podcast clearly (4.2 against Chatterbox's 3.0 and IndexTTS's 2.7, preferred in 8 of 10 fragments) and Chatterbox narration by a little (4.0 against 3.5). Without CosyVoice installed, auto uses Chatterbox. Any engine can still be chosen per video with --engine.

Before that, Chatterbox and IndexTTS split the benchmark (details in docs/v2-plan.md): IndexTTS fits its slots far better (41/42 clips without atempo, against 30/42), and by ear it was preferred for the male voice but not for the female one, where Chatterbox sounded steadier (IndexTTS sometimes sounds as if trembling). On those two single-voice videos --engine indextts suited the low voice, Chatterbox the high one.

CosyVoice 3, benchmarked later on the same material, speaks fastest (~16 chars/s), so nearly every clip fits as is (42/42 in the benchmark; none of the 41 clips of the real dubs of both videos needed atempo), with a native accent and the least tremor on the male voice. It clones cross-lingually (reference audio, no transcript) and once in a while garbles a sentence; the quality check catches it (in the female test video, one sentence ended on edge-tts after every retry).

auto used to pick by the reference's pitch (IndexTTS below 165 Hz): a rule from two voices that the podcast session retired, since IndexTTS came last in a conversation. The log line and tts.json record the engine and how it was chosen.

The engines fit clips to their slot differently. Chatterbox has no length control, so clips that don't fit are sped up with atempo (up to --max-tempo) and, if that's not enough, shortened by Claude. IndexTTS (duration_factor) and CosyVoice (speed, which stretches the spectrogram before the vocoder) take a duration factor, and the order is:

  1. Speak each clip with the factor its calibrated speaking rate predicts for the slot (IndexTTS's rate is measured at factor 0.85, so translations aren't cut more than needed), never below --min-factor (0.7), and correct it once with the measured duration.
  2. If it still doesn't fit, ask Claude for a shorter sentence.
  3. Only if the shortened sentence doesn't fit either, compress below --min-factor, down to 0.5, noted in overflow.log; atempo covers whatever is left.

tts.json records the engine and the factor of each clip. The workers log to work/indextts.log and work/cosyvoice.log.

Each engine keeps its own license: see Licenses.

Licenses

OpenDub's own code, this repository, is under the MIT license. It covers OpenDub's code only. The voice engines and models run in their own venvs, are downloaded from their authors when installed (uv sync --extra …, setup-engine, Hugging Face), are not redistributed here, and each keeps its own license:

Component License Notes
Chatterbox Multilingual V3 (Resemble AI) MIT Its output carries Resemble AI's inaudible Perth watermark
IndexTTS-2.5 (bilibili) bilibili Model Use License Agreement Allows this personal use; separate license above 100 M monthly users; outputs may not train other models; revocable (details)
CosyVoice 3 (Alibaba FunAudioLLM) Apache 2.0 Code and weights
Nemotron 3 Diarization (NVIDIA) OpenMDW 1.1 Several voices
HTDemucs (demucs) MIT Separation
faster-whisper and the Whisper models MIT Transcription, quality check
yt-dlp Unlicense Download

Two services are used online: edge-tts talks to Microsoft's read-aloud voices (not an official API, under Microsoft's terms), and the translation sends the transcript to Anthropic (Claude) under your subscription or API terms. OpenDub is a personal tool: it is not packaged or supported for distribution.

Responsible use

OpenDub is a tool for personal use: watching in Spanish videos you'd otherwise watch in English. Don't publish dubs made with cloned voices of other people.

  • Clone a voice only with the consent of the person it belongs to, and don't present a cloned voice as that person speaking.
  • Dub only what you may adapt (your own videos, content whose license allows it, personal study): a dub is a derivative work of its source, and the rights of the original video stay with its owners.

Known limitations

  • If a sentence still doesn't fit at 1.25× after two shortening rounds, the next clip is delayed instead of overlapping; the drift is recovered at the next pause.
  • On a Mac, faster-whisper runs on the CPU (CTranslate2 doesn't use Metal): small transcribes about 2 minutes of audio in ~20 s.
  • edge-tts relies on an unofficial Microsoft service and could stop working.
  • Voice cloning follows a single speaker (the one in the reference); videos with several voices are all dubbed with that one.
  • On Linux, whisper uses the GPU only when cuBLAS and cuDNN can be loaded: they come with uv sync --extra separate (PyTorch's CUDA wheels) or a system CUDA install; otherwise it falls back to the CPU with a warning. Tested on an RTX 5080: large-v3 transcribes 2.5 minutes of audio in ~10 s.

Benchmarks

The scripts behind the engine choices (generation, CER, accent, voice similarity, F0, jitter and 3–9 Hz tremor) are in docs/bench/; uv run docs/bench/run.py <name> scores a new model against all the others.

Tests

uv run pytest

They need ffmpeg and the web extra, but no GPU and no voice engine: every test that would reach one replaces it. GitHub Actions runs them on each push and pull request (.github/workflows/ci.yml), and also checks the frontend: its types, and that the committed build in src/opendub/web/static matches what npm run build makes from frontend/ (rebuild and commit it after changing the frontend).