01 Text to speech models
TTS models, end to end
Worked on text to speech models across the stack, from data preprocessing to pipelines that automate training. This included instant voice cloning and the data generation pipelines behind it.
About this visualisation
Most modern neural text to speech systems share one skeleton, and it is the one drawn above. Text is tokenised and pushed through an encoder to produce linguistic features. Those features are conditioned on a speaker latent, a learned representation of how a voice sounds, covering pitch, timbre, energy and delivery. The conditioned representation is then decoded into an acoustic representation such as a mel spectrogram or a sequence of audio codec tokens, and a neural vocoder, classically a GAN such as HiFi-GAN, turns that into the waveform you hear.
Architectures differ mainly in the middle. Autoregressive systems in the style of XTTS generate the acoustic representation one step at a time, feeding each step back into the input, which is the loop the diagram shows. Flow matching and diffusion systems such as F5-TTS instead refine the whole representation in parallel over a series of denoising steps, trading the loop for iteration in place. More recent LLM-centred designs in the style of Qwen TTS predict discrete codec tokens directly and hand them to a codec decoder rather than a mel vocoder. Speaker conditioning varies too, and embedding a few seconds of reference audio into the speaker latent is what makes instant voice cloning possible.
The speaker latents and the manifolds they live on are what make this space interesting to study. The acoustic features of human voices do not fill their space uniformly. They concentrate on a much lower-dimensional manifold, and a good speaker encoder maps every voice to a point on it. Distances on that manifold correspond to how similar two voices sound, moving along it changes pitch, timbre and delivery smoothly, and cloning a voice from a few seconds of audio is the act of locating a new speaker on a manifold learned from many others. How well a system generalises to voices it has never heard depends almost entirely on how well this structure is learned.
The visualisation commits to the parts these systems share, text encoding, speaker conditioning, stepwise generation and vocoding, and draws the autoregressive variant because it is the easiest to see. Colour carries the meaning. Text and speaker have their own hues, the conditioned input sits between them because it is both, generated steps take the warm accent, and the transformer layers stay neutral so they never compete with the data moving through them.
The waveform is drawn from the same samples the play button plays, so the picture and the sound are one piece of data. The sample is generated speech from a commercial neural text to speech system. If the file cannot be loaded the page falls back to a signal synthesised in the browser and says so.
- Pipeline
- Text encoder, speaker conditioning, autoregressive or diffusion decoder, neural vocoder
- Generation
- Four steps shown, each one appended to the input of the next
- Signal
- Generated speech sample, analysed at 22.05 kHz mono
- Scene
- Seven token blocks, one speaker block, three transformer layers, four generated tokens