Abstract.This work proposes FireRedTTS, a foundation text-to-speech framework, to meet the growing demands for personalized and diverse generative speech applications. The framework comprises three parts: data processing, foundation system, and downstream applications. First, we comprehensively present our data processing pipeline, which transforms massive raw audio into a large-scale high-quality TTS dataset with rich annotations and a wide coverage of content, speaking style, and timbre. Then, we propose a language-model-based foundation TTS system. The speech signal is compressed into discrete semantic tokens via a semantic-aware speech tokenizer, and can be generated by a language model from the prompt text and audio. Then, a two-stage waveform generator is proposed to decode them to the high-fidelity waveform. We present two applications of this system: voice cloning for dubbing and human-like speech generation for chatbots. The experimental results demonstrate the solid in-context learning capability of FireRedTTS, which can stably synthesize high-quality speech consistent with the prompt text and audio. For dubbing, FireRedTTS can clone target voices in a zero-shot way for the UGC scenario and adapt to studio-level expressive voice characters in the PUGC scenario via few-shot fine-tuning with 1-hour recording. Moreover, FireRedTTS achieves controllable human-like speech generation in a casual style with paralinguistic behaviors and emotions via instruction tuning, to better serve spoken chatbots.

Contents

• Demo
• System Overview
• Human-Like Speech Generation
• Emotional Speech Generation
• Para-Lingual Speech Generation
• Zero-Shot Voice Cloning
• Zero-Shot In-Context Learning
• Few-Shot Speaker Adaptation
• Ablation Studies
• Prompt Pre-processing for Robust Voice Clone
• Non-Streamable and Streamable Decoder

Demo

System Overview

Figure 1. An overview of the FireRedTTS foundation system with a text-to-speech language model that maps text tokens to semantic tokens, and a two-stage token-to-waveform generator. The semantic tokens are extracted using a semantic-aware speech tokenizer. In Mel decoder, (c) illustrates a flow-matching-based approach that transforms the noise into a spectrogram using semantic tokens as conditions, and (d) depicts a streamable decoder based on a multi-stream language model and a Mel Codec.

Human-Like Speech Generation

Emotions

Paralingustics

Voice Cloning

Zero-Shot In-Context Learning

Few-Shot Speaker Adaptation

Ablation Studies

Prompt Processing for Robust Voice Cloning

Non-Streamable and Streamable Decoder