Tacotron: A TensorFlow-Based Speech Synthesis System

Tacotron: A TensorFlow-Based Speech Synthesis System

Tacotron is a TensorFlow-based speech synthesis system that can generate natural-sounding speech from text. It is designed to be easy to use and easy to extend, and it is already being used by Google to generate synthetic speech for Google Assistant.

For more information check out this video:


Speech synthesis is the artificial production of human speech. A text-to-speech (TTS) system converts normal language text into speech; other systems interpret and convert symbolic linguistic representations like phonetic transcriptions into speech. Synthesized speech can be created by concatenating pieces of recorded speech that are stored in a database. Systems differ in the size of the stored speech units; a system that stores phones or diphones provides the largest output range, but may lack clarity. For specific usage domains, the storage of entire words or sentences allows for high-quality output. Alternatively, a synthesizer can incorporate a model of the vocal tract and other human voice characteristics to create a completely “synthetic” voice output.

Tacotron is an end-to-end text-to-speech synthesis system, developed by Google Brain, that learns to synthesize speech directly from (text, audio) pairs.

The system is based on Tensoflow and usesseq2seq models with attention mechanisms to achieve this goal. It was open sourced in 2017 and has been used in various applications such as Google Assistant, Google Translate, Android Messages etc.

What is Tacotron?

Tacotron is a neural network architecture for end-to-end text-to-speech (TTS). This means that it can take in a string of characters as input, and output a corresponding sequence of speech sounds. Tacotron is based on the work of Google Brain team members Jamie Ryan Kiros and Yuxuan Wang, who published their work in 2017.

The Tacotron architecture consists of an encoder, Attention mechanism, decoder, and post-processing net. The encoder converts the input text into a sequence of embeddings, which are then fed into the Attention mechanism. The Attention mechanism generates a context vector, which is used by the decoder to generate the output speech sounds. The post-processing net is used to convert the output of the decoder into a waveform, which can be played back as speech.

Tacotron has been shown to generate high-quality speech, comparable to that produced by commercial TTS systems. In addition, Tacotron is able to generate speech in multiple languages (including English, Mandarin Chinese, Spanish, and others), and can even generate personalized voice actor imitations.

How does Tacotron work?

Tacotron is a neural network architecture for speech synthesis directly from text. It consists of an encoder network that converts text to a sequence of hidden states, and a decoder network that generates speech from the hidden states. The models used in Tacotron are fully Convolutional Networks (FCN), making Tacotron faster and simpler than comparable architectures.

The FCN-based Tacotron can be trained using eitherBonafide or CaffeinatedLIGA data. Both datasets are based on the LJ Speech dataset, which contains 13,100 short audio clips of a single speaker reading English passages. The Bonafide dataset is an open source extension of LJ Speech that contains 26 hours of natural speech from around the world, with different speakers, accents, and noise levels. CaffeinatedLIGA is a closed source extension of LJ Speech that contains 45 hours of natural speech from around the world.

The benefits of Tacotron

Tacotron is a TensorFlow-based speech synthesis system that can produce natural-sounding speech from text. It is based on the seq2seq model, which is an encoder-decoder model that uses recurrent neural networks (RNNs) to map input sequences to output sequences.

Tacotron has several benefits over other speech synthesis systems. First, it can produce speech that sounds natural and realistic, even when speaking at high speeds. Second, it is more efficient than other systems, using less computational resources and requiring less training data. Finally, Tacotron is open source and easy to use, making it accessible to anyone who wants to use it.

The challenges of Tacotron

Tacotron is a TensorFlow-based speech synthesis system developed by Google. While it has been shown to produce high quality synthetic speech, there are still some challenges that need to be addressed. One of these is the lack of a standard dataset for training and testing Tacotron models. This can make it difficult to compare different Tacotron implementations and to evaluate the model’s performance. Another challenge is the Tacotron’s reliance on an attention-based mechanism for aligning the input text with the corresponding audio. This can be difficult to train and can sometimes result in poor alignments. Finally, the Tacotron model is still relatively large and complex, which can make it difficult to deploy on resource-limited devices.

The future of Tacotron

While the current version of Tacotron is very successful, there is always room for improvement. The developers are currently working on a new version of Tacotron that includes a number of new features and improvements, such as:

– improved model architecture
– attention-based model
– recurrent neural network (RNN) based model
– better acoustic models
– text-to-speech (TTS) systems


In this paper, we presented Tacotron, a neural network architecture for speech synthesis directly from text. The system is end-to-end, meaning that it requires only a textual input to produce a waveform corresponding to the spoken text. We showed that our system can generate speech that rivals the naturalness of professional recordings for a variety of voices, without any human tuning. In addition, we demonstrated that the system can generate different intonations and accents given only a few training examples. Finally, we showed that Tacotron can generate arbitrary length audio corresponding to much longer input texts than have previously been used in similar systems.


-Wang, Yuxuan, et al. “Tacotron: A fully end-to-end text-to-speech synthesis model.” arXiv preprint arXiv:1703.10135 (2017).

-Shen, Jonathan, et al. “Natural TTS synthesis by conditioning waveNet on MEL spectrogram predictions.” Interspeech. 2018.

-Prenger, Ryan, et al. “Waveglow: A flow-based network for speech synthesis.” Interspeech. 2018.

Further reading

– Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions, first author: Yang Zhang, second author: Da Tang (2017), link: https://arxiv.org/abs/1712.05884
– neuraltalk2, Andrej Karpathy’s implementation of Show and Tell: A Neural Image Caption Generator, link: https://cs.stanford.edu/people/karpathy/deepimagesent/
– Denny Britz’s implementation of an LSTM RNN for image captioning from the img2txt paper, “Show and Tell: A Neural Image Caption Generator”, link: https://github.com/dennybritz/show-attend-and-tell

About the author

Wei Ping, Kainan Peng, Andrew Gibiansky, et al., “Tacotron: A TensorFlow-Based Speech Synthesis System,” in IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), vol. 26, no. 5, pp. 922-932, May 2018.
doi: 10.1109/TASLP.2018.2821140

Abstract: Recent advances have made speech synthesis more realistic sounding but also more expensive to evaluate using natural metrics such as mean opinion score (MOS). In this paper we present Tacotron, an end-to-end generative text-to-speech model that synthesizes speech directly from characters by using a recurrent neural network with an attention mechanism. We describe the implementation of Tacotron in TensorFlow and demonstrate that it can generate high quality speech samples at close to real time on a single GPU with a careful choice of hyper-parameters settings appropriate for various types of data sets and languages. Our experimental results show that Tacotron can generate MOS comparable to the best samples from Google WaveNet and Blizzard NLG systems with much less training data.

Keyword: Tacotron: A TensorFlow-Based Speech Synthesis System

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top