Lately, Suno AI has been getting attention as a speech synthesis system. Can anyone explain what machine learning methods are used in this technology? How does the text-to-natural-sounding speech conversion work, and what audio processing stages are involved? I'm curious to know what limitations exist and how they can be mitigated.
What is Suno AI technology and how does it generate audio?
👁️ 140 views💬 3 replies❤️ 0 likes
3 Replies
Suno AI uses a hybrid approach: first, a transformer neural network (usually a GPT-like model) generates a phonetic contour of the text, and then a specialized vocoder (such as WaveGlow or HiFi-GAN) converts it into an audio waveform. In practice, I tried exporting a finished script into their online constructor and noticed that the quality heavily depends on text preprocessing: remove extra punctuation, break long sentences into short phrases, and add pause phrases («…», «—») where a natural breath is needed. This helps the model better match the intonation and avoid a "flat" delivery.
The main limitations are sensitivity to unfamiliar names and rare accents, as well as a limited set of emotions. To compensate, I add phonetic hints in square brackets to the original text (e.g., [ˈdʒeɪn] for the name "Jane") and use post-processing in an audio editor: slight compression and a subtle reverb "wash" make the speech sound more natural and lively. If a specific emotional tone is needed, you can "stitch" together multiple manually selected versions of the same phrase and mix them in a DAW program.
Suno AI uses a transformer-based TTS pipeline (similar to VITS), where a phoneme encoder and duration predictor generate a mel-spectrogram that a neural vocoder then converts into waveform audio. In my own tests, I've noticed the voice can sound a bit monotone with long, complex sentences, so feeding SSML cues or fine-tuning on an expressive dataset helps smooth out the prosody. The biggest limitations are rare word pronunciation and extreme intonation shifts, which you can work around by adding custom pronunciation rules or supplemental style tokens.
When I was experimenting with voice synthesizers to create a demo of my guitar riff, I noticed that Suno AI, like other TTS models, first converts text into phoneme tokens and then generates "living" sound through a neural vocoder. However, at fast vocal tempos, some artifacts can appear, which can be smoothed out with post-processing in an audio editor.