
A team of generative AI researchers created a Swiss Army knife for sound, one that allows users to control the audio output simply using text.
While some AI models can compose a song or modify a voice, none have the dexterity of the new offering.
Called Fugatto (short for Foundational Generative Audio Transformer Opus 1), it generates or transforms any mix of music, voices and sounds described with prompts using any combination of text and audio files.
For example, it can create a music snippet based on a text prompt, remove or add instruments from an existing song, change the accent or emotion in a voice - even let people produce sounds never heard before.
This thing is wild, said Ido Zmishlany, a multi-platinum producer and songwriter - and cofounder of One Take Audio, a member of the NVIDIA Inception program for cutting-edge startups. Sound is my inspiration. It's what moves me to create music. The idea that I can create entirely new sounds on the fly in the studio is incredible.
A Sound Grasp of Audio We wanted to create a model that understands and generates sound like humans do, said Rafael Valle, a manager of applied audio research at NVIDIA and one of the dozen-plus people behind Fugatto, as well as an orchestral conductor and composer.
Supporting numerous audio generation and transformation tasks, Fugatto is the first foundational generative AI model that showcases emergent properties - capabilities that arise from the interaction of its various trained abilities - and the ability to combine free-form instructions.
Fugatto is our first step toward a future where unsupervised multitask learning in audio synthesis and transformation emerges from data and model scale, Valle said.
A Sample Playlist of Use Cases For example, music producers could use Fugatto to quickly prototype or edit an idea for a song, trying out different styles, voices and instruments. They could also add effects and enhance the overall audio quality of an existing track.
The history of music is also a history of technology. The electric guitar gave the world rock and roll. When the sampler showed up, hip-hop was born, said Zmishlany. With AI, we're writing the next chapter of music. We have a new instrument, a new tool for making music - and that's super exciting.
An ad agency could apply Fugatto to quickly target an existing campaign for multiple regions or situations, applying different accents and emotions to voiceovers.
Language learning tools could be personalized to use any voice a speaker chooses. Imagine an online course spoken in the voice of any family member or friend.
Video game developers could use the model to modify prerecorded assets in their title to fit the changing action as users play the game. Or, they could create new assets on the fly from text instructions and optional audio inputs.
Making a Joyful Noise One of the model's capabilities we're especially proud of is what we call the avocado chair, said Valle, referring to a novel visual created by a generative AI model for imaging.
For instance, Fugatto can make a trumpet bark or a saxophone meow. Whatever users can describe, the model can create.
With fine-tuning and small amounts of singing data, researchers found it could handle tasks it was not pretrained on, like generating a high-quality singing voice from a text prompt.
Users Get Artistic Controls Several capabilities add to Fugatto's novelty.
During inference, the model uses a technique called ComposableART to combine instructions that were only seen separately during training. For example, a combination of prompts could ask for text spoken with a sad feeling in a French accent.
The model's ability to interpolate between instructions gives users fine-grained control over text instructions, in this case the heaviness of the accent or the degree of sorrow.
I wanted to let users combine attributes in a subjective or artistic way, selecting how much emphasis they put on each one, said Rohan Badlani, an AI researcher who designed these aspects of the model.
In my tests, the results were often surprising and made me feel a little bit like an artist, even though I'm a computer scientist, said Badlani, who holds a master's degree in computer science with a focus on AI from Stanford.
The model also generates sounds that change over time, a feature he calls temporal interpolation. It can, for instance, create the sounds of a rainstorm moving through an area with crescendos of thunder that slowly fade into the distance. It also gives users fine-grained control over how the soundscape evolves.
Plus, unlike most models, which can only recreate the training data they've been exposed to, Fugatto allows users to create soundscapes it's never seen before, such as a thunderstorm easing into a dawn with the sound of birds singing.
A Look Under the Hood Fugatto is a foundational generative transformer model that builds on the team's prior work in areas such as speech modeling, audio vocoding and audio understanding.
The full version uses 2.5 billion parameters and was trained on a bank of NVIDIA DGX systems packing 32 NVIDIA H100 Tensor Core GPUs.
Fugatto was made by a diverse group of people from around the world, including India, Brazil, China, Jordan and South Korea. Their collaboration made Fugatto's multi-accent and multilingual capabilities stronger.
One of the hardest parts of the effort was generating a blended dataset that contains millions of audio samples used for training. The team employed a multifaceted strategy to generate data and instructions that considerably expanded the range of tasks the model could perform, while achieving more accurate performance and enabling new tasks without requiring additional data.
They also scrutinized existing datasets to reveal new relationships among the dat
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
17/11/2026
Fabric Insights and Fonn Group have announced a partnership to connect editorial...
17/11/2026
The National Association of Black Journalists is offering up a Thanksgiving Eve treat on Wednesday, Nov. 25, in the form of a live streaming event featuring for...
17/11/2026
Bridge Technologies has announced that its VB440 production probe now supports both monitoring and signal generation for Apple ProRes video streams. The additio...
17/11/2026
Canon U.S.A., Inc. recently welcomed 21 Jericho High School students to its Melv...
17/11/2026
Spectrum News has deployed the Amagi platform to support its live news operations. Amagi says the platform lets Spectrum News streamline newsroom workflows, red...
17/11/2026
Microsoft has expanded its use of Haivision's live video contribution ecosys...
17/11/2026
Setanta Sports has deployed Ateme's TITAN Live platform to support its OTT video delivery operations. Ateme's transcoding technology is integrated into ...
17/11/2026
Media Links has joined the KAIROS Alliance Partner program for Panasonic's KAIROS IT/IP live video production platform. The company completed interoperabili...
17/11/2026
Behind The Mic provides a roundup of recent news regarding on-air talent, including new deals, departures, and assignments compiled from press releases and repo...
17/11/2026
In live sports and in life, things go by in the blink of an eye. Savoring the mo...
17/11/2026
Satellite operators are adding a new generation of low-Earth-orbit (LEO) network...
17/11/2026
Clear-Com has delivered five customized test systems for electromagnetic securit...
17/11/2026
Stephen Arnold Music (SAM) has created the sonic identity for WTGL, the women's team golf league from TMRW Sports, created in partnership with the LPGA and ...
17/11/2026
The Professional Audio Manufacturers Alliance (PAMA) has introduced Give Sound a Voice, an initiative aimed at raising awareness of the value of sound among the...
17/11/2026
Fanatics Studios, the sports entertainment joint venture between Fanatics and OBB, has named Jamal Henderson its first President of Film & TV, announced Michael...
17/11/2026
The NFL's Thanksgiving Week games will feature halftime performances by U2, ...
17/11/2026
FanSided, a Minute brand with more than 250 fan communities, has launched a redesigned mobile app that combines team news, analysis, live game coverage, stats a...
17/11/2026
Major League Volleyball (MLV), the longest-running indoor professional volleybal...
17/11/2026
Prime Video will present a 15-hour slate of live sports on Black Friday, Nov. 27...
17/11/2026
Catherine Dior was more than a muse. A Resistance fighter, a survivor of deporta...
17/11/2026
Multiband dynamics plug-in boasts FT1 integration
Freqport's latest plug-in combines multiband upward and downward dynamics with the company's renow...
17/11/2026
Hi-fi experts launch two pro-audio models
Berlin-based hi-fi experts Teufel Audio have recently branched out into the pro-audio world with their first pairs...
17/11/2026
A year of ComposerCloud+ for $99
EastWest's annual Black Friday Sale is already live, and new users are being offered one year of the all-encompassing C...
17/11/2026
Pick your favourite gear from 2026
The annual SOS Awards are now underway, and as always, we're asking readers to help us choose the best pro-audio rele...
17/11/2026
As part of an ongoing series production program for a German automaker, production capacity has been increased at SGL Carbon's Austrian site in Ried im Innk...
17/11/2026
eds3_5_jq(document).ready(function($) { $(#eds_sliderM519).chameleonSlider_2_1({...
17/11/2026
Signiant has acquired the media intelligence technology, Ceivo, developed by Burnlab with founder Aaron Sloman joining the company as Vice President, Product Ma...
17/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
17/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
17/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
17/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
17/11/2026
Berklees 2027 Alumni Grammy Nominees: See the Full List From pop and R&B to jazz, musical theater, and video game scores, Berklee talent is represented across...
17/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
17/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
17/11/2026
LA JOLLA, CA-Two Scripps Research faculty, Assistant Professor Christian Diercks...
17/11/2026
Our Biggest Sale of the Year on Ivory 3!
From November 17th through November 30th, enjoy our best savings of the year on Ivory 3 Products!
Whether it's ou...
17/11/2026
Tuesday 17 November 2026
Sky announces major free upgrade to the Sky Glass and ...
17/11/2026
Tuesday 17 November 2026
Passion, Betrayal and Scandal: Sky Arts Reveals the Ma...
17/11/2026
Exact 3D Video, a web service based in Japan, has launched its English version, ...
17/11/2026
Today, Apple Podcasts named Baby, this is Keke Palmer its Show of the Year, hono...
17/11/2026
NVIDIA's Ian Buck kicked off the company's week at SC26 in Chicago on Monday with new tools for AI agents, software that can plan experiments, run code ...
16/11/2026
The ESPN Edge Innovation Conference once again gave key ESPN technology partners as a chance to take to the stage and discuss the future of sports fan experienc...
16/11/2026
As rights holders rethink who tells their stories and where, IMG is preparing th...
16/11/2026
From the sidelines, sonic savants advocate for greater emphasis on sound as sports and entertainment increasingly merge
SVG Live!, a one-day event on Nov. 10 i...
16/11/2026
Deal combines Yamaha's audio technology with Disguise's visual-productio...
16/11/2026
Creative effects designed to encourage experimentation
Berry Effects is the brainchild of Abe and Ralph, a pair of multi-instrumentalists and audio engineer...
16/11/2026
Physics-based synth gets first major update
Anukari have just released a new version of their innovative physics-based software synth, marking the instrumen...
16/11/2026
Both versions kitted out with new tools & improvements
Hit'n'Mix have just launched the latest major version of their innovative DAW software, with ...
16/11/2026
Dynamic bands, unmasking features & new stereo tools
Launched in 2021, Eventide's SplitEQ plug-in introduced an interesting new approach to EQ, with the...