
A team of generative AI researchers created a Swiss Army knife for sound, one that allows users to control the audio output simply using text.
While some AI models can compose a song or modify a voice, none have the dexterity of the new offering.
Called Fugatto (short for Foundational Generative Audio Transformer Opus 1), it generates or transforms any mix of music, voices and sounds described with prompts using any combination of text and audio files.
For example, it can create a music snippet based on a text prompt, remove or add instruments from an existing song, change the accent or emotion in a voice - even let people produce sounds never heard before.
This thing is wild, said Ido Zmishlany, a multi-platinum producer and songwriter - and cofounder of One Take Audio, a member of the NVIDIA Inception program for cutting-edge startups. Sound is my inspiration. It's what moves me to create music. The idea that I can create entirely new sounds on the fly in the studio is incredible.
A Sound Grasp of Audio We wanted to create a model that understands and generates sound like humans do, said Rafael Valle, a manager of applied audio research at NVIDIA and one of the dozen-plus people behind Fugatto, as well as an orchestral conductor and composer.
Supporting numerous audio generation and transformation tasks, Fugatto is the first foundational generative AI model that showcases emergent properties - capabilities that arise from the interaction of its various trained abilities - and the ability to combine free-form instructions.
Fugatto is our first step toward a future where unsupervised multitask learning in audio synthesis and transformation emerges from data and model scale, Valle said.
A Sample Playlist of Use Cases For example, music producers could use Fugatto to quickly prototype or edit an idea for a song, trying out different styles, voices and instruments. They could also add effects and enhance the overall audio quality of an existing track.
The history of music is also a history of technology. The electric guitar gave the world rock and roll. When the sampler showed up, hip-hop was born, said Zmishlany. With AI, we're writing the next chapter of music. We have a new instrument, a new tool for making music - and that's super exciting.
An ad agency could apply Fugatto to quickly target an existing campaign for multiple regions or situations, applying different accents and emotions to voiceovers.
Language learning tools could be personalized to use any voice a speaker chooses. Imagine an online course spoken in the voice of any family member or friend.
Video game developers could use the model to modify prerecorded assets in their title to fit the changing action as users play the game. Or, they could create new assets on the fly from text instructions and optional audio inputs.
Making a Joyful Noise One of the model's capabilities we're especially proud of is what we call the avocado chair, said Valle, referring to a novel visual created by a generative AI model for imaging.
For instance, Fugatto can make a trumpet bark or a saxophone meow. Whatever users can describe, the model can create.
With fine-tuning and small amounts of singing data, researchers found it could handle tasks it was not pretrained on, like generating a high-quality singing voice from a text prompt.
Users Get Artistic Controls Several capabilities add to Fugatto's novelty.
During inference, the model uses a technique called ComposableART to combine instructions that were only seen separately during training. For example, a combination of prompts could ask for text spoken with a sad feeling in a French accent.
The model's ability to interpolate between instructions gives users fine-grained control over text instructions, in this case the heaviness of the accent or the degree of sorrow.
I wanted to let users combine attributes in a subjective or artistic way, selecting how much emphasis they put on each one, said Rohan Badlani, an AI researcher who designed these aspects of the model.
In my tests, the results were often surprising and made me feel a little bit like an artist, even though I'm a computer scientist, said Badlani, who holds a master's degree in computer science with a focus on AI from Stanford.
The model also generates sounds that change over time, a feature he calls temporal interpolation. It can, for instance, create the sounds of a rainstorm moving through an area with crescendos of thunder that slowly fade into the distance. It also gives users fine-grained control over how the soundscape evolves.
Plus, unlike most models, which can only recreate the training data they've been exposed to, Fugatto allows users to create soundscapes it's never seen before, such as a thunderstorm easing into a dawn with the sound of birds singing.
A Look Under the Hood Fugatto is a foundational generative transformer model that builds on the team's prior work in areas such as speech modeling, audio vocoding and audio understanding.
The full version uses 2.5 billion parameters and was trained on a bank of NVIDIA DGX systems packing 32 NVIDIA H100 Tensor Core GPUs.
Fugatto was made by a diverse group of people from around the world, including India, Brazil, China, Jordan and South Korea. Their collaboration made Fugatto's multi-accent and multilingual capabilities stronger.
One of the hardest parts of the effort was generating a blended dataset that contains millions of audio samples used for training. The team employed a multifaceted strategy to generate data and instructions that considerably expanded the range of tasks the model could perform, while achieving more accurate performance and enabling new tasks without requiring additional data.
They also scrutinized existing datasets to reveal new relationships among the dat
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
07/10/2026
Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...
06/09/2026
June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...
04/08/2026
Dalet, a leading technology and service provider for media-rich organizations, t...
30/07/2026
CP Communications provided RF audio, RF video, communications, RF coordination, ...
30/07/2026
OpenDrives has been included in CRN's 2026 Storage 100 list in the Software-Defined Storage category. The annual list, selected by the CRN editorial team, r...
30/07/2026
LATAM Airlines has selected SES to provide multi-orbit inflight connectivity to its fleet of Airbus and Embraer aircraft. More than 60 aircraft - including Airb...
30/07/2026
Tagboard has launched the Producer API, an open interface that allows key commands in Tagboard's live production environment to be triggered from external d...
30/07/2026
FOX Sports and the New York Racing Association (NYRA) have announced a multi-year agreement with Del Mar Thoroughbred Club that will make FOX Sports the exclusi...
30/07/2026
ABC Commercial has launched four free ad-supported streaming television (FAST) channels on LG Smart TVs across North America, Great Britain, and select countrie...
30/07/2026
The Six Kings Slam exhibition tennis tournament will return to Riyadh on October 21, 22, and 24, streaming live on Netflix at no additional cost to subscribers....
30/07/2026
Central Christian Church in Mt. Vernon, Illinois has deployed Clear-Com's LQ...
30/07/2026
Blackmagic Design has released the UltraStudio Express 3G family, a pair of USB4 capture and playback devices compatible with Mac, Windows, and Linux computers ...
30/07/2026
ESPN has reached a media rights agreement with the Women's Pro Baseball League (WPBL), making ESPN the national streaming home of the league's 2026 seas...
30/07/2026
Most Valuable Promotions (MVP) and the Professional Fighters League (PFL) have announced a merger that will operate under the MVP banner. PFL CEO John Martin wi...
30/07/2026
The Columbus Blue Jackets will simulcast all game broadcasts across television and radio beginning with the 2026-27 NHL season. Under the new format, the televi...
30/07/2026
TMRW Sports has selected Populous as architect for a purpose-built stadium for its professional flag football league, being developed in partnership with the NF...
30/07/2026
SiriusXM has announced SiriusXM Sports Pass, a new subscription plan launching September 1 that bundles the company's sports audio programming into a single...
30/07/2026
Leagues, broadcasters, and technologists gather to tackle cloud economics, distr...
30/07/2026
A major player in the cloud-based solutions, Amazon Web Services (AWS) has conti...
30/07/2026
The NBA will provide production for all local game broadcasts, including pregame and postgame coverage...
30/07/2026
Cosm will open its fourth immersive sports and entertainment venue in downtown Detroit on September 10. Located at 25 Cadillac Square, adjacent to Campus Martiu...
30/07/2026
NFL Network will be available on the ESPN App beginning July 30 for Unlimited plan subscribers. The addition brings NFL Network's live games, studio program...
30/07/2026
NEP Group has added Riedel's SimplyLive Production Suite to NEP Platform, giving customers access to the live production software through NEP's orchestr...
30/07/2026
LOS ANGELES, CA, July 30, 2026 - The nonprofit Sundance Institute today announce...
30/07/2026
A new take on classic algorithmic reverbs
Waves have just introduced a new flagship algorithmic reverb plug-in that offers the company's take on the ico...
30/07/2026
Save up to 30% on virtual instruments & plug-ins
Arturia have recently launched their annual Summer Sale, which sees discounts of up to 30% applied across a...
30/07/2026
ActiveAnalogue technology powers new flagship modular console
A year on from the launch of the Oracle, SSL have revealed a new flagship console which they s...
30/07/2026
Up to 22dBA of passive attenuation
The latest arrival to the beyerdynamic range introduces a new compact monitoring headphone that's been designed to ta...
30/07/2026
As part of its traditional graduation ceremony, SGL Carbon honored the achievements of a total of twelve young professionals who successfully completed their ap...
30/07/2026
Premiere date revealed for SBS Original drama The Airport Chaplain, starring Hug...
30/07/2026
TVNZ and Evergent are celebrating a major TVNZ milestone - the evolution from a free-to-view, ad-supported business into a hybrid model, offering premium sport...
30/07/2026
WebMiere Adds Native VP9/AV1 WebM/MKV Import to Adobe Premiere Pro
Brie Clayton July 29, 2026
0 Comments
Brought to you by KawaiiEngine, WebMiere is a...
30/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
30/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
30/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
30/07/2026
At IBC2026, GlobalM will showcase the latest evolution of its software-defined media platform, enabling broadcasters, telecom operators and media organisations ...
30/07/2026
At the 2026 TAB Show, Digital Alert Systems will highlight and demo the new DAS-Link AIR, an over-the-air common alert protocol (CAP) dissemination system for ...
30/07/2026
OpenDrives Named to the 2026 CRN Storage 100 List
Company Recognized Among the 50 Coolest Software-Defined Storage Vendors for Unlimited Capacity, S3 Hybrid M...
30/07/2026
New Scene Search capability in VIDA Content OS makes video content itself searchable, letting broadcasters and streamers find key moments across millions of ass...
30/07/2026
July 30th, 2026
TRIBECA FILMS ANNOUNCES SUMMER 2026 ACQUISITIONS INCLUDING L...
30/07/2026
Thursday 30 July 2026
Which WSL matches are Sky Sports showing on the 2026/27 opening weekend?
The season will kick off on Friday 4 September with Manchester ...
30/07/2026
Thursday 30 July 2026
Enjoy discounts on the iPhone Air, plus 5Gb speeds with U...
30/07/2026
Broadcaster John Creedon explores Ireland's rich musical heritage Sunday 2 A...
30/07/2026
Back to school means balancing assignments, deadlines and downtime. GeForce NOW makes it easy to have it all.
With cloud gaming, everyday laptops used for clas...
29/07/2026
In this series, Grass Valley explores the seven paradoxes shaping the next era o...
29/07/2026
Thats a wrap on Season 1 of SVG Students to Watch! Across the country, these stu...
29/07/2026
The agreement will move Gotham Sports App subscribers to DAZN during the 2026-27...
29/07/2026
Longtime digital-media leader expands role overseeing conference branding, video...
29/07/2026
By Bailey Pennick
One of the most exciting things about the Sundance Film Festival is having a front-row seat for the bright future of independent filmmaking. ...