A team of generative AI researchers created a Swiss Army knife for sound, one that allows users to control the audio output simply using text.
While some AI models can compose a song or modify a voice, none have the dexterity of the new offering.
Called Fugatto (short for Foundational Generative Audio Transformer Opus 1), it generates or transforms any mix of music, voices and sounds described with prompts using any combination of text and audio files.
For example, it can create a music snippet based on a text prompt, remove or add instruments from an existing song, change the accent or emotion in a voice - even let people produce sounds never heard before.
This thing is wild, said Ido Zmishlany, a multi-platinum producer and songwriter - and cofounder of One Take Audio, a member of the NVIDIA Inception program for cutting-edge startups. Sound is my inspiration. It's what moves me to create music. The idea that I can create entirely new sounds on the fly in the studio is incredible.
A Sound Grasp of Audio We wanted to create a model that understands and generates sound like humans do, said Rafael Valle, a manager of applied audio research at NVIDIA and one of the dozen-plus people behind Fugatto, as well as an orchestral conductor and composer.
Supporting numerous audio generation and transformation tasks, Fugatto is the first foundational generative AI model that showcases emergent properties - capabilities that arise from the interaction of its various trained abilities - and the ability to combine free-form instructions.
Fugatto is our first step toward a future where unsupervised multitask learning in audio synthesis and transformation emerges from data and model scale, Valle said.
A Sample Playlist of Use Cases For example, music producers could use Fugatto to quickly prototype or edit an idea for a song, trying out different styles, voices and instruments. They could also add effects and enhance the overall audio quality of an existing track.
The history of music is also a history of technology. The electric guitar gave the world rock and roll. When the sampler showed up, hip-hop was born, said Zmishlany. With AI, we're writing the next chapter of music. We have a new instrument, a new tool for making music - and that's super exciting.
An ad agency could apply Fugatto to quickly target an existing campaign for multiple regions or situations, applying different accents and emotions to voiceovers.
Language learning tools could be personalized to use any voice a speaker chooses. Imagine an online course spoken in the voice of any family member or friend.
Video game developers could use the model to modify prerecorded assets in their title to fit the changing action as users play the game. Or, they could create new assets on the fly from text instructions and optional audio inputs.
Making a Joyful Noise One of the model's capabilities we're especially proud of is what we call the avocado chair, said Valle, referring to a novel visual created by a generative AI model for imaging.
For instance, Fugatto can make a trumpet bark or a saxophone meow. Whatever users can describe, the model can create.
With fine-tuning and small amounts of singing data, researchers found it could handle tasks it was not pretrained on, like generating a high-quality singing voice from a text prompt.
Users Get Artistic Controls Several capabilities add to Fugatto's novelty.
During inference, the model uses a technique called ComposableART to combine instructions that were only seen separately during training. For example, a combination of prompts could ask for text spoken with a sad feeling in a French accent.
The model's ability to interpolate between instructions gives users fine-grained control over text instructions, in this case the heaviness of the accent or the degree of sorrow.
I wanted to let users combine attributes in a subjective or artistic way, selecting how much emphasis they put on each one, said Rohan Badlani, an AI researcher who designed these aspects of the model.
In my tests, the results were often surprising and made me feel a little bit like an artist, even though I'm a computer scientist, said Badlani, who holds a master's degree in computer science with a focus on AI from Stanford.
The model also generates sounds that change over time, a feature he calls temporal interpolation. It can, for instance, create the sounds of a rainstorm moving through an area with crescendos of thunder that slowly fade into the distance. It also gives users fine-grained control over how the soundscape evolves.
Plus, unlike most models, which can only recreate the training data they've been exposed to, Fugatto allows users to create soundscapes it's never seen before, such as a thunderstorm easing into a dawn with the sound of birds singing.
A Look Under the Hood Fugatto is a foundational generative transformer model that builds on the team's prior work in areas such as speech modeling, audio vocoding and audio understanding.
The full version uses 2.5 billion parameters and was trained on a bank of NVIDIA DGX systems packing 32 NVIDIA H100 Tensor Core GPUs.
Fugatto was made by a diverse group of people from around the world, including India, Brazil, China, Jordan and South Korea. Their collaboration made Fugatto's multi-accent and multilingual capabilities stronger.
One of the hardest parts of the effort was generating a blended dataset that contains millions of audio samples used for training. The team employed a multifaceted strategy to generate data and instructions that considerably expanded the range of tasks the model could perform, while achieving more accurate performance and enabling new tasks without requiring additional data.
They also scrutinized existing datasets to reveal new relationships among the dat
Most recent headlines
11/12/2025
Dalet, a leading provider of cloud-native, end-to-end media workflow solutions, ...
03/12/2025
WASHINGTON The Federal Communications Commission has opened a docket for comments on the proposed $6.2 billion Nexstar acquisition for Tegna and set deadlines f...
03/12/2025
MILAN, Italy Brightcove has released seven new features designed to expand global reach, improve audience engagement, enhance live-streaming quality and streaml...
03/12/2025
NEW YORK Great American Media said it plans to launch Pure Flix Familia, a dedicated Spanish-language platform, in 2026....
03/12/2025
SOUTHPORT, Conn. Main Street Sports Group has announced that the FanDuel Sports Network app is now available directly on Vizio and on smart TVs with Vizio OS. T...
03/12/2025
SAN JOSE, Calif. Harmonic has announced that Telia, the second-largest telecom operator in Norway, is modernizing its broadband network with the company's c...
03/12/2025
GREEN BAY, Wis. Sinclair said Jay Zollar, vice president and general manager of WLUK-WCWF here, will retire Dec. 31 after 26 years running the stations. Station...
02/12/2025
Case Study: How Mid-Atlantic Sports Network Moved to All-IP Distribution in 60 D...
02/12/2025
2025 Sports Broadcasting Hall of Fame: Lee Corso, Coach, Commentator, FirebrandBy Ken Kerschbaumer
Tuesday, December 2, 2025 - 7:00 am
Print This Story | S...
02/12/2025
SVG All-Stars: Dan Nabors, Senior Director, Remote Engineering, TNT SportsThe veteran tech leader is helping guide Warner Bros. Discovery's at-home' re...
02/12/2025
National Lacrosse League Opens Season With New Cloud-Based Official Replay-Revie...
02/12/2025
Platinum White Paper: The Cinematic Look in Live Production - Bridging Aesthetic...
02/12/2025
SVG India Announces Advisory Board; JioStar's Prashant Khanna Named ChairmanBy Ken Kerschbaumer
Tuesday, December 2, 2025 - 11:15 am
Print This Story |...
02/12/2025
SVG Summit 2025 Preview: New AI Production Tools Workshop' Features ESPN, F...
02/12/2025
Written and directed by Hailey Gates, Atropia won the U.S. Grand Jury Prize: Dramatic at the 2025 Sundance Film Festival....
02/12/2025
The lightweight and highly portable CORVUS-RAVEN, recently demonstrated at the V...
02/12/2025
The lightweight, rugged Panther 2 tri-band VSAT delivers high-speed data communications for Internet, VPN connectivity and video transmission over commercial an...
02/12/2025
eds3_5_jq(document).ready(function($) { $(#eds_sliderM519).chameleonSlider_2_1({...
02/12/2025
NGID volunteers in the United States
NGID volunteers in Mexico
NGID volunteers in India
NGID volunteers in Spain
NGID volunteers in Australia
NGID voluntee...
02/12/2025
WASHINGTON The Federal Communications Commission's Media Bureau has issued a reminder that stations in DMAs 111 through 120 must implement its audio descrip...
02/12/2025
SAN DIEGO Sony Electronics today introduced the Alpha 7 V (ILCE-7M5), the fifth generation of its Alpha 7 full-frame mirrorless lineup, powered by the newly dev...
02/12/2025
WASHINGTON The Federal Communications Commission has opened a docket for comments on Nexstar Media Group's proposed $6.2 billion acquisition of Tegna and se...
02/12/2025
WASHINGTON The National Association of Broadcasters has added a trio of media executives, Veronica Rodriguez, Matt Schnaars and Jeff Warshaw, to its board of di...
02/12/2025
MILAN, Italy Brightcove has released seven new features designed to expand global reach, improve audience engagement, enhance live-streaming quality and streaml...
02/12/2025
BCNEXXT, the developers of the advanced playout platform Vipe, today announced a new Service Provider Partner agreement with NPC Media, the Australian managed p...
02/12/2025
DPA Microphones today announces the immediate availability of its new N-Series Digital Wireless Microphone System for customers throughout the U.S. In addition ...
02/12/2025
Hiltron Communications reports a well attended and successful Space Tech Expo exhibition, held at the Messe Bremen from Tuesday November 18 through Thursday Nov...
02/12/2025
The beginning of September saw Riverfire by Australian Retirement Trust, in association with Channel 9 and Triple M, kick off the 2025 Brisbane Festival in spec...
02/12/2025
SGM Lighting is pleased to announce an exciting new partnership with ACT Entertainment, under which ACT will serve as the exclusive distributor for SGM's ne...
02/12/2025
Harmonic (NASDAQ: HLIT) today announced that Telia, the second largest telecom operator in Norway, is modernizing its broadband network with the company's i...
02/12/2025
European cloud data protection specialist Keepit is expanding its sales network in the DACH region and has gained TIM AG, one of the leading value-added distrib...
02/12/2025
Featured products at ISE 2026
This year s ISE 2026 theme "Push Beyond" reflects Calrec s direction of travel. The company is dedicated to supporting the AV an...
02/12/2025
Czech Television, the national public broadcaster, has deployed one of the largest LiveU live production projects in the region for live sports and news coverag...
02/12/2025
Test & measurement innovator, Leader Electronics, has announced the release of v8.2 software for its ZEN and ZEN-W Series, bringing advanced cinematic camera se...
02/12/2025
Global provider of video streaming software and services, Accedo, and OTT platform provider, Magine Pro, have signed a binding agreement to carve out and merge ...
02/12/2025
With proven success in broadcast, OTT, and content supply chains, K.S Avinash set to scale Skandha's service-first approach across APAC
Skandha Media Servi...
02/12/2025
AJA Video Systems has joined the RAVENNA community, a network of partners devoted to advancing the RAVENNA standard for distributing real-time audio over intern...
02/12/2025
HUNT VALLEY, Md. Sinclair and its wholly-owned subsidiary One Media Technologies have released a new report focused on how ATSC 3.0, the IP-based NextGen TV sta...
02/12/2025
WASHINGTON The Federal Communications Commission's Media Bureau has issued a reminder that stations in DMAs 111 through 120 must implement its audio descrip...
02/12/2025
WASHINGTON Members of the Advanced Television Systems Committee have elected four industry veterans to three-year terms on the organization's board of direc...
02/12/2025
A new survey from S&P Global Market Intelligence Kagan highlights how important social media is for younger consumers, with Gen Z spending 5.1 hours a day on so...
02/12/2025
Faculty Notes: Fall/Winter 2025 Recent accomplishments, releases, and events by Berklee faculty.
December 1, 2025
By
Editorial Staff
The cast of Kind of M...
02/12/2025
Today, Mistral AI announced the Mistral 3 family of open-source multilingual, multimodal models, optimized across NVIDIA supercomputing and edge platforms.
M...
02/12/2025
Ring In The Holidays With Our Biggest Sale Ever On Ivory II Pianos!From now until December 31st, you can save 50% or more on every Ivory II piano and collection...
02/12/2025
Back to All News
Made in America: How Stranger Things' Electrified the US Economy
Entertainment
02 December 2025
GlobalUnited States
Link copied to cl...
02/12/2025
Back to All News
Culinary Class Wars' Season 2 Trailer Teases New Rules, R...
02/12/2025
--
The International Electrotechnical Commission (IEC), the International Organization for Standardization (ISO), and the International Telecommunication Unio...
02/12/2025
Harmonic's cOS Platform Will Redefine Subscribers' Quality of Experience, Lowering Churn SAN JOSE, Calif. - Dec. 2, 2025 - Harmonic (NASDAQ: HLIT) today...
02/12/2025
At AWS re:Invent, NVIDIA and Amazon Web Services expanded their strategic collab...
A team of generative AI researchers created a Swiss Army knife for sound, one that allows users to control the audio output simply using text.
