Sony Pixel Power calrec Sony

Now Hear This: World's Most Flexible Sound Machine Debuts

25/11/2024

A team of generative AI researchers created a Swiss Army knife for sound, one that allows users to control the audio output simply using text.

While some AI models can compose a song or modify a voice, none have the dexterity of the new offering.

Called Fugatto (short for Foundational Generative Audio Transformer Opus 1), it generates or transforms any mix of music, voices and sounds described with prompts using any combination of text and audio files.

For example, it can create a music snippet based on a text prompt, remove or add instruments from an existing song, change the accent or emotion in a voice - even let people produce sounds never heard before.

This thing is wild, said Ido Zmishlany, a multi-platinum producer and songwriter - and cofounder of One Take Audio, a member of the NVIDIA Inception program for cutting-edge startups. Sound is my inspiration. It's what moves me to create music. The idea that I can create entirely new sounds on the fly in the studio is incredible.

A Sound Grasp of Audio We wanted to create a model that understands and generates sound like humans do, said Rafael Valle, a manager of applied audio research at NVIDIA and one of the dozen-plus people behind Fugatto, as well as an orchestral conductor and composer.

Supporting numerous audio generation and transformation tasks, Fugatto is the first foundational generative AI model that showcases emergent properties - capabilities that arise from the interaction of its various trained abilities - and the ability to combine free-form instructions.

Fugatto is our first step toward a future where unsupervised multitask learning in audio synthesis and transformation emerges from data and model scale, Valle said.

A Sample Playlist of Use Cases For example, music producers could use Fugatto to quickly prototype or edit an idea for a song, trying out different styles, voices and instruments. They could also add effects and enhance the overall audio quality of an existing track.

The history of music is also a history of technology. The electric guitar gave the world rock and roll. When the sampler showed up, hip-hop was born, said Zmishlany. With AI, we're writing the next chapter of music. We have a new instrument, a new tool for making music - and that's super exciting.

An ad agency could apply Fugatto to quickly target an existing campaign for multiple regions or situations, applying different accents and emotions to voiceovers.

Language learning tools could be personalized to use any voice a speaker chooses. Imagine an online course spoken in the voice of any family member or friend.

Video game developers could use the model to modify prerecorded assets in their title to fit the changing action as users play the game. Or, they could create new assets on the fly from text instructions and optional audio inputs.

Making a Joyful Noise One of the model's capabilities we're especially proud of is what we call the avocado chair, said Valle, referring to a novel visual created by a generative AI model for imaging.

For instance, Fugatto can make a trumpet bark or a saxophone meow. Whatever users can describe, the model can create.

With fine-tuning and small amounts of singing data, researchers found it could handle tasks it was not pretrained on, like generating a high-quality singing voice from a text prompt.

Users Get Artistic Controls Several capabilities add to Fugatto's novelty.

During inference, the model uses a technique called ComposableART to combine instructions that were only seen separately during training. For example, a combination of prompts could ask for text spoken with a sad feeling in a French accent.

The model's ability to interpolate between instructions gives users fine-grained control over text instructions, in this case the heaviness of the accent or the degree of sorrow.

I wanted to let users combine attributes in a subjective or artistic way, selecting how much emphasis they put on each one, said Rohan Badlani, an AI researcher who designed these aspects of the model.

In my tests, the results were often surprising and made me feel a little bit like an artist, even though I'm a computer scientist, said Badlani, who holds a master's degree in computer science with a focus on AI from Stanford.

The model also generates sounds that change over time, a feature he calls temporal interpolation. It can, for instance, create the sounds of a rainstorm moving through an area with crescendos of thunder that slowly fade into the distance. It also gives users fine-grained control over how the soundscape evolves.

Plus, unlike most models, which can only recreate the training data they've been exposed to, Fugatto allows users to create soundscapes it's never seen before, such as a thunderstorm easing into a dawn with the sound of birds singing.

A Look Under the Hood Fugatto is a foundational generative transformer model that builds on the team's prior work in areas such as speech modeling, audio vocoding and audio understanding.

The full version uses 2.5 billion parameters and was trained on a bank of NVIDIA DGX systems packing 32 NVIDIA H100 Tensor Core GPUs.

Fugatto was made by a diverse group of people from around the world, including India, Brazil, China, Jordan and South Korea. Their collaboration made Fugatto's multi-accent and multilingual capabilities stronger.

One of the hardest parts of the effort was generating a blended dataset that contains millions of audio samples used for training. The team employed a multifaceted strategy to generate data and instructions that considerably expanded the range of tasks the model could perform, while achieving more accurate performance and enabling new tasks without requiring additional data.

They also scrutinized existing datasets to reveal new relationships among the dat
LINK: https://blogs.nvidia.com/blog/fugatto-gen-ai-sound-model/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

07/10/2026

Dalet Flex LTS Delivers Smarter Media Operations from Ingest to Distribution

Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...

06/09/2026

Dolby and MagentaTV Bring Fans Closer to the FIFA World Cup 2026 in Germany with Dolby Vision and Dolby Atmos

June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...

27/08/2026

Federal Appeals Court Blocks Some FCC Political Ad Discount Rules

Share Copy link Facebook X Linkedin Bluesky Email...

27/08/2026

Zixi To Demo Hybrid Workflows At IBC2026

Share Copy link Facebook X Linkedin Bluesky Email...

27/08/2026

Study: Americans Spend More Time Watching YouTube Than Live TV

Share Copy link Facebook X Linkedin Bluesky Email...

27/08/2026

Magewell To Highlight Newly Released Director Plus At IBC2026

Share Copy link Facebook X Linkedin Bluesky Email...

27/08/2026

PlayBox Neo continues to shape the media landscape by ini...

PlayBox Neo Stand 7.C29 IBC2026 - 11 14 September Well-established global specialists in media playout, automation, and streaming solutions, PlayBox Neo ...

27/08/2026

Rise Academy Invites the Industry to Join Race 4 The Futu...

Rise Academy, the broadcast and media skills charity dedicated to delivering practical media technology learning experiences, careers resources and work experie...

27/08/2026

Mediagenix Takes Title Management Beyond the Catalog at I...

Mediagenix, a global leader in smart content solutions to profitably connect the right content to the right audience, will showcase the latest evolution of its ...

27/08/2026

Screen Australia opens applications for Global Fellowships Program

Screen Australia opens applications for Global Fellowships Program 27 August 2026 Applications are now open for the Screen Australia Global Fellowships progra...

26/08/2026

New Validation Booster FAQ: What applicants need to know

Applicants interested in the Media Innovation Europe 3.0 Validation Booster call can now consult a new FAQ document addressing practical questions raised during...

26/08/2026

NWSL Moves Sunday Night Package to Roku, Expands Slate to 25 Matches in 2027

Roku Sports Channel will stream eight matches free beginning Aug. 30 as league shifts Sunday-night games away from Victory+...

26/08/2026

Inside Mountain Media Center's Hybrid-Cloud Architecture for Live Event Production

Evertz and Mobile TV Group detail how on-premises infrastructure and cloud resou...

26/08/2026

IBC 2026: Sennheiser Group to Showcase Broadcast Audio Portfolio

The Sennheiser Group will demonstrate its broadcast audio portfolio at IBC2026 (Hall 8, Booth D50), covering wireless microphone workflows, studio monitoring, a...

26/08/2026

Appear and TVTEL Deploy X20 Platform for Centralized VAR Signal Transport in Chile

Appear and Transmisiones Televisivas TVTEL Limitada have announced the deploymen...

26/08/2026

Minitech Adds Leader LPX500 and LT4670 to Rental Inventory

Leader Electronics has announced that Danish broadcast rental company Minitech has purchased a Leader LPX500 hybrid IP/SDI waveform monitor and LT4670 sync puls...

26/08/2026

IBC 2026: Encompass to Showcase Cloud Media Services

Encompass will demonstrate its Altitude media services at IBC2026 (IAMT Hub, Diamond Lounge, Hall 3, RAI Amsterdam, September 11-14). Key products on show incl...

26/08/2026

Massillon Washington High School First to Implement Daktronics Camino Platform

Daktronics has partnered with Massillon Washington High School in Ohio to install a new LED video display and implement the Daktronics Camino show control platf...

26/08/2026

Clear-Com Appoints Venuetech as Middle East Partner

Clear-Com has appointed Dubai-based Venuetech LLC as its main partner for the Middle East. Venuetech distributes professional audiovisual solutions across the r...

26/08/2026

NBC Sports NOW Debuts Two New Shows Beginning August 31

NBC Sports NOW, NBC Sports' free 24/7 streaming channel, will launch a new weekday lineup beginning Monday, August 31, including two new shows. Not So Fast...

26/08/2026

IBC 2026: Calrec to Preview ImPulseV and Argo Updates

Calrec will demonstrate updates to its ImPulseV DSP software and Argo ecosystem at IBC2026 (Stand 8.C47), including previews aligned with the EBU's Dynamic ...

26/08/2026

IBC 2026: Imagine Communications to Showcase AI Capabilities

Imagine Communications will demonstrate AI applications across its ad tech and video infrastructure portfolios at IBC2026 (Stand 1.B73, RAI Amsterdam, September...

26/08/2026

IBC 2026: Dalet to Showcase AI and Workflow Automation

Dalet will demonstrate updates to Dalet Flex, Dalet Pyramid, and Dalia, its agentic AI platform, at IBC2026 (Hall 7, Stand 7.A43). The company will also partici...

26/08/2026

Nashville Predators Tap ViewLift for New DTC Service, Pairing Streaming With Scripps Sports Broadcast Deal

Team-owned platform will stream every locally carried Predators game beginning t...

26/08/2026

Little League Baseball World Series 2026: ESPN Restores RailCam Coverage at Lamade Stadium, Upscales Cameras at Volunteer Stadium

The 12-day tournament receives the broadcaster's full onsite-studio presence...

26/08/2026

SOMA Laboratory launch the Enigma

A synth with no controls? If you're into weird and wonderful instruments, then you may already be familiar with Warsaw-based SOMA Laboratory - if you...

26/08/2026

Dreamtonics unveil Instrument X

Physical modelling orchestral instrument announced Following their recent cryptic hints, Dreamtonics have officially revealed the latest addition to their s...

26/08/2026

Bertram Brossardt visits SGL Carbon in Meitingen

Discussions at the Meitingen site on competitiveness, energy supply, and fuel cell technology Bertram Brossardt, CEO of the Bavarian metal and electrical emplo...

26/08/2026

Rohde & Schwarz USA celebrates grand opening of new manufacturing center in Frederick, Maryland

Rohde & Schwarz USA celebrates grand opening of new manufacturing center in Fred...

26/08/2026

Designing Agentic Media Workflows Humans Can Trust

For large media companies, the content archive is a gold mine. There are thousands of hours of existing, high-value assets just waiting to be tapped for highlig...

26/08/2026

Major Live Sports Events Bolster Broadcast in Nielsen's Q2 2026 Ad Supported Gauge

Ad Supported Viewing Represents 71.5% of TV in Q2 Streaming Captures Over 48% o...

26/08/2026

DAZN Accelerates U.S. Expansion with Acquisition of EverPass Media

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

Truthset Brings Data Rated Audiences to FreeWheel

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

Broadpeak Targets CTV Shopping with Click2Cart Launch

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

SMPTE Announces 2026 Honorees

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

MediaKind Beam selected by CNBC UK for cloud-connected li...

MediaKind today announced that CNBC UK has selected Beam to modernize its contribution and distribution infrastructure, bringing greater flexibility, resilience...

26/08/2026

Telestream Unveils the Next Phase of UP at IBC2026

Telestream, a global leader in media workflow technologies, today announced the next evolution of Telestream UP, expanding the cloud-native platform introduced ...

26/08/2026

SMPTE Announces 2026 Honorees

SMPTE, the home for media professionals, technologists and engineers, today announced its 2026 Honorees, recognizing individuals and organizations whose contrib...

26/08/2026

Broadpeak launches Click2Cart to mark new era in shoppabl...

New solution bridges connected TV (CTV) and ecommerce to make shoppable performance advertising easier to scale for streaming platforms and broadcasters Broadp...

26/08/2026

Scripps Research and collaborators awarded $19.5 million to establish an open-access autonomous chemistry laboratory

LA JOLLA, CA-Recent advances in robotics, automation and artificial intelligence...

26/08/2026

How WeFX Made From's' Monsters Feel Uncomfortably Real

How WeFX Made From's' Monsters Feel Uncomfortably Real Brie Clayton August 26, 2026 0 Comments From eight-foot dolls and feathered birds to R...

26/08/2026

Eleanor Roosevelt High School Elevates Student Rallies with Blackmagic Design

Eleanor Roosevelt High School Elevates Student Rallies with Blackmagic Design Brie Clayton August 26, 2026 0 Comments Student led production team uses...

26/08/2026

Grass Valley Advances Integrated Live Production for Unif...

K-Frame and Maverik X innovations extend familiar production control into AMPP, alongside new replay, audio, commentary and communications capabilities. At IBC...

26/08/2026

Magewell Expands Director and Pro Convert IP Lines at IBC...

New products continue to enable any signal, anywhere for creators...

26/08/2026

AFP and Dalet Bring Trusted Content into Every Newsroom

Dalet, a leading technology and service provider for media-rich organizations, today announced a new integrated newsroom experience with Agence France-Presse (A...

26/08/2026

Bolin Technology and ITDev Join AIMS

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

Enco to Highlight Caption, Translation Enhancements at IBC2026

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

ZOO Digital Chooses Interra Systems BATON for Quality Control

Share Copy link Facebook X Linkedin Bluesky Email...