Sony Pixel Power calrec Sony

Now Hear This: World's Most Flexible Sound Machine Debuts

25/11/2024

A team of generative AI researchers created a Swiss Army knife for sound, one that allows users to control the audio output simply using text.

While some AI models can compose a song or modify a voice, none have the dexterity of the new offering.

Called Fugatto (short for Foundational Generative Audio Transformer Opus 1), it generates or transforms any mix of music, voices and sounds described with prompts using any combination of text and audio files.

For example, it can create a music snippet based on a text prompt, remove or add instruments from an existing song, change the accent or emotion in a voice - even let people produce sounds never heard before.

This thing is wild, said Ido Zmishlany, a multi-platinum producer and songwriter - and cofounder of One Take Audio, a member of the NVIDIA Inception program for cutting-edge startups. Sound is my inspiration. It's what moves me to create music. The idea that I can create entirely new sounds on the fly in the studio is incredible.

A Sound Grasp of Audio We wanted to create a model that understands and generates sound like humans do, said Rafael Valle, a manager of applied audio research at NVIDIA and one of the dozen-plus people behind Fugatto, as well as an orchestral conductor and composer.

Supporting numerous audio generation and transformation tasks, Fugatto is the first foundational generative AI model that showcases emergent properties - capabilities that arise from the interaction of its various trained abilities - and the ability to combine free-form instructions.

Fugatto is our first step toward a future where unsupervised multitask learning in audio synthesis and transformation emerges from data and model scale, Valle said.

A Sample Playlist of Use Cases For example, music producers could use Fugatto to quickly prototype or edit an idea for a song, trying out different styles, voices and instruments. They could also add effects and enhance the overall audio quality of an existing track.

The history of music is also a history of technology. The electric guitar gave the world rock and roll. When the sampler showed up, hip-hop was born, said Zmishlany. With AI, we're writing the next chapter of music. We have a new instrument, a new tool for making music - and that's super exciting.

An ad agency could apply Fugatto to quickly target an existing campaign for multiple regions or situations, applying different accents and emotions to voiceovers.

Language learning tools could be personalized to use any voice a speaker chooses. Imagine an online course spoken in the voice of any family member or friend.

Video game developers could use the model to modify prerecorded assets in their title to fit the changing action as users play the game. Or, they could create new assets on the fly from text instructions and optional audio inputs.

Making a Joyful Noise One of the model's capabilities we're especially proud of is what we call the avocado chair, said Valle, referring to a novel visual created by a generative AI model for imaging.

For instance, Fugatto can make a trumpet bark or a saxophone meow. Whatever users can describe, the model can create.

With fine-tuning and small amounts of singing data, researchers found it could handle tasks it was not pretrained on, like generating a high-quality singing voice from a text prompt.

Users Get Artistic Controls Several capabilities add to Fugatto's novelty.

During inference, the model uses a technique called ComposableART to combine instructions that were only seen separately during training. For example, a combination of prompts could ask for text spoken with a sad feeling in a French accent.

The model's ability to interpolate between instructions gives users fine-grained control over text instructions, in this case the heaviness of the accent or the degree of sorrow.

I wanted to let users combine attributes in a subjective or artistic way, selecting how much emphasis they put on each one, said Rohan Badlani, an AI researcher who designed these aspects of the model.

In my tests, the results were often surprising and made me feel a little bit like an artist, even though I'm a computer scientist, said Badlani, who holds a master's degree in computer science with a focus on AI from Stanford.

The model also generates sounds that change over time, a feature he calls temporal interpolation. It can, for instance, create the sounds of a rainstorm moving through an area with crescendos of thunder that slowly fade into the distance. It also gives users fine-grained control over how the soundscape evolves.

Plus, unlike most models, which can only recreate the training data they've been exposed to, Fugatto allows users to create soundscapes it's never seen before, such as a thunderstorm easing into a dawn with the sound of birds singing.

A Look Under the Hood Fugatto is a foundational generative transformer model that builds on the team's prior work in areas such as speech modeling, audio vocoding and audio understanding.

The full version uses 2.5 billion parameters and was trained on a bank of NVIDIA DGX systems packing 32 NVIDIA H100 Tensor Core GPUs.

Fugatto was made by a diverse group of people from around the world, including India, Brazil, China, Jordan and South Korea. Their collaboration made Fugatto's multi-accent and multilingual capabilities stronger.

One of the hardest parts of the effort was generating a blended dataset that contains millions of audio samples used for training. The team employed a multifaceted strategy to generate data and instructions that considerably expanded the range of tasks the model could perform, while achieving more accurate performance and enabling new tasks without requiring additional data.

They also scrutinized existing datasets to reveal new relationships among the dat
LINK: https://blogs.nvidia.com/blog/fugatto-gen-ai-sound-model/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

07/10/2026

Dalet Flex LTS Delivers Smarter Media Operations from Ingest to Distribution

Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...

06/09/2026

Dolby and MagentaTV Bring Fans Closer to the FIFA World Cup 2026 in Germany with Dolby Vision and Dolby Atmos

June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...

26/08/2026

Bertram Brossardt visits SGL Carbon in Meitingen

Discussions at the Meitingen site on competitiveness, energy supply, and fuel cell technology Bertram Brossardt, CEO of the Bavarian metal and electrical emplo...

26/08/2026

Rohde & Schwarz USA celebrates grand opening of new manufacturing center in Frederick, Maryland

Rohde & Schwarz USA celebrates grand opening of new manufacturing center in Fred...

26/08/2026

Designing Agentic Media Workflows Humans Can Trust

For large media companies, the content archive is a gold mine. There are thousands of hours of existing, high-value assets just waiting to be tapped for highlig...

26/08/2026

Major Live Sports Events Bolster Broadcast in Nielsen's Q2 2026 Ad Supported Gauge

Ad Supported Viewing Represents 71.5% of TV in Q2 Streaming Captures Over 48% o...

26/08/2026

DAZN Accelerates U.S. Expansion with Acquisition of EverPass Media

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

Truthset Brings Data Rated Audiences to FreeWheel

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

Broadpeak Targets CTV Shopping with Click2Cart Launch

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

SMPTE Announces 2026 Honorees

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

MediaKind Beam selected by CNBC UK for cloud-connected li...

MediaKind today announced that CNBC UK has selected Beam to modernize its contribution and distribution infrastructure, bringing greater flexibility, resilience...

26/08/2026

Telestream Unveils the Next Phase of UP at IBC2026

Telestream, a global leader in media workflow technologies, today announced the next evolution of Telestream UP, expanding the cloud-native platform introduced ...

26/08/2026

SMPTE Announces 2026 Honorees

SMPTE, the home for media professionals, technologists and engineers, today announced its 2026 Honorees, recognizing individuals and organizations whose contrib...

26/08/2026

Broadpeak launches Click2Cart to mark new era in shoppabl...

New solution bridges connected TV (CTV) and ecommerce to make shoppable performance advertising easier to scale for streaming platforms and broadcasters Broadp...

26/08/2026

Scripps Research and collaborators awarded $19.5 million to establish an open-access autonomous chemistry laboratory

LA JOLLA, CA-Recent advances in robotics, automation and artificial intelligence...

26/08/2026

How WeFX Made From's' Monsters Feel Uncomfortably Real

How WeFX Made From's' Monsters Feel Uncomfortably Real Brie Clayton August 26, 2026 0 Comments From eight-foot dolls and feathered birds to R...

26/08/2026

Eleanor Roosevelt High School Elevates Student Rallies with Blackmagic Design

Eleanor Roosevelt High School Elevates Student Rallies with Blackmagic Design Brie Clayton August 26, 2026 0 Comments Student led production team uses...

26/08/2026

Grass Valley Advances Integrated Live Production for Unif...

K-Frame and Maverik X innovations extend familiar production control into AMPP, alongside new replay, audio, commentary and communications capabilities. At IBC...

26/08/2026

Magewell Expands Director and Pro Convert IP Lines at IBC...

New products continue to enable any signal, anywhere for creators...

26/08/2026

AFP and Dalet Bring Trusted Content into Every Newsroom

Dalet, a leading technology and service provider for media-rich organizations, today announced a new integrated newsroom experience with Agence France-Presse (A...

26/08/2026

Bolin Technology and ITDev Join AIMS

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

Enco to Highlight Caption, Translation Enhancements at IBC2026

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

ZOO Digital Chooses Interra Systems BATON for Quality Control

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

Gomez Criticizes FCC's Media Bureau on Political Advertising

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

Tedial to Demo AI-Driven Media Supply Chain Solutions at IBC 2026

Share Copy link Facebook X Linkedin Bluesky Email...

26/08/2026

AIMS Welcomes Bolin Technology and ITDev as Newest Member...

PTZ Camera Manufacturer and FPGA Design Specialist Join Alliance to Advance IPMX and Open AV-over-IP Standards The Alliance for IP Media Solutions (AIMS) today...

26/08/2026

ZOO Digital Selects Interra Systems BATON for File-Based...

BATON offers ZOO Digital advanced audio and video quality analysis, and scalable architecture Interra Systems, a leading provider of end-to-end quality assuran...

26/08/2026

Scottish Rugby Expands Managed Coach Communications Partn...

Riedel Communications today announced the continued expansion of its managed coach communications partnership with Scottish Rugby Union. Now entering its third ...

26/08/2026

Zixi Demonstrates Hybrid Workflows and Proven Live Video...

Zixi brings Broadcaster V19 and showcases interoperable platform on Stand 5.A76 Zixi, a leader in live video delivery and workflow orchestration, will be exhib...

26/08/2026

Scripps Research professor Jeffery Kelly elected to the European Academy of Engineering

LA JOLLA, CA-Jeffery Kelly, the H. Lutcher Brown Professor of Chemistry at Scrip...

26/08/2026

Indie Developer Launches Orty, aWindows App That Helps Freelance VideoEditors Find Better Clients

Indie Developer Launches Orty, aWindows App That Helps Freelance VideoEditors Fi...

26/08/2026

BTS on Sinners and Is God Is Shot with Pocket Cinema Camera 6K Pro

BTS on Sinners and Is God Is Shot with Pocket Cinema Camera 6K Pro Brie Clayton August 25, 2026 0 Comments Award winning filmmaker Travis Henry elevat...

26/08/2026

Sky greenlights second chapter of acclaimed Original crime drama Under Salt Marsh

Rafe Spall reprises his role as Detective Eric Bull in the six-part sequel, crea...

26/08/2026

New Sky DocumentaryThe Pembrokeshire Murders(w/t) sheds fresh light on a landmark case involving two double murders

A high-profile police investigation led to John Cooper's conviction. Now, tw...

26/08/2026

AFP and Dalet Bring Trusted Content into Every Newsroom

Dalet, a leading technology and service provider for media-rich organizations, today announced a new integrated newsroom experience with Agence France-Presse (A...

26/08/2026

COMING SOON TO RT

As RT marks 100 years of public service broadcasting in Ireland, today we announce a season of fresh, original content, while preparing to move into a second c...

25/08/2026

Open, Sustainable, Predictable: What Utah Scientific Is Bringing to IBC 2026

Open, Sustainable, Predictable: What Utah Scientific Is Bringing to IBC 2026 As broadcast leaders prepare to gather at the RAI Amsterdam for IBC 2026, the main ...

25/08/2026

The Big Shift: Inside NC State's Phased Move From SDI to SMPTE ST 2110

NC State's Matthew Byrd breaks down how an IP-based infrastructure is delivering greater flexibility and scalability for Wolfpack productions...

25/08/2026

Monumental Sports Network Adds DAZN as Distribution Partner for Monumental+ in the DMV

Deal expands the DTC reach in-market of Capitals, Wizards, Mystics games and oth...

25/08/2026

Jacksonville Jaguars Carlos Caceres on Producing Shows at an Ever-Evolving EverBank Stadium

The two-year renovation period will create a stadium of the future for the NFL f...

25/08/2026

Behind The Mic: Genie Bouchard Joins ESPN for 2026 US Open; NBC Sports Names College Football Announce Hosts

Behind The Mic provides a roundup of recent news regarding on-air talent, includ...

25/08/2026

LD Systems Installs L-Acoustics A Series at Shell Energy Stadium in Houston

Shell Energy Stadium in Houston has installed a new L-Acoustics A Series loudspeaker system, designed and installed by LD Systems, a Clair Global brand. The 20,...

25/08/2026

FloCollege Expands Distribution and Adds Features for 2026-27 Season

FloCollege is expanding its distribution and product offerings for the 2026-27 season, covering more than 20 NCAA partner conferences across Division I, II, and...

25/08/2026

TNA Wrestling and REVOLT Announce Content Partnership

TNA Wrestling and REVOLT have announced a content partnership bringing exclusive TNA Xplosion matches and content from TNA's library to REVOLT. Our indust...

25/08/2026

Gracenote: FAST Sports Programming Growth Outpaces Channel Growth

Gracenote, Nielsen's content intelligence business, has released its Q3 2026 Data Hub analysis showing that FAST sports programming is growing at more than ...

25/08/2026

Amagi Appoints Thomas dHrouville Head of Sales for France, Benelux, and Nordics

Amagi has appointed Thomas d'H rouville as Head of Sales for France, Benelux, and Nordics. Based in Paris, he will lead Amagi's growth strategy across t...

25/08/2026

IPC Appoints CCATech as Channel Partner in Brazil

IPC has appointed CCATech as its channel partner in Brazil, expanding the company's presence in Latin America. CCATech was founded in 2023 by Carlos Abrah o...