Sony Pixel Power calrec Sony

Now Hear This: World's Most Flexible Sound Machine Debuts

25/11/2024

A team of generative AI researchers created a Swiss Army knife for sound, one that allows users to control the audio output simply using text.

While some AI models can compose a song or modify a voice, none have the dexterity of the new offering.

Called Fugatto (short for Foundational Generative Audio Transformer Opus 1), it generates or transforms any mix of music, voices and sounds described with prompts using any combination of text and audio files.

For example, it can create a music snippet based on a text prompt, remove or add instruments from an existing song, change the accent or emotion in a voice - even let people produce sounds never heard before.

This thing is wild, said Ido Zmishlany, a multi-platinum producer and songwriter - and cofounder of One Take Audio, a member of the NVIDIA Inception program for cutting-edge startups. Sound is my inspiration. It's what moves me to create music. The idea that I can create entirely new sounds on the fly in the studio is incredible.

A Sound Grasp of Audio We wanted to create a model that understands and generates sound like humans do, said Rafael Valle, a manager of applied audio research at NVIDIA and one of the dozen-plus people behind Fugatto, as well as an orchestral conductor and composer.

Supporting numerous audio generation and transformation tasks, Fugatto is the first foundational generative AI model that showcases emergent properties - capabilities that arise from the interaction of its various trained abilities - and the ability to combine free-form instructions.

Fugatto is our first step toward a future where unsupervised multitask learning in audio synthesis and transformation emerges from data and model scale, Valle said.

A Sample Playlist of Use Cases For example, music producers could use Fugatto to quickly prototype or edit an idea for a song, trying out different styles, voices and instruments. They could also add effects and enhance the overall audio quality of an existing track.

The history of music is also a history of technology. The electric guitar gave the world rock and roll. When the sampler showed up, hip-hop was born, said Zmishlany. With AI, we're writing the next chapter of music. We have a new instrument, a new tool for making music - and that's super exciting.

An ad agency could apply Fugatto to quickly target an existing campaign for multiple regions or situations, applying different accents and emotions to voiceovers.

Language learning tools could be personalized to use any voice a speaker chooses. Imagine an online course spoken in the voice of any family member or friend.

Video game developers could use the model to modify prerecorded assets in their title to fit the changing action as users play the game. Or, they could create new assets on the fly from text instructions and optional audio inputs.

Making a Joyful Noise One of the model's capabilities we're especially proud of is what we call the avocado chair, said Valle, referring to a novel visual created by a generative AI model for imaging.

For instance, Fugatto can make a trumpet bark or a saxophone meow. Whatever users can describe, the model can create.

With fine-tuning and small amounts of singing data, researchers found it could handle tasks it was not pretrained on, like generating a high-quality singing voice from a text prompt.

Users Get Artistic Controls Several capabilities add to Fugatto's novelty.

During inference, the model uses a technique called ComposableART to combine instructions that were only seen separately during training. For example, a combination of prompts could ask for text spoken with a sad feeling in a French accent.

The model's ability to interpolate between instructions gives users fine-grained control over text instructions, in this case the heaviness of the accent or the degree of sorrow.

I wanted to let users combine attributes in a subjective or artistic way, selecting how much emphasis they put on each one, said Rohan Badlani, an AI researcher who designed these aspects of the model.

In my tests, the results were often surprising and made me feel a little bit like an artist, even though I'm a computer scientist, said Badlani, who holds a master's degree in computer science with a focus on AI from Stanford.

The model also generates sounds that change over time, a feature he calls temporal interpolation. It can, for instance, create the sounds of a rainstorm moving through an area with crescendos of thunder that slowly fade into the distance. It also gives users fine-grained control over how the soundscape evolves.

Plus, unlike most models, which can only recreate the training data they've been exposed to, Fugatto allows users to create soundscapes it's never seen before, such as a thunderstorm easing into a dawn with the sound of birds singing.

A Look Under the Hood Fugatto is a foundational generative transformer model that builds on the team's prior work in areas such as speech modeling, audio vocoding and audio understanding.

The full version uses 2.5 billion parameters and was trained on a bank of NVIDIA DGX systems packing 32 NVIDIA H100 Tensor Core GPUs.

Fugatto was made by a diverse group of people from around the world, including India, Brazil, China, Jordan and South Korea. Their collaboration made Fugatto's multi-accent and multilingual capabilities stronger.

One of the hardest parts of the effort was generating a blended dataset that contains millions of audio samples used for training. The team employed a multifaceted strategy to generate data and instructions that considerably expanded the range of tasks the model could perform, while achieving more accurate performance and enabling new tasks without requiring additional data.

They also scrutinized existing datasets to reveal new relationships among the dat
LINK: https://blogs.nvidia.com/blog/fugatto-gen-ai-sound-model/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

06/09/2026

Dolby and MagentaTV Bring Fans Closer to the FIFA World Cup 2026 in Germany with Dolby Vision and Dolby Atmos

June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...

04/08/2026

Dalet Announces Commercial Availability of Dalia, Bringing Media-Aware Agentic AI to Enterprise Productions

Dalet, a leading technology and service provider for media-rich organizations, t...

04/07/2026

Detective Conan: Fallen Angel of the Highway Opens in Dolby Cinemas Across Japan, Presented in Dolby Atmos and Dolby ...

April 7 2026, 19:00 (PDT) Detective Conan: Fallen Angel of the Highway Opens in...

18/06/2026

Ratings Roundup: Knicks-Spurs NBA Finals Is Most Watched Since Jordan Bulls Era; FIFA World Cup Opens Big

Ratings Roundup is a rundown of recent rating news and is derived from press rel...

18/06/2026

InfoComm 2026: PTZOptics Debuts Healthcare Visual Reasoning Integration with LayerJot

PTZOptics has unveiled new Visual Reasoning demonstrations at InfoComm 2026 (Boo...

18/06/2026

IBC 2026: Announces New Future Tech Ignite Initiative, Expanded Exhibitor Participation

IBC2026 will take place at the RAI Amsterdam from September 11-14, bringing toge...

18/06/2026

InfoComm 2026 Holds Inaugural Media Day with Product Announcements from 19 Exhibitors

InfoComm 2026 held its first-ever Media Day on June 17, providing journalists an...

18/06/2026

Info Comm 2026: FOR-A America Announces TAA-Compliant LED Display Package

FOR-A America has announced a Trade Agreements Act (TAA)-compliant LED display solution combining Alfalite's Litepix LED displays and Brompton Technology...

18/06/2026

SVG GameDay, Ep. 20: Cleveland Browns Kyle Millen - Run of Shows & Rock n Roll

In-venue and creative video staffers at the professional and collegiate level have one major thing in common: the intensity and attention to detail ramps up dur...

18/06/2026

International Federation of American Football, TMRW Sports Partner on Global Growth of Flag Football

The International Federation of American Football (IFAF) and TMRW Sports have an...

18/06/2026

AJA Debuts Io Xpand Thunderbolt 5 Expansion Chassis for KONA and Corvid PCIe I/O Cards

AJA Video Systems has unveiled Io Xpand, a Thunderbolt 5-enabled PCIe expansion ...

18/06/2026

ESPN Marks 30th Anniversary of WNBA's Inaugural Game With Liberty-Sparks Broadcast

ESPN has announced its coverage plans for the 30th anniversary of the WNBA's...

18/06/2026

FOX Sports' Big Noon Kickoff Heads to London for Union Jack Classic From Wembley Stadium

FOX Sports' Big Noon Kickoff will broadcast live from Wembley Stadium in Lon...

18/06/2026

InfoComm 2026: Show Opens With Microsoft Keynote, Media Day, and New Industry Initiatives

InfoComm 2026 opened on Wednesday at the Las Vegas Convention Center, bringing t...

18/06/2026

SVG New Sponsor Spotlight: Akta's Matt Smith on Building AI Into the Foundation of the Video Workflow

As media companies look to deliver more live, VOD, and snackable sports content ...

18/06/2026

2026 Sundance Film Festival: Local Lens

Top L-R: Take Me Home, The Lake Bottom L-R: TheyDream, Union County Free Summer Screening Series Announced Screenings for the Local Utah Community at...

18/06/2026

iamReverb gets an update

Improvements & new IR content iamReverb Audio have just launched a free update that kits their convolution reverb plug-in out with some new features and int...

18/06/2026

Two notes unveil Genome 2.0

Modelling suite gains improved captures, iOS support & more Two notes Audio Engineering have just announced the launch of Genome 2.0, a significant update t...

18/06/2026

VSL update Vienna Ensemble Pro 8

New AI assistance feature, video overhaul & more VSL have just announced the launch of Vienna Ensemble Pro 8.1 and 8.1V, a pair of major updates to their ev...

18/06/2026

The Gauge: Poland | May 2026

The average daily TV screen time in May was 3 hours and 36 minutes, marking a clear decrease of 15 minutes compared to April. This trend proved to be much stron...

18/06/2026

Gracenote and PubMatic Bring Curated Live Sports and Content-Level Deals to Programmatic CTV

Integration embeds Gracenote content intelligence, including contextual segments...

18/06/2026

AJA Debuts Io Xpand Thunderbolt 5 Expansion Chassis for...

AJA Video Systems unveiled Io Xpand, a high-performance Thunderbolt 5-enabled PCIe expansion chassis for AJA KONA and Corvid video and audio I/O cards. As dema...

18/06/2026

Providius Announces Providius Direct for Faster AV-over-I...

New NVRT-driven workflow enables on-demand traffic mirroring and guided troubleshooting for AV teams, integrators, and support organizations ahead of InfoComm 2...

18/06/2026

Mavis Studio Brings New Muscle to Mobile Production on iP...

At InfoComm 2026, Mavis announced a major update to Mavis Studio, its live production app for the iPad, with new features designed to make professional AV produ...

18/06/2026

Harmonic Completes Divestiture of Video Business to Media...

Transaction Positions Harmonic as a Pure-Play Broadband Company Harmonic Inc. (NASDAQ: HLIT), the worldwide leader in virtualized broadband solutions, today a...

18/06/2026

ACT Entertainment and NETGEAR Partner to Power the Futur...

ACT Entertainment and NETGEAR have announced a strategic partnership that establishes verified interoperability between NETGEAR Switches and key technologies fr...

18/06/2026

IBC2026 creates new pathways from innovation to real-worl...

IBC2026 is set to bring the global media, entertainment and technology community together at the RAI Amsterdam from 11 14 September, enabling industry players f...

18/06/2026

PLIANT TECHNOLOGIES DEBUTS CREWCOM FLEX THE WORLDS FIRST...

Pliant Technologies is proud to introduce CrewCom Flex , the world's first frameless matrix intercom system and the only complete matrix IP-based intercom s...

18/06/2026

XR Sports Alliance Strengthens its Capabilities with New...

The XR Sports Alliance (XRSA) has announced that a new cohort of members has joined the strategic initiative: ActionStreamer, Antigravity, Creative Artists Agen...

18/06/2026

Hollywood Filmmakers Launch AI Production Platform Cascade

Share Copy link Facebook X Linkedin Bluesky Email...

18/06/2026

ATVA Blasts Deltavision Media for Demanding 'Egregious' Retrans Fees

Share Copy link Facebook X Linkedin Bluesky Email...

18/06/2026

MediaKind Completes Merger with Harmonic's Video Business

Share Copy link Facebook X Linkedin Bluesky Email...

18/06/2026

AJA Unveils Io Xpand Thunderbolt 5 Expansion Chassis At InfoComm 2026

Share Copy link Facebook X Linkedin Bluesky Email...

18/06/2026

Linda May Han Oh Receives Guggenheim Fellowship

Linda May Han Oh Receives Guggenheim Fellowship The Berklee professor, bassist, and composer will use the fellowship to debut Dreams of Knowing, an interdisci...

18/06/2026

Techtel delivers Haivision live streaming system for TAFE NSW St Leonards, enabling real-world broadcast training

Techtel delivers Haivision live streaming system for TAFE NSW St Leonards, enabl...

18/06/2026

The Great Roaming Rinse

New study reveals 10 hidden data drainers costing Brits hundreds abroad - and the holiday hotspots where you could get rinsed the mostThursday 18 June 2026 The...

18/06/2026

How to watch the 2026/27 Scottish Premiership season on Sky Sports

Thursday 18 June 2026 How to watch the 2026/27 Scottish Premiership season on Sky Sports Which matches are Sky Sports showing on the 2026/27 Scottish Premiers...

18/06/2026

Sky Sale: Latest deals now on, with discounts on iPhone Air & 2.5Gbps speeds

Thursday 18 June 2026 Sky Sale: Latest deals now on, with discounts on iPhone Air & 2.5Gbps speeds The latest deals have dropped from Sky Mobile, the award-wi...

18/06/2026

FOX Advertising and Toonstar Team to Create New Opportunities for Brands in Digital-First Animation

FOX Advertising and Toonstar Team to Create New Opportunities for Brands in Digi...

18/06/2026

Arqiva secures WTA Tier-4 accreditation

Arqiva's Crawley Court and Chalfont Grove teleports re-certified at highest World Teleport Association standard 18 June 2026, Winchester, UK - Arqiva, the ...

18/06/2026

Apple announces changes to iOS in Brazil - UPDATE - Posted on 18 June 2026

Apple today announced changes impacting iOS apps in Brazil that reflect a recent agreement with Brazil's competition regulator, the Conselho Administrativo ...

18/06/2026

Sync and Stream: GeForce NOW Connects to Members' Game Libraries Across Devices

Play favorite titles from popular game libraries, keep progress synced and jump ...

18/06/2026

At Cannes Lions, NVIDIA Partners Reshape Advertising and Marketing With AI

The digital era gave the advertising and marketing industry speed; the AI era is giving it autonomous operations. For companies building next-generation techn...

17/06/2026

EVS Achieves EcoVadis Gold Medal, Ranking in Top 5% of Companies Globally

EVS has announced it has received the EcoVadis Gold Medal for sustainability performance, ranking among the top 5% of companies globally in the Technology/Mid-S...

17/06/2026

Chyron Releases Weather 2.4 with Updated DataFlow Module

Chyron has released Weather 2.4, an update to its weather suite for broadcasters and meteorologists. The release focuses on enhancements to the DataFlow module,...

17/06/2026

VSiN Launches Best Bets TV, a 24/7 FAST Channel

VSiN, The Sports Betting Network, has announced the launch of Best Bets TV, a free ad-supported streaming TV (FAST) channel. The 24/7 channel is currently avail...

17/06/2026

Snapchat and Team Whistle Launch World Cup Creator Program Across Miami, New York, and Los Angeles

DAZN's Team Whistle and Snap Inc. have announced a creator program centered ...

17/06/2026

LiveU Supporting Broadcast and Public Safety Operations for Summer of Soccer

LiveU is providing video transmission technology for broadcasters, production companies, and public safety agencies across North America's busy Summer of So...

17/06/2026

NATAS Announces Two New Board Members and COO Promotion

The National Academy of Television Arts and Sciences (NATAS) has announced that Laurens Grant and Jacob Ullman have joined its Board of Directors. Chief of Staf...