Sony Pixel Power calrec Sony

Now Hear This: World's Most Flexible Sound Machine Debuts

25/11/2024

A team of generative AI researchers created a Swiss Army knife for sound, one that allows users to control the audio output simply using text.

While some AI models can compose a song or modify a voice, none have the dexterity of the new offering.

Called Fugatto (short for Foundational Generative Audio Transformer Opus 1), it generates or transforms any mix of music, voices and sounds described with prompts using any combination of text and audio files.

For example, it can create a music snippet based on a text prompt, remove or add instruments from an existing song, change the accent or emotion in a voice - even let people produce sounds never heard before.

This thing is wild, said Ido Zmishlany, a multi-platinum producer and songwriter - and cofounder of One Take Audio, a member of the NVIDIA Inception program for cutting-edge startups. Sound is my inspiration. It's what moves me to create music. The idea that I can create entirely new sounds on the fly in the studio is incredible.

A Sound Grasp of Audio We wanted to create a model that understands and generates sound like humans do, said Rafael Valle, a manager of applied audio research at NVIDIA and one of the dozen-plus people behind Fugatto, as well as an orchestral conductor and composer.

Supporting numerous audio generation and transformation tasks, Fugatto is the first foundational generative AI model that showcases emergent properties - capabilities that arise from the interaction of its various trained abilities - and the ability to combine free-form instructions.

Fugatto is our first step toward a future where unsupervised multitask learning in audio synthesis and transformation emerges from data and model scale, Valle said.

A Sample Playlist of Use Cases For example, music producers could use Fugatto to quickly prototype or edit an idea for a song, trying out different styles, voices and instruments. They could also add effects and enhance the overall audio quality of an existing track.

The history of music is also a history of technology. The electric guitar gave the world rock and roll. When the sampler showed up, hip-hop was born, said Zmishlany. With AI, we're writing the next chapter of music. We have a new instrument, a new tool for making music - and that's super exciting.

An ad agency could apply Fugatto to quickly target an existing campaign for multiple regions or situations, applying different accents and emotions to voiceovers.

Language learning tools could be personalized to use any voice a speaker chooses. Imagine an online course spoken in the voice of any family member or friend.

Video game developers could use the model to modify prerecorded assets in their title to fit the changing action as users play the game. Or, they could create new assets on the fly from text instructions and optional audio inputs.

Making a Joyful Noise One of the model's capabilities we're especially proud of is what we call the avocado chair, said Valle, referring to a novel visual created by a generative AI model for imaging.

For instance, Fugatto can make a trumpet bark or a saxophone meow. Whatever users can describe, the model can create.

With fine-tuning and small amounts of singing data, researchers found it could handle tasks it was not pretrained on, like generating a high-quality singing voice from a text prompt.

Users Get Artistic Controls Several capabilities add to Fugatto's novelty.

During inference, the model uses a technique called ComposableART to combine instructions that were only seen separately during training. For example, a combination of prompts could ask for text spoken with a sad feeling in a French accent.

The model's ability to interpolate between instructions gives users fine-grained control over text instructions, in this case the heaviness of the accent or the degree of sorrow.

I wanted to let users combine attributes in a subjective or artistic way, selecting how much emphasis they put on each one, said Rohan Badlani, an AI researcher who designed these aspects of the model.

In my tests, the results were often surprising and made me feel a little bit like an artist, even though I'm a computer scientist, said Badlani, who holds a master's degree in computer science with a focus on AI from Stanford.

The model also generates sounds that change over time, a feature he calls temporal interpolation. It can, for instance, create the sounds of a rainstorm moving through an area with crescendos of thunder that slowly fade into the distance. It also gives users fine-grained control over how the soundscape evolves.

Plus, unlike most models, which can only recreate the training data they've been exposed to, Fugatto allows users to create soundscapes it's never seen before, such as a thunderstorm easing into a dawn with the sound of birds singing.

A Look Under the Hood Fugatto is a foundational generative transformer model that builds on the team's prior work in areas such as speech modeling, audio vocoding and audio understanding.

The full version uses 2.5 billion parameters and was trained on a bank of NVIDIA DGX systems packing 32 NVIDIA H100 Tensor Core GPUs.

Fugatto was made by a diverse group of people from around the world, including India, Brazil, China, Jordan and South Korea. Their collaboration made Fugatto's multi-accent and multilingual capabilities stronger.

One of the hardest parts of the effort was generating a blended dataset that contains millions of audio samples used for training. The team employed a multifaceted strategy to generate data and instructions that considerably expanded the range of tasks the model could perform, while achieving more accurate performance and enabling new tasks without requiring additional data.

They also scrutinized existing datasets to reveal new relationships among the dat
LINK: https://blogs.nvidia.com/blog/fugatto-gen-ai-sound-model/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

16/11/2026

Rohde & Schwarz Japan demonstrates multi-vendor ASA-ML communication with highperformance oscilloscope

Rohde & Schwarz Japan demonstrates multi-vendor ASA-ML communication with high p...

16/11/2026

CNAO and Hitachi launch advanced rotating proton therapy system in Italy

November 16, 2026 Hitachi system enables proton beams to be delivered from 360 degrees around the patient, expanding treatment capacity and supporting more pre...

16/11/2026

NBC Celebrates Centennial

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

16/11/2026

Apple TV Adding Podcasts

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

16/11/2026

Fabric Insights Partners With Fonn Group on Enhancing Media Workflows

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

16/11/2026

SDVI Names Marcy Lefkovitz as VP of Strategic Accounts

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

16/11/2026

Monthly Streaming Costs Up, Internet Fees Down for 2026

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

16/11/2026

Hexagon Introduces Camera-Mounted 3D Capture and Tracking System

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

16/11/2026

Bridge Technologies Adds Apple ProRes Support to VB440 Probe

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

16/11/2026

Rebuilding Broadcast on Air: Inside the 2026 SMPTE Media Technology Summit

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

16/11/2026

Hiltron HMAM Chosen by Major Satcom Service Provider

Hiltron Communications has received a repeat order from one of Europes longest established communications satellite operators. Central to this project is a Hilt...

16/11/2026

Fabric Insights and Fonn Group Partner to Unlock Media Wo...

Fabric Insights and Fonn Group today announce a partnership to connect editorial planning and media operations across their complementary product portfolios. ...

16/11/2026

Density 1.0 brings a complete color grading workflow to Final Cut Pro

Density 1.0 brings a complete color grading workflow to Final Cut Pro Brie Clayton November 16, 2026 0 Comments Filmmaker Pascal Basel's plugin pr...

16/11/2026

Berklee Students to Perform at Boston's CITGO Sign Relighting

Berklee Students to Perform at Boston's CITGO Sign Relighting Kuper Walker, Ollie Mann, and dreamcorp. will perform at the free public celebration in Kenm...

16/11/2026

The Hollywood Reporter Names Berklee Among World's Best Music Schools

The Hollywood Reporter Names Berklee Among World's Best Music Schools The annual list spotlights Berklee's screen scoring programs and new master'...

16/11/2026

Taurus Rooms - A Five-Year Vision Proven In Meeting Rooms

Five years ago, Lightware launched the original Taurus UCX with a clear vision: USB-C would become the central connection in meeting rooms. The goal was simple:...

16/11/2026

Sonniss: bundles over 80% below their libraries' prices, plus up to 60% off over 2,700 sound effects libraries

Sonniss: bundles over 80% below their libraries' prices, plus up to 60% off ...

16/11/2026

Singaporean Comedy Luck My Life Shot with Blackmagic URSA Cine 17K 65

Singaporean Comedy Luck My Life Shot with Blackmagic URSA Cine 17K 65 Brie Clayton November 16, 2026 0 Comments Blackmagic PYXIS 12K and DaVinci Resol...

16/11/2026

Can Digital Product Passports help meet climate goals?

As countries gather to discuss climate targets at COP 31 this week, one question that is at the heart of debates is how to measure the environmental impact of t...

16/11/2026

NVIDIA Announces Rubin HPC - A New GPU to Accelerate Agents for Science, Engineering and Analytics

Announced today at SC26, NVIDIA Rubin HPC is a new GPU in the NVIDIA Vera Rubin ...

16/11/2026

CUDA at 20: How the Platform Transformed Scientific Computing

This week at SC26, NVIDIA is marking two decades of CUDA by honoring the researchers, developers and engineers who helped make GPU computing a foundation for mo...

16/11/2026

NVIDIA CUDA-X Libraries Give Agents the Tools to Accelerate Scientific Discovery

With expanded NVIDIA CUDA-X libraries, including cuPhoton, cuPotential and cuSpa, researchers can sift through vast amounts of data to accelerate research, from...

16/11/2026

NVIDIA Sweeps MLPerf Training and Endpoints as Vera Rubin Debuts With Leading Performance

In its MLPerf benchmarks debut, with results released today, the NVIDIA Vera Rub...

16/11/2026

NVIDIA Expands Slurm to Bring Quantum Computing Into Supercomputing

Quantum processor deployments at supercomputing centers have grown nearly sevenfold over the past five years, with nearly 30 supercomputing centers around the w...

16/11/2026

Lambda Boosts AI Factory Resiliency and Efficiency With NVIDIA Photonics

To train an AI model, thousands of GPUs have to work together, requiring a network that's fast, efficient and resilient. Lambda is tackling the network pie...

15/11/2026

Yamaha launch MGX, URX & CC1 updates

Mixers, interfaces & DAW controller receive new firmware Yamaha have just launched a trio of firmware updates that bring some powerful new functionality to ...

15/11/2026

Can VideoProc AI Really Reduce Noise and Enhance Blurry, Old Footage? Real-World Tests

Can VideoProc AI Really Reduce Noise and Enhance Blurry, Old Footage? Real-World...

15/11/2026

AI Day Seoul Brings Together Korea's Industry Leaders, Startups and Developer Communities

Over two days at COEX, more than 2,000 attendees converged for AI Day Seoul, bri...

14/11/2026

MLB Networks David Maybrown, 57, Passes Away

Longtime MLB Network employee David Maybrown passed away Thursday at the age of 57. David joined MLB Network in 2009 shortly after its launch, beginning as an i...

14/11/2026

Rotary Exciter & Resonator Clamps from ADDAC System

Modular innovators introduce two guitar-based products ADDAC System's recently introduced Four String Series makes it possible to integrate physical str...

14/11/2026

Marcelo Dai's Live Session Captured with URSA Cine Immersive

Marcelo Dai's Live Session Captured with URSA Cine Immersive Brie Clayton November 14, 2026 0 Comments Artist takes viewers on an immersive journe...

14/11/2026

Political Ads Spiked a Week Before Midterm Elections, Nielsen Says

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

14/11/2026

FCC Sends 2027 Upper C-Band Auction Plan Into Motion

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

14/11/2026

ZEISS Showcases Latest Cine Lenses at TorunCAMERIMAGE 202...

Newest Lenses, Horizon Anamorphic Seminar & Guided Lens Character Experience Highlight Creative Possibilities At the 2026 Toru CAMERIMAGE Film Festival, ZEISS...

14/11/2026

Sonnet Announces Black Friday Sale on Select Products

Sonnet Announces Black Friday Sale on Select Products Brie Clayton November 14, 2026 0 Comments WHAT: Sonnet Technologies is having a sale on a select...

14/11/2026

Berklee Alumni Shine at the 2026 Latin Grammy Awards

Berklee Alumni Shine at the 2026 Latin Grammy Awards This year's winners included songwriters, engineers, and performers who contributed to acclaimed reco...

13/11/2026

E1 Series Brings Its Remote-First Broadcast Model to Miami for Season Finale

The UIM E1 Series wraps its third season this week with the Bombay Sapphire E1 Miami GP, Nov. 11-14, on Biscayne Bay at the Miami Seaplane Base on Watson Island...

13/11/2026

SVG GameDay, Ep. 39: Carolina Hurricanes Christine Williams & Jason Danby - Celebrating a Stanley Cup in Raleigh

In-venue and creative video staffers at the professional and collegiate level ha...

13/11/2026

Benson Boone to Headline Entertainment at Discover NHL Winter Classic

Singer-songwriter Benson Boone will headline entertainment for the Discover NHL Winter Classic between the Utah Mammoth and Colorado Avalanche at Rice-Eccles St...

13/11/2026

Blackmagic Design Announces Ambisonic Microphone for Immersive Spatial Audio

Blackmagic Design has announced the Blackmagic Ambisonic Microphone, a 7th order ambisonic microphone array that captures spatial audio for immersive video prod...

13/11/2026

The Studio B&H and Riedel to Power SMPTE MTS Solutions Hub

SMPTE has announced that The Studio B&H and Riedel Communications will provide the video infrastructure and communications technology for the Solutions Hub at t...

13/11/2026

HPA Sets 2027 Tech Retreat Theme Around a Changing Industry

The Hollywood Professional Association (HPA) has announced the theme and program focus for the 2027 HPA Tech Retreat, which takes place Feb. 14-18, 2027, in La ...

13/11/2026

Football Manager 27 Touch Joins Apple Arcade, with Plants vs. Zombies, Prince of Persia Added Dec. 3

Football Manager 27 Touch is available on Apple Arcade starting today. The game ...

13/11/2026

MotoGP and ESPN Announce Two-Year Media Rights Agreement for U.S. and Latin America

MotoGP has announced a two-year media rights agreement with ESPN covering the Un...

13/11/2026

Apple Sports Adds UFC Ahead of UFC 334

Apple Sports, the free iPhone app, has added the Ultimate Fighting Championship (UFC) to its lineup of supported sports, in time for Saturday's UFC 334. The...

13/11/2026

NBA Names Fanatics Studios Multiyear Creative and Event Production Partner for NBA All-Star

The NBA and Fanatics Studios, the joint venture between Fanatics and entertainme...

13/11/2026

From ManningCast to McAfee to Star Wars: ESPN, Disney Pull Back the Curtain on Super Bowl LXI Plans

ESPN held its annual EDGE Innovation Conference last night and offered up an ove...

13/11/2026

Battle in the Bronx: YES Network Presents 2026 Cortaca Jug at Yankee Stadium

Regional sports network gives Ithaca College, SUNY Cortland students a platform to thrive One of the biggest games in Division III college football will be pla...