Sony Pixel Power calrec Sony

Now Hear This: World's Most Flexible Sound Machine Debuts

25/11/2024

A team of generative AI researchers created a Swiss Army knife for sound, one that allows users to control the audio output simply using text.

While some AI models can compose a song or modify a voice, none have the dexterity of the new offering.

Called Fugatto (short for Foundational Generative Audio Transformer Opus 1), it generates or transforms any mix of music, voices and sounds described with prompts using any combination of text and audio files.

For example, it can create a music snippet based on a text prompt, remove or add instruments from an existing song, change the accent or emotion in a voice - even let people produce sounds never heard before.

This thing is wild, said Ido Zmishlany, a multi-platinum producer and songwriter - and cofounder of One Take Audio, a member of the NVIDIA Inception program for cutting-edge startups. Sound is my inspiration. It's what moves me to create music. The idea that I can create entirely new sounds on the fly in the studio is incredible.

A Sound Grasp of Audio We wanted to create a model that understands and generates sound like humans do, said Rafael Valle, a manager of applied audio research at NVIDIA and one of the dozen-plus people behind Fugatto, as well as an orchestral conductor and composer.

Supporting numerous audio generation and transformation tasks, Fugatto is the first foundational generative AI model that showcases emergent properties - capabilities that arise from the interaction of its various trained abilities - and the ability to combine free-form instructions.

Fugatto is our first step toward a future where unsupervised multitask learning in audio synthesis and transformation emerges from data and model scale, Valle said.

A Sample Playlist of Use Cases For example, music producers could use Fugatto to quickly prototype or edit an idea for a song, trying out different styles, voices and instruments. They could also add effects and enhance the overall audio quality of an existing track.

The history of music is also a history of technology. The electric guitar gave the world rock and roll. When the sampler showed up, hip-hop was born, said Zmishlany. With AI, we're writing the next chapter of music. We have a new instrument, a new tool for making music - and that's super exciting.

An ad agency could apply Fugatto to quickly target an existing campaign for multiple regions or situations, applying different accents and emotions to voiceovers.

Language learning tools could be personalized to use any voice a speaker chooses. Imagine an online course spoken in the voice of any family member or friend.

Video game developers could use the model to modify prerecorded assets in their title to fit the changing action as users play the game. Or, they could create new assets on the fly from text instructions and optional audio inputs.

Making a Joyful Noise One of the model's capabilities we're especially proud of is what we call the avocado chair, said Valle, referring to a novel visual created by a generative AI model for imaging.

For instance, Fugatto can make a trumpet bark or a saxophone meow. Whatever users can describe, the model can create.

With fine-tuning and small amounts of singing data, researchers found it could handle tasks it was not pretrained on, like generating a high-quality singing voice from a text prompt.

Users Get Artistic Controls Several capabilities add to Fugatto's novelty.

During inference, the model uses a technique called ComposableART to combine instructions that were only seen separately during training. For example, a combination of prompts could ask for text spoken with a sad feeling in a French accent.

The model's ability to interpolate between instructions gives users fine-grained control over text instructions, in this case the heaviness of the accent or the degree of sorrow.

I wanted to let users combine attributes in a subjective or artistic way, selecting how much emphasis they put on each one, said Rohan Badlani, an AI researcher who designed these aspects of the model.

In my tests, the results were often surprising and made me feel a little bit like an artist, even though I'm a computer scientist, said Badlani, who holds a master's degree in computer science with a focus on AI from Stanford.

The model also generates sounds that change over time, a feature he calls temporal interpolation. It can, for instance, create the sounds of a rainstorm moving through an area with crescendos of thunder that slowly fade into the distance. It also gives users fine-grained control over how the soundscape evolves.

Plus, unlike most models, which can only recreate the training data they've been exposed to, Fugatto allows users to create soundscapes it's never seen before, such as a thunderstorm easing into a dawn with the sound of birds singing.

A Look Under the Hood Fugatto is a foundational generative transformer model that builds on the team's prior work in areas such as speech modeling, audio vocoding and audio understanding.

The full version uses 2.5 billion parameters and was trained on a bank of NVIDIA DGX systems packing 32 NVIDIA H100 Tensor Core GPUs.

Fugatto was made by a diverse group of people from around the world, including India, Brazil, China, Jordan and South Korea. Their collaboration made Fugatto's multi-accent and multilingual capabilities stronger.

One of the hardest parts of the effort was generating a blended dataset that contains millions of audio samples used for training. The team employed a multifaceted strategy to generate data and instructions that considerably expanded the range of tasks the model could perform, while achieving more accurate performance and enabling new tasks without requiring additional data.

They also scrutinized existing datasets to reveal new relationships among the dat
LINK: https://blogs.nvidia.com/blog/fugatto-gen-ai-sound-model/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

04/08/2026

Dalet Announces Commercial Availability of Dalia, Bringing Media-Aware Agentic AI to Enterprise Productions

Dalet, a leading technology and service provider for media-rich organizations, t...

04/07/2026

Detective Conan: Fallen Angel of the Highway Opens in Dolby Cinemas Across Japan, Presented in Dolby Atmos and Dolby ...

April 7 2026, 19:00 (PDT) Detective Conan: Fallen Angel of the Highway Opens in...

01/06/2026

Dolby Sets the New Standard for Premium Entertainment at CES 2026

January 6 2026, 05:30 (PST) Dolby Sets the New Standard for Premium Entertainment at CES 2026 Throughout the week, Dolby brings to life the latest innovatio...

14/05/2026

Sweetwater and Airstream Unveil Mobile Dolby Atmos Recording Studio

Sweetwater and Airstream have announced a custom-built Dolby Atmos mobile recording studio inside an Airstream trailer, set to tour music festivals, schools, tr...

14/05/2026

American Association of Professional Baseball Expands Broadcast Distribution for 2026 Season

The American Association of Professional Baseball (AAPB) has announced a new par...

14/05/2026

ESPN to Establish Week-Long Super Bowl LXI Broadcast Center on Santa Monica Beach

ESPN has announced plans to transform Santa Monica Beach into a broadcast hub du...

14/05/2026

Amagi Announces Major Upgrade to CLOUDPORT Cloud Broadcast Platform

Amagi has announced a significant update to Amagi CLOUDPORT, its cloud-based broadcast playout platform. The update includes 250-plus features shipped in FY25-2...

14/05/2026

Clear-Com to Showcase New Products and Platform Updates at InfoComm 2026

Clear-Com will exhibit at InfoComm 2026 (Booth N7005, June 17-19, Las Vegas Convention Center), introducing a new product that builds on Arcadia Central Station...

14/05/2026

Ikegami to Exhibit at BroadcastAsia 2026 with Two New Viewfinder Premieres

Ikegami will exhibit at BroadcastAsia 2026 (Stand 5D3-1, Singapore Expo, May 20-22), introducing two new viewfinders alongside its existing camera, control, and...

14/05/2026

dB Broadcast Delivers IP-Based OB Trucks for Cloudbass Featuring Grass Valley LDX 100 Cameras

Grass Valley has announced that dB Broadcast has delivered new IP-based outside ...

14/05/2026

NAGRAVISION and WPBSA Launch Play Snooker Digital Platform

NAGRAVISION, a Kudelski Group company, has announced a partnership with the World Professional Billiards and Snooker Association (WPBSA) to launch Play Snooker,...

14/05/2026

Belden To Acquire RUCKUS Networks for Approximately $1.85 Billion

Belden Inc. has announced a definitive agreement to acquire RUCKUS Networks from Vistance Networks for approximately $1.85 billion. The transaction has been app...

14/05/2026

NVIDIA Releases Content Localization Blueprint for AI-Assisted Dubbing and Speech Translation

NVIDIA has released the Content Localization Blueprint, a modular reference arch...

14/05/2026

Disney+ To Stream Banana Bowl Championship Live This October

Disney has announced that Disney will be the exclusive U.S. streaming home of the Banana Bowl, the Banana Ball league season championship, streaming live this ...

14/05/2026

Arkona and Manifold To Exhibit on Magna Systems Stand at BroadcastAsia 2026

Arkona technologies and technology partner manifold will demonstrate their production solutions on the Magna Systems and Engineering stand (Booth 5D1-1) at Broa...

14/05/2026

Haivision Webinar: New Broadcast Contribution Products Featuring Minor League Baseball Case Study

Haivision will host a webinar on Thursday, May 21 at 10 a.m. ET / 4 p.m. CET cov...

14/05/2026

The CW Network and ESPN Announce ACC Sublicense Agreement Through 2030-31 Season

The CW Network and ESPN have announced a sublicense broadcast agreement for The CW to televise ACC football and men's and women's college basketball gam...

14/05/2026

Detroit Pistons Ink Local Media Rights Deal With Scripps Sports

The agreement marks Scripps Sports' first NBA local rights deal...

14/05/2026

Report: Broadcasting Among Hardest-Hit Industries as AI Reshapes the Workforce

A new report from education-technology company Wiingy testing post-ChatGPT predictions against three years of real-world data has identified broadcasting as one...

14/05/2026

Madonna, Shakira, and BTS to Headline First-Ever FIFA World Cup Final Halftime Show

Global Citizen and FIFA have announced that Madonna, Shakira, and BTS will headl...

14/05/2026

Sundance Institute Names 2026 Episodic Lab Fellows

LOS ANGELES, CA, May 14, 2026 - The nonprofit Sundance Institute announced today the cohort selected for the 2026 Episodic Lab program, taking place at Dunaway ...

14/05/2026

Spotify Expands Music Access for Young Listeners, Extending Managed Accounts to Free Tier

At Spotify, we're focused on making every listening experience feel intentio...

14/05/2026

Spotify Brings Nashville's Songwriting Community Together for Mental Health Summit

Spotify recently welcomed songwriters, artists, executives, and music students t...

14/05/2026

Sonuscore update Lux Orchestral Strings

New articulations, ostinatos, Motion Scoring Articulation Sets & more Sonuscore's flagship cinematic string library has just been treated to a significa...

14/05/2026

Pr Recording & Residence

World-class studio opens on T rkiye's Aegean coast P r Recording & Residence have announced their official opening, introducing a new world-class reside...

14/05/2026

Nugen Audio update DialogCheck

Now supports channel layouts up to 9.1.6 Nugen Audio have just released an update for their AI-powered dialogue intelligibility and compliance tool. Set to ...

14/05/2026

UVI release Orchestral Suite 2

New recordings & one-key chord tool UVI have just announced the release of Orchestral Suite 2, a ground-up redesign of their all-in-one symphonic orchestra ...

14/05/2026

Rob Papen launch eXplorer 11

Two new arrivals & expanded factory content Rob Papen's all-encompassing plug-in and virtual instrument collection has just been treated to another upda...

14/05/2026

FSK Audio release QRonicle

Embed QR codes into DAW sessions FSK Audio's latest plug-in doesn't process audio, but serves as an organisational tool that allows QR codes to be e...

14/05/2026

SBS Board appoints Jane Palfreyman Managing Director

SBS Board appoints Jane Palfreyman Managing Director 13 May, 2026 Media releases The Special Broadcasting Service (SBS) Board of Directors is pleased to an...

14/05/2026

Transforming bold ideas into market-ready productions: Digital Originals returns

Transforming bold ideas into market-ready productions: Digital Originals returns 14 May, 2026 Media releases SBS, NITV and Screen Australia have announced ...

14/05/2026

Australia Uncovered Returns to SBS with Bold New Season Featuring John Safran on Free Speech and Powerful Stories of Justice and Survival

Australia Uncovered Returns to SBS with Bold New Season Featuring John Safran on...

14/05/2026

Rohde & Schwarz transforms spectrum complexity into situational awareness and effective countermeasures at AOC Europe 2026

Rohde & Schwarz transforms spectrum complexity into situational awareness and ef...

14/05/2026

Rohde & Schwarz and Quantum Systems join forces to redefine EW and C-UAS-enabled uncrewed operations

Rohde & Schwarz and Quantum Systems join forces to redefine EW and C-UAS-enabled...

14/05/2026

Rohde & Schwarz showcases STANAG aligned ARDRONIS Counter UAS capability at NATO Technical Interoperability Exercise 2026

Rohde & Schwarz showcases STANAG aligned ARDRONIS Counter UAS capability at NATO...

14/05/2026

Code of Silence wins Best Drama Series at the 2026 BAFTA's!

Code of Silence has won the BAFTA for Best Drama Series at Sunday night's ceremony at the Royal Festival Hall. The series, starring Rose Ayling-Ellis and w...

14/05/2026

The Wraith Shield Advantage: Transforming L3Harris Radios into AI-Enabled Counter-UAS Sensors

Soldiers equipped with Falcon IV radios will soon gain a sense-and-protect capa...

14/05/2026

Getting into the Space Nuclear Power Game with Next-Generation Technology

Artists concept of the L3Harris Next Gen RTG in flight configuration, designed to provide 250 watts of reliable power for decades-long missions in deep space....

14/05/2026

Vivid Broadcast builds remote production network around Calrec

Vivid Broadcast was embracing remote production long before it became the industry norm. Now, with Calrec's True Control 2.0-enabled Argo M and Type R conso...

14/05/2026

Nielsen data shows NZ vehicle advertisers are shifting gears as fuel pressures make EVs and hybrids an increasingly attractive option

Car ad spend rises sharply in March as more auto buyers turn to electric, hybrid...

14/05/2026

Is This the Year for Agentic AI's Breakout in Broadcast?

Share Copy link Facebook X Linkedin Bluesky Email...

14/05/2026

CBS LA, Los Angeles Rams Ink New TV Deal

Share Copy link Facebook X Linkedin Bluesky Email...

14/05/2026

CueScripts CueiT 4 0 Wins Futures Best of Show Award Pres...

CueScript's CueiT 4.0 Wins Future's Best of Show Award, Presented at 2026 NAB Show by TV Tech CueScript, a leading international developer of professio...

14/05/2026

Expert-Led Education Sessions and Development of Online T...

Expert-Led Education Sessions and Development of Online Training Program Accelerate IPMX Adoption and Deployment The Alliance for IP Media Solutions (AIMS) to...

14/05/2026

Klvr rechargeable battery launches in the USA to cut cost...

Klvr is launching in the United States with a professional-grade rechargeable battery solution that cuts costs and improves performance across live entertainmen...

14/05/2026

Shooting into the depths of Bedlam with URSA Cine 17K 65

Shooting into the depths of Bedlam with URSA Cine 17K 65 Brie Clayton May 14, 2026 0 Comments Indie feature film paired digital 65mm capture with a Bl...

14/05/2026

WeMakeColor expands with Baselight, becoming hybrid color facility

WeMakeColor expands with Baselight, becoming hybrid color facility Caroline Shawley May 14, 2026 0 Comments Boutique Mexican-based studio integrates B...

14/05/2026

Berklee's Summer in the City Returns with Free Concerts Throughout Boston Area

Berklee's Summer in the City Returns with Free Concerts Throughout Boston Ar...

14/05/2026

Chelsey Green Named to Billboard's 2026 Women in Music List

Chelsey Green Named to Billboard's 2026 Women in Music List The Berklee professor and chair of the Recording Academy Board of Trustees joins other high-pr...