Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, software, tools and accelerations for RTX PC users.AI has changed computing forever. The spotlight has most recently been on generative AI, but AI-accelerated, NVIDIA RTX-powered tools have also been key in improving gaming, content creation and productivity over the years.
The NVIDIA Broadcast app is one example, using Tensor Cores on a local RTX GPU to seamlessly improve audio and video quality. Paired with the NVIDIA encoder (NVENC) built into GeForce RTX and NVIDIA RTX GPUs, the app makes it easy to get started as a livestreamer or to look professional during video conference calls.
The Stream Dream
High-quality livestreaming traditionally required expensive hardware. Many livestreamers relied on software CPU encoding using the x264 software library, which often impacted gameplay quality. This led many to use a dual-PC setup, with one PC focused on gaming and content and the other on encoding the stream. It was complicated to assemble, difficult to troubleshoot and often cost-prohibitive for budding livestreamers.
NVENC is here to help. It's a dedicated hardware video encoder on NVIDIA GPUs that processes the encoding, freeing up the rest of the system to focus on game and content performance. Industry-leading streaming apps like Open Broadcaster Software (OBS) are adding support for NVENC, paving the way for a new generation of broadcasters on popular platforms like Twitch and YouTube.
Meanwhile, NVIDIA Maxine helps solve the issue of expensive equipment. It includes free, AI-powered features like virtual green screens and webcam-based augmented reality tracking that eliminate the need for special equipment like physical green screens or motion- capture suits. Broadcasters first got to experience the technology at TwitchCon 2019, where they tested OBS live on the show floor with an AI-accelerated green screen on a GeForce RTX 2080 GPU.
Maxine's AI-powered effects debuted for RTX users in the RTX Voice beta, and moved into the NVIDIA Broadcast app.
Now Showing: NVIDIA Broadcast NVIDIA Broadcast offers AI-powered features that improve audio and video quality for a variety of use cases. It's user-friendly, works in any app and is a breeze to set up.
It includes:
Noise and Acoustic Echo Removal: AI eliminates unwanted background noise from both the mic and inbound audio at the touch of a button.
Virtual Backgrounds: Features like Background Removal, Replacement and Blur help customize backgrounds without the need for expensive equipment or complex lighting setups.
Eye Contact: AI helps make it appear as though a streamer is looking directly at the camera, even when they're glancing off camera or taking notes.
Auto Frame: Dynamically tracks movements in real time, automatically cropping and zooming moving objects regardless of their position.
Vignette: AI applies a darkening effect to the corners of camera images, providing visual contrast to draw attention to the center of the video and adding stylistic flair.
Video Noise Removal: Removes visual noise from low-light situations for a cleaner picture.
NVIDIA Broadcast works by creating a virtual camera, microphone or speaker in Windows so that users can set up their devices once and use them in any broadcasting, video conferencing or voice chat apps, including Discord, Google Meet, Microsoft Teams, OBS Studio, Slack, Webex and Zoom.
Those with an NVIDIA GeForce RTX, TITAN RTX, NVIDIA RTX or Quadro RTX GPU can use their GPU's dedicated Tensor Cores to help the app's AI networks run in real time.
The same AI-powered technology in NVIDIA Broadcast is also available to app developers as a software development kit. Audiovisual technology company Elgato includes Maxine's AI audio noise removal technology in its Wave Link software, while VTube Studio - a popular app for connecting a 3D model to a webcam for streaming as an animated character - offers an RTX-accelerated model tracker plug-in as a free download. Independent developer Xaymar uses NVIDIA Maxine in his VoiceFX plug-in.
Content creators can use this plug-in or Elgato's virtual studio technology (VST) filter to clean up noise and echo from recordings in post-processing in video editing suites like Adobe Premiere Pro or in digital audio workstations like Ableton Live and Adobe Audition.
(Not) Hearing Is Believing Since its release, NVIDIA Broadcast has been used by millions.
I've utilized the video noise removal and background replacement the most, said Mr_Vudoo, a Twitch personality and broadcaster. The eye contact feature was very interesting and quite honestly took me by surprise at how well it worked.
Unmesh Dinda, host of the YouTube channel PiXimperfect, demonstrated NVIDIA Broadcast's noise-canceling and echo-removal AI features in an extreme scenario. He set an electric fan whirring directly into his microphone and donned a helmet that was intensely hammered on. Even with these loud sounds in the background, Dinda could be heard crystal clear with Broadcast's noise-removal feature turned on. The video has racked up more than 12 million views.
NVIDIA Broadcast is also a useful tool for the growing remote workforce. In an article, Tom's Hardware editor-in-chief Avram Piltch detailed his testing of the app's noise reduction features against noisy air conditioners, lawn-mowing neighbors and even a robot-wielding, tantrum-throwing child. Broadcast's AI audio filters prevailed every time:
I got my eight-year-old to fake throwing a fit right behind me and, once I enabled noise removal, every whine of I'm not going to bed' went silent (at least on the recording), said Piltch. To double the challenge, we had him throw a tantrum while carrying
