Sony Pixel Power calrec Sony

TOPS of the Class: Decoding AI Performance on RTX AI PCs and Workstations

12/06/2024

Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, software, tools and accelerations for RTX PC users.

The era of the AI PC is here, and it's powered by NVIDIA RTX and GeForce RTX technologies. With it comes a new way to evaluate performance for AI-accelerated tasks, and a new language that can be daunting to decipher when choosing between the desktops and laptops available.

While PC gamers understand frames per second (FPS) and similar stats, measuring AI performance requires new metrics.

Coming Out on TOPS The first baseline is TOPS, or trillions of operations per second. Trillions is the important word here - the processing numbers behind generative AI tasks are absolutely massive. Think of TOPS as a raw performance metric, similar to an engine's horsepower rating. More is better.

Compare, for example, the recently announced Copilot+ PC lineup by Microsoft, which includes neural processing units (NPUs) able to perform upwards of 40 TOPS. Performing 40 TOPS is sufficient for some light AI-assisted tasks, like asking a local chatbot where yesterday's notes are.

But many generative AI tasks are more demanding. NVIDIA RTX and GeForce RTX GPUs deliver unprecedented performance across all generative tasks - the GeForce RTX 4090 GPU offers more than 1,300 TOPS. This is the kind of horsepower needed to handle AI-assisted digital content creation, AI super resolution in PC gaming, generating images from text or video, querying local large language models (LLMs) and more.

Insert Tokens to Play TOPS is only the beginning of the story. LLM performance is measured in the number of tokens generated by the model.

Tokens are the output of the LLM. A token can be a word in a sentence, or even a smaller fragment like punctuation or whitespace. Performance for AI-accelerated tasks can be measured in tokens per second.

Another important factor is batch size, or the number of inputs processed simultaneously in a single inference pass. As an LLM will sit at the core of many modern AI systems, the ability to handle multiple inputs (e.g. from a single application or across multiple applications) will be a key differentiator. While larger batch sizes improve performance for concurrent inputs, they also require more memory, especially when combined with larger models.

The more you batch, the more (time) you save. RTX GPUs are exceptionally well-suited for LLMs due to their large amounts of dedicated video random access memory (VRAM), Tensor Cores and TensorRT-LLM software.

GeForce RTX GPUs offer up to 24GB of high-speed VRAM, and NVIDIA RTX GPUs up to 48GB, which can handle larger models and enable higher batch sizes. RTX GPUs also take advantage of Tensor Cores - dedicated AI accelerators that dramatically speed up the computationally intensive operations required for deep learning and generative AI models. That maximum performance is easily accessed when an application uses the NVIDIA TensorRT software development kit (SDK), which unlocks the highest-performance generative AI on the more than 100 million Windows PCs and workstations powered by RTX GPUs.

The combination of memory, dedicated AI accelerators and optimized software gives RTX GPUs massive throughput gains, especially as batch sizes increase.

Text-to-Image, Faster Than Ever Measuring image generation speed is another way to evaluate performance. One of the most straightforward ways uses Stable Diffusion, a popular image-based AI model that allows users to easily convert text descriptions into complex visual representations.

With Stable Diffusion, users can quickly create and refine images from text prompts to achieve their desired output. When using an RTX GPU, these results can be generated faster than processing the AI model on a CPU or NPU.

That performance is even higher when using the TensorRT extension for the popular Automatic1111 interface. RTX users can generate images from prompts up to 2x faster with the SDXL Base checkpoint - significantly streamlining Stable Diffusion workflows.

ComfyUI, another popular Stable Diffusion user interface, added TensorRT acceleration last week. RTX users can now generate images from prompts up to 60% faster, and can even convert these images to videos using Stable Video Diffuson up to 70% faster with TensorRT.

TensorRT acceleration can be put to the test in the new UL Procyon AI Image Generation benchmark, which delivers speedups of 50% on a GeForce RTX 4080 SUPER GPU compared with the fastest non-TensorRT implementation.

TensorRT acceleration will soon be released for Stable Diffusion 3 - Stability AI's new, highly anticipated text-to-image model - boosting performance by 50%. Plus, the new TensorRT-Model Optimizer enables accelerating performance even further. This results in a 70% speedup compared with the non-TensorRT implementation, along with a 50% reduction in memory consumption.

Of course, seeing is believing - the true test is in the real-world use case of iterating on an original prompt. Users can refine image generation by tweaking prompts significantly faster on RTX GPUs, taking seconds per iteration compared with minutes on a Macbook Pro M3 Max. Plus, users get both speed and security with everything remaining private when running locally on an RTX-powered PC or workstation.

The Results Are in and Open Sourced But don't just take our word for it. The team of AI researchers and engineers behind the open-source Jan.ai recently integrated TensorRT-LLM into its local chatbot app, then tested these optimizations for themselves.

Source: Jan.ai The researchers tested its implementation of TensorRT-LLM against the open-source llama.cpp inference engine across a variety of GPUs and CPUs used by the community. They found that TensorRT is 30-70% faster than llam
LINK: https://blogs.nvidia.com/blog/ai-decoded-tops/...
See more stories from nvidia

North America Stories

20/07/2024

Comcast NBCU to Provide Service Members with Free Streaming of Olympics

DALLAS Comcast NBCUniversal has announced that once again, service members and honorably discharged Veterans worldwide will enjoy free access to NBCUniversal...

20/07/2024

SES Expands Partnership with Ateme for Sports and Live Events

PARIS Ateme, a global provider of video compression, delivery, and streaming solutions, says longtime partner SES is integrating its technology to improve and s...

20/07/2024

Google Named the Official Search AI Partner of Team USA

NEW YORK Google, Team USA and NBCUniversal have announced a sponsorship agreement naming Google as the Official Search AI Partner of Team USA....

20/07/2024

LiveU, Pente Networks Launch LiveU Private Connectivity

HACKENSACK, N.J. As broadcasters and news organizations look for more reliable connectivity for their coverage of this year's elections, LiveU is reporting ...

20/07/2024

CBS Sports and Serie A Renew U.S. Media Rights Agreement

NEW YORK Following a recent deal for English Football League rights, CBS Sports continues to bulk up on international soccer rights, with a multi-platform, two-...

20/07/2024

Optimum Reaches Multi-Year Extension with Gray Media

NEW YORK Altice USA's Optimum pay TV operations and Gray Media have agreed to a multi-year extension of their retransmission consent agreement....

20/07/2024

FOR-A to Spotlight Software-Defined IP Solutions At IBC Show

BRUGHERIO, Italy FOR-A will show spotlight software-defined IP solutions and hybrid production at the IBC Show, Sept. 13-16, at the RAI Amsterdam Convention Cen...

20/07/2024

Perifery Introduces AI+ 2.0 Suite of Tools

FT. LAUDERDALE, Fla. Perifery has launched AI+ 2.0, an AI-powered suite for media professionals to revitalize their existing content libraries with AI-generated...

20/07/2024

nxtedition unveils XR mixed reality studio control and ne...

Innovator in live production environments, nxtedition, will showcase its agile microservices-based approach to storytelling on stand 7.A02 at IBC2024 from 13-16...

20/07/2024

Viaccess-Orca IBC 2024 Exhibitor Preview

In today s competitive landscape, the diverse needs and distinct value-chain challenges within the video delivery ecosystem demand effective, flexible solutions...

20/07/2024

Broadpeak Brings Dynamic Ad Insertion Innovations and New...

Broadpeak , a global expert in video delivery with more than 150 customers serving more than 200 million end users worldwide, will highlight how it is elevating...

20/07/2024

Interra Systems to Present Companys Cutting-Edge Media QC...

Interra Systems, the leading provider of end-to-end quality assurance solutions to the digital media industry, will showcase its latest innovations in content-a...

20/07/2024

WRC Promoter Chooses Moments Lab to Drive Innovation in M...

Leading AI and video search company Moments Lab (formerly Newsbridge), is pleased to announce its partnership with WRC Promoter GmbH, the organization responsib...

20/07/2024

LiveU and Pente Networks Democratize Private Cellular Net...

The race to the US Presidential Election continues to heat up further this summer with the recent Republican National Convention this week in Wisconsin and the ...

20/07/2024

Cinegy Brings Cloud Monetization and Delivery and AI-powe...

Launch of new Air playout, automation and delivery platform bundles, Air Pack and Air Infinity; new AI-powered automatic subtitling; first IBC showing of Cinegy...

20/07/2024

Cine Gear Expo Names 2024 Tech Award and Film Competition...

Nearly one-hundred companies submitted and presented their newest technologies for the Cine Gear Expo 2024 Technical Awards. Cine Gear's team of expert judg...

20/07/2024

Matthews Studio Equipment Debuts Patriot Wrench

Matthews Studio Equipment announces new Patriot Wrench, a breakthrough tool for on set efficiency. The versatile wrench features the seven most common, industry...

20/07/2024

ViewNexa powers AZA Group and BlaCon Medias content portf...

Bitcentral, the leading provider of professional media solutions for broadcast and digital video, today announces that AZA Group, a specialist in broadcast engi...

20/07/2024

Perifery Launches AI solution to Revitalize and Monetize...

Perifery , a division of DataCore, has today announced the launch of AI 2.0, a transformational AI-powered solution suite designed to enable media professional...

20/07/2024

CVP Showcases New Production Solutions at Summer Kit Fest

Happening on 24-25 July, the Summer Kit Fest is the perfect event to get hands-on with the newest kit from over 50 top brands CVP, one of Europe's leading...

20/07/2024

FOR-A Showcases Software-Defined IP Solutions and Hybrid...

FOR-A, a leading manufacturer of broadcast and production technology, will present its latest innovations in IP-based and hybrid workflows at IBC2024. The compa...

20/07/2024

Cine Gear Expo Names 2024 Tech Award & Film Competition Winners

Cine Gear Expo Names 2024 Tech Award & Film Competition Winners Brie Clayton July 19, 2024 0 Comments Nearly one-hundred companies submitted and prese...

19/07/2024

Sundance Institute Announces Finalists in RFP Process

Six cities selected to move forward in the next phase of the process...

19/07/2024

L3Harris launches PilotApp - redefining flight data intelligence for aviation safety and efficiency

L3Harris Commercial Aviation has launched PilotApp, designed to empower pilots w...

19/07/2024

L3Harris Secures Avionics Contract with Air India for Next-Generation Voice and Data Recorders

L3Harris Technologies announce a landmark agreement with Air India to become lea...

19/07/2024

Comcast to Offer 'Enhanced 4K Coverage of Paris Olympics on USA Network

PHILADELPHIA Comcast has unveiled new details of its plans for offering enhanced 4K from Xfinity and said that its first enhanced 4K feeds will be available as...

19/07/2024

Sky News UK Among Global Broadcasters Hit by IT Outage

Sky News UK and Sky Sports News have been taken off air by what's being described as the biggest IT outage of all time'....

19/07/2024

FCC Adopts R&O To Make Closed Captioning Settings Easy To Access

WASHINGTON, D.C. Watching television for those with hearing-impairments will become a bit easier following adoption July 18 of a Federal Communications Commissi...

19/07/2024

Perifery Launches AI+ 2.0 to Revitalize and Monetize Media Content

Perifery Launches AI 2.0 to Revitalize and Monetize Media Content Brie Clayton July 19, 2024 0 Comments Advanced AI Software Suite Enables Users to L...

19/07/2024

Comcast NBCU Will Provide Free Olympics Stream for Military Community

Comcast NBCUniversal said it is working with the Army & Air Force Exchange Service to provide military community members with free streaming of NBCU's cover...

19/07/2024

Tom Fenton, Dean of American Foreign Correspondents,' Has Died

Tom Fenton, former CBS News correspondent, died July 16 in Novato, California. He was 94....

19/07/2024

Local News Close-Up: Capital Gains in Albany, New York

The lawmakers who make up the state legislature in Albany, New York, headed home at the end of June, but there's still plenty going on in the Capital Region...

19/07/2024

Access Hollywood' Heads to Paris

Access Hollywood and Access Daily with Mario & Kit will start previewing the 2024 Paris Summer Olympics starting Monday, July 22 with a week of special programm...

19/07/2024

NBCU Transforms Rockefeller Center Into Paris-Themed Olympic Hub

NBCUniversal is turning Rockefeller Center, its base in midtown Manhattan, into what it calls a hub for Team USA fans during the Olympics. That includes Parisia...

19/07/2024

Anchor Tom Garris Jumps From WTAE Pittsburgh to WMUR Manchester (NH)

Tom Garris, weekend morning anchor and weekday reporter at WTAE Pittsburgh, is moving to WMUR in Manchester, New Hampshire. Both are part of Hearst Television a...

19/07/2024

Comcast to Offer Enhanced 4K Coverage of Paris Olympics on USA Network

PHILADELPHIA Comcast has unveiled new details of its plans for offering enhanced 4K from Xfinity and said that its first enhanced 4K feeds will be available as...

19/07/2024

A Smart Z-Finder Review: Transforming the small screen of a smartphone into a powerful tool

The Zacuto Smart Z-Finder is an innovative viewfinder designed for smartphone fi...

19/07/2024

Altice USA Launches $30 a Month 'Entertainment TV'

NEW YORK Altice USA's Optimum is launching Entertainment TV, a new internet TV package of 80 plus channels for $30 a month that is available exclusively on ...

19/07/2024

Calrec Celebrates Diamond Jubilee with New Audio Tech at IBC 2024

As it celebrates its diamond jubilee this year, Calrec has announced that it will be pushing the boundaries of audio broadcasting at IBC 2024 with a full range ...

19/07/2024

CBS Sports Inks Multi-Platform Rights Deal with English Football League

NEW YORK CBS Sports and the English Football League (EFL) have announced an exclusive, multi-year, multi-platform rights agreement that will see CBS Sports offe...

19/07/2024

Netflix Subs Hit 277.6M as Revenue and Profits Spike

LOS GATOS, Calif. Netflix posted very strong Q2 2024 financials, with global Netflix subs growing 16.5% to 277.65 million, revenue up 17% and operating income s...

19/07/2024

ALIBI Music Sets the Tone for True Crime with New Underscores and Drones

ALIBI Music Sets the Tone for True Crime with New Underscores and Drones Brie Clayton July 18, 2024 0 Comments These seven production music albums rat...

19/07/2024

Christmas is Coming to Ting Park and Guy Stadium This Weekend

Only a Few More Chances to see the Salamanders and Yard Gnomes This Year The Holly Springs Salamanders have another big weekend coming up at Ting Park! Join t...

19/07/2024

Paramount Advertising Launches Self-Service Platform for Smaller Businesses

Paramount Advertising said it launched its self-serve ad buying platform, designed to attract more ad dollars from small and mid-sized businesses and other mark...

19/07/2024

Brian Lesser Returns To GroupM as Global CEO

Brian Lesser was named Global CEO of GroupM, the big media buying company that is part of WPP....

19/07/2024

KBLR Las Vegas Names Katia Gutirrez News Anchor and Multimedia Journalist

Katia Guti rrez has been promoted to news anchor and multimedia journalist at KBLR Las Vegas, focused on Noticiero Telemundo at the station. She starts in the n...

19/07/2024

Season 2 of Frasier' Sees Crane Back at Seattle Radio Station

Comedy Frasier returns for Season 2 on Thursday, September 19 on Paramount Plus. Two episodes are out that day, before they drop weekly on Thursdays....

19/07/2024

Total TV Ad Impressions Down 3.73% in First Half: iSpot Report

Total TV ad impressions on streaming and linear TV dipped 3.73% to 4.23 trillion in the first half of 2024, according to a new report from iSpot.tv....

19/07/2024

Insurer Progressive Had Most National and Local Ad Impressions, AdImpact Reports

Insurance company Progressive led all advertisers in national and local impressions in June and July, according to AdImpact's new TV intelligence platform, ...

19/07/2024

Francis Ford Coppola, Grateful Dead, Bonnie Raitt Get Kennedy Center Honors

Filmmaker Francis Ford Coppola, jam band the Grateful Dead, singer-songwriter Bonnie Raitt, jazz performer Arturo Sandoval and New York theater The Apollo will ...