Sony Pixel Power calrec Sony

TOPS of the Class: Decoding AI Performance on RTX AI PCs and Workstations

12/06/2024

Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, software, tools and accelerations for RTX PC users.

The era of the AI PC is here, and it's powered by NVIDIA RTX and GeForce RTX technologies. With it comes a new way to evaluate performance for AI-accelerated tasks, and a new language that can be daunting to decipher when choosing between the desktops and laptops available.

While PC gamers understand frames per second (FPS) and similar stats, measuring AI performance requires new metrics.

Coming Out on TOPS The first baseline is TOPS, or trillions of operations per second. Trillions is the important word here - the processing numbers behind generative AI tasks are absolutely massive. Think of TOPS as a raw performance metric, similar to an engine's horsepower rating. More is better.

Compare, for example, the recently announced Copilot+ PC lineup by Microsoft, which includes neural processing units (NPUs) able to perform upwards of 40 TOPS. Performing 40 TOPS is sufficient for some light AI-assisted tasks, like asking a local chatbot where yesterday's notes are.

But many generative AI tasks are more demanding. NVIDIA RTX and GeForce RTX GPUs deliver unprecedented performance across all generative tasks - the GeForce RTX 4090 GPU offers more than 1,300 TOPS. This is the kind of horsepower needed to handle AI-assisted digital content creation, AI super resolution in PC gaming, generating images from text or video, querying local large language models (LLMs) and more.

Insert Tokens to Play TOPS is only the beginning of the story. LLM performance is measured in the number of tokens generated by the model.

Tokens are the output of the LLM. A token can be a word in a sentence, or even a smaller fragment like punctuation or whitespace. Performance for AI-accelerated tasks can be measured in tokens per second.

Another important factor is batch size, or the number of inputs processed simultaneously in a single inference pass. As an LLM will sit at the core of many modern AI systems, the ability to handle multiple inputs (e.g. from a single application or across multiple applications) will be a key differentiator. While larger batch sizes improve performance for concurrent inputs, they also require more memory, especially when combined with larger models.

The more you batch, the more (time) you save. RTX GPUs are exceptionally well-suited for LLMs due to their large amounts of dedicated video random access memory (VRAM), Tensor Cores and TensorRT-LLM software.

GeForce RTX GPUs offer up to 24GB of high-speed VRAM, and NVIDIA RTX GPUs up to 48GB, which can handle larger models and enable higher batch sizes. RTX GPUs also take advantage of Tensor Cores - dedicated AI accelerators that dramatically speed up the computationally intensive operations required for deep learning and generative AI models. That maximum performance is easily accessed when an application uses the NVIDIA TensorRT software development kit (SDK), which unlocks the highest-performance generative AI on the more than 100 million Windows PCs and workstations powered by RTX GPUs.

The combination of memory, dedicated AI accelerators and optimized software gives RTX GPUs massive throughput gains, especially as batch sizes increase.

Text-to-Image, Faster Than Ever Measuring image generation speed is another way to evaluate performance. One of the most straightforward ways uses Stable Diffusion, a popular image-based AI model that allows users to easily convert text descriptions into complex visual representations.

With Stable Diffusion, users can quickly create and refine images from text prompts to achieve their desired output. When using an RTX GPU, these results can be generated faster than processing the AI model on a CPU or NPU.

That performance is even higher when using the TensorRT extension for the popular Automatic1111 interface. RTX users can generate images from prompts up to 2x faster with the SDXL Base checkpoint - significantly streamlining Stable Diffusion workflows.

ComfyUI, another popular Stable Diffusion user interface, added TensorRT acceleration last week. RTX users can now generate images from prompts up to 60% faster, and can even convert these images to videos using Stable Video Diffuson up to 70% faster with TensorRT.

TensorRT acceleration can be put to the test in the new UL Procyon AI Image Generation benchmark, which delivers speedups of 50% on a GeForce RTX 4080 SUPER GPU compared with the fastest non-TensorRT implementation.

TensorRT acceleration will soon be released for Stable Diffusion 3 - Stability AI's new, highly anticipated text-to-image model - boosting performance by 50%. Plus, the new TensorRT-Model Optimizer enables accelerating performance even further. This results in a 70% speedup compared with the non-TensorRT implementation, along with a 50% reduction in memory consumption.

Of course, seeing is believing - the true test is in the real-world use case of iterating on an original prompt. Users can refine image generation by tweaking prompts significantly faster on RTX GPUs, taking seconds per iteration compared with minutes on a Macbook Pro M3 Max. Plus, users get both speed and security with everything remaining private when running locally on an RTX-powered PC or workstation.

The Results Are in and Open Sourced But don't just take our word for it. The team of AI researchers and engineers behind the open-source Jan.ai recently integrated TensorRT-LLM into its local chatbot app, then tested these optimizations for themselves.

Source: Jan.ai The researchers tested its implementation of TensorRT-LLM against the open-source llama.cpp inference engine across a variety of GPUs and CPUs used by the community. They found that TensorRT is 30-70% faster than llam
LINK: https://blogs.nvidia.com/blog/ai-decoded-tops/...
See more stories from nvidia

Most recent headlines

04/08/2024

Dalet Appoints Santiago Solanas as CEO to Lead Next Era of Growth and Innovation

Dalet, a leading technology and service provider for media-rich organizations, is excited to announce Santiago Solanas as its new Chief Executive Officer (CEO)....

14/07/2024

Gray Television Stations To Air Seattle Kraken Games in Alaska

Gray Television has made a deal with Tegna and the Seattle Kraken to air Kraken National Hockey League games in two Alaska markets....

14/07/2024

Chili's Sponsors Summer Movies on Vizio's WatchFree Plus

Chili's has signed up as the exclusive presenting sponsor of a Hot Summer Movies collection of films available on demand on Vizio's WatchFree Plus str...

14/07/2024

Live From MLB All-Star 2024: With Full Slate of Events in Arlington, MLB Network Settles In for Multi-Day MLB Draft at Cowtown Coliseum

Live From MLB All-Star 2024: With Full Slate of Events in Arlington, MLB Network...

14/07/2024

Live From MLB All-Star 2024: League's Global Events Team Runs to the Rodeo for MLB Draft in Fort Worth

Live From MLB All-Star 2024: League's Global Events Team Runs to the Rodeo f...

14/07/2024

Spacecom Secures $4.2 Million Satellite Communication Services Deal

BACK TO PRESS RELEASES Spacecom Secures $4.2 Million Satellite Communication Services Deal 14.07.2024 Spacecom Secures $4.2 Million Satellite Communication ...

13/07/2024

Skywalkers: A Love Story Takes Love to Unimaginable Heights

Jeff Zimbalist, Ivan Beerkus, Angela Nikolau, and Maria Bukhonina at the premiere of Skywalkers: A Love Story at The Ray Theatre in Park City, Utah, on Januar...

13/07/2024

Invitation for South African filmmakers to submit films for the 97th Annual Academy Awards (Oscars) International Feature Film category.

The National Film and Video Foundation (NFVF), an agency of the Department of Sp...

13/07/2024

VideoAmp Integrates Snap Inc. Ad Inventory

NEW YORK VideoAmp has announced a wide-ranging collaboration with Snap Inc. that will see the social technology platform integrate its first-party data, video, ...

13/07/2024

RAI Moves up Launch of DVB-T2 HEVC Broadcasts

The Italian public broadcaster RAI will start broadcasting on Multiplex B in the DVB-T2 HEVC standard on August 28, according to Marco Rossignoli, president of ...

13/07/2024

Mile-High AI: NVIDIA Research to Present Advancements in Simulation and Gen AI at SIGGRAPH

Mile-High AI: NVIDIA Research to Present Advancements in Simulation and Gen AI a...

13/07/2024

Future Today Adds Studio Content to Fawesome

Future Today said it has signed licencing deals with Sony Pictures Entertainment, Samuel Goldwyn Films and Gravitas Ventures that will add fresh content to its ...

13/07/2024

VideoAmp To Integrate Data, Inventory From Snap Into Ad-Planning Tools

VideoAmp said it made a deal enabling the measurement company to integrate Snap's first-party data, video and augmented-reality ad inventory into VideoAmp&#...

13/07/2024

ABC Renews Bachelor in Paradise' for Season 10

ABC has ordered a tenth season of Bachelor in Paradise. The season will air in 2025 on the network and stream the next day on Hulu....

13/07/2024

Bill Belichick Joins Inside the NFL' for Season 2 on The CW

Bill Belichick, the legendary head coach who led the New England Patriots to eight Super Bowl wins, is joining Inside the NFL, which starts its second season on...

13/07/2024

Allen Media Group's HBCU Go Set To Kick Off College Football Season

Allen Media Group's HBCU Go said it will televise 26 college football games played by Historically Black Colleges and Universities....

13/07/2024

Backed By Top Producers, Exploding Kittens' Card Game Adapted to Netflix Series

Exploding Kittens, an animated series based on the card game, debuts on Netflix ...

13/07/2024

ABC Owned Stations, Ava DuVernay Partner on Our America' Special

ABC Owned Television Stations premieres Our America: Hidden Stories with Ava DuVernay starting Sunday, July 14. The one-hour special sheds light on the interco...

13/07/2024

Shania Twain To Host People's Choice Country Awards' on NBC

Shania Twain will host the People's Choice Country Awards, which happens Thursday, September 26 on NBC and Peacock. The two-hour special will be live at the...

12/07/2024

My byte-size week at Calrec

My Byte-size week at Calrec By Erin Crowther My name is Erin, and I am a student at Ryburn Sixth Form. This week I completed my work experience at Calrec Audi...

12/07/2024

Fox Sports to Create Virtual Reality Experience for Copa Amrica Final

CAMPBELL, Calif. Fox Sports and YBVR have announced that they will be creating an immersive virtual reality experience for the Copa Am rica Final on Meta Quest,...

12/07/2024

Gray TV Stations in Alaska to Air Seattle Kraken Games

TYSONS, Va. Tegna Inc., the Seattle Kraken and Gray Media have announced that Gray's KAUU in Anchorage and KYEX in Juneau will expand the Kraken broadcast n...

12/07/2024

Hallmark To Launch New Subscription Streaming Service

PASADENA, Calif. Hallmark has unveiled Hallmark+, a subscription streaming television service that will combine the company's brand of lifestyle experience ...

12/07/2024

Going the extra Milo: MRMC unveils new rig

The Super Milo can achieve speeds of up to 3 m/s on track and 5 m/s with combined camera movement By Matthew Corrigan Published: July 12, 2024 The Super M...

12/07/2024

TGI Sport acquires virtual advertising company Supponor

The deal is scheduled to close mid-summer, financial details have not been revealed By Jenny Priestley Published: July 12, 2024 The deal is scheduled to c...

12/07/2024

What the Clearhaven Deal Means for Zixi's Future

In recent years, as TV companies and producers pushed to launch streaming services and deploy IP infrastructures, Zixi has been at the center of that tectonic t...

12/07/2024

Ateme To Focus on Maximizing Output, Minimizing Costs At IBC 2024

PARIS Ateme will feature technologies pushing the boundaries of transformation, monetization and experience at IBC 2024, Sept. 13-16, at the RAI Convention Cent...

12/07/2024

DNEG Group Completes Prime Focus Technologies Acquisition

The DNEG Group has completed acquisition of Prime Focus Technologies....

12/07/2024

Wimbledon and Hawk-Eye Innovations Extend Tech Agreement

LONDON The All England Lawn Tennis Club and Sony's Hawk-Eye Innovations have announced a new multi-year agreement that will see Hawk-Eye continue to provide...

12/07/2024

Vision2 Marketing to Rep FOR-A as Part of Acceleration in...

As part of a more expansive push into the U.S. market, FOR-A announces its partnership with Vision2 Marketing, a manufacturers' rep for pro audio, video, li...

12/07/2024

KTRK Optimizes Look of Live Streams and Broadcasts with S...

Houston broadcaster KTRK-TV has been on the scene for 70 years and is known for community outreach as well as investigative journalism. Journalist alumni includ...

12/07/2024

QuickLink Expands its US Presence With Two New Seasoned T...

QuickLink, the leading global provider of multi-camera video productions and remote contributions, announces a significant growth strategy with two high-level p...

12/07/2024

Imagine Communications Names Don Durand SVP Global Sales...

Imagine Communications has appointed international media industry executive Don Durand to the position of senior vice president of global sales for its ad tech ...

12/07/2024

BH Telecom selects Appear to deliver Moja TV service to s...

Appear, the global leader in live production technology, has been selected by BH Telecom to enhance the live channel processing software for its Moja TV service...

12/07/2024

Ikegami Announces New Additions to its Range of Broadcast...

Ikegami announces two new additions to its range of broadcast quality picture monitors: the 24-inch HLM-2460WA and 18.5 inch HLM-1860WR. These will make their E...

12/07/2024

Vizrt focuses on BroadcastAV market with PTZ3 PLUS camera...

Vizrt, the leader in real-time graphics and live production solutions for content creators, brings the BroadcastAV market sharply into focus with today's la...

12/07/2024

Czech TV Live Streams Prague Marathon with Resilient Deje...

Local production company Livecast counts on Dejero Smart Blending Technology for connectivity on the move Dejero recently provided Czech production company Li...

12/07/2024

LTN playout solution powers European Football Championshi...

LTN, the industry leader in transformative media technology and video transport solutions, announces the successful delivery of live and pre-recorded coverage t...

12/07/2024

Ateme to Showcase Cutting-Edge Solutions to Maximise Outp...

Ateme's solutions deliver unprecedented video experiences and content monetisation while optimizsng workflows through hybrid on-premises and cloud systems. ...

12/07/2024

Lightware Announces New Leadership in the Enterprise Team

Lightware Visual Engerineering, a leading manufacturer of connectivity solutions for the professional integrated systems market and a pioneer in signal manageme...

12/07/2024

Anton Bauer Salt-E Dog Wins Innovation Impact Award at Ro...

Anton/Bauer has announced that its pioneering sustainable power product, Salt-E Dog, has been awarded the prestigious Innovation Impact Award at the Royal Telev...

12/07/2024

Teatro TV Browser by iWedia Surpasses 40 million Devices...

iWedia, a leading innovator in software solutions for connected TV devices, is thrilled to announce a significant milestone for its flagship product, the Teatro...

12/07/2024

Tim Belcher joins Light Iron as new Managing Director

Light Iron, the post-production creative-services division of Panavision, has named industry veteran Tim Belcher the company's new Managing Director. Belche...

12/07/2024

Screen Australia and ABC announce new comedy series Optics

12 07 2024 - Media release Screen Australia and ABC announce new comedy series Optics Optics. Photo credit: Joel Pratley ABC and Screen Australia have annou...

12/07/2024

COW Job Listing: Need a Drone Shot in Boardman, Oregon

COW Job Listing: Need a Drone Shot in Boardman, Oregon Brie Clayton July 11, 2024 0 Comments Need a Drone shot July 9, 2024COW Job Listing: Freelanc...

12/07/2024

American Underground Celebrates Pride & Juneteenth with Free, Public Learning Events

American Underground, Capitol Broadcasting's co-working community division, ...

12/07/2024

5 Session-Saving Tips for Studio One

By Craig Anderton I admit it: the following tips are based on personally embarrassing experiences. I like to work fast to keep the creative juices flowing, but...

12/07/2024

Fox Sports Heads to Berlin for Euro Final; Wraps Up Massive Month of International Soccer Coverage

Fox Sports Heads to Berlin for Euro Final; Wraps Up Massive Month of Internation...

12/07/2024

Euro 2024: UEFA Gets Set for Spain v England Final

Euro 2024: UEFA gets set for Spain v England final By George Bevir Friday, July 12, 2024 - 11:19 Print This Story When England take on Spain on Sunday in ...

12/07/2024

Euro 2024: Inside UEFA's International Broadcast Centre

Euro 2024: Inside UEFA's International Broadcast Centre By George Bevir Friday, July 12, 2024 - 08:00 Print This Story The IBC in Leipzig, Germany (Ph...