Sony Pixel Power calrec Sony

Wide Open: NVIDIA Accelerates Inference on Meta Llama 3

18/04/2024

NVIDIA today announced optimizations across all its platforms to accelerate Meta Llama 3, the latest generation of the large language model (LLM).

The open model combined with NVIDIA accelerated computing equips developers, researchers and businesses to innovate responsibly across a wide variety of applications.

Trained on NVIDIA AI Meta engineers trained Llama 3 on computer clusters packing 24,576 NVIDIA H100 Tensor Core GPUs, linked with RoCE and NVIDIA Quantum-2 InfiniBand networks.

To further advance the state of the art in generative AI, Meta recently described plans to scale its infrastructure to 350,000 H100 GPUs.

Putting Llama 3 to Work Versions of Llama 3, accelerated on NVIDIA GPUs, are available today for use in the cloud, data center, edge and PC.

From a browser, developers can try Llama 3 at ai.nvidia.com. It's packaged as an NVIDIA NIM microservice with a standard application programming interface that can be deployed anywhere.

Businesses can fine-tune Llama 3 with their data using NVIDIA NeMo, an open-source framework for LLMs that's part of the secure, supported NVIDIA AI Enterprise platform. Custom models can be optimized for inference with NVIDIA TensorRT-LLM and deployed with NVIDIA Triton Inference Server.

Taking Llama 3 to Devices and PCs Llama 3 also runs on NVIDIA Jetson Orin for robotics and edge computing devices, creating interactive agents like those in the Jetson AI Lab.

What's more, NVIDIA RTX and GeForce RTX GPUs for workstations and PCs speed inference on Llama 3. These systems give developers a target of more than 100 million NVIDIA-accelerated systems worldwide.

Get Optimal Performance with Llama 3 Best practices in deploying an LLM for a chatbot involves a balance of low latency, good reading speed and optimal GPU use to reduce costs.

Such a service needs to deliver tokens - the rough equivalent of words to an LLM - at about twice a user's reading speed which is about 10 tokens/second.

Applying these metrics, a single NVIDIA H200 Tensor Core GPU generated about 3,000 tokens/second - enough to serve about 300 simultaneous users - in an initial test using the version of Llama 3 with 70 billion parameters.

That means a single NVIDIA HGX server with eight H200 GPUs could deliver 24,000 tokens/second, further optimizing costs by supporting more than 2,400 users at the same time.

For edge devices, the version of Llama 3 with eight billion parameters generated up to 40 tokens/second on Jetson AGX Orin and 15 tokens/second on Jetson Orin Nano.

Advancing Community Models An active open-source contributor, NVIDIA is committed to optimizing community software that helps users address their toughest challenges. Open-source models also promote AI transparency and let users broadly share work on AI safety and resilience.

Learn more about how NVIDIA's AI inference platform, including how NIM, TensorRT-LLM and Triton use state-of-the-art techniques such as low-rank adaptation to accelerate the latest LLMs.
LINK: https://blogs.nvidia.com/blog/meta-llama3-inference-acceleration/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

09/10/2026

Dolby Expands Dolby OptiView Platform with New Capabilities at IBC 2026

September 10 2026, 06:00 (PDT) Dolby Expands Dolby OptiView Platform with New Capabilities at IBC 2026 New Sports Intelligence helps providers better unders...

07/10/2026

Dalet Flex LTS Delivers Smarter Media Operations from Ingest to Distribution

Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...

30/09/2026

SVG Students To Watch: Jacquelynn Dupree, Clark Atlanta University

Drawn to the many moving pieces of a live broadcast, this Los Angeles native is building a path in production management and technical operations...

30/09/2026

NHL on TNT Opens New Season in 1080p HDR With Stanley Cup Final on the Horizon

TNT Sports goes big for its season opener in Philadelphia, and REMI workflows at Techwood will again anchor much of regular-season production...

30/09/2026

Katlyn Alexander Named CEO of Qvest North America

Qvest has appointed Katlyn Alexander as CEO of Qvest in North America, effective August 16, 2026. Alexander, who previously served as COO of Qvest US, succeeds ...

30/09/2026

AAPB Wraps 2026 Season with Record Viewership and Expanded Broadcast Footprint

The American Association of Professional Baseball (AAPB) has wrapped its 2026 season, with leaguewide attendance surpassing 1.4 million for the fifth consecutiv...

30/09/2026

Veritone Partners with ATP Media to License Tennis Archive in North America

Veritone has announced a multi-year content licensing agreement with ATP Media, the broadcast production and distribution arm of the ATP Tour, making ATP's ...

30/09/2026

Netflix To Stream WWE Exclusively in Japan Beginning October 1

Netflix has announced it will become the exclusive home for WWE programming in Japan beginning October 1, 2026. The lineup includes Raw, SmackDown, NXT, Premium...

30/09/2026

SES and Sky Extend Satellite Capacity Agreement for UK and Ireland

SES and Sky have announced a new multi-year agreement for direct-to-home satellite services across the United Kingdom and Republic of Ireland, running into the ...

30/09/2026

Big Blue Marble Adds Live Multiview Channels to Cloud Video Kit via AWS Elemental Dynamic Multiview

Big Blue Marble has announced the addition of live multiview channels to its Clo...

30/09/2026

NHRA Announces Multi-Year Media Rights Deals With FOX Sports, The CW, and FloSports

The NHRA has announced multi-year media rights agreements with FOX Sports, The C...

30/09/2026

FOX 5 Atlanta to Broadcast Nashville Predators Games in 2026-27 Season

FOX 5 Atlanta (WAGA-TV) will carry Nashville Predators games during the 2026-27 season as part of Scripps Sports' regional broadcast network for the team. G...

30/09/2026

SVG Sit-Down: Brahma AIs Patrio Cummins on the Possibilities of AI (and the Misconceptions)

AI was a big buzz on the IBC show floor but there weren't a ton of companies...

30/09/2026

PGA TOUR Names Ryan Holcomb Senior Vice President and Head of Original Content

The PGA TOUR has announced the appointment ofRyan Holcomb as Senior Vice President and Head of Original Content, according to The Hollywood Reporter. Holcomb wi...

30/09/2026

ESPN Drops the Puck on Enhanced Audio for the New NHL Season

The sound will be more immersive, with improved design and more mikes on the ice and the players The National Hockey League's season opener on Tuesday nigh...

30/09/2026

TNT Sports' Chris Brown, Lee Mabry on Beginning 2026-27 NHL Season With an Onsite Effort in Philly

TNT Sports is dropping the puck on the 2026-27 NHL season in a big way in Philad...

30/09/2026

International Swimming League Names Gravity Media as Production Partner

The International Swimming League ( ISL ) today announced the appointment of Gravity Media, global leader in live production and content and media services and ...

30/09/2026

Give Me the Backstory: Get to Know Amir Bar-Lev, the Director of The Last First: Winter K2

By Ramona Flume One of the most exciting things about the Sundance Film Festiva...

30/09/2026

Sync Audio release MegaMorph

Add an envelope generator to any plug-in Sync Audio's latest creation is a handy utility plug-in that can be used to combine and modulate other effects ...

30/09/2026

Strymon launch the La Brea

All-analogue tape-style drive joins Series A Strymon's reputation was built on their digital knowhow, so many were surprised (and intrigued) when they a...

30/09/2026

Sonuscore The Pulse: Synth Wave & Epic Atmospheres

Two new expansion packs released Sonuscore have just introduced two new expansion packs for The Pulse, its production engine instrument for modern cinematic...

30/09/2026

Clear-Com's Arcadia Central Station Brings Professional Production Communications to...

eds3_5_jq(document).ready(function($) { $(#eds_sliderM519).chameleonSlider_2_1({...

30/09/2026

AWARN Alliance Urges FCC to Let Broadcasters Tap Full 3.0 Capabilities

Share Copy link Facebook X Linkedin Bluesky Email...

30/09/2026

Senator Asks If All-GOP FCC Would Violate Federal Law

Share Copy link Facebook X Linkedin Bluesky Email...

30/09/2026

Marshall PTZ Cameras Bring Legendary Progressive Rock Exp...

The An Evening with Emerson, Lake and Palmer tour brings one of rock's most iconic bands back to the stage. These performances seamlessly blend live foota...

30/09/2026

Schibsted Media Chooses DHD RX2 and TX2 Mixers for Remote...

Schibsted Media, one of Norways largest and most influential digital media publishers and content producers, has invested in mixers and digital signal processor...

30/09/2026

Big Blue Marble adds live multiview channels to its video...

Big Blue Marble announces the addition of live multiview channels to Cloud Video Kit, its broadcast-grade video processing platform. The feature is powered by A...

30/09/2026

Telekom Slovenije brings local advertising to internation...

Leading video software provider Synamedia today announced that Telekom Slovenije, Slovenia's largest provider of telecommunications and TV services, is open...

30/09/2026

PODCAST EXPO 2026 Recorded Using Blackmagic Design Workflow

PODCAST EXPO 2026 Recorded Using Blackmagic Design Workflow Brie Clayton September 29, 2026 0 Comments Conference featuring 24 sessions over two days ...

30/09/2026

IBC2026 Show Report

IBC2026 Show Report David Kirk September 29, 2026 0 Comments The International Broadcasting Convention has long been Europe's equivalent of the US...

30/09/2026

SGO Unveils Workflows Slate To Enable Automated Burn-ins, Slates

Share Copy link Facebook X Linkedin Bluesky Email...

30/09/2026

Public Advocacy Group Urges Broadcasters to Stop Airing Trump Taxpayer Funded Ad

Share Copy link Facebook X Linkedin Bluesky Email...

30/09/2026

Qvest Promotes Katlyn Alexander to CEO, North America

Share Copy link Facebook X Linkedin Bluesky Email...

30/09/2026

Mavis Camera 8-2 adds iPhone 18 Pro Support with Variable...

Mavis has launched Mavis Camera 8.2, the latest release of its professional camera app for the iPhone and iPad. This update brings support for iPhone 18 Pro and...

30/09/2026

Tribeca Festival Announces 2027 Dates And Calls For Submissions

September 30th, 2026 TRIBECA FESTIVAL ANNOUNCES 2027 DATES AND CALLS FOR SUBMISSIONS Following a Record Box Office Year, the 26th Edition, Presented by OKX,...

30/09/2026

Harmonic Launches Recon Product Family Developed in Partnership with Comcast

New Solution Expands Harmonic's Network Intelligence Portfolio and Brings Comcast-Developed Fiber Monitoring Technology to Broadband Operators, Providing Re...

30/09/2026

A Look Back at (My) 524 Months with Broadcast Cameras

Thank you to Film-TV Video for the original article - you can view it here It was early 1983. I had just turned 21 when I joined Hitachi Denshi's European...

30/09/2026

Claire Brock and Sophie Morris discover how consumers can shop smarter in new RT documentary series

How to Shop Well Now starts Monday at 9.35pm on RT One and RT Player On the...

30/09/2026

From Training to Production, NVIDIA and CoreWeave Close the Loop on Agentic AI

Building on nearly a decade of co-engineering, CoreWeave has built NVIDIA compute, networking and software into a cloud purpose-built for AI that's still re...

29/09/2026

Oak View Group Names Cisco as Official Technology Partner

Oak View Group (OVG) has announced Cisco as an Official Technology Partner, with plans to standardize infrastructure across its venue network on Cisco solutions...

29/09/2026

SportsNet Pittsburgh Wins Three Mid-Atlantic Emmy Awards

SportsNet Pittsburgh (SNP) won three honors at the 2026 Mid-Atlantic Emmy Awards. Program awards went to Bucs BP in the Sports Program Live category (produced ...

29/09/2026

CBC, PWHL Revive Hockey Night in Canada' on Saturdays

The Professional Women's Hockey League has announced home openers for all 12 teams for the 2026-27 season, along with a broadcast partnership with CBC under...

29/09/2026

Manifold Technologies Recognized as MXL Contributor Following IBC Innovation Award

Manifold technologies has been recognized as a contributor to the Media eXchange...

29/09/2026

Fubo Launches My Local Sports RSN Add-On Package

FuboTV has announced My Local Sports, allowing subscribers in select markets to add their local regional sports network (RSN) as an add-on to a Fubo Sports subs...

29/09/2026

ANI Deploys Grass Valley AMPP and Framelight X for News Operations

Asian News International (ANI), a wire agency and video news provider based in India, has deployed Grass Valley's AMPP Media Operating System and Framelight...

29/09/2026

SABC Deploys Ikegami UHK-X600 Cameras for IP-Based Production

South African Broadcasting Corporation (SABC) has deployed Ikegami UHK-X600 cameras as part of an upgrade to IP-based production infrastructure. The deployment ...

29/09/2026

Busy in Bristol: ESPN Headquarters Come Alive With New Studio W, Remote-Production Workflows

REMI, REMCO models will drive the bulk of the broadcaster's NHL productions ...

29/09/2026

NASCAR Productions' Keith D'Alessandro on the Continued Evolution of Its Centralized Production Hub

Three years in, the facility is supporting more centralized race production, com...

29/09/2026

NBC Sports Brings Robust Sunday Night Baseball Production Playbook to MLB Wild Card Round

NBC, Peacock, and NBSN are carrying all four Wild Card series with a production ...