![](//www.4rfv.com/images/no_image.jpg)
NVIDIA today announced optimizations across all its platforms to accelerate Meta Llama 3, the latest generation of the large language model (LLM).
The open model combined with NVIDIA accelerated computing equips developers, researchers and businesses to innovate responsibly across a wide variety of applications.
Trained on NVIDIA AI Meta engineers trained Llama 3 on computer clusters packing 24,576 NVIDIA H100 Tensor Core GPUs, linked with RoCE and NVIDIA Quantum-2 InfiniBand networks.
To further advance the state of the art in generative AI, Meta recently described plans to scale its infrastructure to 350,000 H100 GPUs.
Putting Llama 3 to Work Versions of Llama 3, accelerated on NVIDIA GPUs, are available today for use in the cloud, data center, edge and PC.
From a browser, developers can try Llama 3 at ai.nvidia.com. It's packaged as an NVIDIA NIM microservice with a standard application programming interface that can be deployed anywhere.
Businesses can fine-tune Llama 3 with their data using NVIDIA NeMo, an open-source framework for LLMs that's part of the secure, supported NVIDIA AI Enterprise platform. Custom models can be optimized for inference with NVIDIA TensorRT-LLM and deployed with NVIDIA Triton Inference Server.
Taking Llama 3 to Devices and PCs Llama 3 also runs on NVIDIA Jetson Orin for robotics and edge computing devices, creating interactive agents like those in the Jetson AI Lab.
What's more, NVIDIA RTX and GeForce RTX GPUs for workstations and PCs speed inference on Llama 3. These systems give developers a target of more than 100 million NVIDIA-accelerated systems worldwide.
Get Optimal Performance with Llama 3 Best practices in deploying an LLM for a chatbot involves a balance of low latency, good reading speed and optimal GPU use to reduce costs.
Such a service needs to deliver tokens - the rough equivalent of words to an LLM - at about twice a user's reading speed which is about 10 tokens/second.
Applying these metrics, a single NVIDIA H200 Tensor Core GPU generated about 3,000 tokens/second - enough to serve about 300 simultaneous users - in an initial test using the version of Llama 3 with 70 billion parameters.
That means a single NVIDIA HGX server with eight H200 GPUs could deliver 24,000 tokens/second, further optimizing costs by supporting more than 2,400 users at the same time.
For edge devices, the version of Llama 3 with eight billion parameters generated up to 40 tokens/second on Jetson AGX Orin and 15 tokens/second on Jetson Orin Nano.
Advancing Community Models An active open-source contributor, NVIDIA is committed to optimizing community software that helps users address their toughest challenges. Open-source models also promote AI transparency and let users broadly share work on AI safety and resilience.
Learn more about how NVIDIA's AI inference platform, including how NIM, TensorRT-LLM and Triton use state-of-the-art techniques such as low-rank adaptation to accelerate the latest LLMs.
Most recent headlines
04/08/2024
Dalet, a leading technology and service provider for media-rich organizations, is excited to announce Santiago Solanas as its new Chief Executive Officer (CEO)....
26/07/2024
by Ianeta Le'i
I honestly don't remember the first time I met Bryson Ch...
26/07/2024
(L-R) Louise (Karen Gillan) and Antonina (Margaret Sophie Stein) strike up an un...
26/07/2024
Last month, Spotify's annual Songs of Summer predictions made their return, ...
26/07/2024
Summer is peak season for some of our favorite sports and events. Whether you're rooting for the home team at the ballpark or cheering for your country as i...
26/07/2024
Nine talented young professionals completed their vocational training at SGL Carbon in Meitingen from the beginning of June to mid-July, achieving an overall gr...
26/07/2024
Orders1 of $5.2 billion; book-to-bill of 1.0x
Revenue of $5.3 billion, up 13%
...
26/07/2024
(Editor's note: TVTech reached out to several station groups to determine the impact of the CrowdStrike outage on operations. Many, including NextStar Media...
26/07/2024
NEWPORT BEACH, Calif. Bitcentral has integrated Pixalate's ad fraud protection, privacy and compliance analytics solution Post-Bid Analytics and Reporting i...
26/07/2024
Aldous Huxley's classic novel Brave New World is a science fiction thriller set in a murky future with a highly ordered society. While Huxley and other sc...
26/07/2024
Hollyland Adds Pyro 7 All-in-one Video Monitor, TX and RX to Pyro Series
Brie Clayton July 26, 2024
0 Comments
Seven-inch 2.4/5GHz wireless monitor wi...
26/07/2024
Geddy Lee Asks Are Bass Players Human Too? Graded Using DaVinci Resolve Studio
Brie Clayton July 26, 2024
0 Comments
Blackmagic today announced that t...
26/07/2024
Another reminder of the popularity of ad-supported connected TV comes in a survey by Premion and Advertiser Perceptions which found that 61% of the marketers th...
26/07/2024
Nielsen's Gracenote unit said it is selling its metadata about the content of television shows and movies for use in targeting connected-TV advertising camp...
26/07/2024
CBS premieres 13 Days in Ferguson, a special about the aftermath of a police shooting of a young Black man in 2014, on Friday, August 9. The special features Ce...
26/07/2024
Disney and Warner Bros. Discovery announced the launch of their long-awaited premium streaming bundle for the United States. For $16.99 per month, subscribers w...
26/07/2024
It provides multi-voltage output (14.4V, 28V, and 48V) through two XLR4 outputs and one XLR3, offering power to cameras, monitors, and lighting equipment
By Je...
26/07/2024
The lawsuit submitted by investor Scott Baker claims the deal would cost Paramount shareholders $1.65 billion
By Jenny Priestley
Published: July 26, 2024
...
26/07/2024
Florida-based Video Village Miami provides on-set video assist for local commercials and music videos. Frank Collins established the outfit in 2019 after a 20-y...
26/07/2024
QuickLink, the leading global provider of multi-camera video productions and remote contribution solutions, will present QuickLink StudioPro at IBC 2024 (Stand ...
26/07/2024
Relo Metrics (formerly GumGum Sports), the leading AI-powered sponsorship analytics platform for real-time data decisions, today launched an industry-leading in...
26/07/2024
Anton/Bauer, a leading provider of mobile power systems for the film and television industry, has announced the launch of the VCLX LI 1600. The latest evolution...
26/07/2024
Bitcentral, the leading provider of professional media solutions for broadcast and digital video, today announces that it has integrated Pixalate, the ad fraud ...
26/07/2024
Cobalt Iron Inc., a leading provider of SaaS-based enterprise data protection, today announced that Secure Automated Backup with Compass , a cloud backup-as-a-s...
26/07/2024
VisualOn IBC2024 Exhibitor Preview
Sept. 13-16
Amsterdam
Stand 5.A52
As video streaming consumption surges, service providers face significant challenges wi...
26/07/2024
Quanteec
IBC2024 Exhibitor Preview
Sept. 13-16
RAI Amsterdam
Stand 1.A05
Quanteec, a global provider of video delivery technology for broadcasters, will s...
26/07/2024
Witbe
IBC2024 Exhibitor Preview
Sept. 13-16
Amsterdam
Stand 5.F84
At IBC2024, Witbe will showcase the power of its automated testing and proactive monitori...
26/07/2024
WASHINGTON, D.C. The Federal Communications Commission has announced the launch of its new Mobile Speed Test app. The new app features an enhanced user interfac...
26/07/2024
LONDON In a notable landmark for the use of cloud-based services at the Olympics, Alibaba Cloud and Olympic Broadcasting Services (OBS), has today announced the...
26/07/2024
NEW YORK As part of a $2.5 billion media rights deal with the NBA announced on July 24, NBCUniversal and the NBA have also announced an 11-year agreement for ri...
26/07/2024
BEAVERTON, Ore. Versatile Video Coding (VVC) promises to play a pivotal role in driving advancements in video compression technology, with organizations across ...
26/07/2024
COLUMBUS, Ohio The political news channel PurpleTV has expanded it reach with news that it has launched on Channel 25.1 in Columbus, Ohio on TV Station WXOH, wh...
26/07/2024
NEW YORK As part of a push into the contextual advertising business, Nielsen's Gracenote has announced that it is developing new contextual categories to gi...
26/07/2024
Puget Systems Returns to SIGGRAPH 2024 to Demonstrate its On-Premises Generative...
26/07/2024
Dreamtek Delivers Innovative A/V, Live Production and Post Workflows with Blackm...
26/07/2024
Dejero to bring even smarter' mobile internet connectivity solutions to IBC...
26/07/2024
The 2024 Paris Olympic Games are about to begin, and Capitol Broadcasting is rea...
26/07/2024
Berklee Alumni Nominated for 2024 Emmy Awards Twenty-one Berklee alumni received Emmy nominations this year for music composition, sound mixing, and sound editi...
26/07/2024
Jennifer Lyons, president and general manager of WBBM Chicago, is moving into a ...
26/07/2024
Kennan Oliphant has been named VP of news and station manager at WWJ Detroit. He will report to Jennifer Lyons, who has added a regional GM role that gives her ...
26/07/2024
Charlie Hustle & The Matter of Pete Rose, a four-part docuseries, debuts on HBO and Max July 24. The series details the life of the baseball superstar, includin...
26/07/2024
The 2024 Paris Olympic Games present a unique and exciting opportunity for local...
26/07/2024
Paris 2024: V varo Providing Live Media Transport Services for Leading Internati...
26/07/2024
Paris 2024: BBC Sport on Its Remote Production Workflow for the Summer Games By Heather McLean, SVG Europe
Friday, July 26, 2024 - 7:31 am
Print This Story...
26/07/2024
Paris 2024: BBC Sport Unveils the Olympic Storytelling and Breathtaking Scenery ...
26/07/2024
Paris 2024: Taking a Step Forwards with BBC Sport's Olympic Studio Combining...
26/07/2024
Paris 2024: Warner Bros. Discovery Sports Europe Provides a First Look at Its Ol...
26/07/2024
Live From Paris 2024: NBC Olympics Set for Historic Opening Ceremony Unilateral coverage builds on impressive OBS production as NBC dives deep into Team USA cov...
26/07/2024
Live From Paris 2024: SVG's Onsite Editorial Team Breaks Down Opening Ceremo...
26/07/2024
True-crime documentary, The Body Next Door to launch on Sky Documentaries and NO...