
Businesses across every industry are rolling out AI services this year. For Microsoft, Oracle, Perplexity, Snap and hundreds of other leading companies, using the NVIDIA AI inference platform - a full stack comprising world-class silicon, systems and software - is the key to delivering high-throughput and low-latency inference and enabling great user experiences while lowering cost.
NVIDIA's advancements in inference software optimization and the NVIDIA Hopper platform are helping industries serve the latest generative AI models, delivering excellent user experiences while optimizing total cost of ownership. The Hopper platform also helps deliver up to 15x more energy efficiency for inference workloads compared to previous generations.
AI inference is notoriously difficult, as it requires many steps to strike the right balance between throughput and user experience.
But the underlying goal is simple: generate more tokens at a lower cost. Tokens represent words in a large language model (LLM) system - and with AI inference services typically charging for every million tokens generated, this goal offers the most visible return on AI investments and energy used per task.
Full-stack software optimization offers the key to improving AI inference performance and achieving this goal.
Cost-Effective User Throughput Businesses are often challenged with balancing the performance and costs of inference workloads. While some customers or use cases may work with an out-of-the-box or hosted model, others may require customization. NVIDIA technologies simplify model deployment while optimizing cost and performance for AI inference workloads. In addition, customers can experience flexibility and customizability with the models they choose to deploy.
NVIDIA NIM microservices, NVIDIA Triton Inference Server and the NVIDIA TensorRT library are among the inference solutions NVIDIA offers to suit users' needs:
NVIDIA NIM inference microservices are prepackaged and performance-optimized for rapidly deploying AI foundation models on any infrastructure - cloud, data centers, edge or workstations.
NVIDIA Triton Inference Server, one of the company's most popular open-source projects, allows users to package and serve any model regardless of the AI framework it was trained on.
NVIDIA TensorRT is a high-performance deep learning inference library that includes runtime and model optimizations to deliver low-latency and high-throughput inference for production applications.
Available in all major cloud marketplaces, the NVIDIA AI Enterprise software platform includes all these solutions and provides enterprise-grade support, stability, manageability and security.
With the framework-agnostic NVIDIA AI inference platform, companies save on productivity, development, and infrastructure and setup costs. Using NVIDIA technologies can also boost business revenue by helping companies avoid downtime and fraudulent transactions, increase e-commerce shopping conversion rates and generate new, AI-powered revenue streams.
Cloud-Based LLM Inference To ease LLM deployment, NVIDIA has collaborated closely with every major cloud service provider to ensure that the NVIDIA inference platform can be seamlessly deployed in the cloud with minimal or no code required. NVIDIA NIM is integrated with cloud-native services such as:
Amazon SageMaker AI, Amazon Bedrock Marketplace, Amazon Elastic Kubernetes Service
Google Cloud's Vertex AI, Google Kubernetes Engine
Microsoft Azure AI Foundry coming soon, Azure Kubernetes Service
Oracle Cloud Infrastructure's data science tools, Oracle Cloud Infrastructure Kubernetes Engine
Plus, for customized inference deployments, NVIDIA Triton Inference Server is deeply integrated into all major cloud service providers.
For example, using the OCI Data Science platform, deploying NVIDIA Triton is as simple as turning on a switch in the command line arguments during model deployment, which instantly launches an NVIDIA Triton inference endpoint.
Similarly, with Azure Machine Learning, users can deploy NVIDIA Triton either with no-code deployment through the Azure Machine Learning Studio or full-code deployment with Azure Machine Learning CLI. AWS provides one-click deployment for NVIDIA NIM from SageMaker Marketplace and Google Cloud provides a one-click deployment option on Google Kubernetes Engine (GKE). Google Cloud provides a one-click deployment option on Google Kubernetes Engine, while AWS offers NVIDIA Triton on its AWS Deep Learning containers.
The NVIDIA AI inference platform also uses popular communication methods for delivering AI predictions, automatically adjusting to accommodate the growing and changing needs of users within a cloud-based infrastructure.
From accelerating LLMs to enhancing creative workflows and transforming agreement management, NVIDIA's AI inference platform is driving real-world impact across industries. Learn how collaboration and innovation are enabling the organizations below to achieve new levels of efficiency and scalability.
Serving 400 Million Search Queries Monthly With Perplexity AI Perplexity AI, an AI-powered search engine, handles over 435 million monthly queries. Each query represents multiple AI inference requests. To meet this demand, the Perplexity AI team turned to NVIDIA H100 GPUs, Triton Inference Server and TensorRT-LLM.
Supporting over 20 AI models, including Llama 3 variations like 8B and 70B, Perplexity processes diverse tasks such as search, summarization and question-answering. By using smaller classifier models to route tasks to GPU pods, managed by NVIDIA Triton, the company delivers cost-efficient, responsive service under strict service level agreements.
Through model parallelism, which splits LLMs across GPUs, Perplexity achieved a threefold cost reduction while maintaining low latency and high accurac
North America Stories
04/07/2025
By Lucy Spicer
Editor's note: We are deeply saddened by the passing of Robe...
04/07/2025
Introducing CVP Auctions: Trusted Tech, At Auction-Only Prices
Brie Clayton July 4, 2025
0 Comments
A New Destination for Film and Broadcast Gear at U...
04/07/2025
Unveiling China's Ecological Wonders with URSA Mini Pro 12K
Brie Clayton July 4, 2025
0 Comments
Nature Documentaries Freeze Epic Momentsin Extrem...
04/07/2025
Create Any Planet in After Effects
Graham Quince July 4, 2025
0 Comments
The Advanced 3D renderer in After Effects makes it really easy to import 3D o...
04/07/2025
Custom Gradient Colorize for Resolve Fusion
Simon Ubsdell July 4, 2025
0 Comments
Digging into the theory of the gradient colorize process to build ou...
04/07/2025
Watching live sports on TV these days is a delicate balance for broadcasters how much information is too much?...
04/07/2025
WASHINGTON As Federal Communications Commission Chair Brendan Carr works to achieve his goal of a very busy, productive summer at the FCC, the agency has rele...
04/07/2025
Other World Computing Launches OWC Guardian USB-C Hardware-Encrypted Portable SS...
04/07/2025
CANAL Brings MotoGP to Vision Pro with Blackmagic URSA Cine Immersive
Brie Clayton July 3, 2025
0 Comments
Experience MotoGP like never before as CA...
03/07/2025
To achieve a true level of decisive overmatch, communication systems must be as ...
03/07/2025
The Broadcasters Foundation of America has named Ray Hopkins, president of U.S. distribution at Paramount Global, to its board of directors....
03/07/2025
SIOUX FALLS, S.D. In his first major policy speech, Federal Communications Commission chair Brendan Carr outlined his Build America Agenda for the agency'...
03/07/2025
As the industry is making a major push to transition to NextGen TV and develop new businesses that will fund the next generation of broadcasting, TV Tech's ...
03/07/2025
Despite enabling its subscribers to watch films and TV shows in the highest possible resolution, Netflix has revealed it is taking things old school with the in...
03/07/2025
WASHINGTON In response to news that Paramount Global has agreed to pay $16 million to settle a lawsuit by President Donald Trump, Democrats and other administra...
03/07/2025
WUPPERTAL, Germany Riedel Communications today launched RefSuite for professional sports workflows....
03/07/2025
WEITERSTADT, Germany LYNX Technik, a provider of modular signal processing solutions, has announced that founder and longtime CEO Winfried Deckelmann is steppin...
03/07/2025
WASHINGTON Representatives from six trade groups representing tech, pay TV, broadband, LPTV and other sectors recently met with Federal Communications Commissio...
03/07/2025
PLYMOUTH, Wis. Dan Whealy has acquired Heartland Video Systems and will be its new owner and president, the systems integrator said....
03/07/2025
Acclaimed filmmaker, educator, and industry veteran Jay Holben has released his most ambitious work to date: The Director's Guide to Everything a sweeping...
03/07/2025
For Los Angeles-based photographer Scout Pruski, documenting the annual Dyke Day LA is a way to honor and give back to the city's legacy queer community and...
03/07/2025
Modular, Travel-Ready Solution Matthews Studio Equipment launches the new Chimera V-Flat Panel Frame Kit, the lightweight and modular, ultra-durable alternativ...
03/07/2025
Dalet, a leading technology and service provider for media-rich organizations, today announced the release of Dalet Flex 2025.6 LTS (long-term support), introdu...
03/07/2025
Lightware's LARA Wizard makes it quicker and easier than ever to set up audio-visual (AV) room automation and control configurations, ensuring anyone, from ...
03/07/2025
Alfalite, the only European manufacturer of LED screens, has obtained both the NCAGE Code (NATO Commercial and Government Entity) and TAA (Trade Agreements Act)...
03/07/2025
nxtedition is implementing a new solution for WIN Network, based in its headquarters in Wollongong, New South Wales, Australia. The nxtedition integrated platfo...
03/07/2025
ORS Group and Insys Video Technologies today announce the launch of a new joint brand identity: Big Blue Marble. The move reflects the evolution of a long-stand...
03/07/2025
Evergent, a global leader in customer lifecycle management and monetization solutions for subscription-based businesses, today announced the successful results ...
03/07/2025
LYNX Technik, provider of modular signal processing solutions today announces an executive leadership transition, effective July 1, 2025. Markus Motzko has been...
03/07/2025
Chaos Releases V-Ray for Blender
Brie Clayton July 2, 2025
0 Comments
Production-Proven Renderer V-Ray Arrives Natively in Blender for the First Time;...
03/07/2025
Calvin University Adds Blackmagic Replay to Broadcasts and Streams
Brie Clayton July 2, 2025
0 Comments
Replay workflow joins 14 Blackmagic Design cam...
03/07/2025
Berklee's Gospel Performance Program Welcomes Jonathan Nelson as the Herb Al...
03/07/2025
Back to All News
Get Ready for Even More Action! Netflix Has Confirmed the Thir...
03/07/2025
Back to All News
Get ready for even more action! Netflix has confirmed the thir...
03/07/2025
In today's fragmented media landscape, advertisers expect more than just reach - they want seamless cross-platform campaigns that deliver measurable results...
03/07/2025
The forecast this month is showing a 100% chance of epic gaming. Catch the scorching lineup of 20 titles coming to the cloud, which gamers can play whether indo...
02/07/2025
Hege Wik and Odin appear in FOLKTALES by Heidi Ewing and Rachel Grady, an official selection of the 2025 Sundance Film Festival. Courtesy of Sundance Institut...
02/07/2025
(Stephen Lovekin/Shutterstock for Sundance Film Festival)
By Patty Consolazio
This one's for any person who's ever felt put down, left out, marginaliz...
02/07/2025
NEW YORK Paramount Global has agreed to pay $16 million to settle a lawsuit by President Donald Trump, ending a long-running legal saga that could clear the way...
02/07/2025
WASHINGTON The Federal Communications Commission has approved the transfer of licenses for five TV stations from Sinclair to Rincon Broadcasting, which announce...
02/07/2025
As the media industry faces an uncertain business climate and rapidly changing consumer habits, TV Tech's most popular stories from the first six months of ...
02/07/2025
PORTSMOUTH, N.H. New data shows that U.S. homes now own an average of two smart TVs, with Samsung and LG dominating the market but Roku and FireTV continue to g...
02/07/2025
LOS ANGELES Funds managed by Oaktree Capital Management have acquired FilmRise, an independent digital streaming content distributor, and merged it with Shout! ...
02/07/2025
WASHINGTON The Advanced Television Systems Committee named Luiz Fausto vice president of standards development, succeeding ATSC vice president Jerry Whitaker, w...
02/07/2025
Boston Conservatory and the Sphinx Organization Join Forces to Make Classical Ed...
02/07/2025
NEW AMSTERDAM CREATOR DAVID SCHULNER JOINS CREATIVE TEAM OF NEW FOX DRAMA SERIE...
02/07/2025
Progressive realism: Aurora on taking up the broadcast reins for World RX alongs...
02/07/2025
Great speed: New host broadcaster Aurora on getting the World RX 2025 season pro...
02/07/2025
Seat of its pants: Aurora boxes clever and looks ahead for the FIA as new host b...
02/07/2025
UEFA Women's Euro 2025: EBU, BBC Sport and Sunset Vine talk coverage plans, ...