Sony Pixel Power calrec Sony

Fast, Low-Cost Inference Offers Key to Profitable AI

23/01/2025

Businesses across every industry are rolling out AI services this year. For Microsoft, Oracle, Perplexity, Snap and hundreds of other leading companies, using the NVIDIA AI inference platform - a full stack comprising world-class silicon, systems and software - is the key to delivering high-throughput and low-latency inference and enabling great user experiences while lowering cost.

NVIDIA's advancements in inference software optimization and the NVIDIA Hopper platform are helping industries serve the latest generative AI models, delivering excellent user experiences while optimizing total cost of ownership. The Hopper platform also helps deliver up to 15x more energy efficiency for inference workloads compared to previous generations.

AI inference is notoriously difficult, as it requires many steps to strike the right balance between throughput and user experience.

But the underlying goal is simple: generate more tokens at a lower cost. Tokens represent words in a large language model (LLM) system - and with AI inference services typically charging for every million tokens generated, this goal offers the most visible return on AI investments and energy used per task.

Full-stack software optimization offers the key to improving AI inference performance and achieving this goal.

Cost-Effective User Throughput Businesses are often challenged with balancing the performance and costs of inference workloads. While some customers or use cases may work with an out-of-the-box or hosted model, others may require customization. NVIDIA technologies simplify model deployment while optimizing cost and performance for AI inference workloads. In addition, customers can experience flexibility and customizability with the models they choose to deploy.

NVIDIA NIM microservices, NVIDIA Triton Inference Server and the NVIDIA TensorRT library are among the inference solutions NVIDIA offers to suit users' needs:

NVIDIA NIM inference microservices are prepackaged and performance-optimized for rapidly deploying AI foundation models on any infrastructure - cloud, data centers, edge or workstations.

NVIDIA Triton Inference Server, one of the company's most popular open-source projects, allows users to package and serve any model regardless of the AI framework it was trained on.

NVIDIA TensorRT is a high-performance deep learning inference library that includes runtime and model optimizations to deliver low-latency and high-throughput inference for production applications.

Available in all major cloud marketplaces, the NVIDIA AI Enterprise software platform includes all these solutions and provides enterprise-grade support, stability, manageability and security.

With the framework-agnostic NVIDIA AI inference platform, companies save on productivity, development, and infrastructure and setup costs. Using NVIDIA technologies can also boost business revenue by helping companies avoid downtime and fraudulent transactions, increase e-commerce shopping conversion rates and generate new, AI-powered revenue streams.

Cloud-Based LLM Inference To ease LLM deployment, NVIDIA has collaborated closely with every major cloud service provider to ensure that the NVIDIA inference platform can be seamlessly deployed in the cloud with minimal or no code required. NVIDIA NIM is integrated with cloud-native services such as:

Amazon SageMaker AI, Amazon Bedrock Marketplace, Amazon Elastic Kubernetes Service

Google Cloud's Vertex AI, Google Kubernetes Engine

Microsoft Azure AI Foundry coming soon, Azure Kubernetes Service

Oracle Cloud Infrastructure's data science tools, Oracle Cloud Infrastructure Kubernetes Engine

Plus, for customized inference deployments, NVIDIA Triton Inference Server is deeply integrated into all major cloud service providers.

For example, using the OCI Data Science platform, deploying NVIDIA Triton is as simple as turning on a switch in the command line arguments during model deployment, which instantly launches an NVIDIA Triton inference endpoint.

Similarly, with Azure Machine Learning, users can deploy NVIDIA Triton either with no-code deployment through the Azure Machine Learning Studio or full-code deployment with Azure Machine Learning CLI. AWS provides one-click deployment for NVIDIA NIM from SageMaker Marketplace and Google Cloud provides a one-click deployment option on Google Kubernetes Engine (GKE). Google Cloud provides a one-click deployment option on Google Kubernetes Engine, while AWS offers NVIDIA Triton on its AWS Deep Learning containers.

The NVIDIA AI inference platform also uses popular communication methods for delivering AI predictions, automatically adjusting to accommodate the growing and changing needs of users within a cloud-based infrastructure.

From accelerating LLMs to enhancing creative workflows and transforming agreement management, NVIDIA's AI inference platform is driving real-world impact across industries. Learn how collaboration and innovation are enabling the organizations below to achieve new levels of efficiency and scalability.

Serving 400 Million Search Queries Monthly With Perplexity AI Perplexity AI, an AI-powered search engine, handles over 435 million monthly queries. Each query represents multiple AI inference requests. To meet this demand, the Perplexity AI team turned to NVIDIA H100 GPUs, Triton Inference Server and TensorRT-LLM.

Supporting over 20 AI models, including Llama 3 variations like 8B and 70B, Perplexity processes diverse tasks such as search, summarization and question-answering. By using smaller classifier models to route tasks to GPU pods, managed by NVIDIA Triton, the company delivers cost-efficient, responsive service under strict service level agreements.

Through model parallelism, which splits LLMs across GPUs, Perplexity achieved a threefold cost reduction while maintaining low latency and high accurac
LINK: https://blogs.nvidia.com/blog/ai-inference-platform/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

09/10/2026

Dolby Expands Dolby OptiView Platform with New Capabilities at IBC 2026

September 10 2026, 06:00 (PDT) Dolby Expands Dolby OptiView Platform with New Capabilities at IBC 2026 New Sports Intelligence helps providers better unders...

07/10/2026

Dalet Flex LTS Delivers Smarter Media Operations from Ingest to Distribution

Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...

29/09/2026

Netflix Deploys Advanced Sony PTZ Cameras For Wonka's The Golden Ticket' Production

Share Copy link Facebook X Linkedin Bluesky Email...

29/09/2026

Massachusetts Broadcasters Association Names Gene Lavanchy 2026 Broadcaster of the Year

Share Copy link Facebook X Linkedin Bluesky Email...

29/09/2026

ESPN Reports Its Most-Watched WNBA Regular Season

Share Copy link Facebook X Linkedin Bluesky Email...

29/09/2026

Scripps Taps Digital News Veteran Sean McGarvy to Lead News Strategy

Share Copy link Facebook X Linkedin Bluesky Email...

29/09/2026

Gray Raises Political Ad Revenue Estimates

Share Copy link Facebook X Linkedin Bluesky Email...

29/09/2026

Edward Czarnecki Uses Game Theory to Assess ATSC 3.0 Transition Policies

Share Copy link Facebook X Linkedin Bluesky Email...

29/09/2026

First Baptist Nashville Drastically Upgrades Production C...

Clear-Com announced that First Baptist Nashville has upgraded its production communications with a Clear-Com intercom solution as part of a comprehensive sanct...

29/09/2026

Blackmagic Cinema Camera 6K Captured Action Shots for Sketch

Blackmagic Cinema Camera 6K Captured Action Shots for Sketch Brie Clayton September 28, 2026 0 Comments Camera's portable design and dynamic range...

29/09/2026

Kean Production Services Deploys AJA ColorBox to Deliver Consistent, Accurate Color Across In-venue Displays at Live Events

Kean Production Services Deploys AJA ColorBox to Deliver Consistent, Accurate Co...

29/09/2026

Berklee Alumni and Faculty Nominated for the 2026 Latin Grammy Awards

Berklee Alumni and Faculty Nominated for the 2026 Latin Grammy Awards A total of 28 Berklee alumni and faculty earned nominations across 20 categories this ye...

29/09/2026

Berklee's K-Pop & Beyond Explores Connections Between K-Pop and Hip-Hop

Berklee's K-Pop & Beyond Explores Connections Between K-Pop and Hip-Hop Now in its fifth year, the series brings together artists, industry leaders, and s...

28/09/2026

NHL Names Adobe as an Official Creative, Marketing and AI Partner

The NHL has announced Adobe as an Official Creative, Marketing and AI Partner beginning with the 2026-27 season. The partnership will provide the league and its...

28/09/2026

MultiDyne to Showcase Expanded Portfolio Including MRMC Robotics at CABSAT 2026

MultiDyne will exhibit at CABSAT 2026 (October 5-7, Dubai Exhibition Centre), showcasing an expanded technology portfolio following its June 2026 acquisition of...

28/09/2026

ARRI and HONOR Launch Co-Branded HONOR Magic9 Pro Max

ARRI and HONOR have announced the HONOR Magic9 Pro Max, a flagship smartphone carrying ARRI branding on the device and packaging as the first product-level expr...

28/09/2026

Net Insight Adds Nimbra 1020G to Pan-Asian Live Media Trial Network

Net Insight has added the Nimbra 1020G to its pan-Asian live media trial network, giving customers and partners in the region a way to evaluate ST 2110 and JPEG...

28/09/2026

Vindral Named Recipient of 2026 Technology and Engineering Emmy Award

Vindral has been named a recipient of a 2026 Technology and Engineering Emmy Award by the National Academy of Television Arts and Sciences in the category of We...

28/09/2026

Interra Systems to Demonstrate QC and Monitoring Portfolio at CABSAT 2026

Interra Systems will exhibit at CABSAT 2026 (Stand B2-37, Dubai Exhibition Centre, October 5-7), demonstrating its BATON QC platform, ORION monitoring solutions...

28/09/2026

PGA TOUR's Michael Raimondo on Agentic Production, ShotLink Automation, and the Evolution of PGA TOUR Studios

Two years into its new production facility, the TOUR is expanding centralized wo...

28/09/2026

Game Creek Videos Epic Building Run Continues With ESPNs Countdown Unit, Upgrades to Four Others

In addition to rolling out a new mobile unit for ESPN's Monday Night Countd...

28/09/2026

Bleacher Report and WWE Reach Multi-Year Global Content Partnership

Bleacher Report and WWE have announced a multi-year content partnership giving Bleacher Report global highlight rights across WWE's programming slate, inclu...

28/09/2026

Behind The Mic: NBC Sports Shares MLB Wild Card Series Announcing Booth; ESPN Reveals NHL 26-27 Season Talent

Behind The Mic provides a roundup of recent news regarding on-air talent, includ...

28/09/2026

Fubo Adds Four NHL Regional Sports Networks for 2026-27 Season

FuboTV has announced a multi-year distribution agreement with the NHL to carry four regional sports networks covering the Carolina Hurricanes, Columbus Blue Jac...

28/09/2026

Program Productions Names Alan Ostfield President and CEO

Program Productions (PPI) has announced the appointment of Alan Ostfield as President and Chief Executive Officer, effective September 28, 2026. Ostfield succee...

28/09/2026

Ha-chan, Shake Your Booty! Encourages Us to Keep Dancing

Director Josef Kubota Wladyka attends the premiere of Ha-Chan, Shake Your Booty! with the film's cast and crew at Eccles Theatre on January 22, 2026, in P...

28/09/2026

Scaler Music introduce Scaler EQ Pro

EQ plug-in can now be played via MIDI Scaler Music have recently released a new and improved version of their patented musical EQ plug-in, kitting it out wi...

28/09/2026

The Asura from Animal Factory Amps

New synth developed in collaboration with philterSoup Boutique effects pedal and Eurorack manufacturer Animal Factory Amps have announced that the new compa...

28/09/2026

Rohde & Schwarz adds an integrated multi-channel pulse analysis option to its FSWX signal and spectrum analyzer

Rohde & Schwarz adds an integrated multi-channel pulse analysis option to its FS...

28/09/2026

SBE Announces 2026 Ennes Scholarship Recipients

Share Copy link Facebook X Linkedin Bluesky Email...

28/09/2026

Spectrum to Demo AI Computing at the Network's Edge at SCTE TechExpo 26

Share Copy link Facebook X Linkedin Bluesky Email...

28/09/2026

Autoscript launches new Voice Controller prompting soluti...

Autoscript, the leading global provider of professional teleprompting solutions, today announced Voice Controller for WinPlus-IP, a new voice-controlled prompti...

28/09/2026

IBC 2026 Technical Conference Highlights

IBC 2026 Technical Conference Highlights David Kirk September 28, 2026 0 Comments Hero image courtesy Deposit Photos David Kirk reports from the 11-1...

28/09/2026

First Baptist Nashville Taps Clear-Com for Production Coms

Share Copy link Facebook X Linkedin Bluesky Email...

28/09/2026

IEEE Announces Award for Leadership in Developing HEVC/H.265

Share Copy link Facebook X Linkedin Bluesky Email...

28/09/2026

Nakadi Introduces Live to Theater Service Powered by Pate...

Nakadi, a global streaming technology innovator dedicated to engineering a new era of digital media delivery, today announced its Live to Theater service based ...

28/09/2026

Sky launches Full Fibre 1.6 Gbps for millions of homes

Monday 28 September 2026 Sky launches Full Fibre 1.6 Gbps for millions of homes Sky has launched a new Full Fibre 1.6 Gbps product, rolling out from today, gi...

28/09/2026

Arvato Systems Highlights Its Role for Automakers and Suppliers

Arvato Systems Highlights It's Role for Automotive Manufacturers and Suppliers TISAX Arvato Systems has successfully completed the TISAX Level 3 assess...

28/09/2026

Crimecall returns for a brand-new series

Crimecall, presented by Carla O'Brien, returns tonight for a brand new series at 9.35pm on RT One and RT Player. In tonight's episode, alongside appe...

28/09/2026

RT SHORT STORY COMPETITION 2026 - SHORTLISTED STORIES ANNOUNCED

CELEBRATING 40 YEARS AT THE HEART OF NEW IRISH WRITING LIVE FROM POETRY IRELAND, DUBLIN AND ON RT RADIO 1 FROM 7PM DETAILS: www.rte.ie/writing | FOLLOW: #r...

27/09/2026

Boss launch the EX-4 Effects Expander

New compact pedal packs in 120 effects The latest multi-effects pedal from Boss is a compact unit that promises to give any pedalboard a massive boost in s...

26/09/2026

ESPN Operations, Production Teams Plan for Every Scenario as WNBA Playoffs Tip Off

With short turnaround between game selections, ESPN's operations team is pla...

26/09/2026

Prime Video Scales Up for First WNBA Postseason, Adding Cameras, Analysts, Coach Mics, and NBA Mobile Units

Prime Video will use GCV's Bird and Magic mobile units for its inaugural WNB...

26/09/2026

NBC Sports Tips Off First WNBA Postseason in 20+ Years Using Sunday Night Basketball as a Production Baseline

Studio show moves on-site as network models playoff coverage on its NBA flagship...

26/09/2026

Ratings Roundup: Cowboys-Commanders Leads NFL Week 2 With 27.2 Million, College GameDay Posts Best Week 3 Ever

Ratings Roundup is a rundown of recent ratings news and is derived from press re...

26/09/2026

From Sketch to Strut, Unzipped Returns to the Sundance Film Festival at 30

Douglas Keeve attends the 2025 Sundance Film Festival screening of Unzipped at the Egyptian Theatre. (Photo by Andrew H. Walker/Shutterstock for Sundance Film...

26/09/2026

Native Instruments update Maschine

Launched alongside three new expansion packs In celebration of their 30th anniversary, Native Instruments have released a major update for their Maschine 3 ...

26/09/2026

YouTube Embraces Microdramas with Features for Episodic Shorts

Share Copy link Facebook X Linkedin Bluesky Email...

26/09/2026

Civil Rights Groups Urge Sen. Cruz to Delay Thumann Severs FCC Confirmation

Share Copy link Facebook X Linkedin Bluesky Email...