Sony Pixel Power calrec Sony

Fast, Low-Cost Inference Offers Key to Profitable AI

23/01/2025

Businesses across every industry are rolling out AI services this year. For Microsoft, Oracle, Perplexity, Snap and hundreds of other leading companies, using the NVIDIA AI inference platform - a full stack comprising world-class silicon, systems and software - is the key to delivering high-throughput and low-latency inference and enabling great user experiences while lowering cost.

NVIDIA's advancements in inference software optimization and the NVIDIA Hopper platform are helping industries serve the latest generative AI models, delivering excellent user experiences while optimizing total cost of ownership. The Hopper platform also helps deliver up to 15x more energy efficiency for inference workloads compared to previous generations.

AI inference is notoriously difficult, as it requires many steps to strike the right balance between throughput and user experience.

But the underlying goal is simple: generate more tokens at a lower cost. Tokens represent words in a large language model (LLM) system - and with AI inference services typically charging for every million tokens generated, this goal offers the most visible return on AI investments and energy used per task.

Full-stack software optimization offers the key to improving AI inference performance and achieving this goal.

Cost-Effective User Throughput Businesses are often challenged with balancing the performance and costs of inference workloads. While some customers or use cases may work with an out-of-the-box or hosted model, others may require customization. NVIDIA technologies simplify model deployment while optimizing cost and performance for AI inference workloads. In addition, customers can experience flexibility and customizability with the models they choose to deploy.

NVIDIA NIM microservices, NVIDIA Triton Inference Server and the NVIDIA TensorRT library are among the inference solutions NVIDIA offers to suit users' needs:

NVIDIA NIM inference microservices are prepackaged and performance-optimized for rapidly deploying AI foundation models on any infrastructure - cloud, data centers, edge or workstations.

NVIDIA Triton Inference Server, one of the company's most popular open-source projects, allows users to package and serve any model regardless of the AI framework it was trained on.

NVIDIA TensorRT is a high-performance deep learning inference library that includes runtime and model optimizations to deliver low-latency and high-throughput inference for production applications.

Available in all major cloud marketplaces, the NVIDIA AI Enterprise software platform includes all these solutions and provides enterprise-grade support, stability, manageability and security.

With the framework-agnostic NVIDIA AI inference platform, companies save on productivity, development, and infrastructure and setup costs. Using NVIDIA technologies can also boost business revenue by helping companies avoid downtime and fraudulent transactions, increase e-commerce shopping conversion rates and generate new, AI-powered revenue streams.

Cloud-Based LLM Inference To ease LLM deployment, NVIDIA has collaborated closely with every major cloud service provider to ensure that the NVIDIA inference platform can be seamlessly deployed in the cloud with minimal or no code required. NVIDIA NIM is integrated with cloud-native services such as:

Amazon SageMaker AI, Amazon Bedrock Marketplace, Amazon Elastic Kubernetes Service

Google Cloud's Vertex AI, Google Kubernetes Engine

Microsoft Azure AI Foundry coming soon, Azure Kubernetes Service

Oracle Cloud Infrastructure's data science tools, Oracle Cloud Infrastructure Kubernetes Engine

Plus, for customized inference deployments, NVIDIA Triton Inference Server is deeply integrated into all major cloud service providers.

For example, using the OCI Data Science platform, deploying NVIDIA Triton is as simple as turning on a switch in the command line arguments during model deployment, which instantly launches an NVIDIA Triton inference endpoint.

Similarly, with Azure Machine Learning, users can deploy NVIDIA Triton either with no-code deployment through the Azure Machine Learning Studio or full-code deployment with Azure Machine Learning CLI. AWS provides one-click deployment for NVIDIA NIM from SageMaker Marketplace and Google Cloud provides a one-click deployment option on Google Kubernetes Engine (GKE). Google Cloud provides a one-click deployment option on Google Kubernetes Engine, while AWS offers NVIDIA Triton on its AWS Deep Learning containers.

The NVIDIA AI inference platform also uses popular communication methods for delivering AI predictions, automatically adjusting to accommodate the growing and changing needs of users within a cloud-based infrastructure.

From accelerating LLMs to enhancing creative workflows and transforming agreement management, NVIDIA's AI inference platform is driving real-world impact across industries. Learn how collaboration and innovation are enabling the organizations below to achieve new levels of efficiency and scalability.

Serving 400 Million Search Queries Monthly With Perplexity AI Perplexity AI, an AI-powered search engine, handles over 435 million monthly queries. Each query represents multiple AI inference requests. To meet this demand, the Perplexity AI team turned to NVIDIA H100 GPUs, Triton Inference Server and TensorRT-LLM.

Supporting over 20 AI models, including Llama 3 variations like 8B and 70B, Perplexity processes diverse tasks such as search, summarization and question-answering. By using smaller classifier models to route tasks to GPU pods, managed by NVIDIA Triton, the company delivers cost-efficient, responsive service under strict service level agreements.

Through model parallelism, which splits LLMs across GPUs, Perplexity achieved a threefold cost reduction while maintaining low latency and high accurac
LINK: https://blogs.nvidia.com/blog/ai-inference-platform/...
See more stories from nvidia

North America Stories

19/11/2026

Samba Combines Streaming, Linear TV Data

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

19/11/2026

Charting the Progress of Brazil TV 3.0 with VVC

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

19/11/2026

Skydance to Combine Warner Bros. Television Group and Paramount Television Studios

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , er...

19/11/2026

Australia's Seven Selects RTS Comms for New Headquarters

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

19/11/2026

Richard Burki Expands the Possibilities of Immersive Audi...

As Founder and Creative Director of Future Phonic Studios, the Netherlands' first Dolby Atmos music studio, Richard Burki has spent more than a decade pushi...

19/11/2026

Boris FX Black Friday Sale - 25% Off Everything

Boris FX Black Friday Sale - 25% Off Everything Brie Clayton November 19, 2026 0 Comments Check out the Boris FX Black Friday sale on Monday, November...

19/11/2026

Virtual Production Finds Its Middle Ground with Blackmagic Design

Virtual Production Finds Its Middle Ground with Blackmagic Design Brie Clayton November 19, 2026 0 Comments How Neon Vision is using PYXIS 12K and Ult...

19/11/2026

FL Studio Announces Black Friday Discounts of Up to 50% Across Its Software Lineup

FL Studio Announces Black Friday Discounts of Up to 50% Across Its Software Line...

19/11/2026

TBS Drama Series Tagusari Brothers Uses DaVinci Resolve Studio

TBS Drama Series Tagusari Brothers Uses DaVinci Resolve Studio Brie Clayton November 19, 2026 0 Comments TBS ACT streamlines post production for broad...

19/11/2026

Beeble launches SwitchX 3.0 with dynamic environments and greater creative control

Beeble launches SwitchX 3.0 with dynamic environments and greater creative contr...

19/11/2026

NDI Releases Latest Version With Recording, Playback APIs

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

19/11/2026

Newsmax Asks Court to Block Foxs Acquisition of Roku

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

19/11/2026

HPA to Honor The Odyssey Director Christopher Nolan with Judges Award for Creativity and Innovation

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , er...

19/11/2026

Australian Professional Leagues Renews Partnership with B...

Five-year deployment has automated APL's media supply chain from fixture to archive, managing over 200,000 assets Blue Lucy, the workflow orchestration and...

19/11/2026

AES Adds Dialogue Intelligibility to Loudness Website

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

19/11/2026

Rise Awards 2026 Celebrate Inspirational Women in Broadcast

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

19/11/2026

BPS Trial at Dominion Energy Substation Proves Successful

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

19/11/2026

Wisycom and DPA Prove Essential for Demanding and Unpredi...

iStreamU LLC, a rapidly growing production and systems integration company founded by longtime Producer/Director Ivan Gonzalez, has standardized its audio workf...

19/11/2026

Seven Scripps Research scientists named to Clarivate's 2026 list of Highly Cited Researchers

LA JOLLA, CA-Clarivate, a global information and analytics company, named seven ...

19/11/2026

Season's Streamings and Savings: Black Friday Deals Come to GeForce NOW

GeForce NOW is kicking off Black Friday with more reasons to make time for gaming. Six-month memberships are back for a limited time. Save 30% on the first six-...

18/11/2026

SVG TranSPORT 2026: All Sessions Now Available To Watch on SVG PLAY

Held in New York City, the event explored the latest technologies and trends shaping the future of live-sports transmission and distribution...

18/11/2026

Samsung Takes the Gold at Your Olympic Games Dakar; Galaxy Phones Play Role in Coverage

The Youth Olympic Games Dakar 2026 opened in Senegal on October 31, giving young...

18/11/2026

DIRECTV teams with Gracenote to enhance the sports programming experiences that fans want

Strategic renewal builds on existing engagement, supports DIRECTV's objectiv...

18/11/2026

SMPTE Publishes Engineering Report on Content Authenticity

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Senate Commerce Committee Approves Thumann Severs for FCC Commissioner

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

wTVision Qualifies NEOiD PTZ Cameras for Virtual Sets, AR

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Spanish Media Group Deploys Lawo IP Audio Infrastructure

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Library of American Broadcasting Foundation Honors 2026 Giants of Broadcasting & Electronic Arts

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , er...

18/11/2026

Monthly Subscription Streaming Service Prices Up Just 8% Since 2022

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Spectrum News Streamlines Workflow With Amagi

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Media Links Joins Panasonic Kairos Partner Program

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Signiant Acquires Burnlab's Ceivo AI-Driven Media Platform

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Democrats Urge Delay of Thumann Severs FCC Confirmation

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Hisong Unveils AirStudio Muse Audio Capture System

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Spectrum Reach Taps Debbie Presser to Lead Sales at AdLink New York

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Nationwide EAS Test Runs Without Apparent Hiccups

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

SMPTE Identifies Key Requirements for Content Provenance...

SMPTE, the home for media professionals, technologists and engineers, today announced the publication of an engineering report with the summarized results of an...

18/11/2026

Beeble launches SwitchX 3.0 with dynamic environments and greater creative control

Beeble recently launched SwitchX 3.0, a major update to its AI-powered video-to-...

18/11/2026

Film London to lead new industry PhD program in partnership with Brunel University

Film London will lead and manage a long-term industry research program in partne...

18/11/2026

Hexagon launches MANTIS to capture 3D scene data as cameras roll

Hexagon, the global leader in measurement technologies, recently launched MANTIS by Leica Geosystems, a camera-mounted 3D capture and tracking system to give fi...

18/11/2026

NVIDIA RTX Spark Compact Desktops Out Now

New NVIDIA RTX Spark compact desktops from ASUS, Dell and MSI are available today, bringing local AI agents, image and video generation, and model fine-tuning t...

17/11/2026

Fabric Insights and Fonn Group Partner to Connect Newsroom Planning and Media Operations

Fabric Insights and Fonn Group have announced a partnership to connect editorial...

17/11/2026

NABJ Online Seminar Highlights Black Sports Production Legends on Nov. 25

The National Association of Black Journalists is offering up a Thanksgiving Eve treat on Wednesday, Nov. 25, in the form of a live streaming event featuring for...

17/11/2026

Bridge Technologies Adds Apple ProRes Support to VB440 Production Probe

Bridge Technologies has announced that its VB440 production probe now supports both monitoring and signal generation for Apple ProRes video streams. The additio...

17/11/2026

Canon U.S.A. Hosts Jericho High School Journalism Students at Melville Headquarters

Canon U.S.A., Inc. recently welcomed 21 Jericho High School students to its Melv...

17/11/2026

Amagi to Power Live Newscasts for Spectrum News

Spectrum News has deployed the Amagi platform to support its live news operations. Amagi says the platform lets Spectrum News streamline newsroom workflows, red...

17/11/2026

Microsoft Expands Haivision Video Contribution Deployment, Adds Makito ONE for Global Live Events

Microsoft has expanded its use of Haivision's live video contribution ecosys...

17/11/2026

Setanta Sports Deploys Ateme TITAN Live for OTT Sports Streaming

Setanta Sports has deployed Ateme's TITAN Live platform to support its OTT video delivery operations. Ateme's transcoding technology is integrated into ...

17/11/2026

Media Links Joins Panasonic's KAIROS Alliance Partner Program

Media Links has joined the KAIROS Alliance Partner program for Panasonic's KAIROS IT/IP live video production platform. The company completed interoperabili...

17/11/2026

Behind The Mic: Dan Hicks, Iconic NBC Sports Broadcaster, Passes Away at 64

Behind The Mic provides a roundup of recent news regarding on-air talent, including new deals, departures, and assignments compiled from press releases and repo...