Sony Pixel Power calrec Sony

Fast, Low-Cost Inference Offers Key to Profitable AI

23/01/2025

Businesses across every industry are rolling out AI services this year. For Microsoft, Oracle, Perplexity, Snap and hundreds of other leading companies, using the NVIDIA AI inference platform - a full stack comprising world-class silicon, systems and software - is the key to delivering high-throughput and low-latency inference and enabling great user experiences while lowering cost.

NVIDIA's advancements in inference software optimization and the NVIDIA Hopper platform are helping industries serve the latest generative AI models, delivering excellent user experiences while optimizing total cost of ownership. The Hopper platform also helps deliver up to 15x more energy efficiency for inference workloads compared to previous generations.

AI inference is notoriously difficult, as it requires many steps to strike the right balance between throughput and user experience.

But the underlying goal is simple: generate more tokens at a lower cost. Tokens represent words in a large language model (LLM) system - and with AI inference services typically charging for every million tokens generated, this goal offers the most visible return on AI investments and energy used per task.

Full-stack software optimization offers the key to improving AI inference performance and achieving this goal.

Cost-Effective User Throughput Businesses are often challenged with balancing the performance and costs of inference workloads. While some customers or use cases may work with an out-of-the-box or hosted model, others may require customization. NVIDIA technologies simplify model deployment while optimizing cost and performance for AI inference workloads. In addition, customers can experience flexibility and customizability with the models they choose to deploy.

NVIDIA NIM microservices, NVIDIA Triton Inference Server and the NVIDIA TensorRT library are among the inference solutions NVIDIA offers to suit users' needs:

NVIDIA NIM inference microservices are prepackaged and performance-optimized for rapidly deploying AI foundation models on any infrastructure - cloud, data centers, edge or workstations.

NVIDIA Triton Inference Server, one of the company's most popular open-source projects, allows users to package and serve any model regardless of the AI framework it was trained on.

NVIDIA TensorRT is a high-performance deep learning inference library that includes runtime and model optimizations to deliver low-latency and high-throughput inference for production applications.

Available in all major cloud marketplaces, the NVIDIA AI Enterprise software platform includes all these solutions and provides enterprise-grade support, stability, manageability and security.

With the framework-agnostic NVIDIA AI inference platform, companies save on productivity, development, and infrastructure and setup costs. Using NVIDIA technologies can also boost business revenue by helping companies avoid downtime and fraudulent transactions, increase e-commerce shopping conversion rates and generate new, AI-powered revenue streams.

Cloud-Based LLM Inference To ease LLM deployment, NVIDIA has collaborated closely with every major cloud service provider to ensure that the NVIDIA inference platform can be seamlessly deployed in the cloud with minimal or no code required. NVIDIA NIM is integrated with cloud-native services such as:

Amazon SageMaker AI, Amazon Bedrock Marketplace, Amazon Elastic Kubernetes Service

Google Cloud's Vertex AI, Google Kubernetes Engine

Microsoft Azure AI Foundry coming soon, Azure Kubernetes Service

Oracle Cloud Infrastructure's data science tools, Oracle Cloud Infrastructure Kubernetes Engine

Plus, for customized inference deployments, NVIDIA Triton Inference Server is deeply integrated into all major cloud service providers.

For example, using the OCI Data Science platform, deploying NVIDIA Triton is as simple as turning on a switch in the command line arguments during model deployment, which instantly launches an NVIDIA Triton inference endpoint.

Similarly, with Azure Machine Learning, users can deploy NVIDIA Triton either with no-code deployment through the Azure Machine Learning Studio or full-code deployment with Azure Machine Learning CLI. AWS provides one-click deployment for NVIDIA NIM from SageMaker Marketplace and Google Cloud provides a one-click deployment option on Google Kubernetes Engine (GKE). Google Cloud provides a one-click deployment option on Google Kubernetes Engine, while AWS offers NVIDIA Triton on its AWS Deep Learning containers.

The NVIDIA AI inference platform also uses popular communication methods for delivering AI predictions, automatically adjusting to accommodate the growing and changing needs of users within a cloud-based infrastructure.

From accelerating LLMs to enhancing creative workflows and transforming agreement management, NVIDIA's AI inference platform is driving real-world impact across industries. Learn how collaboration and innovation are enabling the organizations below to achieve new levels of efficiency and scalability.

Serving 400 Million Search Queries Monthly With Perplexity AI Perplexity AI, an AI-powered search engine, handles over 435 million monthly queries. Each query represents multiple AI inference requests. To meet this demand, the Perplexity AI team turned to NVIDIA H100 GPUs, Triton Inference Server and TensorRT-LLM.

Supporting over 20 AI models, including Llama 3 variations like 8B and 70B, Perplexity processes diverse tasks such as search, summarization and question-answering. By using smaller classifier models to route tasks to GPU pods, managed by NVIDIA Triton, the company delivers cost-efficient, responsive service under strict service level agreements.

Through model parallelism, which splits LLMs across GPUs, Perplexity achieved a threefold cost reduction while maintaining low latency and high accurac
LINK: https://blogs.nvidia.com/blog/ai-inference-platform/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

04/08/2026

Dalet Announces Commercial Availability of Dalia, Bringing Media-Aware Agentic AI to Enterprise Productions

Dalet, a leading technology and service provider for media-rich organizations, t...

04/07/2026

Detective Conan: Fallen Angel of the Highway Opens in Dolby Cinemas Across Japan, Presented in Dolby Atmos and Dolby ...

April 7 2026, 19:00 (PDT) Detective Conan: Fallen Angel of the Highway Opens in...

01/06/2026

Dolby Sets the New Standard for Premium Entertainment at CES 2026

January 6 2026, 05:30 (PST) Dolby Sets the New Standard for Premium Entertainment at CES 2026 Throughout the week, Dolby brings to life the latest innovatio...

21/05/2026

CBS Sports Expands WNBA Tip-Off Show To Cover Half of 20-Game, Regular-Season Package

Game Creek Video Columbia and Celtic, NEP Supershooter 8 will house onsite produ...

21/05/2026

SVG Students To Watch: Christina Zelin, Rowan University

Freshly graduated, this upstart producer, director, and camera operator is already working as an AP on videoboard shows for the Philadelphia Phillies In the li...

21/05/2026

Media Links Announces Channel Partnership with Clearcast Asia Ahead of BroadcastAsia 2026

Media Links has announced a channel partnership with Clearcast Asia, a broadcast...

21/05/2026

SiriusXM and NASCAR Announce Multi-Year Broadcasting Agreement Renewal

SiriusXM and NASCAR have announced a multi-year renewal of their broadcasting agreement. SiriusXM will continue to carry live broadcasts of every NASCAR Cup Ser...

21/05/2026

Audio-Technica Hosts High-Density Wireless Microphone Demo at Technica House in New York City

Audio-Technica held a demonstration event at its Technica House location in New ...

21/05/2026

RTL Deutschland Selects Ateme Frame-Rate Conversion Technology for Live Event Workflows

Ateme has announced that RTL Deutschland has selected Ateme's software-based...

21/05/2026

BCC Live Deploys LiveU LU900Q for Record-Breaking IRONMAN Texas Broadcast

LiveU has announced that BCC Live deployed the LU900Q intelligent production unit for the first time during the 2026 Memorial Hermann IRONMAN Texas North Americ...

21/05/2026

Mark Aitken to Receive 2026 ATSC Mark Richer Industry Leadership Medal

ATSC has announced that Mark Aitken, President of ONE Media and Senior VP of Advanced Technology at Sinclair Broadcast Group, will receive the 2026 Mark Richer ...

21/05/2026

BBright Outlines MXL Integration Strategy for Software-Defined Broadcast Workflows

BBright has published a technical analysis of the Media eXchange Layer (MXL), de...

21/05/2026

Esports World Cup 2026 to Be Held in Paris, France

The Esports Foundation has announced that the 2026 Esports World Cup (EWC) will be hosted in Paris, France, from July 6 through August 23. The event marks the f...

21/05/2026

Chyron Announces PRIME Scorebug and Expanded Chyron LIVE Scorebug Capabilities

Chyron has announced PRIME Scorebug, a scorebug solution built on the PRIME Platform for on-premises sports production, and has expanded Chyron LIVE with purpos...

21/05/2026

Media Links Integrates Xscend Platform with DataMiner at BroadcastAsia 2026

Media Links has announced the integration of its Xscend IP transport platform with Skyline Communications' DataMiner xOps platform. The integration will be ...

21/05/2026

SVG New Sponsor Spotlight: Nova Lume CEO Jim Casey on Bringing Flexible IP Intercom to Live Production

As live sports productions continue to demand more flexible, scalable, and cost-...

21/05/2026

SVG Rewind: ESPN's Use of POVORA Wireless Tilt Control CapCam Gives College Football Fans a First-Person View of the Game

In advance of this year's Sports Emmy Awards, SVG is taking a deep dive into...

21/05/2026

Join Shade in Miami and Atlanta for Post-Production Networking Events

Hey Miami & Atlanta post-production folks! Shade is hosting a free private suite at a Braves game (6/2) and Marlins game (6/5) and have about a dozen extra tic...

21/05/2026

Phoenix Suns, Mercury, and Gray Media Extend Broadcast Partnership Through 2030

The Suns and Mercury become the first NBA and WNBA teams to make games available under a single broadcast partner across both over-the-air and streaming....

21/05/2026

Apple TV, MLS to Produce First Major Pro Sporting Event Shot Entirely on iPhone 17 Pros

iPhones are part of the the regular production rotation for Friday Night Baseba...

21/05/2026

SVG Rewind: AIQ: Where Data Meets Dirt Technology Brings Real-Time Data Analysis to Rodeo

In advance of this year's Sports Emmy Awards, SVG is taking a deep dive into...

21/05/2026

Filmmakers' Favorites: Sundance Film Festival Alums on Welcome to the Dollhouse

Heather Matarazzo as Dawn Wiener in Todd Solondz's Welcome to the Dollhouse...

21/05/2026

Create, Control, and Personalize Your Listening Across Every Moment

Spotify has always been about helping you find something you want to listen to. And over the years, we've learned your taste and the moments that matter to ...

21/05/2026

You Know Every Song. We Saved You Two Tickets. Introducing Reserved.

Getting concert tickets today can feel like a race you're set up to lose. You show up at the right time, refresh endlessly, and still miss out. Too often, ...

21/05/2026

Books on Spotify: New Updates for Listeners and Authors

In 2022, Spotify entered a new chapter by introducing audiobooks to our platform. Since then, we've grown our catalog to include more than 700,000 titles, e...

21/05/2026

Investor Day 2026 Remarks From Spotify Co-CEOs Alex Norstrm and Gustav Sderstrm

Opening remarks ALEX Good morning everyone, I'm Alex [Norstr m]. GUSTAV And I'm Gustav [S derstr m]. ALEX Whether you've been following our j...

21/05/2026

Spotify's 2026 Investor Day Recap: Raising Ambition for the Next Era of Media

Today, Spotify hosted our third Investor Day in New York City, offering the fina...

21/05/2026

Spotifys Investor Day 2026: Ambitioniert in die nchste ra der Medien

Spotify hat heute seinen dritten Investor Day in New York City veranstaltet und der Finanzwelt tiefere Einblicke in das Gesch ft, die Produktstrategie und die l...

21/05/2026

Rcapitulatif de l'Investor Day 2026 de Spotify : Des ambitions renouveles pour la prochaine re des mdias

Aujourd'hui, Spotify a organis son troisi me Investor Day New York. En pl...

21/05/2026

Il riepilogo dell'Investor Day 2026 di Spotify: eleviamo le ambizioni per la prossima era dei media

Oggi, a New York City, Spotify ha presentato il suo terzo Investor Day, offrendo...

21/05/2026

Resumen del Investor Day 2026 de Spotify: elevando la ambicin para la prxima era de los medios

Hoy Spotify celebr su tercer Investor Day en Nueva York, donde ofrecimos a la c...

21/05/2026

Rangkuman Investor Day Spotify 2026: Meningkatkan Ambisi untuk Era Media Berikutnya

Hari ini, Spotify menyelenggarakan Investor Day yang ketiga di New York City, me...

21/05/2026

2026 (Investor Day):

2026 : (Investor Day) , , . ...

21/05/2026

2026 :

20 2026 ...

21/05/2026

Resumo do Investor Day 2026 do Spotify: elevando a ambio para a prxima era da mdia

Hoje, o Spotify realizou seu terceiro Investor Day em Nova York, oferecendo co...

21/05/2026

2026: Spotify

Spotify Investor Day ...

21/05/2026

Spotify'n 2026 Investor Day zeti: Medyann Yeni a in Byk Hedefler

Spotify bug n, 20'nci y l d n m m z kutlad m z bu y lda, finans camias na, i modelimiz, r n stratejimiz ve uzun vadeli vizyonumuz hakk nda daha detayl ...

21/05/2026

Cinematic Motion & Ethereal Flows for Sonuscore's The Score

Two new Story Packs join orchestral instrument line-up Sonuscore have just introduced two new additions to The Score, marking the instrument's first maj...

21/05/2026

Heavyocity release Oblivion Drums

30,000 samples, 99 presets & 504 loops Heavyocity are well known for their hard-hitting cinematic instruments, and their latest release is no exception to t...

21/05/2026

Rohde & Schwarz AI powered voice to data: The future of air traffic control takes flight at Airspace World 2026

Rohde & Schwarz AI powered voice to data: The future of air traffic control take...

21/05/2026

Redefining Persistent and Affordable Airpower for Special Operations Forces

SKY RAIDER II INTERNATIONAL's modular open systems architecture delivers expanded operational reach and mission flexibility....

21/05/2026

From Payload to Platform: Autonomous ISR Where It Actually Matters

ASO-enabled WESCAM MX-10 systems conduct systematic wide-area maritime search patterns, autonomously managing sensor scan operations to expand coverage, reduce ...

21/05/2026

Sports content accounts for fastest-growing portion of top global SVOD catalogs

HBO Max, a new addition to Gracenote Data Hub, is home to the most sports programming among major streamers NEW YORK May 21, 2026 New analysis by Gracenote...

21/05/2026

Study: Nearly Half of U.S. Viewers Watch Video With Captions

Share Copy link Facebook X Linkedin Bluesky Email...

21/05/2026

Apple TV to Capture MLS Game Entirely on iPhone 17 Pro

Share Copy link Facebook X Linkedin Bluesky Email...

21/05/2026

MPTS 2026 Draws Record Numbers for Landmark 10th Annivers...

The UK's leading event for the creative industries united thousands of professionals for two days of networking, debate, industry insight and getting hands-...

21/05/2026

Recreating Doug Trumbull's Slitscan VFX - After Effects Mastery

Recreating Doug Trumbull's Slitscan VFX - After Effects Mastery Graham Quince May 21, 2026 0 Comments In this After Effects tutorial, I'm divi...

21/05/2026

Cavalry: An Array of Fun Stuff

Cavalry: An Array of Fun Stuff Simon Ubsdell May 21, 2026 0 Comments Arrays are a really powerful feature of Cavalry and here we'll go over some o...

21/05/2026

Chyron Launches PRIME Scorebug

Share Copy link Facebook X Linkedin Bluesky Email...