Sony Pixel Power calrec Sony

Fast, Low-Cost Inference Offers Key to Profitable AI

23/01/2025

Businesses across every industry are rolling out AI services this year. For Microsoft, Oracle, Perplexity, Snap and hundreds of other leading companies, using the NVIDIA AI inference platform - a full stack comprising world-class silicon, systems and software - is the key to delivering high-throughput and low-latency inference and enabling great user experiences while lowering cost.

NVIDIA's advancements in inference software optimization and the NVIDIA Hopper platform are helping industries serve the latest generative AI models, delivering excellent user experiences while optimizing total cost of ownership. The Hopper platform also helps deliver up to 15x more energy efficiency for inference workloads compared to previous generations.

AI inference is notoriously difficult, as it requires many steps to strike the right balance between throughput and user experience.

But the underlying goal is simple: generate more tokens at a lower cost. Tokens represent words in a large language model (LLM) system - and with AI inference services typically charging for every million tokens generated, this goal offers the most visible return on AI investments and energy used per task.

Full-stack software optimization offers the key to improving AI inference performance and achieving this goal.

Cost-Effective User Throughput Businesses are often challenged with balancing the performance and costs of inference workloads. While some customers or use cases may work with an out-of-the-box or hosted model, others may require customization. NVIDIA technologies simplify model deployment while optimizing cost and performance for AI inference workloads. In addition, customers can experience flexibility and customizability with the models they choose to deploy.

NVIDIA NIM microservices, NVIDIA Triton Inference Server and the NVIDIA TensorRT library are among the inference solutions NVIDIA offers to suit users' needs:

NVIDIA NIM inference microservices are prepackaged and performance-optimized for rapidly deploying AI foundation models on any infrastructure - cloud, data centers, edge or workstations.

NVIDIA Triton Inference Server, one of the company's most popular open-source projects, allows users to package and serve any model regardless of the AI framework it was trained on.

NVIDIA TensorRT is a high-performance deep learning inference library that includes runtime and model optimizations to deliver low-latency and high-throughput inference for production applications.

Available in all major cloud marketplaces, the NVIDIA AI Enterprise software platform includes all these solutions and provides enterprise-grade support, stability, manageability and security.

With the framework-agnostic NVIDIA AI inference platform, companies save on productivity, development, and infrastructure and setup costs. Using NVIDIA technologies can also boost business revenue by helping companies avoid downtime and fraudulent transactions, increase e-commerce shopping conversion rates and generate new, AI-powered revenue streams.

Cloud-Based LLM Inference To ease LLM deployment, NVIDIA has collaborated closely with every major cloud service provider to ensure that the NVIDIA inference platform can be seamlessly deployed in the cloud with minimal or no code required. NVIDIA NIM is integrated with cloud-native services such as:

Amazon SageMaker AI, Amazon Bedrock Marketplace, Amazon Elastic Kubernetes Service

Google Cloud's Vertex AI, Google Kubernetes Engine

Microsoft Azure AI Foundry coming soon, Azure Kubernetes Service

Oracle Cloud Infrastructure's data science tools, Oracle Cloud Infrastructure Kubernetes Engine

Plus, for customized inference deployments, NVIDIA Triton Inference Server is deeply integrated into all major cloud service providers.

For example, using the OCI Data Science platform, deploying NVIDIA Triton is as simple as turning on a switch in the command line arguments during model deployment, which instantly launches an NVIDIA Triton inference endpoint.

Similarly, with Azure Machine Learning, users can deploy NVIDIA Triton either with no-code deployment through the Azure Machine Learning Studio or full-code deployment with Azure Machine Learning CLI. AWS provides one-click deployment for NVIDIA NIM from SageMaker Marketplace and Google Cloud provides a one-click deployment option on Google Kubernetes Engine (GKE). Google Cloud provides a one-click deployment option on Google Kubernetes Engine, while AWS offers NVIDIA Triton on its AWS Deep Learning containers.

The NVIDIA AI inference platform also uses popular communication methods for delivering AI predictions, automatically adjusting to accommodate the growing and changing needs of users within a cloud-based infrastructure.

From accelerating LLMs to enhancing creative workflows and transforming agreement management, NVIDIA's AI inference platform is driving real-world impact across industries. Learn how collaboration and innovation are enabling the organizations below to achieve new levels of efficiency and scalability.

Serving 400 Million Search Queries Monthly With Perplexity AI Perplexity AI, an AI-powered search engine, handles over 435 million monthly queries. Each query represents multiple AI inference requests. To meet this demand, the Perplexity AI team turned to NVIDIA H100 GPUs, Triton Inference Server and TensorRT-LLM.

Supporting over 20 AI models, including Llama 3 variations like 8B and 70B, Perplexity processes diverse tasks such as search, summarization and question-answering. By using smaller classifier models to route tasks to GPU pods, managed by NVIDIA Triton, the company delivers cost-efficient, responsive service under strict service level agreements.

Through model parallelism, which splits LLMs across GPUs, Perplexity achieved a threefold cost reduction while maintaining low latency and high accurac
LINK: https://blogs.nvidia.com/blog/ai-inference-platform/...
See more stories from nvidia

North America Stories

04/07/2025

Give Me the Backstory: Get to Know Tadashi Nakamura, the Filmmaker Behind Third Act

By Lucy Spicer Editor's note: We are deeply saddened by the passing of Robe...

04/07/2025

Introducing CVP Auctions: Trusted Tech, At Auction-Only Prices

Introducing CVP Auctions: Trusted Tech, At Auction-Only Prices Brie Clayton July 4, 2025 0 Comments A New Destination for Film and Broadcast Gear at U...

04/07/2025

Unveiling China's Ecological Wonders with URSA Mini Pro 12K

Unveiling China's Ecological Wonders with URSA Mini Pro 12K Brie Clayton July 4, 2025 0 Comments Nature Documentaries Freeze Epic Momentsin Extrem...

04/07/2025

Create Any Planet in After Effects

Create Any Planet in After Effects Graham Quince July 4, 2025 0 Comments The Advanced 3D renderer in After Effects makes it really easy to import 3D o...

04/07/2025

Custom Gradient Colorize for Resolve Fusion

Custom Gradient Colorize for Resolve Fusion Simon Ubsdell July 4, 2025 0 Comments Digging into the theory of the gradient colorize process to build ou...

04/07/2025

NESN Gives Red Sox-Reds Matchup a Retro Makeover

Watching live sports on TV these days is a delicate balance for broadcasters how much information is too much?...

04/07/2025

FCC Announces Tentative Agenda for July Open Meeting

WASHINGTON As Federal Communications Commission Chair Brendan Carr works to achieve his goal of a very busy, productive summer at the FCC, the agency has rele...

04/07/2025

Other World Computing Launches OWC Guardian USB-C Hardware-Encrypted Portable SSD

Other World Computing Launches OWC Guardian USB-C Hardware-Encrypted Portable SS...

04/07/2025

CANAL+ Brings MotoGP to Vision Pro with Blackmagic URSA Cine Immersive

CANAL Brings MotoGP to Vision Pro with Blackmagic URSA Cine Immersive Brie Clayton July 3, 2025 0 Comments Experience MotoGP like never before as CA...

03/07/2025

Advanced Communication Systems Unleash a Tactical Advantage in Modern Combat Operations

To achieve a true level of decisive overmatch, communication systems must be as ...

03/07/2025

Paramount's Ray Hopkins Joins Broadcasters Foundation Board

The Broadcasters Foundation of America has named Ray Hopkins, president of U.S. distribution at Paramount Global, to its board of directors....

03/07/2025

FCC Chair Carr Lays Out Infrastructure Policy Agenda

SIOUX FALLS, S.D. In his first major policy speech, Federal Communications Commission chair Brendan Carr outlined his Build America Agenda for the agency'...

03/07/2025

TV Tech's Top NextGen TV Stories of 2025

As the industry is making a major push to transition to NextGen TV and develop new businesses that will fund the next generation of broadcasting, TV Tech's ...

03/07/2025

Netflix Rolls Out AV1 Film Grain Synthesis Streams for Film Content

Despite enabling its subscribers to watch films and TV shows in the highest possible resolution, Netflix has revealed it is taking things old school with the in...

03/07/2025

FCC's Gomez, Sen. Markey, Critics Blast Paramount-Trump Settlement

WASHINGTON In response to news that Paramount Global has agreed to pay $16 million to settle a lawsuit by President Donald Trump, Democrats and other administra...

03/07/2025

Riedel Communications Unveils RefSuite Sports Ecosystem

WUPPERTAL, Germany Riedel Communications today launched RefSuite for professional sports workflows....

03/07/2025

LYNX Technik Appoints New Executive Leadership Team

WEITERSTADT, Germany LYNX Technik, a provider of modular signal processing solutions, has announced that founder and longtime CEO Winfried Deckelmann is steppin...

03/07/2025

CTA, NCTA, LPTVBA Meet With FCC to Oppose NAB's ATSC 3.0 Petition

WASHINGTON Representatives from six trade groups representing tech, pay TV, broadband, LPTV and other sectors recently met with Federal Communications Commissio...

03/07/2025

Dan Whealy Acquires Heartland Video Systems

PLYMOUTH, Wis. Dan Whealy has acquired Heartland Video Systems and will be its new owner and president, the systems integrator said....

03/07/2025

Filmmaker Jay Holben Releases The Directors Guide to Ever...

Acclaimed filmmaker, educator, and industry veteran Jay Holben has released his most ambitious work to date: The Director's Guide to Everything a sweeping...

03/07/2025

Capturing the Moment - Pruski Photographs Dyke Day LA wit...

For Los Angeles-based photographer Scout Pruski, documenting the annual Dyke Day LA is a way to honor and give back to the city's legacy queer community and...

03/07/2025

New Reinvented Chimera V-Flat Panel Frame Kit

Modular, Travel-Ready Solution Matthews Studio Equipment launches the new Chimera V-Flat Panel Frame Kit, the lightweight and modular, ultra-durable alternativ...

03/07/2025

Dalet Flex LTS Release Elevates Content Workflows with Ne...

Dalet, a leading technology and service provider for media-rich organizations, today announced the release of Dalet Flex 2025.6 LTS (long-term support), introdu...

03/07/2025

Lightware launches LARA Wizard to help integrators simpli...

Lightware's LARA Wizard makes it quicker and easier than ever to set up audio-visual (AV) room automation and control configurations, ensuring anyone, from ...

03/07/2025

Alfalite obtains NCAGE and TAA certifications strengtheni...

Alfalite, the only European manufacturer of LED screens, has obtained both the NCAGE Code (NATO Commercial and Government Entity) and TAA (Trade Agreements Act)...

03/07/2025

WIN streamlines complex live production with nxtedition

nxtedition is implementing a new solution for WIN Network, based in its headquarters in Wollongong, New South Wales, Australia. The nxtedition integrated platfo...

03/07/2025

ORS Group and Insys Video Technologies unite under one br...

ORS Group and Insys Video Technologies today announce the launch of a new joint brand identity: Big Blue Marble. The move reflects the evolution of a long-stand...

03/07/2025

Evergent Proves Cross-Industry Power of AI for Japanese G...

Evergent, a global leader in customer lifecycle management and monetization solutions for subscription-based businesses, today announced the successful results ...

03/07/2025

LYNX Technik Appoints New Executive Leadership Team to Dr...

LYNX Technik, provider of modular signal processing solutions today announces an executive leadership transition, effective July 1, 2025. Markus Motzko has been...

03/07/2025

Chaos Releases V-Ray for Blender

Chaos Releases V-Ray for Blender Brie Clayton July 2, 2025 0 Comments Production-Proven Renderer V-Ray Arrives Natively in Blender for the First Time;...

03/07/2025

Calvin University Adds Blackmagic Replay to Broadcasts and Streams

Calvin University Adds Blackmagic Replay to Broadcasts and Streams Brie Clayton July 2, 2025 0 Comments Replay workflow joins 14 Blackmagic Design cam...

03/07/2025

Berklee's Gospel Performance Program Welcomes Jonathan Nelson as the Herb Alpert Visiting Artist

Berklee's Gospel Performance Program Welcomes Jonathan Nelson as the Herb Al...

03/07/2025

Get Ready for Even More Action! Netflix Has Confirmed the Third Season of the Brazilian Original Series 'Criminal Code'

Back to All News Get Ready for Even More Action! Netflix Has Confirmed the Thir...

03/07/2025

Get ready for even more action! Netflix has confirmed the third season of the Brazilian original series Criminal Code

Back to All News Get ready for even more action! Netflix has confirmed the thir...

03/07/2025

Centralized Cross-Platform Campaigns and Unified Sales Workflows with WO Fusion

In today's fragmented media landscape, advertisers expect more than just reach - they want seamless cross-platform campaigns that deliver measurable results...

03/07/2025

GeForce NOW's 20 July Games Bring the Heat to the Cloud

The forecast this month is showing a 100% chance of epic gaming. Catch the scorching lineup of 20 titles coming to the cloud, which gamers can play whether indo...

02/07/2025

Release Rundown: What to Watch in July, From FOLKTALES to Oh, Hi!

Hege Wik and Odin appear in FOLKTALES by Heidi Ewing and Rachel Grady, an official selection of the 2025 Sundance Film Festival. Courtesy of Sundance Institut...

02/07/2025

IGUALADA Beats the Odds in the Ultimate Underdog Tale

(Stephen Lovekin/Shutterstock for Sundance Film Festival) By Patty Consolazio This one's for any person who's ever felt put down, left out, marginaliz...

02/07/2025

Paramount Settles Trump Lawsuit for $16 Million

NEW YORK Paramount Global has agreed to pay $16 million to settle a lawsuit by President Donald Trump, ending a long-running legal saga that could clear the way...

02/07/2025

FCC Approves Sale of Sinclair Stations to Rincon

WASHINGTON The Federal Communications Commission has approved the transfer of licenses for five TV stations from Sinclair to Rincon Broadcasting, which announce...

02/07/2025

TV Tech's Top Stories of 2025

As the media industry faces an uncertain business climate and rapidly changing consumer habits, TV Tech's most popular stories from the first six months of ...

02/07/2025

Average U.S. Home Now Has Two Smart TVs

PORTSMOUTH, N.H. New data shows that U.S. homes now own an average of two smart TVs, with Samsung and LG dominating the market but Roku and FireTV continue to g...

02/07/2025

Oaktree Acquires FilmRise

LOS ANGELES Funds managed by Oaktree Capital Management have acquired FilmRise, an independent digital streaming content distributor, and merged it with Shout! ...

02/07/2025

ATSC Names Luiz Fausto Vice President of Standards Development

WASHINGTON The Advanced Television Systems Committee named Luiz Fausto vice president of standards development, succeeding ATSC vice president Jerry Whitaker, w...

02/07/2025

Boston Conservatory and the Sphinx Organization Join Forces to Make Classical Education Accessible

Boston Conservatory and the Sphinx Organization Join Forces to Make Classical Ed...

02/07/2025

World RX of Sweden: Aurora's Matt Beal on Taking Up the Broadcast Reins Alongside the FIA

Progressive realism: Aurora on taking up the broadcast reins for World RX alongs...

02/07/2025

World RX of Sweden: Getting the Production Together With NEP Finland, Cloudbass Graphics

Great speed: New host broadcaster Aurora on getting the World RX 2025 season pro...

02/07/2025

World RX of Sweden: Host Broadcaster Aurora Looks Ahead to This Weekend's Second Race of the Season in Hljes

Seat of its pants: Aurora boxes clever and looks ahead for the FIA as new host b...

02/07/2025

UEFA Women's Euro 2025: EBU, BBC Sport, Sunset+Vine Talk Coverage Plans, Distribution and Maintaining Momentum

UEFA Women's Euro 2025: EBU, BBC Sport and Sunset Vine talk coverage plans, ...