
As AI models evolve and adoption grows, enterprises must perform a delicate balancing act to achieve maximum value.
That's because inference - the process of running data through a model to get an output - offers a different computational challenge than training a model.
Pretraining a model - the process of ingesting data, breaking it down into tokens and finding patterns - is essentially a one-time cost. But in inference, every prompt to a model generates tokens, each of which incur a cost.
That means that as AI model performance and use increases, so do the amount of tokens generated and their associated computational costs. For companies looking to build AI capabilities, the key is generating as many tokens as possible - with maximum speed, accuracy and quality of service - without sending computational costs skyrocketing.
As such, the AI ecosystem has been working to make inference cheaper and more efficient. Inference costs have been trending down for the past year thanks to major leaps in model optimization, leading to increasingly advanced, energy-efficient accelerated computing infrastructure and full-stack solutions.
According to the Stanford University Institute for Human-Centered AI's 2025 AI Index Report, the inference cost for a system performing at the level of GPT-3.5 dropped over 280-fold between November 2022 and October 2024. At the hardware level, costs have declined by 30% annually, while energy efficiency has improved by 40% each year. Open-weight models are also closing the gap with closed models, reducing the performance difference from 8% to just 1.7% on some benchmarks in a single year. Together, these trends are rapidly lowering the barriers to advanced AI.
As models evolve and generate more demand and create more tokens, enterprises need to scale their accelerated computing resources to deliver the next generation of AI reasoning tools or risk rising costs and energy consumption.
What follows is a primer to understand the concepts of the economics of inference, enterprises can position themselves to achieve efficient, cost-effective and profitable AI solutions at scale.
Key Terminology for the Economics of AI Inference Knowing key terms of the economics of inference helps set the foundation for understanding its importance.
Tokens are the fundamental unit of data in an AI model. They're derived from data during training as text, images, audio clips and videos. Through a process called tokenization, each piece of data is broken down into smaller constituent units. During training, the model learns the relationships between tokens so it can perform inference and generate an accurate, relevant output.
Throughput refers to the amount of data - typically measured in tokens - that the model can output in a specific amount of time, which itself is a function of the infrastructure running the model. Throughput is often measured in tokens per second, with higher throughput meaning greater return on infrastructure.
Latency is a measure of the amount of time between inputting a prompt and the start of the model's response. Lower latency means faster responses. The two main ways of measuring latency are:
Time to First Token: A measurement of the initial processing time required by the model to generate its first output token after a user prompt.
Time per Output Token: The average time between consecutive tokens - or the time it takes to generate a completion token for each user querying the model at the same time. It's also known as inter-token latency or token-to-token latency.
Time to first token and time per output token are helpful benchmarks, but they're just two pieces of a larger equation. Focusing solely on them can still lead to a deterioration of performance or cost.
To account for other interdependencies, IT leaders are starting to measure goodput, which is defined as the throughput achieved by a system while maintaining target time to first token and time per output token levels. This metric allows organizations to evaluate performance in a more holistic manner, ensuring that throughput, latency and cost are aligned to support both operational efficiency and an exceptional user experience.
Energy efficiency is the measure of how effectively an AI system converts power into computational output, expressed as performance per watt. By using accelerated computing platforms, organizations can maximize tokens per watt while minimizing energy consumption.
How the Scaling Laws Apply to Inference Cost The three AI scaling laws are also core to understanding the economics of inference:
Pretraining scaling: The original scaling law that demonstrated that by increasing training dataset size, model parameter count and computational resources, models can achieve predictable improvements in intelligence and accuracy.
Post-training: A process where models are fine-tuned for accuracy and specificity so they can be applied to application development. Techniques like retrieval-augmented generation can be used to return more relevant answers from an enterprise database.
Test-time scaling (aka long thinking or reasoning ): A technique by which models allocate additional computational resources during inference to evaluate multiple possible outcomes before arriving at the best answer.
While AI is evolving and post-training and test-time scaling techniques become more sophisticated, pretraining isn't disappearing and remains an important way to scale models. Pretraining will still be needed to support post-training and test-time scaling.
Profitable AI Takes a Full-Stack Approach In comparison to inference from a model that's only gone through pretraining and post-training, models that harness test-time scaling generate multiple tokens to solve a complex problem. This results in more accurate and relevant model outputs - but
More from Nvidia
22/05/2025
GeForce NOW is turning up the heat this summer with a hot new deal. For a limited time, save 40% on six-month Performance memberships and enjoy premium GeForce ...
21/05/2025
As robots increasingly make their way to the largest enterprises' manufacturing plants and warehouses, the need for access to critical business and operatio...
20/05/2025
Industrial AI is transforming how factories operate, innovate and scale.
The convergence of AI, simulation and digital twins is poised to unlock new levels of ...
19/05/2025
Agentic AI is redefining scientific discovery and unlocking research breakthroughs and innovations across industries. Through deepened collaboration, NVIDIA and...
19/05/2025
Across robot training and development, NVIDIA Research is uncovering breakthroughs in areas such as multimodal generative AI and synthetic data generation.
The...
19/05/2025
Generative AI is transforming PC software into breakthrough experiences - from digital humans to writing assistants, intelligent agents and creative tools.
NVI...
18/05/2025
Electricity. The Internet. Now it's time for another major technology, AI, to sweep the globe.
NVIDIA founder and CEO Jensen Huang took the stage at a pack...
18/05/2025
Empowering engineering teams with more tools for building AI factories, NVIDIA t...
18/05/2025
The age of video analytics AI agents is here.
Video is one of the defining feat...
18/05/2025
TSMC, Cadence, KLA, Siemens and Synopsys are advancing semiconductor manufacturi...
18/05/2025
NVIDIA is highlighting significant momentum for its new Grace CPU C1 this week at the COMPUTEX trade show in Taipei, with a strong showing of support from key o...
18/05/2025
Researchers across Taiwan are tackling complex challenges in AI development, climate science and quantum computing. Their work will soon be boosted by a new sup...
18/05/2025
Leading healthcare organizations across the globe are using agentic AI, robotics...
18/05/2025
Quantum computing promises to shorten the path to solving some of the world'...
15/05/2025
Editor's note: This post is part of Into the Omniverse, a series focused on ...
15/05/2025
AI is creating value for everyone - from researchers in drug discovery to quantitative analysts navigating financial market changes.
The faster an AI system ca...
15/05/2025
Steel clashes and war drums thunder as a new age of battle dawns - one that will test even the mightiest Slayer.
This GFN Thursday, DOOM: The Dark Ages - the b...
14/05/2025
Think tap to pay - but smarter and safer. Visa is tapping into AI to enhance services for its global network of customers, focused on fraud prevention, personal...
14/05/2025
Electronic music icon Don Diablo is known for pushing the boundaries of music, v...
13/05/2025
As the manufacturing industry faces challenges - such as labor shortages, reshor...
13/05/2025
Editor's note: This post is part of the AI On blog series, which explores the latest techniques and real-world applications of agentic AI, chatbots and copi...
12/05/2025
NVIDIA today received multiple accolades at COMPUTEX's Best Choice Awards, in recognition of innovation across the company.
The NVIDIA GeForce RTX 5090 GPU...
08/05/2025
Artificial intelligence is helping identify and treat diseases faster with better results for humankind. Natural disasters like wildfires are next.
Fires in th...
08/05/2025
Calling all wiseguys - 2K's acclaimed Mafia franchise is available to stream...
08/05/2025
As AI use cases continue to expand - from document summarization to custom software agents - developers and enthusiasts are seeking faster, more flexible ways t...
07/05/2025
A new supercomputer offered by Cadence, a leading provider of technology for ele...
07/05/2025
Enterprises across industries are exploring AI to rethink problem-solving and re...
02/05/2025
With graduation season approaching, a new cohort of students is embarking on next steps, aiming to use their passions and skills to make a real, tangible impact...
01/05/2025
May brings more than just rainbows and sunshine - it's also time for fresh adventures and epic battles. This GFN Thursday spotlights 20 can't-miss games...
01/05/2025
For Nicolas Simon, advancing the field of robotics is a personal mission that could change his siblings' lives.
Two-thirds of Simon's family members us...
30/04/2025
The quick-service restaurant (QSR) industry is being reinvented by AI.
For exam...
30/04/2025
AI-powered image generation has progressed at a remarkable pace - from early exa...
28/04/2025
As enterprises increasingly adopt AI, securing AI factories - where complex, agentic workflows are executed - has never been more critical.
NVIDIA is bringing ...
28/04/2025
Agentic AI is redefining the cybersecurity landscape - introducing new opportunities that demand rethinking how to secure AI while offering the keys to addressi...
28/04/2025
Oracle has stood up and optimized its first wave of liquid-cooled NVIDIA GB200 N...
24/04/2025
Advancing AI requires a full-stack approach, with a powerful foundation of computing infrastructure - including accelerated processors and networking technologi...
24/04/2025
Get the controllers ready and clear the calendar - it's a jam-packed GFN Thu...
23/04/2025
As AI models evolve and adoption grows, enterprises must perform a delicate balancing act to achieve maximum value.
That's because inference - the process ...
23/04/2025
Financial services has long been at the forefront of adopting technological innovations. Today, generative AI and agentic systems are redefining the industry, f...
23/04/2025
AI is rapidly reshaping what's possible on a PC - whether for real-time image generation or voice-controlled workflows. As AI capabilities grow, so does the...
23/04/2025
An AI agent is only as accurate, relevant and timely as the data that powers it....
22/04/2025
Whether at sea, land or in the sky - even outer space - NVIDIA technology is helping research scientists and developers alike explore and understand oceans, wil...
22/04/2025
Traditionally, data centers have relied on air cooling - where mechanical chillers circulate chilled air to absorb heat from servers, helping them maintain opti...
22/04/2025
About 15% of the world's population - over a billion people - are affected by neurological disorders, from commonly known diseases like Alzheimer's and ...
17/04/2025
Gold prospecting in Venezuela has led to a malaria resurgence, but researchers h...
17/04/2025
As the days grow longer and the flowers bloom, GFN Thursday brings a fresh lineup of games to brighten the week.
Dive into thrilling hunts and dark fantasy adv...
16/04/2025
Isomorphic Labs is reimagining the drug discovery process with an AI-first approach. At the heart of this work is a new way of thinking about biology.
Max Jade...
16/04/2025
Editor's note: This post is part of Into the Omniverse, a series focused on how developers, 3D practitioners, and enterprises can transform their workflows ...
15/04/2025
The final days of the AI Mathematical Olympiad's latest competition were a transcontinental relay for team NVIDIA.
Every evening, two team members on oppos...
15/04/2025
Every company and country wants to grow and create economic opportunity - but they need virtually limitless intelligence to do so. Working with its ecosystem pa...