
Just as there are widely understood empirical laws of nature - for example, what goes up must come down, or every action has an equal and opposite reaction - the field of AI was long defined by a single idea: that more compute, more training data and more parameters makes a better AI model.
However, AI has since grown to need three distinct laws that describe how applying compute resources in different ways impacts model performance. Together, these AI scaling laws - pretraining scaling, post-training scaling and test-time scaling, also called long thinking - reflect how the field has evolved with techniques to use additional compute in a wide variety of increasingly complex AI use cases.
The recent rise of test-time scaling - applying more compute at inference time to improve accuracy - has enabled AI reasoning models, a new class of large language models (LLMs) that perform multiple inference passes to work through complex problems, while describing the steps required to solve a task. Test-time scaling requires intensive amounts of computational resources to support AI reasoning, which will drive further demand for accelerated computing.
What Is Pretraining Scaling? Pretraining scaling is the original law of AI development. It demonstrated that by increasing training dataset size, model parameter count and computational resources, developers could expect predictable improvements in model intelligence and accuracy.
Each of these three elements - data, model size, compute - is interrelated. Per the pretraining scaling law, outlined in this research paper, when larger models are fed with more data, the overall performance of the models improves. To make this feasible, developers must scale up their compute - creating the need for powerful accelerated computing resources to run those larger training workloads.
This principle of pretraining scaling led to large models that achieved groundbreaking capabilities. It also spurred major innovations in model architecture, including the rise of billion- and trillion-parameter transformer models, mixture of experts models and new distributed training techniques - all demanding significant compute.
And the relevance of the pretraining scaling law continues - as humans continue to produce growing amounts of multimodal data, this trove of text, images, audio, video and sensor information will be used to train powerful future AI models.
Pretraining scaling is the foundational principle of AI development, linking the size of models, datasets and compute to AI gains. Mixture of experts, depicted above, is a popular model architecture for AI training. What Is Post-Training Scaling? Pretraining a large foundation model isn't for everyone - it takes significant investment, skilled experts and datasets. But once an organization pretrains and releases a model, they lower the barrier to AI adoption by enabling others to use their pretrained model as a foundation to adapt for their own applications.
This post-training process drives additional cumulative demand for accelerated computing across enterprises and the broader developer community. Popular open-source models can have hundreds or thousands of derivative models, trained across numerous domains.
Developing this ecosystem of derivative models for a variety of use cases could take around 30x more compute than pretraining the original foundation model.
Developing this ecosystem of derivative models for a variety of use cases could take around 30x more compute than pretraining the original foundation model.
Post-training techniques can further improve a model's specificity and relevance for an organization's desired use case. While pretraining is like sending an AI model to school to learn foundational skills, post-training enhances the model with skills applicable to its intended job. An LLM, for example, could be post-trained to tackle a task like sentiment analysis or translation - or understand the jargon of a specific domain, like healthcare or law.
The post-training scaling law posits that a pretrained model's performance can further improve - in computational efficiency, accuracy or domain specificity - using techniques including fine-tuning, pruning, quantization, distillation, reinforcement learning and synthetic data augmentation.
Fine-tuning uses additional training data to tailor an AI model for specific domains and applications. This can be done using an organization's internal datasets, or with pairs of sample model input and outputs.
Distillation requires a pair of AI models: a large, complex teacher model and a lightweight student model. In the most common distillation technique, called offline distillation, the student model learns to mimic the outputs of a pretrained teacher model.
Reinforcement learning, or RL, is a machine learning technique that uses a reward model to train an agent to make decisions that align with a specific use case. The agent aims to make decisions that maximize cumulative rewards over time as it interacts with an environment - for example, a chatbot LLM that is positively reinforced by thumbs up reactions from users. This technique is known as reinforcement learning from human feedback (RLHF). Another, newer technique, reinforcement learning from AI feedback (RLAIF), instead uses feedback from AI models to guide the learning process, streamlining post-training efforts.
Best-of-n sampling generates multiple outputs from a language model and selects the one with the highest reward score based on a reward model. It's often used to improve an AI's outputs without modifying model parameters, offering an alternative to fine-tuning with reinforcement learning.
Search methods explore a range of potential decision paths before selecting a final output. This post-training technique can iteratively improve the model's responses
More from Nvidia
08/05/2025
Artificial intelligence is helping identify and treat diseases faster with better results for humankind. Natural disasters like wildfires are next.
Fires in th...
08/05/2025
Calling all wiseguys - 2K's acclaimed Mafia franchise is available to stream...
08/05/2025
As AI use cases continue to expand - from document summarization to custom software agents - developers and enthusiasts are seeking faster, more flexible ways t...
07/05/2025
A new supercomputer offered by Cadence, a leading provider of technology for ele...
07/05/2025
Enterprises across industries are exploring AI to rethink problem-solving and re...
02/05/2025
With graduation season approaching, a new cohort of students is embarking on next steps, aiming to use their passions and skills to make a real, tangible impact...
01/05/2025
May brings more than just rainbows and sunshine - it's also time for fresh adventures and epic battles. This GFN Thursday spotlights 20 can't-miss games...
01/05/2025
For Nicolas Simon, advancing the field of robotics is a personal mission that could change his siblings' lives.
Two-thirds of Simon's family members us...
30/04/2025
The quick-service restaurant (QSR) industry is being reinvented by AI.
For exam...
30/04/2025
AI-powered image generation has progressed at a remarkable pace - from early exa...
28/04/2025
As enterprises increasingly adopt AI, securing AI factories - where complex, agentic workflows are executed - has never been more critical.
NVIDIA is bringing ...
28/04/2025
Agentic AI is redefining the cybersecurity landscape - introducing new opportunities that demand rethinking how to secure AI while offering the keys to addressi...
28/04/2025
Oracle has stood up and optimized its first wave of liquid-cooled NVIDIA GB200 N...
24/04/2025
Advancing AI requires a full-stack approach, with a powerful foundation of computing infrastructure - including accelerated processors and networking technologi...
24/04/2025
Get the controllers ready and clear the calendar - it's a jam-packed GFN Thu...
23/04/2025
As AI models evolve and adoption grows, enterprises must perform a delicate balancing act to achieve maximum value.
That's because inference - the process ...
23/04/2025
Financial services has long been at the forefront of adopting technological innovations. Today, generative AI and agentic systems are redefining the industry, f...
23/04/2025
AI is rapidly reshaping what's possible on a PC - whether for real-time image generation or voice-controlled workflows. As AI capabilities grow, so does the...
23/04/2025
An AI agent is only as accurate, relevant and timely as the data that powers it....
22/04/2025
Whether at sea, land or in the sky - even outer space - NVIDIA technology is helping research scientists and developers alike explore and understand oceans, wil...
22/04/2025
Traditionally, data centers have relied on air cooling - where mechanical chillers circulate chilled air to absorb heat from servers, helping them maintain opti...
22/04/2025
About 15% of the world's population - over a billion people - are affected by neurological disorders, from commonly known diseases like Alzheimer's and ...
17/04/2025
Gold prospecting in Venezuela has led to a malaria resurgence, but researchers h...
17/04/2025
As the days grow longer and the flowers bloom, GFN Thursday brings a fresh lineup of games to brighten the week.
Dive into thrilling hunts and dark fantasy adv...
16/04/2025
Isomorphic Labs is reimagining the drug discovery process with an AI-first approach. At the heart of this work is a new way of thinking about biology.
Max Jade...
16/04/2025
Editor's note: This post is part of Into the Omniverse, a series focused on how developers, 3D practitioners, and enterprises can transform their workflows ...
15/04/2025
The final days of the AI Mathematical Olympiad's latest competition were a transcontinental relay for team NVIDIA.
Every evening, two team members on oppos...
15/04/2025
Every company and country wants to grow and create economic opportunity - but they need virtually limitless intelligence to do so. Working with its ecosystem pa...
15/04/2025
CoreWeave today became one of the first cloud providers to bring NVIDIA GB200 NV...
14/04/2025
NVIDIA is working with its manufacturing partners to design and build factories that, for the first time, will produce NVIDIA AI supercomputers entirely in the ...
11/04/2025
As a teenager, Bradley Rothenberg was obsessed with CAD: computer-aided design software.
Before he turned 30, Rothenberg channeled that interest into building ...
10/04/2025
NVIDIA this week recognized the contributions of partners in Europe, the Middle ...
10/04/2025
As AI-powered tools continue to evolve, NVIDIA GeForce RTX 50 Series and NVIDIA ...
10/04/2025
Get ready to explore the Deep South. South of Midnight, the action-adventure gam...
09/04/2025
Necessity is the mother of invention. And sometimes, what a person really needs ...
09/04/2025
NVIDIA is collaborating with Google Cloud to bring agentic AI to enterprises seeking to locally harness the Google Gemini family of AI models using the NVIDIA B...
06/04/2025
This National Robotics Week, NVIDIA highlighted the pioneering technologies that...
03/04/2025
The Nintendo Switch 2, unveiled April 2, takes performance to the next level, powered by a custom NVIDIA processor featuring an NVIDIA GPU with dedicated RT Cor...
03/04/2025
Real-time AI is unlocking new possibilities in media and entertainment, improving viewer engagement and advancing intelligent content creation.
At NAB Show, a...
03/04/2025
Editor's note: This post is part of the AI On blog series, which explores the latest techniques and real-world applications of agentic AI, chatbots and copi...
03/04/2025
GeForce NOW isn't fooling around.
This month, 21 games are joining the cloud gaming library of over 2,000 titles. Whether chasing epic adventures, testing ...
02/04/2025
In the latest MLPerf Inference V5.0 benchmarks, which reflect some of the most c...
02/04/2025
AI is rapidly transforming how organizations solve complex challenges.
The early stages of enterprise AI adoption focused on using large language models to cre...
02/04/2025
Video editing workflows are getting a lot more colorful.
Adobe recently announc...
31/03/2025
Advances in physical AI are enabling organizations to embrace embodied AI across...
27/03/2025
A new resident is moving into the cloud - KRAFTON's inZOI joins the 2,000+ games in the GeForce NOW cloud gaming library.
Plus, members can get ready for a...
26/03/2025
The reliability of the electric grid is critical.
From handling demand surges and evolving power needs to preventing infrastructure failures that can cause wil...
25/03/2025
Generative AI is unlocking new capabilities for PCs and workstations, including ...
20/03/2025
The power and utilities sector keeps the lights on for the world's populatio...
20/03/2025
Time to sharpen the blade. GeForce NOW brings a legendary addition to the cloud: Ubisoft's highly anticipated Assassin's Creed Shadows is now available ...