Sony Pixel Power calrec Sony

How Scaling Laws Drive Smarter, More Powerful AI

12/02/2025

Just as there are widely understood empirical laws of nature - for example, what goes up must come down, or every action has an equal and opposite reaction - the field of AI was long defined by a single idea: that more compute, more training data and more parameters makes a better AI model.

However, AI has since grown to need three distinct laws that describe how applying compute resources in different ways impacts model performance. Together, these AI scaling laws - pretraining scaling, post-training scaling and test-time scaling, also called long thinking - reflect how the field has evolved with techniques to use additional compute in a wide variety of increasingly complex AI use cases.

The recent rise of test-time scaling - applying more compute at inference time to improve accuracy - has enabled AI reasoning models, a new class of large language models (LLMs) that perform multiple inference passes to work through complex problems, while describing the steps required to solve a task. Test-time scaling requires intensive amounts of computational resources to support AI reasoning, which will drive further demand for accelerated computing.

What Is Pretraining Scaling? Pretraining scaling is the original law of AI development. It demonstrated that by increasing training dataset size, model parameter count and computational resources, developers could expect predictable improvements in model intelligence and accuracy.

Each of these three elements - data, model size, compute - is interrelated. Per the pretraining scaling law, outlined in this research paper, when larger models are fed with more data, the overall performance of the models improves. To make this feasible, developers must scale up their compute - creating the need for powerful accelerated computing resources to run those larger training workloads.

This principle of pretraining scaling led to large models that achieved groundbreaking capabilities. It also spurred major innovations in model architecture, including the rise of billion- and trillion-parameter transformer models, mixture of experts models and new distributed training techniques - all demanding significant compute.

And the relevance of the pretraining scaling law continues - as humans continue to produce growing amounts of multimodal data, this trove of text, images, audio, video and sensor information will be used to train powerful future AI models.

Pretraining scaling is the foundational principle of AI development, linking the size of models, datasets and compute to AI gains. Mixture of experts, depicted above, is a popular model architecture for AI training. What Is Post-Training Scaling? Pretraining a large foundation model isn't for everyone - it takes significant investment, skilled experts and datasets. But once an organization pretrains and releases a model, they lower the barrier to AI adoption by enabling others to use their pretrained model as a foundation to adapt for their own applications.

This post-training process drives additional cumulative demand for accelerated computing across enterprises and the broader developer community. Popular open-source models can have hundreds or thousands of derivative models, trained across numerous domains.

Developing this ecosystem of derivative models for a variety of use cases could take around 30x more compute than pretraining the original foundation model.

Developing this ecosystem of derivative models for a variety of use cases could take around 30x more compute than pretraining the original foundation model.

Post-training techniques can further improve a model's specificity and relevance for an organization's desired use case. While pretraining is like sending an AI model to school to learn foundational skills, post-training enhances the model with skills applicable to its intended job. An LLM, for example, could be post-trained to tackle a task like sentiment analysis or translation - or understand the jargon of a specific domain, like healthcare or law.

The post-training scaling law posits that a pretrained model's performance can further improve - in computational efficiency, accuracy or domain specificity - using techniques including fine-tuning, pruning, quantization, distillation, reinforcement learning and synthetic data augmentation.

Fine-tuning uses additional training data to tailor an AI model for specific domains and applications. This can be done using an organization's internal datasets, or with pairs of sample model input and outputs.

Distillation requires a pair of AI models: a large, complex teacher model and a lightweight student model. In the most common distillation technique, called offline distillation, the student model learns to mimic the outputs of a pretrained teacher model.

Reinforcement learning, or RL, is a machine learning technique that uses a reward model to train an agent to make decisions that align with a specific use case. The agent aims to make decisions that maximize cumulative rewards over time as it interacts with an environment - for example, a chatbot LLM that is positively reinforced by thumbs up reactions from users. This technique is known as reinforcement learning from human feedback (RLHF). Another, newer technique, reinforcement learning from AI feedback (RLAIF), instead uses feedback from AI models to guide the learning process, streamlining post-training efforts.

Best-of-n sampling generates multiple outputs from a language model and selects the one with the highest reward score based on a reward model. It's often used to improve an AI's outputs without modifying model parameters, offering an alternative to fine-tuning with reinforcement learning.

Search methods explore a range of potential decision paths before selecting a final output. This post-training technique can iteratively improve the model's responses
LINK: https://blogs.nvidia.com/blog/ai-scaling-laws/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

06/09/2026

Dolby and MagentaTV Bring Fans Closer to the FIFA World Cup 2026 in Germany with Dolby Vision and Dolby Atmos

June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...

04/08/2026

Dalet Announces Commercial Availability of Dalia, Bringing Media-Aware Agentic AI to Enterprise Productions

Dalet, a leading technology and service provider for media-rich organizations, t...

04/07/2026

Detective Conan: Fallen Angel of the Highway Opens in Dolby Cinemas Across Japan, Presented in Dolby Atmos and Dolby ...

April 7 2026, 19:00 (PDT) Detective Conan: Fallen Angel of the Highway Opens in...

26/06/2026

SVG GameDay, Ep. 21: Minnesota Vikings Allan Wertheimer - Large-Scale Shows in Minny

In-venue and creative video staffers at the professional and collegiate level ha...

26/06/2026

Strike Fighter League Announces Second Online Tournament, Set for July 25 in Las Vegas

Strike Fighter League (SFL), a professional air combat digital sport combining f...

26/06/2026

InfoComm 2026: Wisycom Announces MPR60 Firmware Update, MATF Antenna Matrix, and PFL RFoF Box

Wisycom has announced three new additions to its professional wireless ecosystem...

26/06/2026

Eurovision Services Inaugurates Expanded Master Control Room in Madrid

Eurovision Services inaugurated an expanded Master Control Room (MCR) in Madrid on June 1, 2026, building on a broadcast hub the company has operated in the cit...

26/06/2026

Midco Sports and University of North Dakota Renew Broadcast and Sponsorship Partnership

Midco Sports and the University of North Dakota (UND) have announced a two-year ...

26/06/2026

G&D and VuWall Appoint Vutec as Exclusive South Africa Distributor

Guntermann and Drunck (G&D) and VuWall, both part of the Panoptec Technologies Group, have appointed Vutec (Pty) Ltd as exclusive distributor for their KVM and ...

26/06/2026

Visit Seattle Launches Drone Scoreboard at Space Needle for FIFA World Cup 2026

Visit Seattle, the official destination marketing organization for Seattle and King County, has launched what it describes as the world's first drone scoreb...

26/06/2026

CP Communications Provides RF and Wireless Support for 2026 NBA Draft at Barclays Center

CP Communications provided RF video, audio, and crew communications support for ...

26/06/2026

Reimagined MoonPay X Games League Kicks Off With Three-Day Event in Sacramento

Produced by longtime partner Echo Entertainment, the action-sports property is now a team-based year-round league The inaugural season of the MoonPay X Games L...

26/06/2026

MultiDyne Acquires MRMC, Expands into Camera Robotics and Motion Control

The deal establishes MultiDyne Robotics and Motion Control, maintaining the well-known MRMC brand.MultiDyne Video & Fiber Optic Systems has acquired the assets ...

26/06/2026

TNT Sports Heads Into Year 2 of NASCAR Return With New NEP Truck, Expanded In-Car Experience

PX1 will debut at Sonoma as TNT leans into super-slo-mo, drones, SMT data integr...

26/06/2026

Ratings Roundup: USMNT-Australia Draws 23M Viewers; Mexico-South Korea Is Most-Watched Spanish-Language Soccer Match Ever

Ratings Roundup is a rundown of recent rating news and is derived from press rel...

26/06/2026

David Kuckhermann brings calabash to Celemony Tonalic

Virtual session musician plug-in gains new percussion options Celemony's latest update for their virtual session musician platform complements the exist...

26/06/2026

Softube unveil the Console 1 Compact

Half-size model joins Console 1 line-up Shortly after the release of their new Flow Studio controller, Softube have announced the launch of another new surf...

26/06/2026

ELT Group and Rohde & Schwarz sign a cooperation agreement to explore commercial opportunities in electromagnetic warfare and defense

ELT Group and Rohde & Schwarz sign a cooperation agreement to explore commercial...

26/06/2026

Lightware Powers Teddy Swims UK And Europe Tour With Adva...

For Teddy Swims sold-out I've Tried Everything But Therapy tour, event technology specialists, PRG, provided video, automation and lighting across 19 date...

26/06/2026

Taurus TPN powers AV workflows at NurnbergMesse

Modern exhibition and event venues face the challenge of seamlessly integrating traditional conference technology, professional broadcast workflows and IP-based...

26/06/2026

FCC Adopts New Cybersecurity Requirements for Alerting Systems

Share Copy link Facebook X Linkedin Bluesky Email...

26/06/2026

Study: Roku Most Used But Not Highest Rated Streaming Platform

Share Copy link Facebook X Linkedin Bluesky Email...

26/06/2026

Samsung Ads Announces First Shoppable CTV Partners

Share Copy link Facebook X Linkedin Bluesky Email...

26/06/2026

Gray Media Names Annie Cordell General Manager of WMBF

Share Copy link Facebook X Linkedin Bluesky Email...

26/06/2026

Neko Oji: The Guy That Got Reincarnated as a Cat Edited with DaVinci Resolve Studio

Neko Oji: The Guy That Got Reincarnated as a Cat Edited with DaVinci Resolve Stu...

26/06/2026

Adobe to Acquire Topaz Labs

Adobe to Acquire Topaz Labs Brie Clayton June 25, 2026 0 Comments Adobe has seen strong demand for its AI products for creatives, including Adobe Fire...

26/06/2026

Berklee Students Earn Dedicated Section at Raindance Film Festival in London

Berklee Students Earn Dedicated Section at Raindance Film Festival in London Five documentary short films produced in the Africana Studies Department screen a...

26/06/2026

Prison Wives of TikTok is Locked In for U and U&W

Flicker Productions to produce five-part docu-reality series following women who have fallen for men in prison and have become TikTok sensations, with brands an...

26/06/2026

Automating post-production workflows with Baselight, Daylight, Nara & FilmLight API. New York. 8 July 2026

Catch up on the latest developments across Baselight and Daylight v7, Nara and F...

26/06/2026

DFT installs second Polar HQ at China News Film Confirming Position as China's Leading 8K Film Preservation Partner

26. June 2026 News DFT is pleased to announce that a second Polar HQ film s...

26/06/2026

New documentary Freedom Founder: Thomas McKean and the American Revolution comes to RT

New documentary Freedom Founder: Thomas McKean and the American Revolution airs ...

25/06/2026

Launching a Career in Broadcast Engineering: Academic Paths and Essential Certifications

Launching a Career in Broadcast Engineering: Academic Paths and Essential Certif...

25/06/2026

SVG Students To Watch: Jude Kieffer, Ball State University

This superstar shooter/storyteller from Central Indiana hopes to make his mark in the blossoming sports-documentary and -features space In the live-sports-vid...

25/06/2026

Presidio and NHL Renew Multiyear North American Technology Partnership

Presidio and the National Hockey League have announced a multiyear renewal of their North American partnership. Presidio will remain an Official Technology Inno...

25/06/2026

Strike Fighter League Hits the Industry as First Professional Air Combat Sport

Strike Fighter League (SFL) is the world's first professional air combat digital sport that combines elite human performance and physical immersion with cut...

25/06/2026

Rise Reveals 2026 Worldwide Mentoring Cohorts to Support Future Industry Leaders

Rise, the award-winning advocacy group for gender diversity in the broadcast and media technology sector, is pleased to announce the global mentoring cohort for...

25/06/2026

MLB Network To Air American Association of Professional Baseball All-Star Game for First Time on July 15

The 2026 American Association of Professional Baseball (AAPB) All-Star Game will...

25/06/2026

Mediaproxy Partners with HVS for U.S. Broadcast Market

Mediaproxy has named Heartland Video Systems (HVS) as its exclusive partner for US television broadcasting. The Wisconsin-based systems integrator will represen...

25/06/2026

Backblaze Inks Five-Year Multi-Exabyte Data Storage Agreement with CoreWeave

Backblaze has formed an agreement with CoreWeave to create The Essential Cloud for AI. Under the multi-exabyte, $335 million agreement, Backblaze will provide...

25/06/2026

Clear-Com FreeSpeak Cell Tested by RTL Deutschland on 5G Network at Nrburgring

Clear-Com has announced the successful deployment and testing of FreeSpeak Cell by RTL Deutschland during a live event production at the N rburgring race circui...

25/06/2026

Mobile TV Group Launches Full-Stack MTVG Production Platform, Powers Angels Broadcast Television

Mobile TV Group (MTVG) has announced the launch of the MTVG Production Platform,...

25/06/2026

Sony Pictures Entertainment Announces $100 Million Investment in Cosm

Sony Pictures Entertainment (SPE) has announced a $100 million strategic investment in Cosm as lead investor in the company's Series C financing round, acqu...

25/06/2026

FOX Sports Renews Concacaf Gold Cup Rights and Adds Nations League Through 2029

FOX Sports and Concacaf have announced a multi-year media rights agreement making FOX Sports the U.S. English-language home of the Concacaf Gold Cup and Concaca...

25/06/2026

InfoComm 2026: Daktronics and Grass Valley Win rAVe Pubs Best Solution for Large Venue or Live Events

Daktronics and Grass Valley have received the rAVe Pubs Best Solution for Large ...

25/06/2026

Music Production for Women announce Soundlab 2026

Six free workshops across two days Global music education platform Music Production for Women (MPW), have just announced a brand new and highly anticipated ...

25/06/2026

CIOKS launch the DC7 v2

Popular pedalboard PSU gets an upgrade The DC7 v2 is a new and improved version of CIOKS' renowned effects pedal PSU, and is said to be the thinnest, mo...

25/06/2026

Rev Ocean reverb from Arturia

Optimised for lush, enveloping sounds Described as an instantly rewarding reverb , the latest addition to Arturia's range of creative effects plug-ins ...

25/06/2026

Just 48 hours until GearExpo UK!

27 June 2026, Westminster University Harrow Campus GearExpo UK is now upon us, with just two days to go until 150 of the worlds top pro-audio brands and ind...

25/06/2026

The Name You Know, The Lineup You'll Love - SBS2 Returns

The Name You Know, The Lineup You'll Love - SBS2 Returns 25 June, 2026 Media releases SBS Viceland rebrands as SBS2 on Friday 21 August, bringing the c...