How Are Foundation Models Used in Gaming?
02/11/2023
From creating lifelike characters that convey emotions to transforming simple text into captivating imagery, foundation models are becoming essential in accelerating developer workflows while reducing overall costs. These powerful AI models have unlocked a realm of possibilities, empowering designers and game developers to build higher-quality gaming experiences.
What Are Foundation Models? A foundation model is a neural network that's trained on massive amounts of data - and then adapted to tackle a wide variety of tasks. They're capable of enabling a range of general tasks, such as text, image and audio generation. Over the last year, the popularity and use of foundation models has rapidly increased, with hundreds now available.
For example, GPT-4 is a large multimodal model developed by OpenAI that can generate human-like text based on context and past conversations. Another, DALL-E 3, can create realistic images and artwork from a description written in natural language.
Powerful foundation models like NVIDIA NeMo and Edify model in NVIDIA Picasso make it easy for companies and developers to inject AI into their existing workflows. For example, using the NVIDIA NeMo framework, organizations can quickly train, customize and deploy generative AI models at scale. And using NVIDIA Picasso, teams can fine-tune pretrained Edify models with their own enterprise data to build custom products and services for generative AI images, videos, 3D assets, texture materials and 360 HDRi.
How Are Foundation Models Built? Foundation models can be used as a base for AI systems that can perform multiple tasks. Organizations can easily and quickly use a large amount of unlabeled data to create their own foundation models.
The dataset should be as large and diverse as possible, as too little data or poor-quality data can lead to inaccuracies - sometimes called hallucinations - or cause finer details to go missing in generated outputs.
Next, the dataset must be prepared. This includes cleaning the data, removing errors and formatting it in such a way that the model can understand it. Bias is a pervasive issue when preparing a dataset, so it's important to measure, reduce and tackle these inconsistencies and inaccuracies.
Training a foundational model can be time-consuming, especially given the size of the model and the amount of data required. Hardware like NVIDIA A100 or H100 Tensor Core GPUs, along with high-performance data systems like the NVIDIA DGX SuperPOD, can accelerate training. For example, ChatGPT-3 was trained on over 1,000 NVIDIA A100 GPUs over about 34 days.
The three requirements of a successful foundation model. After training, the foundation model is evaluated on quality, diversity and speed. There are several methods for evaluating performance, for example:
Tools and frameworks that quantify how well the model predicts a sample of text
Metrics that compare generated outputs with one or more references and measure the similarities between them
Human evaluators who assess the quality of the generated output on various criteria
Once the model passes the relevant tests and evaluations, it can then be deployed for production.
Exploring Foundation Models in Games Pretrained foundation models can be leveraged by middleware, tools and game developers both during production and at run-time. To train a base model, resources and time are necessary - alongside a certain level of expertise. Currently, many developers within the gaming industry are exploring off-the-shelf models, but need custom solutions that fit their specific use cases. They need models that are trained on commercially safe data and optimized for real-time performance - without exorbitant costs of deployment. The difficulty of meeting these requirements has slowed adoption of foundation models.
However, innovation within the generative AI space is swift, and once major hurdles are addressed, developers of all sizes - from startups to AAA studios - will use foundation models to gain new efficiencies in game development and accelerate content creation. Additionally, these models can help create completely new gameplay experiences.
The top industry use cases are centered around intelligent agents and AI-powered animation and asset creation. For example, many creators today are exploring models for creating intelligent non-playable characters, or NPCs.
Custom LLMs fine-tuned with the lingo and lore of specific games can generate human-like text, understand context and respond to prompts in a coherent manner. They're designed to learn patterns and language structures and understand game state changes - evolving and progressing alongside the player in the game.
As NPCs become increasingly dynamic,real-time animation and audio that sync with their responses will be needed. Developers are using NVIDIA Riva to create expressive character voices using speech and translation AI. And designers are tapping NVIDIA Audio2Face for AI-powered facial animations.
Foundation models are also being used for asset and animation generation. Asset creation during the pre-production and production phases of game development can be time-consuming, tedious and expensive.
With state-of-the-art diffusion models, developers can iterate more quickly, freeing up time to spend on the most important aspects of the content pipeline, such as developing higher-quality assets and iterating. The ability to fine-tune these models from a studio's own repository of data ensures the outputs generated are similar to the art styles and designs of their previous games.
Foundation models a
LINK: | https://blogs.nvidia.com/blog/2023/11/02/foundation-models-gaming/... |
See more stories from nvidia |
Most recent headlines
04/08/2024
Dalet Appoints Santiago Solanas as CEO to Lead Next Era of Growth and Innovation
Dalet, a leading technology and service provider for media-rich organizations, is excited to announce Santiago Solanas as its new Chief Executive Officer (CEO)....
03/06/2024
Dalet and Veritone Reach Agreement to Distribute, Transact and Monetize Media Archives
Dalet, a leading technology and service provider for media-rich organizations, a...
30/05/2024
When to Upgrade Software or Firmware
When to Upgrade Software or Firmware This post is from our blog and news archive. The information may be out of date. Please contact us for further information ...
30/05/2024
The City of San Jose Chooses Utah ScientificAgain!
The City of San Jose Chooses Utah Scientific Again! This post is from our blog and news archive. The information may be out of date. Please contact us for furth...
30/05/2024
Empowering Media Leaders: Transformative Insights from the Executive Learning Series
Navigating Strategy, Ethics, and Innovation in the evolving European media lands...
30/05/2024
Give Me the Backstory: Get to Know Thea Hvistendahl, the Filmmaker Behind Handling the Undead
By Bailey Pennick One of the most exciting things about the Sundance Film Festi...
30/05/2024
CHOOSE CHICAGO, CITY OF CHICAGO ANNOUNCE PROGRAMMING DETAILS OF SUNDANCE INSTITUTE X CHICAGO 2024 (June 28 - 30)
Tickets are now available for the first-of-its-kind event in the United States f...
30/05/2024
UNICEF and Spotify's Award-Winning Mental Health Hub, Our Minds Matter, Comes to Latin America
Mental health and well-being are fundamental components of a child's healthy...
30/05/2024
After 64 days, the winner of Alone Australia has been revealed
After 64 days, the winner of Alone Australia has been revealed 29 May, 2024 Media releases *Contains Spoilers* After 64 days alone in the extreme and wild...
30/05/2024
Alone Australia finale delivers strongly as SBS confirms it will join VOZ streaming
Alone Australia finale delivers strongly as SBS confirms it will join VOZ stream...
30/05/2024
Alone Australia finale delivers strongly for national broadcaster SBS with 4.2m tuning in across the season
Alone Australia finale delivers strongly for national broadcaster SBS with 4.2m ...
30/05/2024
Lockheed Martin Canada Awards L3Harris the Integrated Communications System Contract
Photo credit: BAE...
30/05/2024
The New Standard for Unmanned Ground Vehicle ISR
L3Harris' WESCAM MX -10 RSTA provides advanced ISR and targeting capabilities for land-based platforms including Unmanned Ground Vehicles....
30/05/2024
AccuWeather Inks Deal With Comcast Technology Solutions For Channel Origination
DENVER AccuWeather has selected Comcast Technology Solutions' (CTS') Managed Channel Origination (MCO) to create, manage and distribute linear TV, on-de...
30/05/2024
Samsung, LG Adopt IAB Software Kit
NEW YORK The global body that sets technical standards for digital advertising has expanded the reach of its measurement software development kit to include Sam...
30/05/2024
Amazon Celebrates 10th Anniversary of Fire TV Devices with an AI Upgrade
As Amazon celebrates the 10th anniversary of the launch of the first Fire TV devices in 2014, the company has unveiled new AI-powered search capabilities for th...
30/05/2024
Finding the sustainability balance with personalised video streaming
As the expansion of personalised video streaming continues apace, JUMPs CEO and co-founder, Jer nimo Macan s and Fran ois Polarczyk, sustainability director at ...
30/05/2024
EditShare boosts sales direction with alumnus Grant Carro...
EditShare, the technology leader that enables storytellers to create and manage collaborative workflows at every stage from storyboard to screening, has appoint...
30/05/2024
COW Featured Resume: Paula Zimmerman - Video Editor - Motion Designer
COW Featured Resume: Paula Zimmerman - Video Editor - Motion Designer Brie Clayton May 30, 2024 0 Comments Paula Zimmerman Looking for work? Sign up...
30/05/2024
Palme d'Or Winner Anatomy of a Fall Finished with DaVinci Resolve Studio
Palme d'Or Winner Anatomy of a Fall Finished with DaVinci Resolve Studio Brie Clayton May 30, 2024 0 Comments Winner of the Palme d'Or at the ...
30/05/2024
Real-Time Workflow Masterclass
Real-Time Workflow Masterclass Michael Cioni May 30, 2024 0 Comments Would you rather have your workflow move at the same speed it always has, or woul...
30/05/2024
Ending a Loop Expression in After Effects
Ending a Loop Expression in After Effects Graham Quince May 30, 2024 0 Comments This question comes up quite a bit on forums and it seems to trip peop...
30/05/2024
Parks: Prime Video Has Lowest Churn Rate
DALLAS Consumers who subscribe to streaming services are the least likely to cancel Prime Video among all major providers, according to Parks Associates' St...
30/05/2024
SMPTE/RIS-OSVP tests model for circle of confusion'
According to Camera and Lens Metadata committee, modern digital cinema cameras are not accurately representing the usable depth of field achieved with various l...
30/05/2024
Meet the vice president of product management
Dave MacKinnon, vice president of product management at Clear-Com, explains the value of collaboration and networking in building a media career By Matthew Cor...
30/05/2024
That Station Launches New App, Website, Filled with Features, Music, Interviews and More
That Station's app and website just got a major upgrade. The station crew co...
30/05/2024
SWR Deploys Rohde & Schwarz Pixel Power Software Playout Solution
MUNICH, Germany Regional German broadcaster S dwestrundfunk (SWR) has deployed the Rohde & Schwarz Pixel Power graphics and playout solution....
30/05/2024
Hollyland Technology Unveils Pyro Wireless Video Transmission Series
IRVINE, Calif. Hollyland Technology has launched Pyro, a wireless video transmission system designed for the multi-person, mobile transmission and monitoring re...
30/05/2024
AIMS Opens Call for Presentations for Media-Over-IP Pavilion at AES New York
AIMS has announced it will once again collaborate with the Audio Engineering Society (AES) to bring the popular Media-Over-IP Pavilion to the AES New York show,...
30/05/2024
FCC Announces Opportunity for LPTV Stations to Change Channels
WASHINGTON, D.C. The Federal Communications Commission (FCC) Media Bureau has announced that beginning on August 20, 2024, Class A television, Low Power Televis...
30/05/2024
Comcast's StreamSaver Streaming Bundle Goes Live
Comcast has officially launched its discounted $15-a-month StreamSaver streaming bundle of Netflix, Peacock and Apple TV+ services. The previously announced bun...
30/05/2024
AMG Launches TV Stations, Local Now FAST Channels on Amazon's Fire TV Channels
LOS ANGELES Allen Media Group (AMG) is launching three of its streaming brands o...
30/05/2024
Viant Integrates with Google Cloud's BigQuery Data Clean Rooms
IRVINE, Calif. The ad tech company Viant Technology Inc. has announced a new integration with Google Cloud's BigQuery data clean rooms that enables the seam...
30/05/2024
Marshall Brings Selection of New and Proven AV Solutions...
Marshall Electronics will highlight several of its newest product offerings at InfoComm 2024 (Booth C8982), including the CV612 auto-tracking PTZ camera and the...
30/05/2024
Triveni Digital Streamlines TV Operations at the 2024 ATS...
Triveni Digital today announced that the company will showcase its end-to-end ATSC 3.0 offering at the 2024 ATSC NEXTGEN Broadcast Conference, June 12-14 in Was...
30/05/2024
Riedel Bolero Empowers Student Production Excellence at O...
Riedel Communications today announced that Orange County School of the Arts (OCSA) has successfully integrated the Bolero wireless intercom system into their st...
30/05/2024
SWR moves to software playout with integrated Pixel Power...
Rohde & Schwarz has implemented its Pixel Power graphics and playout solution at S dwestrundfunk (SWR), the regional German broadcaster based in Baden-Baden. SW...
30/05/2024
Media Asset Management vs Inventory Management
From their inception, media asset management (MAM) systems have been a crucial component of the media supply chain. Traditionally MAM systems service customers ...
30/05/2024
Disguise launches new generation of EX media servers to p...
Disguise has announced the next evolution of its hardware solutions with the launch of the new generation of EX media servers, following the recently launched R...
30/05/2024
PlayBox Neo Highlights Smart Media Playout Innovations at...
PlayBox Neo, a globally active producer of broadcast media playout and channel branding solutions, sustained its industry event visibility at three events durin...
30/05/2024
EMG Gravity Media Announces Martyn Edwards as General Man...
EMG / Gravity Media, the leading force in production and content, media services and facilities, is pleased to announce the appointment of Martyn Edwards as the...
30/05/2024
Screen Australia, Australians in Film and VicScreen announce FUTURE VISION
30 05 2024 - Media release Screen Australia, Australians in Film and VicScreen announce FUTURE VISION Lee Sung Jin and Joanna Calo Australians in Film, Scree...
30/05/2024
AI Benefits Come at a Cost
AI Benefits Come at a Cost Brie Clayton May 29, 2024 0 Comments After this, there is no turning back. You take the blue pill the story ends, you wa...
30/05/2024
Morgan Spurlock, Super Size Me' Director, Has Died
Morgan Spurlock, director of Super Size Me, died May 24 in New York. He was 53 and had been battling cancer....
30/05/2024
Departing Longtime Local Anchors Share Their Lessons Learned
A pair of beloved local anchors with extraordinary runs at their stations are finally stepping down. Tom Wills signs off at WJXT Jacksonville May 31, 49 years a...
30/05/2024
Inscape Founder Zeev Neumeier Launches GraySwan To Optimize CTV Ads
Zeev Neumeier, who founded Inscape and was a pioneer in automatic content recognition, is beta-testing his new venture, GraySwan....
30/05/2024
Robert De Niro To Get Service to America Award From NABLF
Robert De Niro will get the 2024 Service to America Leadership Award from the National Association of Broadcasters Leadership Foundation. The award is presented...
30/05/2024
AGT' is Back, With More Golden Buzzers and Youngest Contestant Ever
Season 19 of America's Got Talent is on NBC starting May 28. The new season features more Golden Buzzers, which send a standout act directly to the live sho...
30/05/2024
The Quiz With Balls', Game Show That Pits Brains Against Boldness, Starts on Fox
Jay Pharoah hosts The Quiz with Balls, a game show that Fox says pits brains ag...
30/05/2024
Netflix's Bridgerton' Keeps Top Spot in TVision Power Score Rankings
Netflix's Bridgerton (season three) repeated as the top show in TVision's Power Score ranking of programs on connected TV for the week of May 20....