Sony Pixel Power calrec Sony

AI Takes a Bow: Interactive GLaDOS Robot Among 9 Winners in Challenge


YouTube robotics influencer Dave Niewinski has developed robots for everything from driveable La-Z-Boy chairs to an AI-guided cornhole tosser and horse-drawn chariot racing.

His recent Interactive Animatronic GLaDOS project was among nine winners in the Hackster AI Innovation Challenge. About 100 contestants vied for prizes from NVIDIA and Sparkfun by creating open-source projects to advance the use of AI in edge computing, robotics and IoT.

Niewinski won first place in the generative AI applications category for his innovative robot based on the GLaDOS guide from game series Portal, the first-person puzzle platform from video game developer Valve.

Other top winners included contestants Andrei Ciobanu and Allen Tao, who took first prize in the generative AI models for the edge and AI at the edge applications categories, respectively. Ciobanu used generative AI to help virtually try on clothes, while Tao developed a ROS-based robot to map the inside of a home to help find things.

Harnessing LLMs for Robots Niewinski builds custom applications for robotics at his Armoury Labs business in Waterloo, Ontario, Canada, where he uses the NVIDIA Jetson platform for edge AI and robotics, creating open-source tutorials and YouTube videos following his experiences.

He built his interactive GLaDOS robot to create a personal assistant for himself in the lab. It handles queries using Transformer-based speech recognition, text-to-speech, and large language models (LLMs) running onboard an NVIDIA Jetson AGX Orin, which interfaces with a robot arm and camera for interactions.

GLaDOS can track his whereabouts in the lab, move in different directions to face him and respond quickly to queries.

I like doing things with robots that people will look at and say it's not what they had immediately expected, he said.

He wanted the assistant to sound like the original GLaDOS from Portal and respond quickly. Fortunately, the gaming company Valve has put all of the voice lines from Portal and Portal 2 on its website, allowing Niewinski to download the audio to help train a model.

Using Jetson, your average question-and-answer stuff runs pretty quick for speech, he said.

Niewinski used NVIDIA's open-source NeMo toolkit to fine-tune a voice for GLaDOS, training a spectrogram generator network called FastPitch and HiFiGAN vocoder network to refine the audio quality.

Both networks are deployed on Orin with NVIDIA Riva to enable speech recognition and synthesis that's been optimized to run at many times the real-time rate of speech, so that it can run alongside the LLM while maintaining a smooth, interactive delivery.

For generating realistic responses from GLaDOS, Niewinski uses a locally hosted LLM called OpenChat that he runs in Docker from jetson-containers, saying that it was a drop-in replacement for OpenAI's API. All of this AI is running on the Jetson module, using the latest open-source ML software stack built with CUDA and JetPack.

To enable GLaDOS to move, Niewinski developed the interactions for a Unitree Z1 robotic arm. It has a stereo camera and models for seeing and tracking a human speaking and a 3D-printed GLaDOS head and body shell around the arm.

Trying on Generative AI for Fashion Fit Winner Ciobanu, based in Romania, aimed to improve the virtual clothing try-on experience with the help of generative AI, taking a top prize for his EdgeStyle: Fashion Preview at the Edge.

He used AI models such as YOLOv5, SAM and OpenPose to extract and refine data from images and videos. Then he used Stable Diffusion to generate the images, which he said was key to achieving accurate virtual try-ons.

This system taught the model how clothes fit different poses on people, which he said enhanced the realism of the try-ons.

It's quite handy as it allows users to see how clothes would look on them without actually trying them on, said Ciobanu.

The NVIDIA JetPack SDK provided all the tools needed to run AI models smoothly on the Jetson Orin, he said.

It's super-helpful to have a stable set of tools, especially when you're dealing with AI tech that keeps changing, said Ciobanu. It really cut down on the time and hassle for us developers, letting us focus more on the cool stuff we're building instead of getting stuck on tech issues.

Finding Lost Items With Robot Assistance Winner Tao, based in Ontario, Canada, created a robot to lessen the burden of searching for things lost around the house. His An Eye for an Item project took top honors at the Hackster challenge.

Finding lost objects is a chore, and recent developments in zero-shot object detection and LLMs make it feasible for a computer to detect arbitrary objects for us based on textual or pictorial descriptions, presenting an opportunity for automation, said Tao.

Tao said he needed robot computing capabilities to catalog objects in any unstructured environment - whether a living room or large warehouse. And he needed it to also perform real-time calculations for localization to help with navigation, as well as running inference on larger object detection models.

Jetson Orin was a perfect fit, supporting all functionality from text and image queries into NanoDB, to real-time odometry feedback, including leveraging Isaac ROS' hardware-accelerated AprilTag detections for drift correction, he said.

Other winners of the AI Innovation Challenge include:

George Profenza, Escalator people tracker, 2nd place, Generative AI Applications category

Dimiter Kendri, Cooking meals with a local AI assistant using Jetson AGX Orin, 3rd place, Generative AI Applications category

Vy Phan, ClearWaters Underwater Image Enhancement with Generative AI, 2nd place, Generative AI Models category

Huy Mai, Realtime Language Segment Anything on Jetson Orin, 2nd place, Generative AI Models category

Fakhrur Razi, Autonomous Int
See more stories from nvidia

More from Nvidia


EvolutionaryScale Debuts With ESM3 Generative AI Model for Protein Design

Generative AI has revolutionized software development with prompt-based code generation - protein design is next. EvolutionaryScale today announced the release...


Crack the Case With Tell Me Why' and As Dusk Falls' on GeForce NOW

Sit back and settle in for some epic storytelling. Tell Me Why and As Dusk Falls - award-winning, narrative-driven games from Xbox Studios - add to the 1,900+ g...


Decoding How NVIDIA AI Workbench Powers App Development

Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible and showcases new hardware, softwar...


Light Bulb Moment: NVIDIA CEO Sees Bright Future for AI-Powered Electric Grid

The electric grid and the utilities managing it have an important role to play in the next industrial revolution that's being driven by AI and accelerated c...


NVIDIA Advances Physical AI at CVPR With Largest Indoor Synthetic Dataset

NVIDIA contributed the largest ever indoor synthetic dataset to the Computer Vision and Pattern Recognition (CVPR) conference's annual AI City Challenge - h...


NVIDIA Research Wins CVPR Autonomous Grand Challenge for End-to-End Driving

Making moves to accelerate self-driving car development, NVIDIA was today named an Autonomous Grand Challenge winner at the Computer Vision and Pattern Recognit...


Seamless in Seattle: NVIDIA Research Showcases Advancements in Visual Generative AI at CVPR

NVIDIA researchers are at the forefront of the rapidly advancing field of visual...


Believe in Something Unconventional, Something Unexplored,' NVIDIA CEO Tells Caltech Grads

NVIDIA founder and CEO Jensen Huang on Friday encouraged Caltech graduates to pu...


The Proudest Refugee': How Veronica Miller Charts Her Own Path at NVIDIA

When she was five years old, Veronica Miller (n e Teklai) and her family left their homeland of Eritrea, in the Horn of Africa, to escape an ongoing war with Et...


NVIDIA Releases Open Synthetic Data Generation Pipeline for Training Large Language Models

NVIDIA today announced Nemotron-4 340B, a family of open models that developers ...


Cloud Ahoy! Treasure Awaits With Sea of Thieves' on GeForce NOW

Set sail for adventure, pirates. Sea of Thieves makes waves in the cloud this week. It's an adventure-filled GFN Thursday with four new games joining the Ge...


Every Company's Data is Their Gold Mine,' NVIDIA CEO Says at Databricks Data + AI Summit

Accelerated computing is transforming data processing and analytics for enterpri...


Scaling to New Heights: NVIDIA MLPerf Training Results Showcase Unprecedented Performance and Elasticity

The full-stack NVIDIA accelerated computing platform has once again demonstrated...


Nerding About NeRFs: How Neural Radiance Fields Transform 2D Images Into Hyperrealistic 3D Models

Let's talk about NeRFs - no, not the neon-colored foam dart blasters, but ne...


TOPS of the Class: Decoding AI Performance on RTX AI PCs and Workstations

Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, softwa...


Why Accelerated Data Processing Is Crucial for AI Innovation in Every Industry

Across industries, AI is supercharging innovation with machine-powered computation. In finance, bankers are using AI to detect fraud more quickly and keep accou...


Here Comes a New Challenger: Street Fighter 6' Joins GeForce NOW

Capcom's latest entry in the iconic Street Fighter series, Street Fighter 6, punches its way into the cloud this GFN Thursday. The game, along with Ubisoft&...


Yotta CEO Sunil Gupta on Supercharging India's Fast-Growing AI Market

India's AI market is expected to be massive. Yotta Data Services is setting its sights on supercharging it. In this episode of NVIDIA's AI Podcast, Suni...


Creativity Accelerated: New RTX-Powered AI Hardware and Software Announced at COMPUTEX

NVIDIA launched NVIDIA Studio at COMPUTEX in 2019. Five years and more than 500 ...


SAP and NVIDIA Create AI for The Most Valuable Language,' CEOs Unveil at Sapphire Orlando

German enterprise cloud leader SAP is harnessing generative AI and industrial di...


NVIDIA and Cisco Weave Fabric for Generative AI

Building and deploying AI applications at scale requires a new class of computing infrastructure - one that can handle the massive amounts of data, compute powe...


Digital Bank Debunks Financial Fraud With Generative AI

European neobank bunq is debunking financial fraudsters with the help of NVIDIA accelerated computing and AI. Dubbed the bank of the free, bunq offers online...


Foxconn Trains Robots, Streamlines Assembly With NVIDIA AI and Omniverse

Foxconn operates more than 170 factories around the world - the latest one a virtual plant pushing the state of the art in industrial automation. It's the ...


Taiwan Electronics Giants Drive Industrial Automation With NVIDIA Metropolis and NIM

Taiwan's leading consumer electronics giants are making advances with AI aut...


KServe Providers Dish Up NIMble Inference in Clouds and Data Centers

Deploying generative AI in the enterprise is about to get easier than ever. NVIDIA NIM, a set of generative AI inference microservices, works with KServe, open...


Accelerate Everything,' NVIDIA CEO Says Ahead of COMPUTEX

Generative AI is reshaping industries and opening new opportunities for innovation and growth, NVIDIA founder and CEO Jensen Huang said in an address ahead of ...


Power Tool: Generative AI Tracks Typhoons, Tames Energy Use

Weather forecasters in Taiwan had their hair blown back when they saw a typhoon up close, created on a computer that slashed the time and energy needed for the ...


NVIDIA Grace Hopper Superchip Accelerates Murex MX.3 Analytics Performance, Reduces Power Consumption

After the 2008 financial crisis and increased risk-management regulations that f...


Elevate Your Expertise: NVIDIA Introduces AI Infrastructure and Operations Training and Certification

NVIDIA has introduced a self-paced course, called AI Infrastructure and Operatio...


GeForce NOW Brings the Heat With World of Warcraft'

World of Warcraft comes to the cloud this week, part of the 17 games joining the GeForce NOW library, with seven available to stream this week. Plus, it's ...


Riding the Wayve of AV 2.0, Driven by Generative AI

Generative AI is propelling AV 2.0, a new era in autonomous vehicle technology characterized by large, unified, end-to-end AI models capable of managing various...


Tidy Tech: How Two Stanford Students Are Building Robots for Handling Household Chores

Imagine having a robot that could help you clean up after a party - or fold heap...


Decoding How NVIDIA RTX AI PCs and Workstations Tap the Cloud to Supercharge Generative AI

Editor's note: This post is part of the AI Decoded series, which demystifies...


NVIDIA Scoops Up Wins at COMPUTEX Best Choice Awards

Building on more than a dozen years of stacking wins at the COMPUTEX trade show's annual Best Choice Awards, NVIDIA was today honored with BCAs for its late...


Senua's Story Continues: GeForce NOW Brings Senua's Saga: Hellblade II' to the Cloud

Every week, GFN Thursday brings new games to the cloud, featuring some of the la...


Into the Omniverse: SoftServe and Continental Drive Digitalization With OpenUSD and Generative AI

Editor's note: This post is part of Into the Omniverse, a series focused on ...


Watt a Win: NVIDIA Sweeps New Ranking of World's Most Energy-Efficient Supercomputers

In the latest ranking of the world's most energy-efficient supercomputers, k...


New Performance Optimizations Supercharge NVIDIA RTX AI PCs for Gamers, Creators and Developers

NVIDIA today announced at Microsoft Build new AI performance optimizations and i...


NVIDIA Expands Collaboration With Microsoft to Help Developers Build, Deploy AI Applications Faster

If optimized AI workflows are like a perfectly tuned orchestra - where each comp...


A Superbloom of Updates in the May Studio Driver Gives Fresh Life to Content Creation

Editor's note: This post is part of our In the NVIDIA Studio series, which c...


Every Company to Be an Intelligence Manufacturer,' Declares NVIDIA CEO Jensen Huang at Dell Technologies World

AI heralds a new era of innovation for every business in every industry, NVIDIA ...


Fight for Honor in Men of War II' on GFN Thursday

Whether looking for new adventures, epic storylines or games to play with a friend, GeForce NOW members are covered. Start off with the much-anticipated sequel...


NVIDIA, Teradyne and Siemens Gather in the City of Robotics' to Discuss Autonomous Machines and AI

Senior executives from NVIDIA, Siemens and Teradyne Robotics gathered this week ...


Fire It Up: Mozilla Firefox Adds Support for AI-Powered NVIDIA RTX Video

Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and which showcases new hardware, ...


How Basecamp Research Helps Catalog Earth's Biodiversity

Basecamp Research is on a mission to capture the vastness of life on Earth at an unprecedented scale. Phil Lorenz, CTO at Basecamp Research, discusses using AI ...


Needle-Moving AI Research Trains Surgical Robots in Simulation

A collaboration between NVIDIA and academic researchers is prepping robots for surgery. ORBIT-Surgical - developed by researchers from the University of Toront...


Gemma, Meet NIM: NVIDIA Teams Up With Google DeepMind to Drive Large Language Model Innovation

Large language models that power generative AI are seeing intense innovation - m...


Drug Discovery, STAT! NVIDIA, Recursion Speed Pharma R&D With AI Supercomputer

Described as the largest system in the pharmaceutical industry, BioHive-2 at the Salt Lake City headquarters of Recursion debuts today at No. 35, up more than 1...


Drug Discovery, STAT! NVIDIA, Recursion Speed Pharma R&D With AI Supercomputer

Described as the largest system in the pharmaceutical industry, BioHive-2 at the...


Dial It In: Data Centers Need New Metric for Energy Efficiency

Data centers need an upgraded dashboard to guide their journey to greater energy efficiency, one that shows progress running real-world applications. The formu...