![](//www.4rfv.com/4rfv_news_images/nvidia_scripts/CNJOQUD4Q6G4_0.jpg)
YouTube robotics influencer Dave Niewinski has developed robots for everything from driveable La-Z-Boy chairs to an AI-guided cornhole tosser and horse-drawn chariot racing.
His recent Interactive Animatronic GLaDOS project was among nine winners in the Hackster AI Innovation Challenge. About 100 contestants vied for prizes from NVIDIA and Sparkfun by creating open-source projects to advance the use of AI in edge computing, robotics and IoT.
Niewinski won first place in the generative AI applications category for his innovative robot based on the GLaDOS guide from game series Portal, the first-person puzzle platform from video game developer Valve.
Other top winners included contestants Andrei Ciobanu and Allen Tao, who took first prize in the generative AI models for the edge and AI at the edge applications categories, respectively. Ciobanu used generative AI to help virtually try on clothes, while Tao developed a ROS-based robot to map the inside of a home to help find things.
Harnessing LLMs for Robots Niewinski builds custom applications for robotics at his Armoury Labs business in Waterloo, Ontario, Canada, where he uses the NVIDIA Jetson platform for edge AI and robotics, creating open-source tutorials and YouTube videos following his experiences.
He built his interactive GLaDOS robot to create a personal assistant for himself in the lab. It handles queries using Transformer-based speech recognition, text-to-speech, and large language models (LLMs) running onboard an NVIDIA Jetson AGX Orin, which interfaces with a robot arm and camera for interactions.
GLaDOS can track his whereabouts in the lab, move in different directions to face him and respond quickly to queries.
I like doing things with robots that people will look at and say it's not what they had immediately expected, he said.
He wanted the assistant to sound like the original GLaDOS from Portal and respond quickly. Fortunately, the gaming company Valve has put all of the voice lines from Portal and Portal 2 on its website, allowing Niewinski to download the audio to help train a model.
Using Jetson, your average question-and-answer stuff runs pretty quick for speech, he said.
Niewinski used NVIDIA's open-source NeMo toolkit to fine-tune a voice for GLaDOS, training a spectrogram generator network called FastPitch and HiFiGAN vocoder network to refine the audio quality.
Both networks are deployed on Orin with NVIDIA Riva to enable speech recognition and synthesis that's been optimized to run at many times the real-time rate of speech, so that it can run alongside the LLM while maintaining a smooth, interactive delivery.
For generating realistic responses from GLaDOS, Niewinski uses a locally hosted LLM called OpenChat that he runs in Docker from jetson-containers, saying that it was a drop-in replacement for OpenAI's API. All of this AI is running on the Jetson module, using the latest open-source ML software stack built with CUDA and JetPack.
To enable GLaDOS to move, Niewinski developed the interactions for a Unitree Z1 robotic arm. It has a stereo camera and models for seeing and tracking a human speaking and a 3D-printed GLaDOS head and body shell around the arm.
Trying on Generative AI for Fashion Fit Winner Ciobanu, based in Romania, aimed to improve the virtual clothing try-on experience with the help of generative AI, taking a top prize for his EdgeStyle: Fashion Preview at the Edge.
He used AI models such as YOLOv5, SAM and OpenPose to extract and refine data from images and videos. Then he used Stable Diffusion to generate the images, which he said was key to achieving accurate virtual try-ons.
This system taught the model how clothes fit different poses on people, which he said enhanced the realism of the try-ons.
It's quite handy as it allows users to see how clothes would look on them without actually trying them on, said Ciobanu.
The NVIDIA JetPack SDK provided all the tools needed to run AI models smoothly on the Jetson Orin, he said.
It's super-helpful to have a stable set of tools, especially when you're dealing with AI tech that keeps changing, said Ciobanu. It really cut down on the time and hassle for us developers, letting us focus more on the cool stuff we're building instead of getting stuck on tech issues.
Finding Lost Items With Robot Assistance Winner Tao, based in Ontario, Canada, created a robot to lessen the burden of searching for things lost around the house. His An Eye for an Item project took top honors at the Hackster challenge.
Finding lost objects is a chore, and recent developments in zero-shot object detection and LLMs make it feasible for a computer to detect arbitrary objects for us based on textual or pictorial descriptions, presenting an opportunity for automation, said Tao.
Tao said he needed robot computing capabilities to catalog objects in any unstructured environment - whether a living room or large warehouse. And he needed it to also perform real-time calculations for localization to help with navigation, as well as running inference on larger object detection models.
Jetson Orin was a perfect fit, supporting all functionality from text and image queries into NanoDB, to real-time odometry feedback, including leveraging Isaac ROS' hardware-accelerated AprilTag detections for drift correction, he said.
Other winners of the AI Innovation Challenge include:
George Profenza, Escalator people tracker, 2nd place, Generative AI Applications category
Dimiter Kendri, Cooking meals with a local AI assistant using Jetson AGX Orin, 3rd place, Generative AI Applications category
Vy Phan, ClearWaters Underwater Image Enhancement with Generative AI, 2nd place, Generative AI Models category
Huy Mai, Realtime Language Segment Anything on Jetson Orin, 2nd place, Generative AI Models category
Fakhrur Razi, Autonomous Int
More from Nvidia
25/07/2024
Hey, you. You're finally awake.
It's the summer of Elder Scrolls - whe...
24/07/2024
AI is set to transform the workforce - and the Georgia Institute of Technology's new AI Makerspace is helping tens of thousands of students get ahead of the...
24/07/2024
Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, softwa...
23/07/2024
Generative AI applications have little, or sometimes negative, value without acc...
23/07/2024
Businesses seeking to harness the power of AI need customized models tailored to their specific industry needs.
NVIDIA AI Foundry is a service that enables ent...
22/07/2024
AI and accelerated computing - twin engines NVIDIA continuously improves - are d...
22/07/2024
Team NVIDIA has triumphed at the Amazon KDD Cup 2024, securing first place Friday across all five competition tracks.
The team - consisting of NVIDIANs Ahmet E...
19/07/2024
Research published earlier this month in the science journal Nature used NVIDIA-powered supercomputers to validate a pathway toward the commercialization of qua...
19/07/2024
Research published earlier this month in the science journal Nature used NVIDIA-powered supercomputers to validate a pathway toward the commercialization of qua...
19/07/2024
AI has seen unprecedented growth - spurring the need for new training and educat...
19/07/2024
Research published earlier this month in the science journal Nature used NVIDIA-powered supercomputers to validate a pathway toward the commercialization of qua...
18/07/2024
Mistral AI and NVIDIA today released a new state-of-the-art language model, Mist...
18/07/2024
It's time for a sweet treat - the GeForce NOW Summer Sale offers high-perfor...
17/07/2024
Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, softwa...
16/07/2024
Editor's note: This post is part of our In the NVIDIA Studio series, which c...
15/07/2024
NVIDIA founder and CEO Jensen Huang and Meta founder and CEO Mark Zuckerberg wil...
12/07/2024
NVIDIA is taking an array of advancements in rendering, simulation and generativ...
11/07/2024
Unlock new experiences every GFN Thursday. Whether post-apocalyptic survival adventures, narrative-driven games or vast, open worlds, GeForce NOW always has som...
11/07/2024
Enhancing Japan's AI sovereignty and strengthening its research and development capabilities, Japan's National Institute of Advanced Industrial Science ...
10/07/2024
Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible and showcases new hardware, softwar...
10/07/2024
Improved cancer diagnostics - and improved patient outcomes - could be among the...
09/07/2024
Sphere, a new kind of entertainment medium in Las Vegas, is joining the ranks of legendary circular performance spaces such as the Roman Colosseum and Shakespea...
08/07/2024
Artificial intelligence is transforming the transportation industry, helping dri...
04/07/2024
GeForce NOW is bringing 22 new games to members this month.
Dive into the four titles available to stream on the cloud gaming service this week to stay cool an...
03/07/2024
Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, softwa...
28/06/2024
On a weekday afternoon, Ashwini Ashtankar sat on the bank of the Doodhpathri River, in a valley nestled in the Himalayas. Taking a deep breath, she noticed that...
27/06/2024
Editor's note: This post is part of Into the Omniverse, a series focused on ...
27/06/2024
Get ready to feel some chills, even amid the summer heat. Capcom's award-winning Resident Evil Village brings a touch of horror to the cloud this GFN Thursd...
26/06/2024
Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, softwa...
26/06/2024
Roblox is a colorful online platform that aims to reimagine the way that people ...
25/06/2024
Generative AI has revolutionized software development with prompt-based code generation - protein design is next.
EvolutionaryScale today announced the release...
24/06/2024
Multi-die chips, known as three-dimensional integrated circuits, or 3D-ICs, represent a revolutionary step in semiconductor design. The chips are vertically sta...
20/06/2024
Sit back and settle in for some epic storytelling. Tell Me Why and As Dusk Falls - award-winning, narrative-driven games from Xbox Studios - add to the 1,900+ g...
19/06/2024
Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible and showcases new hardware, softwar...
18/06/2024
The electric grid and the utilities managing it have an important role to play in the next industrial revolution that's being driven by AI and accelerated c...
17/06/2024
NVIDIA contributed the largest ever indoor synthetic dataset to the Computer Vision and Pattern Recognition (CVPR) conference's annual AI City Challenge - h...
17/06/2024
Making moves to accelerate self-driving car development, NVIDIA was today named an Autonomous Grand Challenge winner at the Computer Vision and Pattern Recognit...
17/06/2024
NVIDIA researchers are at the forefront of the rapidly advancing field of visual...
15/06/2024
NVIDIA founder and CEO Jensen Huang on Friday encouraged Caltech graduates to pu...
14/06/2024
When she was five years old, Veronica Miller (n e Teklai) and her family left their homeland of Eritrea, in the Horn of Africa, to escape an ongoing war with Et...
14/06/2024
NVIDIA today announced Nemotron-4 340B, a family of open models that developers ...
13/06/2024
Set sail for adventure, pirates. Sea of Thieves makes waves in the cloud this week. It's an adventure-filled GFN Thursday with four new games joining the Ge...
12/06/2024
Accelerated computing is transforming data processing and analytics for enterpri...
12/06/2024
The full-stack NVIDIA accelerated computing platform has once again demonstrated...
12/06/2024
Let's talk about NeRFs - no, not the neon-colored foam dart blasters, but ne...
12/06/2024
Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, softwa...
07/06/2024
Across industries, AI is supercharging innovation with machine-powered computation. In finance, bankers are using AI to detect fraud more quickly and keep accou...
06/06/2024
Capcom's latest entry in the iconic Street Fighter series, Street Fighter 6, punches its way into the cloud this GFN Thursday. The game, along with Ubisoft&...
05/06/2024
India's AI market is expected to be massive. Yotta Data Services is setting its sights on supercharging it. In this episode of NVIDIA's AI Podcast, Suni...
05/06/2024
NVIDIA launched NVIDIA Studio at COMPUTEX in 2019. Five years and more than 500 ...