Teaching Interns The Joy Of Collecting Trash With NVIDIA Jetson Robots

You’ve just graduated high school and secured yourself with an internship at one of the biggest tech companies in the world. Your first day on the job? Serving tea and coffee, picking up trash, and marshaling people around.

Under normal circumstances, that would be the classic internship that many of us have dreaded and endured.

For a group sent to NVIDIA, such a grueling internship was made much more enjoyable as the students were not to perform the tasks themselves, but create robots to do it for them.

If only I could have seen those faces – the horror of doing the most boring job in the world, but then taught how to do it with the most advanced technology in the world. NVIDIA taught 12 high school ‘Jetson’ interns over seven weeks on how to use embedded computing and AI neural networks to work with a variety of robots, and use them to perform tasks in the real world. Each robot was different, and used for a different task, but were all powered by NVIDIA’s Jetson platform.

Jetson is NVIDIA’s embedded systems DevKit and Module, used to power robots and fringe devices. It was originally powered by a Tegra-based SoC (the same SoC found in the various NVIDIA SHIELD devices) on the Jetson TX1, which has since been updated to the Parker SoC on the Jetson TX2. These kits are ARM Cortex A57 CPUs coupled with NVIDIA’s Pascal graphics processors.

The Jetson boards are like many other dev kits in that they support a multitude of expansion capabilities for connecting other microcontrollers and ICs, through GPIO, UART, I2C, and SPI. The important difference here is that these Jetson boards have fully fledged GPUs that support NVIDIA’s CUDA programming language. With CUDA, developers can leverage AI and deep learning capabilities on a device directly through something like TensorRT, without having to offload to a powerful system over a network connection (although, that’s still possible).

For those interested, the Jetson TX1 Developer Kit SE is available at discount for NVIDIA Developers (which is easy to sign-up for) at $199. These are the Maxwell-based Tegra GPUs, but are more than capable kits for learning about CUDA and deep learning for embedded devices. The TX1 is also the heart of the MIT RACECAR one of the students used as a project listed later in this article.

The Internship

There were three teams, and they each tackled a different problem. The robots had to be autonomous and perform their function under the very tough condition of a changing environment. Team Electron took to the task of transporting food and drink between cubicles. This can be seen as waitering at a restaurant, or for office supply runs for lazy employees.

The robot used was a Turtlebot which navigated around an office to deliver supplies when instructed over slack. The bot would detect what was placed on top of it with multiple cameras, and then take the object to its destination.

While the Turtlebot could navigate around the office just fine because it had a map of the floor plan, the team quickly ran into issues due to the changing environment and equipment that isn’t detailed in the floor plan – such as trash cans (perhaps sabotage from the other team? read below). The bot could then use the Jetson board to figure out what the collision objects were, and move around them to continue on its destination.

Team Trashformers was on refuse collection duty, as they created a robot to pick up trash around the office of some presumable very messy NVIDIA employees. The team trained the image system to recognize Styrofoam cups, pick them up, and place them in a waste area. However, the team ran into issues from a motion standpoint.

The robot used was a Robotis GP that had been fitted with a Jetson module and a webcam; the problem with this is that it made the bot top-heavy. While Jetson can help with the decision-making and vision, applying it to motion controls on a robot is tough. This is when NVIDIA’s Isaac robot simulation comes in, as it could be used to teach the robot to move with the extra mass more effectively.

The third team, Formula 1 Epoch, turned an MIT RACECAR into a fire marshal, guiding people away from danger as well as doing a headcount and tracking obstacles. While this doesn’t sound very complex from a human perspective, this is basically trying to build an autonomous vehicle in miniature. The car required four neural network running in parallel in order to function.

First it needed to check for people by analyzing legs, bodies and clothing. Next it needed to know how to steer, and this required analysis of LiDAR data (another network), camera images for object detection (yet another network), and finally a navigation routine to send the steering signal (a fourth network that combines the values from the three previous networks). All of this needs to be done in near real-time. The students managed to get a system running that was capable of inferencing five actions per second – which may not sound like a lot, but considering this was on a small RC vehicle, it’s quite impressive.

A New Generation For The Next Generation

Back at SIGGRAPH 2017, we called NVIDIA’s AI development an explosive field. In just two years we’ve gone from pre-programmed and manual action robots with ‘smart’ applications that took tens of thousands of man-hours to train, to fully automated, self-contained AIs that teach themselves in a fraction of the time. The growth in the deep learning field is simply incredible, but it still requires a human touch to setup the AIs in the first place.

NVIDIA’s internships are about preparing a new generation of students for the next generation of technology. Working with budding engineers with an enthusiasm for robotics, and teaching them new ways in which to utilize the hardware in front of them. Don’t teach a robot to walk, teach it how to teach itself.

The lucky few this year, got to play around with some cutting edge technology while having their brains crammed with some seriously advanced concepts surrounding deep learning. The students were already familiar with many of the robotics and control systems, but integrating the element of AI and decision-making is a hard programming challenge for anyone.

Summer camp for these students was more than just a chance at high-tech cram school, it was also a chance to explore the development process of an emerging technology. Not only that, but a chance to document it for others. You see, what the students created over the seven weeks wasn’t just some cool tech demos, they also released their code and designs to the public over GitHub. The sources for this article? Also written by the interns at NVIDIA. We’re all looking forward to what these bright minds can bring us in the future.