Impractical Stencil Aperture Designs to Enable M0201 Assembly
Effectiveness of I/O Stencil Aperture Modifications on BTC Void Reduction
Microalloyed Sn-Cu Pb-Free Solder for High Temp
Selective Reflow Rework Process
Impact of Thermal Loading on the Structural Intergrity of 3D TSV Package
Design and Fabrication of Ultra-Thin Flexible Substrate
Influence of PCB Surface Features on BGA Assembly Yield
Last Will and Testament of the BGA Void
Latest Industry News
Print These Electronic Circuits Directly Onto Skin
Compal increasingly asked to diversify production bases
Intel's margins tumble as customers shift to cheaper chips, shares slide 10%
From Foldable Phones to Stretchy Screens
6 Considerations for Integrating Sensors in Vehicles
Bill Gates Says Unhappy Customers Are Good for Your Business. Here's Why.
iPhone 12 review: Upgrade for the camera, not 5G
Apple's shifting supply chain creates boomtowns in rural Vietnam

Breakthrough Gives Robots Human-like 3D Mental Model of their Environment

Breakthrough Gives Robots Human-like 3D Mental Model of their Environment
MIT engineers have developed a representation of spatial perception for robots that is modeled after the way humans perceive and navigate the world.
Technology Briefing


Since the earliest days of computing, people have imagined robotic servants able to follow high-level, Alexa-type commands, such as “Go to the kitchen and fetch me a coffee cup.” But as MIT engineers explain carrying out such high-level tasks means that robots will have to be able to perceive their physical environment as humans do.

In order to function in the world, you need to have a mental model of the environment around you. This is something that’s effortless for humans. But for robots, it’s a painfully hard problem, which requires transforming pixel values that they see through a camera, into an understanding of the world.

Fortunately, these MIT engineers have developed a representation of spatial perception for robots that is modeled after the way humans perceive and navigate the world. The new model, called 3D Dynamic Scene Graphs, enables a robot to quickly generate a 3D map of its surroundings that also includes objects and their semantic labels such as people, rooms, walls, tables, chairs, and other structures that the robot is likely to see in its environment. The model also allows the robot to extract relevant information from the 3D map and to query the location of objects, rooms, or the moving people in its path.

This compressed representation of the environment is useful because it allows a robot to quickly make decisions and plan its path. This is not too far from what we do as humans. If you need to plan a path from your home to work, you don’t plan every single position you need to take. You just think at the level of streets and landmarks, which helps you plan your route faster.

Beyond domestic helpers, the researchers say robots that adopt this new kind of mental model of the environment may also be suited for other high-level jobs, such as working side-by-side with people on a factory floor or exploring a disaster site for survivors.

The research presented recently at the 2020 Robotics: Science and Systems virtual conference.

Why is this important? Until now, robotic vision and navigation have advanced mainly along two routes: the first involves 3D mapping that enables robots to reconstruct their environment in three dimensions as they explore in real-time; and the second uses semantic segmentation, which helps a robot classify features in its environment as semantic objects, such as a car versus a bicycle, which so far is mostly done with 2D images. The new MIT model of spatial perception is the first to generate a 3D map of the environment in real-time, while also labeling objects, people, and structures within that 3D map.

The key component of the team’s new model is Kimera, an open-source library that the team previously developed to simultaneously construct a 3D geometric model of an environment, while encoding the likelihood that an object is, say, a chair versus a desk. Like the mythical creature that is a mix of different animals, the team wanted Kimera to be a mix of mapping and semantic understanding in 3D.

Kimera works by taking in streams of images from a robot’s camera, as well as inertial measurements from onboard sensors, to estimate the trajectory of the robot or camera and to reconstruct the scene as a 3D mesh, all in real-time.

To generate a semantic 3D mesh, Kimera uses an existing neural network trained on millions of real-world images, to predict the label of each set of pixels, and then projects these labels in 3D using a technique known as ray-casting, commonly used in computer graphics for real-time rendering.

The result is a map of a robot’s environment that resembles a dense, three-dimensional mesh, where each face is color-coded as part of the objects, structures, and people within the environment.

If a robot were to rely on this mesh alone to navigate through its environment, it would be a computationally expensive and time-consuming task. So the researchers developed algorithms to construct 3D dynamic “scene graphs” from Kimera’s initial, highly dense, 3D semantic mesh. In the case of the 3D dynamic scene graphs, the associated algorithms abstract, or break down, Kimera’s detailed 3D semantic mesh into distinct semantic layers, such that a robot can “see” a scene through a particular layer, or lens. This layered representation avoids a robot having to make sense of billions of points and faces in the original 3D mesh. Within the layer of objects and people, the researchers have also been able to develop algorithms that track the movement and the shape of humans in the environment in real-time.

This is essentially enabling robots to have mental models similar to the one humans use. And it is expected to impact many applications, including self-driving cars, search and rescue, collaborative manufacturing, and domestic robots.


No comments have been submitted to date.

Submit A Comment

Comments are reviewed prior to posting. You must include your full name to have your comments posted. We will not post your email address.

Your Name

Your Company
Your E-mail

Your Country
Your Comments

Board Talk
Causes of Blowholes
Tips When Moving a Reflow Oven
Assembling Boards with BGAs on Both Sides
Larger Stencil Apertures and Type 4 Paste
5 vs 8-Zone Ovens
Component Moisture Question?
BGA Components and Coplanarity
How To Verify Cleanliness After Rework and Prior to Re-coating?
Ask the Experts
Initial Screen Print Test Board
HASL Surface Finish and Coplanarity
Legend Marking Discoloration
Cleanliness Testing
Stencil Cleaning Frequency
Exposed Copper Risk
Spotting After DI Water Cleaning
ESD Grounding - 1 Meg Ohm Resistor