How A Virtual Scavenger Hunt Could Train Robots To Find Things In Your Home

Artificial intelligence models need to be trained on data in order to be useful. The more data you plug in, the more the automation tends to improve. A new initiative aims to train AI systems to better navigate spaces by sending them on scavenger hunts inside virtual homes.

According to MIT Technology Review, the scavenger hunt initiative, which was co-developed by researchers at Georgia Tech and Facebook, sends AI bots scurrying through virtual homes cluttered with lamps, tables, couches, and other normal furniture. The goal is for the bot to try to find a specific item in the virtual dwelling after being presented with a question using natural language. So, for example, a bot could be asked “What color is the refrigerator” or “Is the bathroom door open?” In order to answer the question, the AI system has to be able to parse it and then set out exploring the digital home in order to determine the answer.

The research was presented at F8, Facebook’s annual developer conference today. As part of the second day of F8 keynotes, several Facebook executives discussed a number of AI initiatives on stage, and the company today unveiled both a system that utilizes public Instagram data to perform world-record image recognition accuracy, and for the first time released a breakdown of many of the ways it uses AI to automatically detect objectionable content.

“The goal ” of the scavenger hunt project, Devi Parikh, a computer scientist at Georgia Tech and Facebook AI Research (FAIR), who developed the contest with her colleague and husband, Dhruv Batra, told MIT Tech Review, “is to build intelligent systems that can see, talk, plan, and reason.”

The research team utilized numerous types of machine learning to train the bots to answer questions about the virtual home. The bots also develop a level of common sense by running multiple times through the digital space looking for the requested object. “For instance,” MIT Tech Review wrote, “over time, the agent learns that cars are usually found in the garage, and it understands that garages can usually be found by going out the front or back door.”

Researchers are increasingly utilizing virtual environments to train AI systems, MIT Tech Review continued. That’s because the method is thought to be useful in expanding the capabilities of such systems as well as helping overcome their fundamental limitations.

Ultimately, it’s not clear what this research will lead to, but it could be useful for building home robots that actually can respond to voice commands. Getting systems like a Roomba to successfully parse words like “living room” or “bedroom” will require significant amounts of training. “We are clearly headed into an age of assistive agents,” Batra told MIT Tech Review. “These things will develop eyes, and after that, they will follow you around.”