Food Videos Teach Robot "chef" How to Cook

The robotic "chef" was taught by researchers to replicate dishes seen in culinary films.

The robotic chef was trained with eight basic salad recipes by the University of Cambridge academics. The robot learned how to create the meal by watching a video of a human preparing it and then replicating the actions on its own.

Also, the robot's culinary repertoire was expanded in little ways with the help of the videos. The robot concluded the trial by creating its own ninth recipe. Their findings, published in IEEE Access, show that videos can be an invaluable and extensive source of data for automated food production and may make the use of robot chefs more practical and affordable.

Cooking is a difficult task for a robot, despite the decades of science fiction featuring robotic chefs. While there are several prototype robot chefs that have been developed by private companies, they are not yet available to the public and are much outclassed by human cooks.

Learning new recipes is easy for human cooks thanks to the observation of others (or videos on YouTube), but teaching a robot to prepare a wide variety of foods is laborious and time-consuming.
The paper's first author, Grzegorz Sochacki, is a member of the Department of Engineering at Cambridge University. "We wanted to see whether we could train a robot chef to learn in the same incremental way that humans can," he said.

Ph.D. student Sochacki and his fellow researchers in Professor Fumiya Iida's Bio-Inspired Robotics Laboratory came up with eight easy salad recipes and recorded themselves preparing them. The team then trained their robot chef using a pre-existing neural network. The veggies and fruits called for in the eight salad recipes were among those already programmed into the neural network's database of known objects.

A knife, materials, and the human demonstrator's arms, hands, and face were all identified by the robot thanks to its frame-by-frame analysis of the footage using computer vision techniques. Recipes and movies were turned into vectors, and the robot used vector algebra to figure out how closely they matched up with each other.

The robot knew which recipe was being cooked by the human chef by accurately identifying the items used and the human's movements. A human demonstration holding a knife and a carrot would lead the robot to believe that the carrot will be cut.

Despite only detecting 83% of the human chef's activities, the robot was able to correctly identify the recipe 93% of the time across all 16 movies it watched. The robot also recognized that little adjustments to a recipe, like doubling the ingredients or making a simple mistake, were just that, and not an entirely new recipe. In addition, the robot recognized a newly demonstrated eighth salad, saved the recipe to its database, and successfully recreated the dish.

Sochacki remarked, "It's amazing how much nuance the robot was able to detect. These recipes aren't complex -- they're essentially chopped fruits and vegetables, but it was really effective at recognizing, for example, that two chopped apples and two chopped carrots is the same recipe as three chopped apples and three chopped carrots."

Unlike food films produced by some social media influencers, which are full of fast cuts and visual effects and rapidly cut between the person cooking the meal and the dish they are preparing, the videos used to train the robot chef is more straightforward and have more focus on the actual preparation of the food. The robot may have trouble recognizing an object like a carrot if the person demonstrating it has their hand wrapped around it. The human would need to hold the carrot so the robot can see the entire vegetable.

Sochacki stated, "Our robot isn't interested in the sorts of food videos that go viral on social media -- they're simply too hard to follow," said Sochacki. "But as these robot chefs get better and faster at identifying ingredients in food videos, they might be able to use sites like YouTube to learn a whole range of recipes."

Engineering and Physical Sciences Research Council (EPSRC) of UK Research and Innovation (UKRI) provided funding for this study