How AI and robotics are reconstructing a 2,000-year-old fresco in Pompeii

When we sit down to solve a jigsaw puzzle, there’s always one thing we take for granted: the picture on the box. Without that point of reference, we’d be pulling our hair out, trying and failing to rebuild a jumbled pile of miscellaneous pieces.

That’s exactly what was happening in the ancient Roman city of Pompeii, where over 10,000 fragmented pieces from 2,000-year-old frescoes have been lying around for decades, waiting for someone to solve the puzzle. Now, a team of scientists led by the Venice-based Italian Institute of Technology may have found a solution: train a robot to do it.

Dubbed “Reconstructing the Past: Artificial Intelligence and Robotics meet Cultural Heritage” (RePAIR), the project is funded by a €3.5-million grant (just under $4 million) from a European Commission that supports high-risk projects working toward “radically new future technologies.” The project will be developed in two phases: first, an algorithm will reconstruct the puzzle digitally, then a pair of robotic hands will put the puzzle (i.e. the frescoes) back together. This is the first time that AI will be used as an archeological tool at such a large scale, and the first time that robotic hands will be put in charge of so many pieces. If the project works, the scientists are hoping to deploy the technology in other cultural heritage sites around the world, like historic churches in Italy, or even the Ancient city of Palmyra in war-torn Syria.

A key part of the project will be teaching the algorithm how to study like an archeologist and think like a puzzle master. The puzzle-solving AI was developed in collaboration with a team at Ben Gurion University of the Negev, in Israel, and it works like an infinitely more intricate version of the popular memory game “find the pair.” The computer software compares all the fragments in pairs and rates their degree of similarity based on the shape of the pieces, how they fit together, plus how the illustrations on the fragments match up. Normally, this process can be done manually (using a computer), but the team is now teaching the algorithm how to compare pieces on its own.

Thanks to a team of archeologists from the University of Lausanne, who have attempted to solve the puzzle in the past, they already have about a dozen reconstructed clusters (about 10 pieces each) that they are feeding the algorithm. If the computer can put those pieces back together, they will know the system can be deployed at a larger scale.

The robot will be deployed in Pompeii sometime next summer, but for now, the scientists are working on several projects in parallel. While one team is building the algorithm, another is 3D scanning a large sample of fragments so they can be put into the database (once the robot is fully operational, it will scan them on its own). And another team is working on the physical infrastructure and robotic hands that will eventually pick up the pieces and rebuild the frescoes. “One day, you will take all the pieces, put them into a room, lock the door, come back after a few days, and you will find the fresco completely reassembled,” says Marcello Pelillo, a professor of computer science and artificial intelligence at the University of Venice. (Though he admits things probably won’t go quite so smoothly.)

The fragments come from two separate rooms (including the ceilings) in a building called the House of Painters at Work, (named because artists were in the middle of painting when Mount Vesuvius erupted in A.D. 79.) They run the gamut from tiny fragments to palm-sized pieces, most of them damaged, and many of them missing. The scientists are working with archeologists and art historians, who will narrow down the pieces to an initial 1,000 they think belong in the same cluster, or at least on the same wall.

Within a year or two, Pelillo says they should have a preliminary image of the frescoes, but a complete and final image will take longer. When the robotic platform is finished, the plan is for the robot to do it all at once: “The robot will do the scanning itself, then after doing the scanning, it will solve the puzzle, then after solving the puzzle it will reassemble it,” says Pelillo. (The set-up will look like a bridge, with two robotic arms suspended from a metal frame.)

At the end of the project, the frescoes will be displayed in the Archeological Park of Pompeii for visitors to experience, but they will inform further research, too. Ariana Traviglia, the director of the IIT Center for Cultural Heritage and Technology, says we can learn a lot from the reconstructed frescoes and the patterns on them. “Frescoes were not like wallpaper, they were not all the same,” she says. “In each of them, the artist was putting something new following the taste of the owner.” Already, Traviglia can tell that the family who lived there was wealthy: “the stucco all around the ceiling is very high quality.”

If the project is successful, the technology can save conservators precious time that can be spent on restoration. And while this project is currently focused on relatively flat fragments, Traviglia wants to take things up a notch and try more complex shapes next, like ancient statues, wrecks, and amphoras.

Eventually, the team could put all of this into practice in other cultural heritage sites, like the host of ancient churches that crumbled during the earthquakes in Central Italy four years ago. “All over the world, we have so many pieces of broken frescoes and broken objects from our past, and they are so tiny that we don’t have the time and personnel to put them together,” she says. “It will really change the things we can do.”