Researchers develop a novel neural network model that effectively reconstructs 3D digital images of relief-type cultural heritage objects from old photos

Relief-type cultural heritage objects are commonly found in many historical sites worldwide, but often suffer from varying levels of damage and deterioration. Traditional methods for image reconstruction require extensive manual labor and specialized knowledge. Now, researchers have developed a novel neural network model that can reconstruct these reliefs as three-dimensional digital images from old photographs containing their pre-damage information. This innovative technology paves the way for accurate digital preservation of valuable cultural heritage objects.

Relief carvings or relief sculptures are cultural heritage objects with figures that protrude from a background such as a wall or slab, creating a sense of depth. Commonly found at historical sites worldwide, these artworks are considered to be of immense historical and cultural value. Unfortunately, many such relief carvings at heritage sites across the world suffer from varying degrees of damage and deterioration over time. While modern 3D scanning and photogrammetry techniques can digitally preserve their current form, they cannot restore the original appearance of these carvings before damage. Additionally, traditional methods for restoring them are laborious, and require extensive manual intervention and specialized knowledge.

One promising method is 3D digital reconstruction of these reliefs from old photos, captured before the damage or deterioration. Unlike 3D sculptures or 2D paintings, reliefs have a shallow depth and are meant to be viewed from the front or either side. This means that a single image can effectively provide most of the information required for 3D digital reconstruction algorithms.

In a breakthrough, a multinational research team led by Professor Satoshi Tanaka from the College of Information Science and Engineering at Ritsumeikan University, Japan, along with Dr. Jiao Pan, from the University of Science and Technology Beijing, China, developed an innovative multi-task neural network for 3D reconstruction and digital preservation of reliefs using old photos. “Previously, we proposed a 3D reconstruction method for old reliefs based on monocular depth estimation from photos. Although we achieved 95% reconstruction accuracy, finer details such as human faces and decorations were still missing. This was due to the high compression of depth values in 2D relief images, making it difficult to extract depth variations along edges. Our new method tackles this by enhancing depth estimation, particularly along soft edges, using a novel edge-detection approach,” explains Prof. Tanaka.

The team also included Prof. Liang Li from Ritsumeikan University and Prof. Xiaojuan Ban from the University of Science and Technology Beijing. Their study was presented orally at the international conference ACM Multimedia 2024, held in Australia in October, and was published in MM ’24: Proceedings of the 32^nd ACM International Conference on Multimedia on October 28, 2024.

The proposed multi-task neural network performs three tasks, namely semantic segmentation, depth estimation, and soft-edge detection, which work together to enhance the accuracy of 3D reconstruction. The core strength of the network lies in its depth estimation, achieved through a novel soft-edge detector and an edge matching module. Unlike the conventional binary edge classification, the soft-edge detector treats edge detection of relief data as a multi-classification task. Edges in relief images not only represent changes in brightness but also variations in curvature, known as “soft edges”. The soft-edge detector determines the degree of “softness” of these edges in relief images, enhancing depth estimation.

The edge matching module comprises two soft-edge detectors that extract multi-class soft-edge maps and a depth map, from an input relief photo. By matching and detecting differences between the two maps, the network focuses more on the soft-edge regions, resulting in more detailed depth estimation. Finally, the network optimizes a dynamic edge-enhanced loss function, which includes loss from all three tasks, and produces clear and detailed 3D images of reliefs.

The researchers applied this innovative model to reconstruct the hidden reliefs of Borobudur Temple. “The wall reliefs on the ground level of the Borobudur Temple, a UNESCO World Heritage Site in Indonesia, are covered by stone walls due to reinforcement work carried out during the Dutch colonial period and cannot be viewed. Our multi-task neural network successfully reconstructed these hidden sections of Borobudur’s ground-level reliefs from surviving old photographs. Through computer visualization and virtual reality, our research now allows virtual exploration of these unseen treasures,” says Prof. Tanaka, highlighting the potential impact of their work.

Discussing the future implications of these findings, he says, “Our technology holds vast potential for preserving and sharing cultural heritage. It opens new opportunities not only for archeologists but also for immersive virtual experiences through VR and metaverse technologies, preserving global heritage for future generations.”

Title of original paper: Reconstructing, Understanding, and Analyzing Relief-Type Cultural Heritage from a Single Old Photo
Conference: ACM Multimedia 2024
DOI: 10.1145/3664647.3681612

About Professor Satoshi Tanaka from Ritsumeikan University, Japan

Dr. Satoshi Tanaka is currently a Professor at the College of Information Science and Engineering at Ritsumeikan University. He obtained his M.S. and Ph.D. degrees from Waseda University in 1982 and 1987, respectively. In addition to being a fellow of the Ritsumeikan Advanced Research Academy (RARA), he is also a member of the Kinugasa Research Organization Art Research Center and the Research Organization of Science and Technology Research Center of Advanced ICT for Medical and Healthcare. He has over 1,700 publications with over 100,000 citations. His research interests include computer graphics, scientific visualization, digital humanities, and medical visualization.

About Dr. Jiao Pan from the University of Science and Technology Beijing

Dr. Jiao Pan is currently a lecturer at the School of Intelligence Science and Technology at the University of Science and Technology Beijing. She received her Ph.D. degree from Professor Satoshi Tanaka’s laboratory at Ritsumeikan University in 2022. The current study is based on research conducted by Dr. Pan during her Ph.D. on 3D reconstruction of buried Borobudur Temple reliefs.