A novel neural network for preserving cultural heritage via 3D image reconstruction

Researchers develop a novel neural network model that effectively reconstructs 3D digital images of relief-type cultural heritage objects from old photos

2024-10-31

(Press-News.org)

Relief carvings or relief sculptures are cultural heritage objects with figures that protrude from a background such as a wall or slab, creating a sense of depth. Commonly found at historical sites worldwide, these artworks are considered to be of immense historical and cultural value. Unfortunately, many such relief carvings at heritage sites across the world suffer from varying degrees of damage and deterioration over time. While modern 3D scanning and photogrammetry techniques can digitally preserve their current form, they cannot restore the original appearance of these carvings before damage. Additionally, traditional methods for restoring them are laborious, and require extensive manual intervention and specialized knowledge.

One promising method is 3D digital reconstruction of these reliefs from old photos, captured before the damage or deterioration. Unlike 3D sculptures or 2D paintings, reliefs have a shallow depth and are meant to be viewed from the front or either side. This means that a single image can effectively provide most of the information required for 3D digital reconstruction algorithms.

In a breakthrough, a multinational research team led by Professor Satoshi Tanaka from the College of Information Science and Engineering at Ritsumeikan University, Japan, along with Dr. Jiao Pan, from the University of Science and Technology Beijing, China, developed an innovative multi-task neural network for 3D reconstruction and digital preservation of reliefs using old photos. “Previously, we proposed a 3D reconstruction method for old reliefs based on monocular depth estimation from photos. Although we achieved 95% reconstruction accuracy, finer details such as human faces and decorations were still missing. This was due to the high compression of depth values in 2D relief images, making it difficult to extract depth variations along edges. Our new method tackles this by enhancing depth estimation, particularly along soft edges, using a novel edge-detection approach,” explains Prof. Tanaka.

The team also included Prof. Liang Li from Ritsumeikan University and Prof. Xiaojuan Ban from the University of Science and Technology Beijing. Their study was presented orally at the international conference ACM Multimedia 2024, held in Australia in October, and was published in MM ’24: Proceedings of the 32nd ACM International Conference on Multimedia on October 28, 2024.

The proposed multi-task neural network performs three tasks, namely semantic segmentation, depth estimation, and soft-edge detection, which work together to enhance the accuracy of 3D reconstruction. The core strength of the network lies in its depth estimation, achieved through a novel soft-edge detector and an edge matching module. Unlike the conventional binary edge classification, the soft-edge detector treats edge detection of relief data as a multi-classification task. Edges in relief images not only represent changes in brightness but also variations in curvature, known as “soft edges”. The soft-edge detector determines the degree of “softness” of these edges in relief images, enhancing depth estimation.

The edge matching module comprises two soft-edge detectors that extract multi-class soft-edge maps and a depth map, from an input relief photo. By matching and detecting differences between the two maps, the network focuses more on the soft-edge regions, resulting in more detailed depth estimation. Finally, the network optimizes a dynamic edge-enhanced loss function, which includes loss from all three tasks, and produces clear and detailed 3D images of reliefs.

The researchers applied this innovative model to reconstruct the hidden reliefs of Borobudur Temple. “The wall reliefs on the ground level of the Borobudur Temple, a UNESCO World Heritage Site in Indonesia, are covered by stone walls due to reinforcement work carried out during the Dutch colonial period and cannot be viewed. Our multi-task neural network successfully reconstructed these hidden sections of Borobudur’s ground-level reliefs from surviving old photographs. Through computer visualization and virtual reality, our research now allows virtual exploration of these unseen treasures,” says Prof. Tanaka, highlighting the potential impact of their work.

Discussing the future implications of these findings, he says, “Our technology holds vast potential for preserving and sharing cultural heritage. It opens new opportunities not only for archeologists but also for immersive virtual experiences through VR and metaverse technologies, preserving global heritage for future generations.”

***

Reference

Title of original paper: Reconstructing, Understanding, and Analyzing Relief-Type Cultural Heritage from a Single Old Photo

Conference: ACM Multimedia 2024

DOI: 10.1145/3664647.3681612

About Ritsumeikan University, Japan

Ritsumeikan University is one of the most prestigious private universities in Japan. Its main campus is in Kyoto, where inspiring settings await researchers. With an unwavering objective to generate social symbiotic values and emergent talents, it aims to emerge as a next-generation research university. It will enhance researcher potential by providing support best suited to the needs of young and leading researchers, according to their career stage. Ritsumeikan University also endeavors to build a global research network as a “knowledge node” and disseminate achievements internationally, thereby contributing to the resolution of social/humanistic issues through interdisciplinary research and social implementation.

Website: http://en.ritsumei.ac.jp/

Ritsumeikan University Research Report: https://www.ritsumei.ac.jp/research/radiant/eng/

About Professor Satoshi Tanaka from Ritsumeikan University, Japan

Dr. Satoshi Tanaka is currently a Professor at the College of Information Science and Engineering at Ritsumeikan University. He obtained his M.S. and Ph.D. degrees from Waseda University in 1982 and 1987, respectively. In addition to being a fellow of the Ritsumeikan Advanced Research Academy (RARA), he is also a member of the Kinugasa Research Organization Art Research Center and the Research Organization of Science and Technology Research Center of Advanced ICT for Medical and Healthcare. He has over 1,700 publications with over 100,000 citations. His research interests include computer graphics, scientific visualization, digital humanities, and medical visualization.

About Dr. Jiao Pan from the University of Science and Technology Beijing

Dr. Jiao Pan is currently a lecturer at the School of Intelligence Science and Technology at the University of Science and Technology Beijing. She received her Ph.D. degree from Professor Satoshi Tanaka’s laboratory at Ritsumeikan University in 2022. The current study is based on research conducted by Dr. Pan during her Ph.D. on 3D reconstruction of buried Borobudur Temple reliefs.

END

[Attachments] See images for this press release:

ELSE PRESS RELEASES FROM THIS DATE:

Sleep apnea contributes to dementia in older adults, especially women

2024-10-31

A common yet underdiagnosed sleep disorder contributes to the development of dementia among adults — particularly women, a Michigan Medicine study suggests. Investigators uncovered this by examining survey and cognitive screening data from more than 18,500 adults to determine the potential effect of known or suspected obstructive sleep apnea on the risk for dementia. Obstructive sleep apnea is a chronic sleep disorder characterized by episodes disrupted or restricted breathing during sleep. For all adults age 50 and older, having known obstructive sleep apnea or its symptoms — as people often do not know they have the problem ...

The silk thread that can turn clothes into charging stations

2024-10-31

Imagine a sweater that powers electronics to monitor your health or charge your mobile phone while running. This development faces challenges because of the lack of materials that both conduct electricity stably and are well suited for textiles. Now a research group, led by Chalmers University of Technology in Sweden, presents an ordinary silk thread, coated with a conductive plastic material, that shows promising properties for turning textiles into electricity generators. Thermoelectric textiles convert temperature differences, for example between our bodies and the surrounding ...

Glaucoma drug shows promise against neurodegenerative diseases, animal studies suggest

2024-10-31

A drug commonly used to treat glaucoma has been shown in zebrafish and mice to protect against the build-up in the brain of the protein tau, which causes various forms of dementia and is implicated in Alzheimer’s disease. Researchers in the UK Dementia Research Institute at the University of Cambridge screened more than 1,400 clinically-approved drug compounds using zebrafish genetically engineered to make them mimic so-called tauopathies. They discovered that drugs known as carbonic anhydrase inhibitors – of which the glaucoma drug methazolamide is one – clear tau build-up and reduce signs of the disease in zebrafish and mice carrying the mutant forms of tau ...

Human proteins identified that explain inter-individual differences in functional brain connectivity

2024-10-31

BIRMINGHAM, Ala. – A long-standing goal of neuroscience is to understand how molecules and cellular structures on a microscale give rise to communication between brain regions at the macroscale. A study published in Nature Neuroscience now identifies, for the first time, hundreds of brain proteins that explain inter-individual differences in functional connectivity and structural covariation in the human brain. “A central goal of neuroscience is to develop an understanding of the brain that ultimately describes the mechanistic basis of human cognition and behavior,” said Jeremy Herskowitz, Ph.D., associate professor in the University of Alabama at Birmingham ...

A newly developed algorithm shows how a gene is expressed at microscopic resolution

2024-10-31

They say a picture is worth a thousand words. A new method, developed by University of Michigan researchers, creates images that are worth many gigabytes of data, which could revolutionize the way biologists study gene expression. Seq-Scope, developed by Jun Hee Lee, Ph.D., Hyun Min Kang, Ph.D., and their colleagues, was first described in Cell in 2021 as the first method to analyze gene expression at sub micrometer-scale spatial resolution. To compare, a single human hair ranges from 20 to 200 micrometers in width. The team has since improved Seq-Scope, making it more versatile, ...

Why elephants never forget but fleas have, well, the attention span of a flea

2024-10-31

Researchers at the Complexity Science Hub and Santa Fe Institute have developed a model to calculate how quickly or slowly an organism should ideally learn in its surroundings. An organism’s ideal learning rate depends on the pace of environmental change and its life cycle, they say. Every day, we wake to a world that is different, and we adjust to it. Businesses face new challenges and competitors and adapt or go bust. In biology, this is a question of survival: every organism, from bacteria to blue whales, faces the challenge of adapting to environments that ...

Childhood neglect associated with stroke, COPD, cognitive impairment, and depression

2024-10-31

Toronto, ON, – New research from the University of Toronto found that childhood neglect, even in the absence of childhood sexual abuse and physical abuse, is linked with a wide range of mental and physical health problems in adulthood. “While a large body of research has established the detrimental impact of childhood physical and sexual abuse on adult health outcomes, much less is known about whether neglect, in the absence of abuse, has similar negative outcomes,” said first author, Linxiao Zhang, a PhD student at the Factor-Inwentash Faculty of Social Work (FIFSW) at the University of Toronto. “Our research underlines the importance of health professionals ...

Landmark 20-year study of climate change impact on permafrost forests

2024-10-31

In perhaps the first long-term study of CO2 fluxes in northern forests growing on permafrost, an Osaka Metropolitan University-led research team has found that climate change increased not only the sources of carbon, but also the CO2 sinks. The 20-year observation from 2003-2022 in the interior of Alaska showed that while CO2 sinks turned into sources during the first decade, the second decade showed a nearly 20% increase in CO2 sinks. Graduate School of Agriculture Associate Professor Masahito Ueyama and colleagues found that warming led to wetness, which in turn aided the growth of black spruce trees. During photosynthesis, the growing trees were using the increasing ...

Researchers take broadband high-resolution frequency combs into the UV

2024-10-31

WASHINGTON — Researchers have developed a new ultrafast laser platform that generates ultra-broadband ultraviolet (UV) frequency combs with an unprecedented one million comb lines, providing exceptional spectral resolution. The new approach, which also produces extremely accurate and stable frequencies, could enhance high-resolution atomic and molecular spectroscopy. Optical frequency combs — which emit thousands of regularly spaced spectral lines — have transformed fields like metrology, spectroscopy and precision timekeeping via optical atomic clocks, earning the 2005 Nobel Prize in Physics. The ...

Not going out is the “new normal” post-Covid, say experts

2024-10-31

Compared with just before the Covid-19 pandemic, people are spending nearly an hour less a day doing activities outside the home, behaviour that researchers say is a lasting consequence of the pandemic. A new study published in the peer-reviewed Journal of the American Planning Association reveals an overall drop since 2019 of about 51 minutes in the daily time spent on out-of-home activities, plus an almost 12-minute reduction in time spent on daily travel such as driving or taking public transportation. The analysis, based on a survey of 34,000 Americans, ...

LAST 30 PRESS RELEASES:

New partnerships to provide travel grants, coursework in genomic approaches to infectious disease for underrepresented aspiring physicians

Off the clothesline, on the grid: MXene nanomaterials enable wireless charging in textiles

How COVID-19 transformed family dinners

New ESO image captures a dark wolf in the sky

New research reveals delayed evolutionary origin of Asteriidae sea stars

A paper-aluminum combo for strong, sustainable packaging