(Press-News.org) How well do the algorithms used in the AI-supported analysis of medical images perform their respective tasks? This depends to a large extent on the metrics used to evaluate their performance. An international consortium led by scientists from the German Cancer Research Center (DKFZ) and the National Center for Tumor Diseases (NCT) in Heidelberg has compiled the knowledge available worldwide on the specific strengths, weaknesses and limitations of the various validation metrics. With "Metrics Reloaded", the researchers are now providing a widely available online tool that supports users in selecting the most suitable algorithm for their task.
More and more areas of medicine are relying on support from artificial intelligence (AI). This is particularly true for the wide range of questions based on the evaluation of image data: for example, doctors search mammograms for the tiny foci of cancer or calculate the volume of a brain tumor based on the tomographic images from an MRI. They use endoscopic images of the intestine to track down polyps, and when evaluating microscopic tissue sections, subtle changes in individual cells have to be detected.
But are the algorithms used for these different types of image analysis really always suitable for the task in hand? This depends to a large extent on which measured variables, referred to as "metrics" in technical terms, they record - and whether these are actually suitable for the task in question.
"We often notice that validation metrics are used that are not at all relevant to the task from a clinical perspective," says Lena Maier-Hein from the DKFZ, citing an example: "When searching for metastases in the brain, it is initially more important that the algorithm detects even the tiniest lesions than that it can define the contours of each individual metastasis with high precision."
Lena Maier-Hein and her colleagues fear that the use of unsuitable validation metrics can hinder scientific progress and delay the introduction of important image analysis methods into clinical practice.
But which metrics are suitable for a given clinical question, taking into account all strengths, weaknesses and limitations? To find out, the DKFZ data scientists used a multi-stage, structured process to survey opinion leaders from academia and industry from over 70 research institutions worldwide. The survey allowed them to gather information that was previously only available in scattered locations around the world.
"With this work, we are making reliable and comprehensive information on the problems and pitfalls associated with validation metrics in image analysis available to experts for the first time," says Annika Reinke, one of the lead authors.
As a structured body of information that can be accessed by researchers from all disciplines, the work aims to increase understanding of a key problem in AI-assisted image analysis. Although the focus is on the analysis of medical images, the information can also be transferred to other areas of image analysis.
In a second paper, the expert consortium led by the Heidelberg researchers now describes "Metrics Reloaded": A comprehensive framework to help physicians and scientists select metrics that are appropriate to the problem. "Metrics Reloaded" can be used as an online tool. "Users are guided through a comprehensive set of questions to create a precise fingerprint of their image analysis problem. The tool also draws attention to specific problems that arise in certain biomedical issues," explains Paul Jäger, one of the senior authors of the two publications
Metrics Reloaded is suitable for all different categories of problems in image analysis, i.e. for the classification of images, object detection or the assignment of individual pixels (semantic segmentation). The tool works completely independently of the image source, so it can be used just as well for CT or MRI images as for microscopic images. Metrics Reloaded is also suitable for image analyses beyond biomedical issues.
"Metrics Reloaded is the first systematic guide that shows users of AI-based image analyses the way to the right algorithm. We hope that Metrics Reloaded will be used as widely as possible as quickly as possible, as this could significantly improve the quality and reliability of the results of AI-supported image analyses. This would also promote confidence in AI-supported image analysis in routine clinical practice," says Minu Tizabi, one of the lead authors.
The projects were funded by Helmholtz Imaging, one of five research platforms initiated by the Helmholtz Information & Data Science Incubator.
Publications:
Reinke, A./Tizabi, M., … Jäger, P., Maier-Hein L.: Understandig Metric-Related Pitfalls in Image Analysis Validation.
Nature Methods 2024, DOI: 10.1038/s41592-023-02150-0
https://www.nature.com/articles/s41592-023-02150-0
Maier-Hein, L./Reinke, A., … Jäger, P.: Metrics Reloaded: Recommendations for Image Analysis Validation.
Nature Methods 2024, DOI 10.1038/s41592-023-02151-z https://www.nature.com/articles/s41592-023-02151-z
With more than 3,000 employees, the German Cancer Research Center (Deutsches Krebsforschungszentrum, DKFZ) is Germany’s largest biomedical research institute. DKFZ scientists identify cancer risk factors, investigate how cancer progresses and develop new cancer prevention strategies. They are also developing new methods to diagnose tumors more precisely and treat cancer patients more successfully. The DKFZ's Cancer Information Service (KID) provides patients, interested citizens and experts with individual answers to questions relating to cancer.
To transfer promising approaches from cancer research to the clinic and thus improve the prognosis of cancer patients, the DKFZ cooperates with excellent research institutions and university hospitals throughout Germany:
National Center for Tumor Diseases (NCT, 6 sites)
German Cancer Consortium (DKTK, 8 sites)
Hopp Children's Cancer Center (KiTZ) Heidelberg
Helmholtz Institute for Translational Oncology (HI-TRON Mainz) - A Helmholtz Institute of the DKFZ
DKFZ-Hector Cancer Institute at the University Medical Center Mannheim
National Cancer Prevention Center (jointly with German Cancer Aid)
The DKFZ is 90 percent financed by the Federal Ministry of Education and Research and 10 percent by the state of Baden-Württemberg. The DKFZ is a member of the Helmholtz Association of German Research Centers.
END
AI-supported image analysis: metrics determine quality
2024-02-12
ELSE PRESS RELEASES FROM THIS DATE:
Innovative decarbonization: UH researcher offers carbon-driven framework to accelerate shift toward net-zero electric power sector
2024-02-12
Houston, Texas, the epicenter of the global energy market, and the University of Houston – the Energy University – are leading the transition towards a cleaner, more sustainable future with innovative solutions.
With three-quarters of U.S. greenhouse gas emissions stemming from burning fossil fuels for energy, it's clear that the key to curbing emissions lies in reimagining how energy is produced and consumed. In Texas, the shift is already well on its way. Technologies like giant batteries, ...
White people more likely to confront authors of racist online posts to set discussion rules
2024-02-12
CHAMPAIGN, Ill. — White people surveyed in a recent study indicated they would be more likely to confront those who post racist content on social media if their objective were to defend the norms for political discussions rather than to change the person’s prejudiced beliefs.
Communication professors Stewart M. Coles of the University of Illinois Urbana-Champaign and Daniel S. Lane of the University of California, Santa Barbara surveyed people during the 2020 U.S. presidential election cycle to explore the conditions under ...
University of Rhode Island Nutrition study to help inform official USDA dietary guidelines
2024-02-12
Ultra-processed foods make up more than half the food average Americans eat. Including frozen and prepared meals, most packaged snacks, desserts and carbonated soft drinks—but also including more innocuous foods—they are often considered the bane of healthy eating, containing little to no nutrition to fuel healthy bodies.
However, “not all processed foods are created equal,” according to University of Rhode Island Nutrition Professor Kathleen Melanson. Evidence to support the assumption that ultra-processed foods are all bad for one’s health is limited, and the nutritional ...
Number of at-risk youth with intellectual disability and autism in the U.S. foster care system is growing
2024-02-12
Youth with foster care involvement have an increased risk for mental health diagnoses, trauma and worse outcomes in adulthood than their peers. Research about how youth with disabilities, including autism and intellectual disability, interact with this system is lacking. Evidence for how youth with autism or intellectual disability in the foster care system access and use services is needed to advance ways to improve their outcomes.
Recently published in the Journal of the American Medical Association (JAMA) Pediatrics, researchers at Drexel University’s A.J. Drexel Autism Institute, in collaboration with George Mason University’s ...
Multiple city hubs, dispersed parks keep metro areas cooler
2024-02-12
ITHACA, N.Y. – Metropolitan areas with multiple city centers and dispersed green spaces mitigate extreme heat more effectively than those with one dominant city, an analysis by Cornell University city planning scholars finds.
Compared to “monocentric” development, “polycentric” spatial patterns better distribute the density of urban cores and curb the sprawl of impervious, heat-absorbing surfaces, according to the analysis of 50 city regions in Germany. Particularly in larger urban areas, polycentric development can moderate the urban heat island effect, ...
Innovation to overcome deficiencies in 3D printing
2024-02-12
The University of Houston is collaborating with Texas A&M University to tackle the challenge hindering the use of Additive Manufacturing (AM), commonly known as 3D printing, for a variety of commercial applications – the need for real-time monitoring and analysis to ensure consistent quality and reproducibility throughout the production process.
At present, quality control and qualification of metal AM parts is mostly carried out through offline inspection and characterization, but ideally, a broad range of sub-surface and bulk microstructural ...
Novel bispecific design improves CAR T–cell immunotherapy for childhood leukemia
2024-02-12
(MEMPHIS, Tenn. – February 12, 2024) St. Jude Children’s Research Hospital scientists improved chimeric antigen receptor (CAR) T–cell immunotherapy for acute myeloid leukemia (AML), demonstrating better efficacy in the lab. To overcome common problems with CAR T cells, the researchers created an additional means for the therapy to find and eliminate cancer cells, using a small peptide. The study also showed how a computational approach incorporating AlphaFold predicted protein models could help ...
Including socioeconomic status of patients in calculation of Medicare readmission penalties would reduce stress on safety-net hospitals
2024-02-12
INDIANAPOLIS -- The Affordable Care Act requires Medicare to issue penalties that reduce payment to hospitals if post-operative readmission rates within 30 days exceed the national average. A new study led by Regenstrief Institute Research Scientist Andrew Gonzalez, M.D., J.D., MPH, reports that including socioeconomic status in the penalty calculation would reduce the amount of readmission penalties for safety-net hospitals, which typically care for the sickest patients. Other factors, including age and sex are ...
Are ammonia engines the way of the future? (video)
2024-02-12
WASHINGTON, Feb. 12, 2024 — Could ammonia engines power the cars of the future? Carmakers like Toyota are working to make this a reality. Ammonia is combustible and holds promise as a relatively low-effort way to decarbonize the internal combustion engine — but the devil’s in the details. Join George as he discovers at least one of those details by burning stuff in his basement. https://youtu.be/KZ_NlnmPQYk?si=BleQF9-aReuttCU4
Reactions is a video series produced by the American Chemical Society and PBS Digital Studios. Subscribe to Reactions at http://bit.ly/ACSReactions ...
Prevalence of young children fed only breast milk in low- and middle-income countries
2024-02-12
About The Study: In this study of 276,000 children ages 6 to 23 months in 92 low- and middle-income countries, 10.4% were zero-food children (i.e., children who did not consume any animal milk, formula, or solid or semisolid food during the last 24 hours). The prevalence of zero-food children underscores the need for targeted interventions to improve infant and young child feeding practices and ensure optimal nutrition during this critical period of development. The issue is particularly urgent in West and Central ...