(Press-News.org) The term "big data" is defined as a huge amount of digital information, so big and so complex that normal database technology cannot process it. It is not only scientific institutes like the nuclear research center CERN that often store huge amounts of data ("Big Data"). Companies like Google and Facebook do this as well, and analyze it to make better strategic decisions for their business. How successful such an attempt can be was shown in a New York Times article published last year. It reported on the US-based company "Target" which, by analyzing the buying patterns of a young woman, knew about her pregnancy before her father did.
The analyzed amount of data is distributed on several servers on the internet. The search queries go to several servers in parallel. Traditional database management systems do not match all use cases. Either they cannot cope with big data, or they overstrain the user. Therefore data analysts love tools which are based on the open-source software framework Apache Hadoop and which use its efficient file system HDFS. Those do not require expert knowledge. "If you are used to the programming language Java, you can already do a lot with it", explains Jens Dittrich, professor of information systems at Saarland University. But he also adds that Hadoop is not able to query big datasets as efficiently as database systems that are designed for parallel processing.
Dittrich's and his colleague's solution is the development of the "Hadoop Aggressive Indexing Library", abbreviated with HAIL. It enables saving enormous amounts of data in HDFS in such a way that queries are answered up to 100 times faster. The researchers use a method which you can already find in a telephone book. So that you do not have to read the complete list of names, the entries are sorted according to surnames. The sorting of the names generates the so-called index.
The researchers generate such an index for the datasets they distribute on several servers. But in contrast to the telephone book, they sort the data according to several criteria at once and store it multiply. "The more criteria you provide, the higher the probability that you find the specified data very fast", Dittrich explains. "To use the telephone book example again, it means that you have six different books. Every one contains a different sorting of the data – according to name, street, ZIP code, city and telephone number. With the right telephone book you can search according to different criteria and will succeed faster." In addition to that, Dittrich and his research group managed to generate the indexes without any additional costs. He and his group members organized the indexing in such a way that no additional computing time and delay is required. Even the additional storage space requirement is low.
INFORMATION:
Computer science research on the campus of Saarland University
The Department of Computer Science is not the only research institution which is exploring new aspects of computer science. Only a few yards from there, you can also find the Max Planck Institute for Computer Science, the Max Planck Institute for Software Systems, the Center for Bioinformatics, the Center for IT-Security, Privacy and Accountability, the German Research Center for Artificial Intelligence, the Intel Visual Computing Institute and the recently renewed Cluster of Excellence "Multimodal Computing and Interaction".
See also:
Website for HAIL:
http://infosys.uni-saarland.de/hadoop++.php
Press pictures:
www.uni-saarland.de/pressefotos
Further questions are answered by:
Prof. Dr. Jens Dittrich
Tel. +49 681 302 70141
E-Mail:
jens.dittrich@infosys.uni-saarland.de
Gordon Bolduan
Science Communication
Cluster of Excellence
Phone: +49 681 302-70741
Cebit booth: +49 511/ 89497024
E-Mail: bolduan@mmci.uni-saarland.de
Big data: Searching in large amounts of data quickly and efficiently
2013-03-01
[Attachments] See images for this press release:
ELSE PRESS RELEASES FROM THIS DATE:
Volcanic aerosols, not pollutants, tamped down recent Earth warming, says CU-Boulder study
2013-03-01
A team led by the University of Colorado Boulder looking for clues about why Earth did not warm as much as scientists expected between 2000 and 2010 now thinks the culprits are hiding in plain sight -- dozens of volcanoes spewing sulfur dioxide.
The study results essentially exonerate Asia, including India and China, two countries that are estimated to have increased their industrial sulfur dioxide emissions by about 60 percent from 2000 to 2010 through coal burning, said lead study author Ryan Neely, who led the research as part of his CU-Boulder doctoral thesis. Small ...
Wild pollinators increase crop fruit set regardless of honey bees
2013-03-01
Changes made by humans to the natural landscapes can often compromise ecosystems, which paradoxically are vital for human survival. Pollination of crops by wild insects is one such vulnerable ecosystem service, as wild insects are declining in many agricultural landscapes. The study, recently published in Science, focused on understanding whether the ongoing loss of wild insects impacts crop harvest. For this purpose, the researchers compared fields with abundant and diverse wild insects to those with degraded assemblages of wild insects across 600 fields at 41 crop systems ...
Pregnancy permanently changes foot size
2013-03-01
A new University of Iowa study confirms what many women have long suspected – that pregnancy permanently changes the size and shape of a woman's feet.
Flat feet are a common problem for pregnant women. The arch of the foot flattens out, possibly due to the extra weight and increased looseness (laxity) of the joints associated with pregnancy. The new study, published in the March issue of the American Journal of Physical Medicine & Rehabilitation, suggests that this loss of arch height is permanent.
"I had heard women reporting changes in their shoe size with pregnancy, ...
Malign environmental combination favors schizophrenia
2013-03-01
The interplay between an infection during pregnancy and stress in puberty plays a key role in the development of schizophrenia, as behaviourists from ETH Zurich demonstrate in a mouse model. However, there is no need to panic.
Around one per cent of the population suffers from schizophrenia, a serious mental disorder that usually does not develop until adulthood and is incurable. Psychiatrists and neuroscientsist have long suspected that adverse enviromental factors may play an important role in the development of schizophrenia. Prenatal infections such as toxoplasmosis ...
New guidelines for standardizing glucose reporting and optimizing clinical decision making in diabetes
2013-03-01
New Rochelle, NY, March 1, 2013—Most adults and children with type 1 diabetes are not in optimal glycemic control, despite advances in insulin formulations and delivery systems and glucose monitoring approaches. Critical barriers to optimal glycemic control remain. A panel of experts in diabetes management and research met to explore these challenges, and their conclusions and recommendations for how to improve care and optimize clinical decision-making are presented in a white paper in Diabetes Technology & Therapeutics (DTT), a peer-reviewed journal from Mary Ann Liebert, ...
Mechanisms regulating inflammation associated with type 2 diabetes, cancer identified
2013-03-01
(Boston) – A study led by researchers at Boston University School of Medicine (BUSM) has identified epigenetic mechanisms that connect a variety of diseases associated with inflammation. Utilizing molecular analyses of gene expression in macrophages, which are cells largely responsible for inflammation, researchers have shown that inhibiting a defined group of proteins could help decrease the inflammatory response associated with diseases such as obesity, type 2 diabetes, cancer and sepsis.
The study, which is published online in the Journal of Immunology, was led by ...
'Where you're treated matters' in terms of cancer survival
2013-03-01
SEATTLE – A study of older patients with advanced head and neck cancers has found that where they were treated significantly influenced their survival.
The study, led by researchers at Fred Hutchinson Cancer Research Center and published in the March 1 online edition of Cancer, found that patients who were treated at hospitals that saw a high number of head and neck cancers were 15 percent less likely to die of their disease as compared to patients who were treated at hospitals that saw a relatively low number of such cancers. The study also found that such patients ...
Illinois town provides a historical foundation for today's bee research
2013-03-01
CHAMPAIGN, Ill. — A study published in the journal Science reveals a decline in bee species since the late 1800s in West Central Illinois. The study could not have been conducted without the work of a 19th-century naturalist, says a co-author of the new research.
Charles Robertson, a self-taught entomologist who studied zoology and botany at Harvard University and the University of Illinois, was one of the first scientists to make detailed records of the interactions of wild bees and the plants they pollinate, says John Marlin, a co-author of the new analysis in Science. ...
New chemical probe provides tool to investigate role of malignant brain tumor domains
2013-03-01
CHAPEL HILL, N.C. – In an article published as the cover story of the March 2013 issue of Nature Chemical Biology, Lindsey James, PhD, research assistant professor in the lab of Stephen Frye, Fred Eshelman Distinguished Professor in the UNC School of Pharmacy and member of the UNC Lineberger Comprehensive Cancer Center, announced the discovery of a chemical probe that can be used to investigate the L3MBTL3 methyl-lysine reader domain. The probe, named UNC1215, will provide researchers with a powerful tool to investigate the function of malignant brain tumor (MBT) domain ...
Study confirms safety of colonoscopy
2013-03-01
Colon cancer develops slowly. Precancerous lesions usually need many years to turn into a dangerous carcinoma. They are well detectable in an endoscopic examination of the colon called colonoscopy and can be removed during the same examination. Therefore, regular screening can prevent colon cancer much better than other types of cancer. Since 2002, colonoscopy is part of the national statutory cancer screening program in Germany for all insured persons aged 55 or older.
However, only one fifth of those eligible actually make use of the screening program. The reasons ...