Study: Models to predict scientists' future impact often fail

(Press-News.org) Contact information: Santo Fortunato
santo.fortunato@aalto.fi
358-504-605-511
Aalto University
Study: Models to predict scientists' future impact often fail

Models universities partially use to forecast scientists' future contributions are not as reliable as previously thought. In a recent study, Aalto University and IMT Institute for Advanced Studies Lucca researchers demonstrate mathematical analysis of past performance cannot reliably determine the future performance. This means that current models dangerously overestimate the predictability and should not be used for career advancement decision process.

"Based on our results, the predictability of current models for real application in recruitment decisions is questionable. Efforts to model future impact need to be aimed more directly at applications in the career advancement decision process," says Professor Santo Fortunato from Aalto University in Finland.

In recent years it has become more common for universities to use quantitative measures for yard-sticking the productivity and impact of individual researchers to help reduce hiring risks. Models thought to be capable of foreseeing a scientist's future impact by way of his or her future 'h-index' have become a common tool in recruitment decisions as well as other scientific evaluation, advancement, and reward processes.

Scientists concluded that great caution should be taken when attempting to forecast an individual's future based on their 'h-index'. Specifically the authors show that it is easy to grossly overestimate the predictability of cumulative measures over a person's entire career.

The study analyzed 762 scientists from three disciplines: physics, biology, and mathematics. By applying future impact models to these careers, a number of subtle, but critical, flaws in current models were identified. Specifically, the 'h-index' contains false autocorrelation, resulting in a significant overestimation of "predictive power". Moreover, the "predictive power" of these models vary greatly with the career age of scientists, producing least accurate estimates for already risk-burdened early career researchers.

Aalto researchers concluded that care must be taken to select the correct measures and methods to evaluate scientific candidates in the future. Increased attention should be paid to the potential shortfalls of quantitative methods when applied to the decision-making process.

The results were published in the Scientific Reports, a research publication from the publishers of Nature. This study was conducted by Raj Kumar Pany, Kimmo Kaski and Santo Fortunato from Aalto University in collaboration along with Orion Penner and Alexander M. Petersen from the IMT Institute for Advanced Studies in Lucca, Italy.



INFORMATION:

For more information, please contact:

Professor Santo Fortunato
Aalto University, School of Science, Dept. of Biomedical Engineering and Computational Science
Email: santo.fortunato@aalto.fi
Tel. +358 50 460 5511

Postdoctoral Researcher Raj Kumar Pan
Aalto University, School of Science, Dept. of Biomedical Engineering and Computational Science
Email: rajkumar.pan@aalto.fi
Tel. +358 50 512 4346