How to measure a researcher’s value
Researchers and universities are increasingly being evaluated based on analyses of statistics on citations. This method has its problems, but citation indices stir interest and fuel debate.
The quality and impact of science is increasingly being assessed based on calculations and analyses of the number of times researchers cite each other’s publications. The number of citations can provide a quality stamp not only on an individual researcher, but also on a scientific journal, research group, university or country.
For example, the Shanghai Ranking, one of the most influential academic rankings, places heavy emphasis on citations. The most highly cited star academics account for as much as 20% of a university’s total score.
Visible or good?
But citation numbers are a difficult assessment tool and open to interpretation. Critics say that citations actually indicate the attention garnered by research rather than its quality; after all, not all visibility is good. “It is sometimes the presumptuous papers of low scientific quality that make it into the highly-cited list, simply because these papers are often contested in other researchers’ writings,” says Chief Information Specialist Maria Forsman from the Helsinki University Library.
Focusing on citation numbers also helps to strengthen mainstream research, as bold, surprising innovations are easily brushed aside. When somebody is considered a pre-eminent in his or her field, this may be enough reason for many to cite him or her, often without even reading the work. “I remember a case in which a wrongly paraphrased second-hand quotation was multiplied in journal articles all over the world because it was simply copied instead of going to the effort of reading the original.”
What about popular science?
When examining citation numbers, one should bear in mind the strengths and weaknesses of each database. Databases may specialise in journals, books or conference publications, and the data they contain may include or exclude popular scientific writing. Databases also differ in terms of how far back they extend.
The science division of Thomson Reuters (previously, the Institute of Scientific Information, or ISI) has long produced the most valued data on citations. Its Web of Knowledge portal provides access to several databases and tools, including the renowned Web of Science. However, new contenders that have entered the fray include Elsevier’s Scopus and the free Google Scholar.
Reuters’ list of the most highly cited researchers, which is vital to the Shanghai Ranking, includes a dozen University of Helsinki researchers, almost half of whom have already earned the status of professor emeritus.
The hard problems of the soft sciences
An additional difficulty with citation databases is the problem of making comparisons between disciplines; the medical and natural sciences are emphasised at the expense of the arts and social sciences. This is particularly confusing for Finns who are used to the Finnish word tiede (science) encompassing all of the above fields.
The prominence of the hard sciences is due to several factors. Humanities scholars publish more in small languages, which limits their readership. What is more, target audiences are not necessarily global. Publications in the field of law, for example, often focus – with good reason – on the content and interpretation of legislation in the researcher’s own country.
Maria Forsman mentions sociologist Pierre Bourdieu whose original German article about different forms of capital stayed widely unrecognised. Three years later, when the same article appeared in English, it quickly became one of the internationally most cited documents of this research theme. Librarians call this a ‘sleeping beauty’ – something valuable that awakens from a long sleep.
Fast articles, slowly monographs
Many subjects in the humanities are also highly specialised or favour monographs and a slow publication pace. One prominent humanities scholar at the University of Helsinki is Professor Jaakko Frösen, a philologist and papyrologist, whose research group developed a method for conserving ancient, charred manuscripts. Frösen has become one of the top scholars in his field by interpreting the damaged documents found in the city of Petra. Yet one is unlikely to find his name on a list of citations.
In fact, the Reuters rankings feature only one scholar in the humanities who has made his career at the University of Helsinki: Risto Näätänen, professor emeritus of psychology. Using brain imaging to study the effect of auditory stimuli, Näätänen developed the theory of mismatch negativity to explain the way in which the brain processes information. As a cognitive scientist, however, Näätänen falls in the “hard” end of the soft science spectrum.
It seems that lists about science publications are like science itself: no absolute truth can be gained but one can try to get as close as possible.
Researchers and Publications
Open Acces at the University of Helsinki
Helsinki University Bulletin HUB