A method to assess individual research outputs

In the cost sensitive world of today, metrics are being employed more readily to measure the research performance of individual faculty members in order to assess the ‘quality’ or ‘excellence’ of their research. Qualitative measures based on what a very small sample of academics think is always going to lead to inaccuracies and (most likely ‘old school’) biases. Even individuals who feel their research has considerable impact, perhaps due to them winning a Nobel Prize or inventing a multimillion dollar export product, are by no means immune to such measurement systems. What someone did ten years ago may arguably be not so relevant today. In the UK, for example, every six or so years the research excellence framework (REF), formerly known as the research assessment exercise (RAE), takes place in which an attempt is made to quantify research quality. Apart from a fractional assessment of variables such as the actual impact of any research output, research income and PhD research students who have been successful, by far the main measure of success is down to the ‘quality’ of research outputs, in other words how good each academic’s papers are. Even if, for some reason, an attempt is made to force this judgement to be more aligned with what a panel of peers think about each of the outputs, nevertheless judgement on each paper is bound to be strongly influenced by the quality of the journal in which that paper has been published and how the paper has been regarded subsequently, the easiest metric being how many times it has been cited – especially now that various services such as Google Scholar readily proffer such information at the touch of a button. It would take a brave panel member indeed to put forward a paper as being of high quality if it was published in a lesser known journal and which, after say five years of being published, had never been cited. They would have quite an uphill struggle. Conversely where a paper has been published by, what is widely regarded as, a top notch 2 K. Warwick and V.M. Becerra journal and has received a large number of citations even though it only appeared six months previously, then the case for high quality is made. Of course these are the extremes, however these two possibilities present a caricature image of the situation. Clearly the most common measure of journal quality which is now widely employed in practice, even though it may be regarded by some as being not politically correct in terms of the standing of a journal, is its impact factor (IF) (Saglen, 1992; Garfield, 1999; Garfield, 2006). The IF is found for a journal each year by calculating the total number of citations in the previous year to papers in that journal which were published in the two years prior to that, then dividing the total figure by the number of papers published in those two years. For example let us consider all the papers published in a particular journal in 2011 and 2012 and see how many times they have been cited in 2013, then divide by the total number of papers published in 2011 and 2012. That gives the IF for that journal for 2014. There is a widely held view, that a journal’s IF gives an indication of quality for the papers published in it. It is normally anticipated that higher IF journals are more difficult to have a paper published in, and that a paper is going to be more highly regarded if it is published in a high IF journal. Notice, however, that what the IF actually depicts is an average value of citations per paper in one year. In other words, the IF is a figure indicative of a general average of previous papers in a journal. So if your paper is published in a particular journal you can expect, on average, that your paper will be cited IF (of that journal) number of times in one year. Whether the IF is in any way a good or bad measure of an individual paper is another thing. Indeed all it is, is a pre-publication measure. Once a paper has been published then post-publication measures, such as the number of times a paper is cited, come into play. Indeed the H-index (Hirsch, 2005) which gives an indication of an individual researcher’s publication performance is based entirely on paper citation numbers and has no direct link with the IFs of the journals in which the papers appeared. So we are faced with IF as a pre-publication measure and citations as a post publication measure. An interesting aspect of this being that in practice studies have shown that in most cases for a published article, the IF of the journal in which it is published does not display strong correlation with the number of times that paper is cited (Finardi, 2013). They are, in effect, two distinctly different measures for the same paper