Towards measuring research supervision quality

Academics do many things. We teach, research, run departments, engage in public debate, and train young researchers. The outputs of some of these activities are routinely measured and thus play an important role in rewards and promotion. The standard and mostly widely followed measures involve publication outputs (see Hudson 2013 and Conley et al. 2011). Other activities are rarely assessed and thus left to the intrinsic motivation of the academic. Research supervision is one.

Our recent work – summarised in this column – proposes a measure of the quality of advice to PhD candidates, the h1-index. Our proposed indicator of research supervision excellence shows that, by and large, good researchers make good advisers; however, there are many people who excel at research but are less good at supervision, and many more that make great supervisors without being well published themselves.

This index could be a useful additional measure of university quality and transparency by assisting students in choosing a university and PhD supervisor. Moreover, if with measurement comes saliency, this could be an additional motivating factor for academics.

A well-known indicator for research output is the h-index, where h is the highest number of papers a researcher has published that have each been cited h times or more. In other words, an academic has an h-index of 5, if she has published 5 papers which have each been cited at least 5 times. The h-index is a simple way to combine the quality and quantity of research output. Quality without quantity is punished, as is quantity without quality. Invented only in 2005 by Jorge Hirsch (Hirsch 2005), it has taken the world of research assessment by storm.

The h-index naturally lends itself to evaluating collectives. Frances Ruane and I (2009) introduce the h1-index to quantify the quality of research supervision. A professor is a good PhD adviser if she trains many good researchers. We define a professor’s h1-index based on the h-indices of her supervisees; a professor with an h1-index of 4 has had 4 PhD graduates each with an h-index of 4 or more.

Data are a challenge. All universities keep records on PhD completions, but few do so in an accessible way, let alone a structured database. Fortunately, academics are snobs and take pride in their lineage. (Jan Tinbergen was my adviser’s adviser’s adviser, so there.) Academic genealogies can be found. We used Alan Deardorff’s family tree of trade economists. There are 65 professors with 4 or more supervisees, for a total of 356 PhDs. Publication and citation data were taken from the Web of Science. The 12 best professors are listed in Table 1.

Table 1. Selected characteristics of the 12 most successful advisers in trade economics.

Like the h-index, the h1-index is a measure of lifetime achievement. It thus favours those with an earlier career start. Like the h-index is roughly proportional to the number of years since first publication, the h1-index is roughly proportional to the number of years since first graduation – see Figure 1. We can thus compare academics of different vintages by their h- and h1-rate, defined as the index over the number of active years.

Figure 1. The h1-index as a function of the year of first graduation

Note: dots are observations; thick central line is the expected value; thin lines are the 95% confidence interval.

Three professors stand out in that they graduate more high quality supervisees per year than most: Tony Venables, Gene Grossman, and Ron Findlay.

By and large, good researchers make good advisers. There is a weak but positive correlation (0.41) between the h-rate and the h1-rate – see Figure 2. However, there are people who are excellent researchers but less good at research supervision. There are also excellent PhD advisers whose own research is nothing special. People good at neither are excluded from the sample.

Figure 2. The h1-rate as a function of the h-rate

Note: dots are observations; line is the least squares fit.

The flamboyant Harry Johnson stands out. He made important contributions to our understanding of international trade. He also trained a fair number of trade economists who themselves made important contributions. And he taught them how to be good PhD advisers themselves. Johnson’s students tend to have high h-indices as well as high h1-indices. Johnson thus has a high h-, h1- and h2-index; he was an excellent researcher and a grand professor.

There are two key problems with the analysis. Restricting attention to trade economists disadvantages those who also work in other fields. We did not control for student quality at the start of the PhD. That is, we did not measure the value added of research advice but rather intrinsic quality plus all education. We can get some idea, however, by looking at the institutions in Table 1. Some teach at top schools and have first pick from PhD candidates. Others do not, but nevertheless succeed in graduating excellent PhDs.

The results here merely illustrate our proposed indicator of excellence in research supervision. More systematic data collection is needed (see http://ideas.repec.org/top/top.person.students.html for another attempt) not only for a more comprehensive assessment but also for a better understanding of what makes a good PhD adviser.

Supervising PhD candidates is an important task. Yet, evaluations rarely go beyond the number of PhD completions with a few anecdotes about placements thrown in. The quality of research supervision can be measured, however, and should – at the moment, quantity is rewarded, and the incentive is thus to lower standards. The indicator proposed here should be interpreted with care. It takes time for a PhD candidate to manifest himself as a good researcher, and thus for a professor to manifest herself as a good supervisor. The h1-index is fine for celebrating past achievements. It should be used for current decisions only to the extent that present quality correlates with past glory.

Conley, John P, Ali Sina Önder, Mario Crucini, and Robert Driskill (2011), “Publication lags and young economists’ publication lags”, VoxEU.org, 24 October.

Deardoff, Alan (undated), “Family Tree of Trade Economists”.

Hirsch, J E (2005), “An index to quantify an individual’s scientific research output”, PNAS, 102: 16569–16572.

Hudson, John (2013), “Challenging times in academia”, VoxEU.org, 11 November.

Ruane, Frances P, and Richard S J Tol (2009), “A Hirsch measure for the quality of research supervision, and an illustration with trade economists”, Scientometrics, 80: 613–624.

This column first appeared on the website of the Economics of Higher Education Network: economicsofhe.org/