Concerns exist that public funding of science is increasingly risk averse, which could impede the long-term advancement of science (Alberts et al. 2014, Petsko 2012) and in turn economic growth (Watzinger and Schnitzer 2019). Such concerns relate not only to a risk bias in the selection of grants by funding agencies, but also to grants not providing sufficient protective space for researchers to engage in risky research (Azoulay et al. 2011, Heinze et al. 2009, Hollingsworth 2004). Funders have countered these criticisms by soliciting submissions of high-risk research to either regular or specially designed programmes. One prominent example is the European Research Council (ERC), which was established by the European Commission in 2007 to support ‘high-gain/high-risk’ research.
Little systematic evidence, however, has been gathered to address the extent to which funding agencies in general penalise risk when awarding grants, and whether grants encourage risk taking (Franzoni et al. 2021). Even less evidence exists for programmes that have risky research as a stated goal, or for programmes such as the ERC, whose structure may encourage risk taking. Several studies have investigated the first years of the ERC by interviewing researchers and evaluators. While Laudel and Gläser (2014) concluded that the ERC constitutes an institutional innovation that enables excellent research, Luukkonen’s (2012) interviews with members and chairs of ERC panels suggested that “despite the ERC’s aims, the peer review process in some ways constrains the promotion of truly innovative research”.
In a new paper (Veugelers et al. 2022), we address two questions: Does the ERC select applicants with a track record of conducting risky research? And does receiving an ERC grant enable grantees to take more risk in their research?
Identifying risky research
We analyse 10,036 applications to ERC Starting, Consolidator, and Advanced grants across all panels and call years from 2007 to 2013. Our sample includes all successful applications (i.e. 4,345 applications that received an ERC grant) and a random sample of 5,691 failed applications. We do not have access to the application proposals, but we do know the lifetime publication record of applicants. We use this publication record to characterise applicants’ inclination towards risky research.
Specifically, we proxy risk using a measure of the novelty of scientific publications developed by Wang et al. (2017). Acknowledging that novelty is not a perfect proxy for risk, we showed in Wang et al. (2017) that novel research has a higher variance of citations with a higher probability of becoming a frequently cited paper – but also a higher probability of being at the bottom of the citation distribution – characteristics one would associate with risky research. We also found that novel research faces delayed recognition and that the major impact of novel research comes from fields outside their own. In addition, novel research is less likely to be published in high-impact journals. These findings may correlate with how selection takes place, shaping risk biases in funding decisions, such as the use of field-specific panels and the use of short-term bibliometric indicators to assess the quality of applications.
Applicants with a track record of conducting novel research are less likely to win ERC grants
To assess potential selection bias against risk at the ERC, we investigate whether applicants with one or more top 1% novel papers in their field would have a significantly lower likelihood of receiving ERC funding. We control for other confounding factors possibly influencing selection, most notably the high-gain (or excellence) profile of the applicant, by tracking whether the applicant has one or more top 1% highly cited papers in their field. We also control for the applicant ’s number of publications, gender, career stage, country, panel, and grant call year.
We find consistent evidence of selection bias against novelty: applicants with a history of novel research are less likely to be selected for funding than those without such a history. This selection penalty also holds among those applicants with a history of highly cited publications. Applicants with highly cited papers and without a history of novel research are significantly more likely to win a grant than highly cited applicants with a history of novelty
We expected the first stage of funding selection to be more prone to selection bias than the second stage given that the panel only evaluates the full proposal at the second stage, while at the first stage simply the applicant’s CV and an extended synopsis of the proposal are assessed. In addition, the first stage has many more applications to evaluate, increasing the likelihood that panel members will use classic bibliometric indicators in their assessment. However, the selection bias is only marginally lower – not statistically significantly lower – in the second stage.
When separating early and advanced career applicants, we find that the negative selection against novelty is larger and statistically more significant for early career applicants (i.e. Starting and Consolidator Grant applicants) than for advanced applicants (i.e. Advanced Grant applicants), suggesting that panel members are less willing to tolerate risk for early career applicants than for established researchers.
Figure 1 Applicants with a track record of highly novel research are less likely to win an ERC grant
ERC grantees are more likely to produce novel research, but this only holds for early-career applicants, partly due to failed early career applicants cutting back on novel research
To investigate funding treatment effects, we employ a difference-in-differences strategy, which accounts for differences between successful and failed applicants introduced by selection.
We find positive treatment effects for early career applicants when compared to applicants who failed at the second stage of funding selection. This positive treatment effect for novel research is due in part to unsuccessful second stage applicants cutting back on novel research. We cautiously interpret this as a strategy on the part of failed junior applicants to prepare for resubmission of their proposal to a research funding system that appears to be in favour of research that can show top citations, but biased against novel research.
We find no significant treatment effects on producing highly novel research for advanced career applicants. It is important to note that our studying period is short, only covering five years after grant call year, while the funding treatment effect may take time to manifest.
Figure 2 Receiving an ERC grant has a positive treatment effect on producing novel research for early career applicants
Policy implications
Our findings of a bias against funding applicants with highly novel profiles, especially early career applicants, are also found by others (Ayoubi et al. 2021, Boudreau et al. 2016, Carayol and Lanoë 2018).
Our results are particularly troublesome as they show that applicants with a history of doing novel research are less likely to be selected even in a programme such as the ERC, which has a goal of supporting high-risk research. Our finding that early career researchers face a higher penalty against risk in funding selection is also consistent with prior studies (Chubin and Hackett 1990, Wang et al. 2018, Whitley and Gläser 2007) and suggests that panel members are less willing to tolerate risk for early career applicants than for advanced careers.
Our results suggest, further, that such bias is hard wired. Having a visible signal that the applicant is capable of producing highly cited impactful research is apparently insufficient to mitigate this bias. Neither, apparently, is clear communication by the funding agency of its mission to support risky research, as the ERC routinely reminds its panel members during briefings. Also worrying are the results showing the ‘treatment’ effect on the early research applicants who failed in stage two and then cut back on novel research, if that is their way to deal with perceived risk bias when reapplying.
We hope this research raises awareness of risk bias in funding evaluations. It certainly calls for further analysis to unpack the sources of risk bias. To identify the mechanisms leading to such bias, which remain elusive, we will need to develop more measures of risky research.
References
Ayoubi, C, M Pezzoni and F Visentin (2021), "Does it pay to do novel science? The selectivity patterns in science funding", Science and Public Policy 48(5), 635–648.
Azoulay, P, J S Graff Zivin and G Manso (2011), "Incentives and creativity: evidence from the academic life sciences", The RAND Journal of Economics 42(3), 527–554.
Boudreau, K J, E C Guinan, K R Lakhani and C Riedl (2016), "Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science", Management Science 62(10), 2765–2783.
Carayol, N and M Lanoë (2018), "The Impact and Design of Project-Based Funding in Science: Lessons from the ANR Experience", Post-Print hal-02274256, HAL.
Chubin, D E and E J Hackett (1990), Peerless Science: Peer review and US science policy, SUNY Press.
Franzoni, C, P Stephan and R Veugelers (2021), "Funding Risky Research", NBER Working Paper No. 28905.
Heinze, T, P Shapira, J D Rogers and J M Senker (2009), "Organizational and institutional influences on creativity in scientific research", Research Policy 38(4), 610–623.
Hollingsworth, R (2004), "Institutionalizing excellence in biomedical research: the case of Rockefeller University", Creating a Tradition of Biomedical Research, Rockefeller University Press.
Laudel, G and J Gläser (2014), "Beyond breakthrough research: Epistemic properties of research and their consequences for research funding", Research Policy 43(7), 1204–1216.
Luukkonen, T (2012), "Conservatism and risk-taking in peer review: Emerging ERC practices", Research Evaluation 21(1), 48-60.
Petsko, G A (2012), "Goodbye, Columbus", Genome Biology 13(5), Article 155.
Veugelers, R, J Wang and P Stephan (2022), "Do funding agencies select and enable risky research: evidence from ERC using novelty as a proxy of risk taking", NBER Working Paper 30320.
Wang, J, Y N Lee and J P Walsh (2018), "Funding model and creativity in science: Competitive versus block funding and status contingency effects", Research Policy 47(6), 1070–1083.
Wang, J, R Veugelers and P E Stephan (2017), "Bias against novelty in science: A cautionary tale for users of bibliometric indicators", Research Policy 46(8), 1416–1436.
Watzinger, M and M Schnitzer (2019), "Standing on the Shoulders of Science", VoxEU.org, 21 June.
Whitley, R and J Gläser (2007), The changing governance of the sciences : the advent of research evaluation systems, Springer.
