While cyber attacks occur for many reasons and in different ways, the exploitation of software vulnerabilities (also known as ‘bugs’) is one of the major attack vectors. ‘Bug bounty’ programmes and platforms utilise crowdsourcing to find these bugs, with the notion that “[g]iven enough eyeballs, all bugs are shallow” (Raymond 1999).
Bug bounty programmes are a structured and legal way for security researchers to be rewarded for finding software vulnerabilities. These programmes enable organisations to get in touch with ethical hackers (hereafter, ‘researchers’) whose cybersecurity expertise and knowledge complements that of the organisations’ own development and testing teams. From the security researchers’ side, these programmes offer an opportunity to be rewarded legally for the vulnerabilities they find.
Bug bounty platforms are ‘two-sided’ markets that host bug bounty programmes and allow organisations to crowdsource parts of their software for security testing. Such platforms provide organisations with access to a large group of talented researchers with a wide range of knowledge and skills, thereby increasing the probability of finding vulnerabilities. The programmes are structured as tournaments, so companies pay monetary rewards only for unique vulnerabilities found. Top researchers enjoy the opportunity to be invited to private programmes where only selected researchers can participate, thereby increasing the probability of being the first to find and report a vulnerability.
Bug bounty programmes and platforms are part of a more general trend to a ‘gig economy’, where enterprises supplement labour, and workers supplement income, with gig work. From the enterprise’s perspective, these platforms provide access to skilled and flexible labour. Gig work platforms create opportunities for workers to access and compete in global job markets. Furthermore, they facilitate ‘bridge employment’ (i.e. temporary employment between career jobs) and provide income opportunities in down times when the market does not accommodate full-time employment. Indeed, studying the ridesharing market, Koustas (2018) finds that, on average, driving for gig platforms replaced 73% of lost income from a main job. Moreover, taking advantage of gig work platforms during bad times helps overcome periods of income volatility. Similarly, Collins et al. (2019) find that workers typically start new platform work in times of a personal income crisis. Additionally, Stanton and Thomas (2022) examine the value of online gig economy platforms and Stabile et al. (2020) examine the effects of COVID-19 on inequality and gig economy workers.
Employing a unique data set provided by Bugcrowd, our recent paper (Zrahia et al. 2022) documents the effect of an exogenous shock – the COVID-19 pandemic – on the market for vulnerabilities within the bug bounty platform. The data cover 2017-2021, and for each year we focus on the three-month period from March to May. The 2020 period corresponds to the first three months of the pandemic and therefore delineate the Covid shock. We examine the impact of the Covid shock on both the demand for vulnerabilities by participating organisations, and the supply of vulnerabilities from active researchers.
To the best of our knowledge, this is the first study to analyse a large, detailed data set of bounty activity on a bug bounty platform which includes data on private programmes, as well as duplicate submissions. Hence, it allows for a more comprehensive analysis, given that private programmes represent more than 90% of newly offered programmes in recent year.
Since it was launched, Bugcrowd’s platform has hosted more than 2,400 programmes offered by more than 1,000 organisations and attracted more than 30,000 active researchers who made at least one submission to a programme. The data set records payments for valid submissions, which are for vulnerabilities within the defined scope of a programme. While only the first researcher to discover a valid vulnerability is awarded a monetary payment, the data set also records duplicate valid submissions. A duplicate valid submission means that the researcher correctly identified a valid vulnerability, but was not first, and therefore did not receive a monetary award. Accounting for both paid submissions and duplicate valid submissions enables us to compute the average payment for valid submissions, which turns out to be a key variable for understanding the effects of the Covid shock.
Our analysis reveals that the Covid shock impacted both the supply and demand for valid vulnerabilities, but impacted supply much more dramatically. On the supply side, the shock greatly increased the number of submissions, and the number of researchers participating on the Bugcrowd platform. This makes sense if the shock reduced the outside opportunity set for researchers who either lost their jobs or were placed on a leave of absence during that period, because these researchers had more time on hand to look for vulnerabilities in bug bounty programmes. On the demand side, there was a much smaller increase in new programmes, possibly because initiating such programmes takes a non-trivial amount of time.
We examine the effect of the Covid shock with a heuristic supply-and-demand model, by defining valid submissions as the relevant product and the average monetary award for valid submissions as the price. The Covid shock substantially shifted the supply curve by greatly increasing the number of active researchers and shifted the demand curve more moderately by slightly increasing the growth in active programmes. These shifts combined to increase greatly the number of valid submissions. Interestingly, this increase in quantity was mostly due to a huge increase in the number of duplicate valid submissions, reflecting the much more significant supply curve shift. Consequently, there was a large decrease in the average equilibrium price for a valid submission, because valid duplicates do not receive a monetary reward.
That is, the Covid shock ‘threw’ the market for vulnerabilities out of a previously more-or-less stable equilibrium. The reduction in average equilibrium price for a valid submission due to the Covid shock presumably dampened the incentives of individual researchers to search for vulnerabilities.
There was a one-in-six chance of being paid for a valid submission in 2020, compared to a slightly larger than a one-in-three chance in 2019 and 2021. The ratio of paid submissions to total valid submissions was in the 48-49% range for 2017-2018 and in the 36-37% range in 2019 and 2021. The latter numbers reflect a slight trend in increased competition among researchers. The dramatic reduction in the paid-to-valid submissions ratio in 2020 (16%) is solely due to the supply side, and its effect on the average equilibrium price for a valid submission (a 55% decline) is twice as large as the residual effect, which is likely due to both supply and demand factors.
Consequently, the following counterfactual argument can be made: if the demand response would have increased in 2020 so as to keep the ratio of paid submissions to total valid submissions in the 36-37% range, as in 2019 and 2021, rather than falling to 16%, the total number of paid submissions would have been more than double the actual number in 2020. The counterfactual implies that there might have been a missed opportunity to examine more software and find more unique vulnerabilities during the Covid shock period.
Furthermore, the Covid shock provides an opportunity to address key public policy issues associated with crowdsourcing and the ‘gig’ economy. Our setup describes what happened in the ‘white’ market for vulnerabilities, when the value of outside options was lowered. An often-mentioned benefit of the gig economy (freelance work as opposed to permanent jobs) is that the response from an external shock should be almost instantaneous on the supply side. Here we show that this was the case, and we quantify effects from the increased supply of new researchers and submissions.
Governmental agencies have begun to use bug bounty programmes. The Cybersecurity & Infrastructure Security Agency (CISA) in the US has announced in 2021 a vulnerability disclosure policy platform. The platform (provided by Bugcrowd and EnDyna), allows agencies to list systems in scope for their vulnerability disclosure policies, so security researchers may try to find vulnerabilities in agency websites and submit reports for analysis.
Additionally, cyber insurance firms have begun to recognise the benefits of firms participating in a bug bounty programme. Marsh, a large global insurance firm, includes participation in a bug bounty platform as part of its “cyber catalyst” programme, which can lead to lower cyber insurance prices. The Marsh cyber catalyst programme identifies security products that reduce cyber risk, and participation in the HackerOne bug bounty platform is included in the set of “certified” products.
References
Collins, B, A Garin, E Jackson, D Koustas and M Paynek (2019), “Is gig work replacing traditional employment? Evidence from two decades of tax returns”, IRS Working Paper.
Koustas, D K (2018), “Consumption Insurance and Multiple Jobs: Evidence from Rideshare Drivers”, Working Paper.
Raymond. E (1999), “The cathedral and the bazaar”, Knowledge, Technology & Policy 12: 23–49.
Stabile, M, B Apouey and I Solal (2020), “COVID-19, inequality, and gig economy workers”, VoxEU.org, 1 April.
Stanton, C and C Thomas (2022), “The value of online gig economy platforms”, VoxEU.org, 15 January.
Zrahia, A, N Gandal, S Markovich and M Riordan (2022), “The Simple Economics of an External Shock on a Crowdsourced “Bug Bounty” Platform”, CEPR Discussion Paper 17443.
