Formation of COVID-19 policy must cope with many substantial uncertainties about the nature of the disease, the dynamics of the pandemic, and behavioural responses. These uncertainties have become well-recognised qualitatively (e.g. Avery et al. 2020), but they have not been well-characterised quantitatively. Credible measurement of COVID-19 uncertainties is needed to make useful predictions of policy impacts and reasonable policy decisions.
Epidemiological models of disease dynamics, sometimes combined with models of macroeconomic dynamics, have been used to reach conclusions about optimal COVID-19 policy. However, researchers have done little to appraise the realism of their models, nor to quantify uncertainties. Hence, there is little basis to trust the policy prescriptions that have been put forward.
I think it misguided to make policy that is optimal in hypothetical scenarios but potentially far from optimal in reality. It is more prudent to approach COVID-19 policy as a problem in decision making under uncertainty. Facing up to uncertainty, one recognises that it not possible to guarantee choice of optimal policies.
While one cannot guarantee optimality under uncertainty, one may still make decisions that are reasonable in well-defined respects. I specifically suggest adaptive diversification of COVID-19 policy. I proposed adaptive policy diversification in two earlier papers (Manski 2009, 2013). Financial diversification is a familiar recommendation for portfolio allocation. Diversification enables an investor facing uncertain asset returns to limit the potential negative consequences of placing ‘all eggs in one basket’. Analogously, policy is diversified if a planner facing uncertainty randomly assigns treatment units (persons or locations) to different policies. At a point in time, diversification avoids gross errors in policymaking. Over time, it yields new evidence about policy impacts, as in a randomised trial. As evidence accumulates, a planner can revise the fraction of treatment units assigned to each policy in accord with the available knowledge. This idea is ‘adaptive diversification’.
In this column, I explain why current modelling cannot deliver a realistically optimal COVID-19 policy, and then discuss adaptive diversification.
Incredible certitude in epidemiological and macroeconomic modelling
Epidemiological modellers have sought to determine COVID-19 policy that would be optimal from a public health perspective if specified models of disease dynamics were accurate and public health were measured in specified ways (e.g Ferguson et al. 2020, IHME COVID-19 Health Service Utilization Forecasting Team 2020). Assessment of COVID-19 policy should consider the full health, economic, and social impacts of alternative policy options. However, epidemiological modelling has only considered impacts on health. A reason may be that epidemiology has been the province of researchers in medicine and public health. Researchers with these backgrounds find it natural to focus on health concerns. They view the economy and social welfare as matters that may be important but that are beyond their purview.
Following the onset of the COVID-19 pandemic, macroeconomists have sought to expand the scope of optimal policy analysis by joining epidemiological models with models of macroeconomic dynamics and by specifying welfare functions that consider both public health and economic outcomes (e.g. Eichenbaum et al. 2020, Acemoglu et al. 2020). However, there is little basis to assess the realism of the models that have been developed.
A serious underlying problem in both epidemiological and macroeconomic modelling is the dearth of evidence available to inform model specification and estimation. Studies of disease and macroeconomic dynamics are largely unable to perform the randomised trials that have been considered the ‘gold standard’ for medical research. Modelling necessarily relies on observational data, which are difficult to interpret. Lacking much evidence, epidemiologists and macroeconomists have developed models that are sophisticated from mathematical and computational perspectives but that have little empirical grounding.
These modelling efforts may perhaps be useful if interpreted cautiously as computational experiments studying policymaking in hypothetical worlds. However, their relevance to the real world is unclear. Models differ considerably in the assumptions they maintain and in the way they use limited available data to estimate parameters. Researchers provide little information that would enable one to assess model realism. They do little to quantify uncertainty in the predictions they offer.
I have persistently argued for forthright communication of uncertainty in research that aims to inform public policy (Manski 2019). I have criticised the prevalent practice of policy analysis with incredible certitude. Exact predictions of policy outcomes are routine; expressions of uncertainty are rare. Yet predictions often are fragile, resting on unsupported assumptions and limited data. Expressing certitude is not credible. Incredible certitude has been prevalent in both epidemiological and economic modelling.
An example of incredible certitude in epidemiological modelling is the March 2020 report of the Imperial College COVID-19 Response Team, which has influenced policy formation in the UK and US (Ferguson et al. 2020). The team forecast the impact of two alternative policy responses to the pandemic, mitigation and suppression, writing (p. 1):
“Two fundamental strategies are possible: (a) mitigation, which focuses on slowing but not necessarily stopping epidemic spread . . . . and (b) suppression, which aims to reverse epidemic growth, reducing case numbers to low levels and maintaining that situation indefinitely.”
The forecasts were based on a modified version of a model developed earlier to support influenza planning. The report provided scant justification for the application of this model to the COVID-19 context and did little to assess uncertainty in the forecasts made. Based on their forecasts, the COVID-19 Response Team recommended suppression as the preferred policy option. They reached this conclusion even though their report only considered the impact of policy on health, with no attention to economic and social consequences.
There is an urgent need for epidemiologists and economists to join forces to develop credible integrated assessment models of epidemics. Even with the best intentions, this will take considerable time. There is some reason to hope that epidemiologists and economists may be able to communicate with one another because they share a common language for mathematical modelling of dynamic processes. However, each group has in the past exhibited considerable insularity, which may impede collaboration. Moreover, neither discipline has shown much willingness to face up to uncertainty when developing and applying models.
There have been frequent calls for adoption of a uniform COVID-19 policy across locations, particularly across the 50 states of the US. For example, an 11 May 2020 editorial in the Washington Post was titled “The patchwork of state reopenings is a deadly game of trial and error”. The text refers to “the peril posed by the hodgepodge of state decisions to reopen quickly, gradually or not at all yet.” While warning against decentralisation of policymaking across the states, the editorial does not propose what a uniform national policy should be.
Calling for a uniform COVID-19 policy across states would be justified if it were clear what constitutes optimal policy and if it were known that the optimal policy is invariant across states. Then each state should adhere to that policy. However, as explained above, we do not know what optimal policy is for any state. It may be that continued suppression is better for some states (or parts of states) and that some version of reopening is better for others, depending on their characteristics. Hence, there is no prima facie case for making policy uniform across states.
It has long been appreciated in the US that uncertainty may justify decentralisation of policymaking, enabling the states to experiment with policy ideas. Supreme Court Justice Louis Brandeis, in his dissent to the 1932 case New York State Ice Co. v. Liebmann (285 U.S. 311), made what has become a famous remark on this theme: “It is one of the happy incidents of the federal system that a single courageous State may, if its citizens choose, serve as a laboratory; and try novel social and economic experiments without risk to the rest of the country.” It has since become common to refer to the states as the laboratories of democracy.
The Brandeis statement expresses the ‘adaptive’ aspect of the theme of adaptive diversification, recognising that policy variation across states stimulates learning about policy impacts. The ‘diversification’ aspect of the theme has been less well appreciated.
To illustrate, consider the choice between suppression and mitigation framed by Ferguson et al. (2020). Suppression may be the better policy if the Imperial College model makes reasonably accurate predictions of COVID-19 health impacts and if the economic impacts ignored by the model are relatively small. On the other hand, mitigation may be the better policy if the model substantially overestimates the COVID-19 health impacts or if the economic impacts ignored by the model are relatively large. Policy diversification, with some locations implementing suppression and others implementing mitigation, gives up the ideal of optimality in order to protect against making a gross error in policy choice.
When diversifying, what fraction of locations should implement each policy option under consideration? This depends on the welfare function that society uses to evaluate options and on the uncertainties that afflict prediction of policy impacts. In Manski (2009), I study adaptive diversification when social welfare is utilitarian, and a planner uses a simple dynamic version of the minimax-regret criterion to cope with uncertainty. The result is a simple diversification rule. Given specification of an appropriate welfare function and characterisation of the relevant uncertainties, it should be possible to adapt this analysis to diversify COVID-19 policy.
Acemoglu, D, V Chernozhukov, I Werning, and M Whinston (2020), “Optimal Targeted Lockdowns in a Multi-Group SIR Model”, NBER Working Paper 27102.
Avery, C, W Bossert, A Clark, G Ellison and S Ellison (2020), “Policy Implications of Models of the Spread of Coronavirus: Perspectives and Opportunities for Economists”, NBER Working Paper 27007.
Eichenbaum, M, S Rebelo, and M Trabandt (2020), “The Macroeconomics of Epidemics”, NBER Working Paper 26882.
Ferguson, N et al. (2020), “Report 9: Impact of Non-pharmaceutical Interventions (NPIs) to Reduce COVID-19 Mortality and Healthcare Demand”, Imperial College London.
IHME COVID-19 Health Service Utilization Forecasting Team (2020). “Forecasting COVID-19 impact on hospital bed-days, ICU-days, ventilatordays and deaths by US state in the next 4 months”, Institute for Health Metrics and Evaluation, University of Washington.
Manski, C (2009), “Diversified Treatment under Ambiguity,” International Economic Review 50: 1013-1041.
Manski, C (2013), Public Policy in an Uncertain World, Harvard University Press.
Manski, C (2019) “Communicating Uncertainty in Policy Analysis”, Proceedings of the National Academy of Sciences 116: 7634-7641.