The rapid increase in earnings inequality in the US has fuelled a public debate over its causes – including technological change in the form of automation, for example – and has created anxiety over the future of work. The occupational landscape has changed dramatically in the late 20th and early 21st centuries: middle-wage occupations have shrunk as a share of total employment, while occupations requiring social and analytic tasks have grown (Autor and Dorn 2013, Deming 2017). While these patterns are well understood, how much individual occupations or job titles have changed over time is less clear. Furthermore, to what extent these changes are driven by new technologies is an open question.
These questions have been tackled, at least partially, from a number of directions. First, case studies of managers (National Research Council 1999) and cashiers (Basker et al. 2012) have documented substantial changes during the 20th century in the mix of tasks that these workers perform. Second, in other countries (notably Germany), repeated surveys have provided valuable insights into occupational change (Spitz-Oener 2006). And, finally, a burgeoning recent literature has examined the text content of online job ads and found substantial within-occupation churn in the mix of tasks and technologies since the beginning of the Great Recession (Hershbein and Kahn 2018, Modestino et al. forthcoming, Deming and Noray 2018). While each of these streams of research indicates that occupations have evolved, it has been difficult to quantify the extent of occupational change over long horizons because existing datasets are not meant to capture systematic changes in job tasks.
Our research aims to fill this gap in measurement, with the hope of better characterising occupational change in the US. We have introduced a new dataset to measure occupational tasks, technologies, and skills in the US over the last six decades of the 20th century. We used the digitised text content of approximately 8.7 million job ads appearing in three major metropolitan newspapers: The Boston Globe, The New York Times, and The Wall Street Journal.
One of our key findings is that the tasks that workers perform on the job have changed radically, most notably in the decline of routine and manual tasks and the increase in non-routine and analytic tasks. We show that existing data sources, which are unable to measure within-occupation adjustments over time, miss a large share of the evolution of job tasks in the labour market. After documenting these key trends, we explore the role of computer technologies in transforming occupational tasks.
A new dataset
The recent digitisation of historic newspapers and new computational tools developed in the field of machine learning have allowed us to create a new dataset to study occupations. Using the text of approximately 8.7 million job ads published in newspapers, we constructed a new dataset measuring a number of job-related elements, including tasks, technologies, and skills, covering 1940–2000. We also mapped job titles that appear in the postings to their standardised occupational codes, so that our data can be linked to widely used datasets like the census.
A dataset based on job-vacancy text offers several distinct advantages over standard datasets for studying occupational change. First, the text appears in the field where firms define the jobs and the characteristics of workers they are seeking. Second, job postings can be studied at higher frequencies, for example, annual or quarterly intervals. Of course, there are challenges such as representativeness and selection into posting ads versus other methods of search. We therefore perform a number of checks on the data to provide evidence against these threats.
Jobs have changed more than we thought
In recent work (Atalay et al. forthcoming), we use these data to study long-term changes in occupational tasks, as measured by word occurrences in the newspaper job ad text.
We find that the frequency of words related to non-routine analytic and non-routine interactive tasks increased, while the frequency of words related to manual tasks (especially routine manual tasks) declined. For example, Figure 1 shows that, over the 20th century, managers became more analytic and interactive, while becoming less routine cognitive and manual. This margin of change – within the manager job title – is unobserved in standard data sources.
Figure 1 Job tasks of managers in the 20th century
Notes: The top panel plots the frequency of different tasks for job ads with a ‘manager’ job title: non-routine analytic, non-routine interactive, non-routine manual, routine cognitive, and routine manual. The bottom panel presents the frequency of mentions of different O*NET Activity measures for managers. In both panels, measures are divided by the average frequency, measured across all job titles and years. Both series apply a local polynomial smoother, using a bandwidth of four years.
The majority of changes in the tasks workers perform have occurred within rather than between occupations. This finding implies that past research, which has been limited by available data, has significantly understated the occupational transformation of the US labour market. It also suggests a larger potential for task changes to account for the growing US inequality (Firpo et al. 2011, Adao et al. 2019).
How computer technologies changed workers’ tasks
In light of these dramatic changes in occupational tasks, the question arises: What role has technological innovation played to drive occupational change?
In an earlier paper (Atalay et al. 2018), we study the effect of introducing new computer technologies related to information and communication. We introduce a new dataset that records the use of specific technologies in each occupation and year – a level of detail unobserved before. These technologies range from office software (Lotus 1-2-3, WordPerfect, Microsoft Word, Excel, PowerPoint), enterprise programming languages (electronic data processing, Sybase), general-purpose programming languages (COBOL, Fortran, Java), and hardware (UNIVAC, IBM 360, IBM 370), among others. Figure 2 illustrates the power of our data to measure technological innovation and plots the number of times these different technologies appeared in job ads over time. We then study the effect of technology adoption on occupational tasks.
Figure 2 Computer technologies entering the labour market
Notes: The figure depicts the smoothed frequencies of which job ads mention each of the 48 computer technologies, eight of which are highlighted in thick, dark lines and 40 of which are depicted by thin, light grey lines.
The key finding is that mentions of new technologies have been accompanied by the increasing intensity of non-routine analytic job tasks. (There are interesting exceptions, including Microsoft Word and Microsoft PowerPoint – technologies that lead to a substitution away from non-routine analytic tasks.) In the paper, we study the relationship between each technology and each task, depicting a complementarity between technologies and analytic job tasks that had not been uncovered in past research.
We also study the impact of new technologies on inequality through a model of worker sorting and find a modest effect of these new technologies on income inequality in the US.
A method to measure the ever-changing nature of work
New data holds tremendous promise for understanding changes in the labour market. Using newly digitised job ads posted in historical newspapers, we generated a new, publicly available dataset that allows researchers, policymakers, and journalists to study the changing nature of work in the US (the data can be accessed at https://occupationdata.github.io).
Our research demonstrates that new methods and sources of measurement can reveal new patterns in the labour market. And, indeed, we document a large transformation of the labour market, in part driven by the introduction of new computer technologies. In future work, these methods could be extended and applied to study online job postings and allow us to peer into the present and future of work.
Author’s note: This column is based on prior research by the authors (Atalay et al. forthcoming, Atalay et al. 2018) which received generous financial support from the Washington Center for Equitable Growth and the Russell Sage Foundation. The views expressed here are those of the authors and do not necessarily reflect the views of the Federal Reserve Bank of Philadelphia or the Federal Reserve System.
References
Adao, R, M Beraja and N Pandalai-Nayar (2019), “Technological transition with skill heterogeneity across generations”, MIT Discussion Paper.
Atalay, E, P Phongthiengtham, S Sotelo and D Tannenbaum (2018), “New technologies and the labor market”, Journal of Monetary Economics 97: 48–67.
Atalay, E, P Phongthiengtham, S Sotelo and D Tannenbaum (forthcoming), “The evolution of work in the US”, American Economic Journal: Applied Economics.
Autor, D H, and D Dorn (2013), “The growth of low-skill service jobs and the polarization of the US labor market”, American Economic Review 103(5): 1553–97.
Basker, E, S Klimek and P Hoang Van (2012), “Supersize it: The growth of retail chains and the rise of the big-box store”, Journal of Economics & Management Strategy 21(3): 541–82.
Council, N R (1999), The changing nature of work: Implications for occupational analysis, Washington, DC: The National Academies Press.
Deming, D J (2017), “The growing importance of social skills in the labor market”, The Quarterly Journal of Economics 132(4): 1593–640.
Deming, D J, and K L Noray (2018), “STEM careers and the changing skill requirements of work”, NBER Working Paper 25065.
Firpo, S, N M Fortin and T Lemieux (2011), “Occupational tasks and changes in the wage structure”, IZA Discussion Paper 5542.
Hershbein, B, and L B Kahn (2018), “Do recessions accelerate routine-biased technological change? Evidence from vacancy postings”, American Economic Review 108(7): 1737–72.
Modestino, A S, D Shoag and B Joshua (forthcoming), “Upskilling: Do employers demand greater skill when workers are plentiful?”, Review of Economics and Statistics.
Spitz-Oener, A (2006), “Technical change, job tasks, and rising educational demands: Looking outside the wage structure”, Journal of Labor Economics 24(2): 235–70.
