The under-representation of women in higher academic ranks is a widespread phenomenon, and it has barely improved over time, especially in some fields of study. According to the European Commission's (2021) She Figures report, the proportion of female students and graduates at the Doctoral level was close to gender parity (around 48%) across Europe in 2018, with a slight improvement compared to 2015. Yet, the proportion of women among full professors was 26%, with the leaky pipeline phenomenon spanning across all fields, including those where men are under-represented at the student level, like humanities.
Within the field of economics, the literature has extensively documented the gender divide in academia (Lundberg 2020, Janys 2022). In the top US departments, women represent just 15% of full and 27% of associate professors (Chevalier 2022). The corresponding figures for Europe are 20% and 32%, respectively (Auriol et al. 2019). In the UK, women made up 33% of lecturers, 27% of senior lecturers/readers and 15% of professors (Bateman et al. 2021). Under-representation of women economists extends to leadership positions in public and private sectors (Hanspach et al. 2021).
Such imbalances appear very early in the career and call for an examination of the obstacles that women encounter since their underrepresentation not only negatively affects gender equality in the profession but also limits the richness and breadth of topics examined (Bayer and Rouse 2016).
The influence of reference letters on career outcomes
The transition from a graduate programme to the labour market is the stepping stone of the career, and the referral process is a key aspect of access to and success in the job market. Though reference letters are considered an essential ingredient of the professional job market, we lack a large-scale assessment of whether the referral process is gender-neutral, both on the advisor (i.e. do female and male advisors talk about different characteristics of candidates in reference letters they write?) and on the candidate side (i.e. are female and male candidates described differently?). Eberhardt et al. (2022) took a step in this direction, showing that there are gender differences in candidates' descriptions. In our recent paper (Baltrunaite et al. 2022), we study whether and how gender differences in reference letters translate into diverging career paths for female and male researchers.
Specifically, we focus on the content of the letters that PhD candidates receive from their advisors when applying to the junior economics job market and, by combining modern text analysis tools with insights from the psychology literature, explore whether such letters reveal implicit gender stereotypes in how candidates are described and who holds such stereotypes. We then estimate the relative contribution of candidate, advisor, and letter characteristics and these potentially existing implicit biases in explaining gender differences in early career outcomes.
The data
To analyse the influence of reference letters and language used therein on career outcomes, we build a novel rich dataset covering ten cohorts of academic job market applicants (2010-2019) to two top institutions hiring on the international job market for economists, for a total of about 8,000 applications and 25,000 reference letters. We recover information on candidates from their anonymised application packages, including their CVs. We also collect information on candidates' current placement, publication and citation records using massive web-scraping techniques on several publicly accessible repositories (e.g. Google Scholar, LinkedIn) and combine those with information on advisors retrieved from the same sources.
In our data, fewer than one-third of applications come from female candidates, and just 15% of the letters are written by female sponsors. These shares remained constant in the ten years considered. Looking at candidates' characteristics, we observe that women generally come from lower ranked institutions, more often Europe-based, and are less likely to have already published any research work before the job market. As expected, there are significant gender differences in terms of field of specialisation of PhD candidates, with women being ten percentage points more likely to specialise in applied microeconomics research than their male peers, who instead tend to choose topics in macroeconomics, finance, theory or quantitative methods more often (Fortin et al. 2021).
Interestingly, at the time of application, male candidates have, on average, a higher number of academic references, both in terms of designated referees and of actual letters uploaded, and their letters are longer by around half a paragraph. Finally, women eventually have lower career success, not much in terms of ranking of the institution of affiliation, but rather in terms of position and publication records.
What do letters say about candidates?
To answer this question, we focus on target words that likely capture some meaningful characteristics of the candidates. To identify them, we resort to established literature in applied psychology, aiming to pin down the presence and magnitude of stereotypes and implicit (gender) biases in the labour market. Following Schmader et al. (2007), Madera et al. (2009) and Chapman et al. (2020), we start from two semantic categories utilised to describe job applicants: ‘standout’ and ‘grindstone’ terms. They represent, respectively, words referring to the candidates' exceptional character (e.g. outstanding, unique, brilliant) and words referring to the effort they put into work (e.g. hardworking, conscientious, diligent). We also consider two other categories identified as indicators of implicit gender stereotypes: ‘agentic’ and ‘communal’ words. The first ones refer to personality traits related to self-confidence and assertiveness (e.g. competitive, ambitious, proactive). The latter, in contrast, refers to personality traits that emphasise a person's ability to sympathise with others (e.g. agreeable, caring, warm).
We employ one tool of modern text analysis that uses a supervised approach, specifically word embeddings, to quantify whether such implicit stereotypes appear in our corpus of reference letters. Specifically, word embeddings are a machine learning tool that allows the representation of a word's meaning through a mathematical object – i.e. a vector in a low dimensional space whose position and relative proximity to other words' vectors capture their semantic similarity in a way that words with similar meanings or semantically related will lie close together. In particular, we measure similarity by cosine distance. We find that the average (cosine) distance between the embedding of standout terms and that of reference to male candidates is lower than that for female candidates, showing that males are more likely to be described in standout terms. The opposite is true for grindstone words, which lie closer to reference to female candidates. These results persist, also controlling for several candidate's characteristics that are meant to proxy for unobserved ability. Considering then the candidate's assertiveness (agentic traits) vis-à-vis her interpersonal skills (communal traits), the gender differences in their use do not appear to be large or robust.
Who drives the differences?
Although female advisors tend to write more about all candidates' personal characteristics, it is male letter writers who do so to a different extent for male and female candidates, putting more emphasis on standout characteristics in the case of male candidates and on grindstone ones for female candidates.
Do these differences influence career outcomes?
When in a regression framework, we study gender differences in early career achievements, like seniority in the position and prestige of the institution of placement, we find that the use of standout (grindstone) terms is positively (negatively) correlated with proxies of career success, even controlling for proxies of candidate's quality. Moreover, these differences in how candidates are described explain a non-negligible part (from 5% to 8%) of the lower success of female PhDs in the economics profession.
Implications
Our analysis reveals that gender biases in the way male and female economists are perceived and described by senior academics do exist and relate to gender differences in the career progressions of young researchers. It also highlights a potential structural flaw in the academic job market process. By heavily relying on reference letters, it puts female candidates in a weaker position to compete. The negative consequences in terms of career progression are particularly relevant in contexts like the economics profession, which is highly male-dominated, especially among senior professionals. Awareness-raising campaigns, guidelines on what language to use in recommendation letters, and the use of open text versus specific areas of comment are some examples of potential areas of change with the goal of securing an inclusive profession.
References
Auriol E, G Freibel and S Wilhelm (2019), “Women in European Economics”, VoxEU.org, 19 November.
Baltrunaite A, A Casarico and L Rizzica (2022), "Women in Economics: the role of gendered references at entry in the profession", CEPR Discussion paper 17474.
Bayer, A and C E Rouse (2016), "Diversity in the economics profession: A new attack on an old problem", Journal of Economic Perspectives 30 (4): 221-42.
Bateman V, D K Gamage, X Liu and E Hengel (2021), "The Gender Imbalance in UK Economics: Royal Economic Society Women's Committee Silver Anniversary Report".
Chapman, B, M Rooney, E Ludmir, D Cruz, A Salcedo, C Pinnix, P Das, R Jagsi, C Thomas and E Holliday (2020), "Linguistic biases in letters of recommendation for radiation oncology residency applicants from 2015 to 2019", Journal of Cancer Education 37(4):965-972.
Chevalier, J (2022), "Report: Committee on the status of women in the economics profession (CSWEP)", AEA Papers and Proceedings 112: 746-67.
Eberhardt, M, G Facchini and V Rueda (2022), "Gender Differences in Reference Letters: Evidence from the Economics Job Market", CEPR discussion paper 16960.
European Commission (2021), SHE Figures 2021. Technical report, Directorate-General for Research and Innovation.
Fortin, N, T Lemieux and M Rehavi (2021), "Gender differences in fields of specialisation and placement outcomes among Phds in economics", AEA Papers and Proceedings 111: 74-79.
Hanspach P, J Palka and V Sondergeld (2021), "Few top positions in economics are held by women", VoxEU.org, 15 February.
Janys, L (2022), "Evidence for a Two-Women Quota in University Departments across Disciplines", Review of Economics and Statistics, forthcoming.
Lundberg, S (2020), "Women in the economics profession: A new eBook", VoxEU.org, 5 Mar.
Madera, J M, M R Hebl and R C Martin (2009), "Gender and letters of recommendation for academia: agentic and communal differences", Journal of Applied Psychology 94(6): 1591-9.
Schmader, T, J Whitehead and V Wysocki (2007), "A linguistic comparison of letters of recommendation for male and female chemistry and biochemistry job applicants", Sex Roles 57: 509-514
