Responses to questions/comments on the career outcomes paper


Q: Why didn't you include the data?

The data include personal, identifying information for individuals so they cannot be shared publicly. However, an anonymized data table has now been included, along with the analysis code.


Q: Where did the data come from? Did you rely on the tables that PhD programs provide?

Only lists of names were obtained from PhD program webpages. Everything else came from a more primary source: for example, the individual's CV, a biography on their university webpage, their ADS affiliation history, and the like. I found that most current PhD program alumni webpages were years out of date (especially for individuals who left the field) so these definitely would not have been a reliable source.


Q: There is extensive data demonstrating a leakier pipeline for women compared to men, and there are few women in our discipline especially at senior levels. Doesn't this mean that there must be something wrong with your results?

The inference of a leaky pipeline in astronomy is based primarily on the gender structure of the field as measured at a fixed point in time (i.e. today): for example, there are relatively many female undergraduates and relatively few female full-professors. Statistics at all levels are worse than 50/50, so a gender asymmetry clearly exists. However, specifically where the pipeline is leaking today is hard to establish based on a fixed-time snapshot of the field. Today's numbers reflect accumulated historical trends over several decades, and there has been a large change in gender dynamics of the field between 1975 (a typical PhD year for a retirement-age professor) and today. My study only addresses the PhD to postdoc to junior-staff segment of the pipeline, and only for 2000-2012 PhDs. It finds no differential leaks over this specific segment over this specific period. There must still be a leak pre-PhD and there may still be a leak post-staff-hire (although I discuss some reasons in the paper why the latter is probably not a large effect.)

I will note also that my conclusion is consistent with some other independent work, including the present gender structure across the three specific levels investigated (PhD, postdoc, junior-staff: all are about 30% women, suggesting no significant gender-dependent attrition over this period). Citations to other studies to this effect are given in the paper. Post-acceptance I was also pointed to Figure A.1 of this paper: https://arxiv.org/abs/1610.08984


Q: Aside from the numbers, there is ample direct evidence of gender bias in astronomy. But you don't see any evidence of that in your study. Doesn't this mean that there must be something wrong with your results?

There is indeed ample evidence of gender bias on numerous fronts, even today. I have witnessed examples of this personally and it clearly remains a problem. However, empirically, these effects do not seem to lead to a discernable increase in the tendency of postdoctoral women to leave the field or a lower rate of being hired into jobs, relative to men (within statistical errors of approximately 30%). There are a few possibilities for why this may be the case: (a) Women who complete PhD's are resilient and able to overcome this bias. While some may indeed give up or otherwise end up leaving the field because of discrimination or sexism, this is uncommon enough that it is within the noise of the study. (b) There is a lot of awareness of the existence of sexism, bias, and the greater adversity faced by women on hiring committees, and thus many formal and informal processes are in place to try to achieve a level playing field. These efforts by and large are working. (c) Women who complete PhDs are better on average than men who complete PhDs - perhaps because the bar is higher for their recommendation letters, or perhaps because only the very best women go into the field but a broader subset of men do. They should by all rights outcompete men at later stages, but discrimination erodes away this expected advantage. (There may be other reasons I haven't thought of.)


Q: The fraction of astronomers staying in the field (66%) seems far too high to be reasonable. Are you sure you are not missing people who left?

This result surprised me also. However it seems to really be the case. This number is similarly high across essentially all PhD programs I looked at, and also agrees with the analyses reported by specific PhD institutes on their own alumni (e.g. the Dinerstein 2011 study for Texas alumni). So, I think it is robust, at least for the study period.

It may not be quite as high as 66% in reality, because a somewhat higher proportion of the omitted individuals whose careers I was completely unable to trace probably did actually leave (as compared to the retained individuals) . However, because only 10% of the sample was omitted, that could drive the numbers down by only a few percent (to 60% in the absolute most extreme possible case that all omitted individuals left the field). Also, I note that this holds only for US astronomy programs. Physics programs and non-US programs may not have the same rate (see below).


Q: Your study is dominated by famous prestigious schools. They don't capture the experience of people outside these programs.

The study is actually based on a broad range of astronomy PhD programs, not just the top schools. For example, using the R-rank from the NRC survey as an indicator of program prestige, the study includes 7 out of the top 10 programs, 6 out of the programs ranked 11-20, and 8 out of the remaining eleven programs (21-31). These tend to be better-known schools, but this is just because it is rare for a PhD program in astronomy to exist at lesser-known schools.

Additionally, another surprising result was that the prestige of the school makes almost no difference in the rates at which their alumni stay in or leave astronomy. It does have a large impact on the type of job within astronomy, however (e.g., R1 university vs. teaching vs. staff scientist).

The study does fail to adequately capture small astronomy groups in larger physics programs, which are a nontrivial fraction of astrophysics PhDs (and more likely to be at smaller universities with fewer resources). It is possible that the experiences of alumni from those programs are different.


Q: Why didn't you include planetary science?

Many planetary science programs are integrated with geology, making them quite different in demographics from astronomy departments (which are stand-alone or integrated with physics). It would be interesting to look at these programs in the future (and at physics and various other research areas). However, this study was time-consuming enough as it was and I couldn't look at everything!


Q: What about non-US PhDs? Why did you focus only on the US?

The US was a practical choice because it is large (allowing for a statistically meaningful sample), because PhD practices and cultural expectations are relatively uniform across the country, and because it is relatively easy in practice to find lists of alumni compared to non-US institutes. It is also what is most familiar to me, being a US PhD myself.

It would be interesting to look at non-US programs in the future. My suspicion is that the fraction remaining within the field will be lower than in the US, because PhD programs in most countries are shorter and because there are relatively few positions at observatories or space agencies outside the US (these are major employers within the US) and less of a tradition of small teaching colleges. However that is speculation. Someone should try to carry out this study and find out.


Q: This work assumes a gender-binary. A footnote paying lip-service to the complexity of this issue is not enough to address the continuing exclusion of non-binary-identifying individuals from society. Nor do you justify why you assume that the number of non-binary individuals is "small".

I'm really sorry about this. I also find it regrettable that questions of gender equity in astronomy are still usually cast in the traditional binary, silently writing out people with more complex experiences. But because my investigations were based primarily on assigning gender based on first names, there was no opportunity to look into other categories, or into the more fluid and/or continuous notions of gender that better reflect people's actual identities. I would also like the field to be more progressive about this in the future. If you have a suggestion for how I could have addressed this more sensitively please feel free to write me.


Q: Why are astronomers wasting their time attempting to do sociology? Leave this to the experts.

If we want to understand why astronomy as a discipline is the way it is and have productive conversations about to make it better, we need reliable data. A good deal of our knowledge comes from studies of other fields or of physics or physical sciences more generally - but gender demographics, employment markets, and department cultures are very different even just in our parent field of physics, to say nothing of engineering, chemistry, or biology. While we can try to make the case to sociologists that our field is worth their attention, if we really want to know the status of our profession we have to do the work ourselves.

Personally, I can say that this is far from my own primary research topic (I am an observational transient astronomer and have little direct experience with gender issues in science and minimal familiarity with the literature before carrying out this project.) However, I was intrigued by the study of Flaherty (2018), who came to the conclusion that women were hired in to faculty jobs faster than men based on a very simple technique of looking at recent hirees' career pasts. I was, however, concerned about the methodology - and after seeing the level of attention his study received decided it was worth a little bit of my time to see if it could be reproduced via a more reliable method of working forward from the PhDs, rather than using Flaherty's method of working backward from faculty (which by construction left his study blind to people who leave the field).

I was not able to reproduce the result, even after greatly expanding the sample with as many PhD programs that I could find. I considered leaving it at that, but the "file-drawer" effect (non-publication of null or counter-intuitive results) is one of the biggest problems in social-science research and I thought in the end that this was something that the community should be aware of. So I continued to expand and retool it with the intention of ensuring it could pass peer review.

I imagine that if the survey had been carried out by a full-time sociologist it would have been stronger. However, I think it is good enough to help inform us on the status of our profession. Additionally, I did at least send the paper to one social-science professor I know to get his opinion and he was quite enthusiastic about it, so I don't think it would compare too poorly to fully-fledged social research.


Q: I still don't trust the data/analysis. How can I be sure this is right?

The best thing to do would be to try to replicate the study yourself. Gathering my data took a lot of work, but you don't have to gather 1000 data points to confirm the same basic result (even just 100 samples should be enough to show the lack of discernible hiring bias within a fraction of about 50 percent, for instance. This data could be gathered in a day or two.) You could also try other input samples, such as ADS PhD thesis records or fellowship listings. My own preliminary investigations were based on the Hubble fellowship listings, which helpfully provides the PhD year and institution for each fellow.