During this past summer, I relocated across the country and searched for a job in my new locale, the San Francisco Bay Area. Few things were clear; I knew I wanted to work in-person; I knew I was going to have more traction in healthcare-adjacent industries; and finally, I knew I wanted statistical reasoning to be part of my job’s core function.1

That third and final piece–prioritizing statistical reasoning–proved most elusive. Whereas on the other items, I found that many companies were hiring data scientists in the health sciences and enthusiastic about hiring in-person candidates. As I had anticipated, I gained interview traction in this space more than any other industry with similar job titles (marketing, advertising, autonomous vehicles, etc.). It struck me as mildly surprising, then, that companies’ core needs were not statistical. Rather, I noticed urgent needs in data numeracy, stakeholder communication, and quality control–I want to acknowledge that all of these skills are also present in the workflow of a competent statistician. Statisticians need to “read” and “write” data fluently, draw conclusions that can be interpreted by colleagues, and ensure the data fed into their models are of the highest possible quality. The distinction between this suite of tools and statistical reasoning, I think, is perhaps a philosophical one.

To me, one of the most appealing parts of statistical reasoning is that it offers some insight into the way the world appears and why it is different from the way the world might actually work.2 Understanding such a difference–whether it be in science, technology, or social behavior–is fundamental to my acquisition of knowledge. As an innately curious person, I routinely seek out these differences and try to verify if they are true.

Here’s an example in the context of a pharmaceutical drug: suppose a recently completed randomized controlled trial failed to find evidence that the drug worked in patients, compared to those who received a placebo control. On its surface, this information might lead one to believe the drug is non-efficacious. Returning to my earlier point, this is the way the world appears.

But reasoning through this data, a statistician could think of many reasons why the trial failed. If further investigation revealed that the trial enrolled 30 patients; and that, by chance, a disproportionately high number of enrolled patients presented with above average disease severity–then we get a glimpse into the way the world might actually work. Specifcally, the drug might be efficacious, but the trial was ill equipped to tell us this truth. Most statisticians would at this point pause and ask for a lot more context, but not one of them would be able to conlcude efficacy based on this information alone; we are instead occupied with a temporary state of ignorance.

The temporary state of ignorance is the statistican’s comfort zone, metaphorically speaking. Companies are willing to spend lots of money trying to generate conclusions, even outside the scientific sense; intellectual property is protected for financial and economic reasons. But the trait I’m getting at–the recognition of an appearance which is superficially deceptive–is a trait that is required for statistical reasoning. And further, I don’t think it is a skillset adequately valued in today’s job market.

To close, I’ll quote from a September 29th post I saw by James Scott.3 He is professor and department chair of Statistics at the University of Texas. He was writing to parents and students who asked him whether AI makes statistics obselete. On this question, he and I align quite a bit:

If you’re making a claim about how some [AI-enabled] system performed, that’s ultimately a claim about what the evidence supports and how much uncertainty remains. Those are exactly the kinds of claims statisticians are trained to evaluate.

Scott is urging parents to see the value in preparing their children for uncertainty. My thesis, on a similar line of thinking, is that companies are temporarily distracted by generating insights through cleaner datasets and faster analysis, but that they are overlooking the reasoning skills required for real knowledge growth.

  1. For context, my only graduate-level education and most of my job experience is in biostatistics. ↩

  2. Here I’m using the term “world” quite loosely, but it suffices to mean any systematic process, like those that govern the biology of infectious diseases. ↩

  3. “Every time I talk to parents and students considering UT…”. Read the full post on LinkedIn. ↩