Weapons of Math Destruction – A Book Review

In her book Weapons of Math Destruction, Cathy O’Neil discusses the impact of constructing and then (often indiscriminately) applying behaviour-focused predictive models during decision making tasks. These days such models can influence who to hire, who to fire, how to administer and improve education or health care, who to provide social services to, how to target potential consumers and how to run political campaigns. She labels predictive models that have a wide spread, pernicious, effect Weapons of Math Destruction (WMDs).

O’Neil provides numerous specific examples of WMDs, describing in some detail their capabilities and negative impacts- which, as mentioned above, span many aspects of adult life. She convincingly argues that, because of access to ever increasing amounts of data about individuals, along with the increasing ability to automate decision making using these models, both the reach of these models and their potential negative impacts are swiftly growing.

Over the course of the book, O’Neil uses these examples to draw out two overarching issues with WMDs, and predictive models more generally. The first issue is that, for any predictive model, too much faith may be put in the recommendations or pronouncements of a construct which is, in fact, poorly made and thus functionally bad, in the sense of it being a bad fit for the system it is trying to model.

O’Neil notes that such a model might generate inaccurate predictions or explanations due to generally poor construction (e.g. it was trained on bad data and is drawing incorrect conclusions as a result, or the model is constructed in such a way that there is too much variability in its predictions). However she also points out one particularly problematic type of bad modeling that can occur when the model uses a proxy measure (e.g. race) to indirectly and incorrectly infer category membership (e.g. criminal). O’Neil provides numerous examples of this and how it can lead to a vicious downward spiral for those so mis-classified. She refers to this as the ‘birds of a feather’ problem, and it is connected to the second issue she raises, as well.

The second issue, although O’Neil does not state it in precisely this way, is that models may be functional but not in line with our ideological goals. In this case, O’Neil argues that if an existing model is behaving in an unfair manner, then either it should not be used, or it should be altered to be more fair, even if its predictions are more technically accurate in the current version.

I think each of these two overarching issues can lead to some very interesting discussions, but I suggest that the first issue is at least a little more clear cut, both with respect to a practical way forward and also along the moral dimension.

The solution here would seem to involve, first, determining if a model is broken and then, second, either finding a way to improve it, or stopping its use entirely, if it can’t be fixed. Granted, neither of these steps are actually simple to do in practice. Nor is the concept of ‘a bad model’ particularly clear cut.

Adding to this challenge is the fact that there are circumstances, as noted by O’Neil, which may leave people with a lack of motivation to take these required steps- for example, circumstances in which the presence of a known bad model might be tolerated because it allows people to avoid making difficult decisions themselves.

However, I think most people would agree that those who wish to continue using broken models for this or other reasons are not behaving appropriately. On top of this, there are at least some relatively clear strategies for identifying and evaluating potentially bad models. O’Neil highlights the importance of tracking actual outcomes relative to model predictions and using this feedback to increase the accuracy of the model.

The second issue is, from my perspective, the far more challenging of the two, as is often the case where ethics and ideologies are involved. In this case, we might have a model that functions well- even extremely well. And yet, despite its efficiency and accuracy, it may contravene what we consider to be appropriate boundaries for decision making and other behaviors.

A fairly clear cut example in this case is the harnessing of predictive models to exploit individuals. Most people would agree (in principal, if not in practice), that exploitation is not ideologically acceptable. O’Neil provides the example here of for-profit universities, which relentlessly comb the internet to gain information that can then be used to target vulnerable individuals and convince them to enroll in said universities, at very high cost to both the government and the enrollees, along with perhaps little benefit or chance of success.

And yet, even such an apparently clear cut example proves challenging. At what point does the right of a company to sell to its customers cross the line into coercion and exploitation?

Similarly, O’Neil raises the disturbing specter of pre-crimes, originally described in the haunting Philip K. Dick story ‘Minority Report’, where individuals who have not yet committed crimes are targeted by police because models have predicted that they likely will commit crimes in the future. Once again, in this situation (currently becoming uncomfortably less science fiction and more science fact) the potentially high accuracy of any such model actually intensifies the moral dilemma.

Here, much more so than in the for-profit university example, a question is raised that cannot be easily avoided: at what point does ‘collateral damage’ become acceptable, if ever? If a model is very highly accurate, except for on rare occasions, do the benefits outweigh the harm?

These questions are uncomfortable and difficult to consider, and veer quickly into deep philosophical territory. Fundamental ethical concepts like utilitarianism and deontology suddenly come into play. Their seeming esotericism can be difficult to reconcile with the real and harmful outcomes that may result from the misapplication of models in this way.

As a systems scientist, I might here suggest taking, instead, a detour into systems theory, and considering the effect that these models are having on systems that were not designed with them in mind. More specifically, we might consider whether or not they are, to an unacceptable degree, distorting the original intent of these systems, and what can be done, in response, to keep the systems operating in a functionally desirable manner.

O’Neil does allude to this approach in two parts of her book. First she raises it when discussing the issue of insurance companies, which serve for her as an example of a system that has been functionally distorted away from its original purpose. A second discussion of this occurs when she proposes that we pay attention to how feedback loops from (often connected) WMDs may be broken to prevent the distortion of the systems involved, and how what might be referred to as ‘constructive’ feed back loops could be incorporated into systems instead. A minor quibble here is that she refers to these constructive feedback loops as ‘positive’ feedback loops, but this is somewhat confusing, as ‘positive feedback’ is a term which has a fairly specific technical meaning in systems theory (somewhat surprisingly, in systems theory, positive feedback loops can be destructive and negative in effect, as well as constructive, and negative feedback loops can be good, in the sense that they can stabilize a system) but her point is a good one nonetheless.

Thus, O’Neil leaves us with many examples to consider and some final questions to chew on. Her book could prove eye opening to those who have not yet considered this issue in depth, and validating for those, like me, who have been looking for a way to better articulate their ongoing concerns and efforts to be an ethical data scientist.

A Generalist (Data Scientist) Confession

The other day I came across a saying, the first part of which spoke maybe a little too strongly to my own experience: A generalist is someone who knows nothing about everything (the second half of the saying being – a specialist is someone who knows everything about nothing).

Combined with some recent forays into new knowledge territories, this led to some navel gazing and self-evaluation on my part – which I admit I’m prone to from time to time.

I’m a bit on the far end of the generalist spectrum in my professional life – over the course of my schooling I’ve been involved with numerous disciplines, and in the process I’ve been exposed to a wide variety of research methods and techniques, including conceptual, statistical and systems analysis, and many types of modeling. I currently find myself falling under the data science label, where I’m doing analysis projects with computer engineers, statisticians and mathematicians, along with a wide variety of subject domain specialists.

Coming out of my recent navel gazing experience, I have to acknowledge that the more you know, (or perhaps it’s more accurate to say, the more areas of knowledge to which you are exposed), the more you start to feel, in your gut, how little you know in comparison to all that there is to know out there. And your awareness of this gap becomes ever more substantive and explicit the farther on you venture.

As an older, hopefully wiser, and perhaps slightly more humble generalist than I was in my younger years, I think there are a number of healthy reactions to such realizations. The first is that, before running pell-mell into new knowledge territories, we generalists should not take for granted our grasp of the knowledge in areas that we already consider familiar. For those that we wish to maintain a hold on, there will always be new developments, new discoveries and shifts in perspective, which will require us to continually renew and refresh our knowledge base.

It’s also easy for our knowledge of fundamental concepts in these areas to become dulled – assuming we understood them in sufficient depth the first time we encountered them, which might have been as far back as our undergraduate degrees! If we do take the time to regularly return to these concepts, we may find, when we do, that they have become more clear, more salient and more meaningful than they were on our first pass. And at this point we can also beneficially relate them to many other aspects of our knowledge and experience, gained along the way.

In terms of entering new knowledge territories, if we are truly dyed-in-the-wool generalists, it will be hard for us to resist exploring these new spaces. That said, perhaps rather than running willy-nilly through them with naïve abandon, we can encourage ourselves to take the time to selectively explore potential new avenues, and combine them strategically and usefully with what we already know, so that we can build upon our existing strengths and knowledge reserves. And perhaps we can take the humble, but realistic approach of starting with the basics, taking the time to really nail the details, and move forward at a modest pace that allows us to integrate all of this with our existing knowledge.

If we do all of these things right, our knowledge base will deepen rather than simply broaden. We will become generalists-who-know-quite-a-bit-about-many-areas-and-also-quite-a-lot-about-certain-specific-areas, or (if we are moving in the other direction) specialists-who-know-a-whole-lot-about-one-area-but-also quite-a-bit-about-several-other-interestingly-related-areas. I admit that’s a little more messy than the original saying. But, then, growth usually is.