Researchers, journalists, activists, and academics have been warning us for some time now about the dangers of biased AI algorithms. If you’re already familiar with this issue, you may be desensitized to the cries of “the sky is falling” and wary of yet another rant on this topic. I will do a bit of obligatory ranting below for the uninitiated, but I promise to also offer some food for thought, beyond the conventional doom and gloom. I hope you’ll bear with me.
As our ability to create predictive algorithms improved over the years and machine learning models surpassed human ability on a variety of tasks, “fairness” was supposedly an important selling point for AI models. Because algorithms have no prejudice — so we were told — we can trust them to make all sorts of decisions for our society. A giant heap of linear algebra doesn’t see skin color or gender, so how can algorithms be prejudiced? It turns out, of course, that the algorithms we create reflect the biases in our society, and often amplify them. AI models are often prejudiced and the consequences of that bias are sometimes severe.
The COMPAS algorithm created by Northpointe Inc. (now Equivant) has become the canonical example illustrating the danger of biased AI models. Because others have done the hard work of analyzing this algorithm for us and writing about it at length, I can jump on the bandwagon and pick on the COMPAS algorithm too. It’s really quite a scandal. The COMPAS algorithm attempts to predict recidivism rates of criminals, and has been used by a number of states and jurisdictions for case management, including to determine whether a person should be detained or released pre-trial, and also for as a sentencing consideration by judges. Because this algorithm is intended for use in our justice system, it was explicitly calibrated to be “fair.” Spoiler: it isn’t fair at all.
If you have the interest, time, and patience, you can read the original research on COMPAS’ bias here:
https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
COMPAS provides an assessment of risk that a defendant will commit new offenses upon release. (it predicts three types of risk actually — check out the article for details.) What the authors of the above ProPublica article found, in addition to the fact that COMPAS isn’t particularly accurate (as low as 20% accuracy at predicting violent recidivism) is as follows:
- Black defendants were misclassified as high-risk at nearly twice the rate of white defendants (45% vs. 23%).
- White defendants were misclassified as low-risk at nearly twice the rate of black defendants (48% vs. 28%).
COMPAS is racist. How did this happen, when this algorithm was explicitly calibrated for “fairness?”
Viewed mathematically, fairness is surprisingly easy to define. So easy, in fact, that I can write out for you countless mathematical definitions of fairness, all of them different! In machine learning we consider an algorithm to be “fair” if the results are independent of certain variables. So, for example, if we can show that an algorithm’s accuracy isn’t correlated to race (i.e. it makes mistakes at the same rate for all ethnic groups) then it’s accuracy is “fair” with respect to race. Spoiler, again: this is exactly the definition that Northpointe Inc. used in creating COMPAS. It worked, in the sense that COMPAS makes mistakes at roughly the same rates for white and black defendants. But the type of errors that COMPAS tends to make for white defendants is different from the type of errors it tends to make for black ones. It tends to overestimate risk for black defendants, and underestimate risk for white defendants, so it’s racist.
We now see clearly that a definition of fairness that we can use to evaluate an AI model depends on two factors: the choice of variables (e.g. race) and the choice of result (e.g. accuracy). You might conclude that had Northpointe Inc. chosen a better result for calibration — such as the rate of overestimation of risk — then COMPAS would have been truly fair.
Not so fast!
Fairness with respect to accuracy (Northpointe’s original definition) is important too. If, for example, it were known that COMPASS is substantially less accurate for white defendants than for black ones, would white defendants then be better able question the algorithm’s assessment and obtain special treatment? On the other hand, if it were known that COMPASS is less accurate for black defendants, would a racist judge be more likely to disregard the algorithm’s assessment in a black defendant’s case? And what if overall accuracy of the model dropped when using the new definition of fairness? Then — although we would no longer be giving an advantage to white defendants — we would arguably have less justice overall, because we’d have more incorrect assessments.
Machine learning researchers tell us that simultaneous optimization of an algorithm for both definitions of fairness we discussed above (fairness with regard to overall accuracy, and fairness with regard to the type of errors made) is mathematically impossible.
We can try to identify all of the different demographic groups that may be impacted by bias in an algorithm: women, people of color, queer folks of all stripes (or spots, if that’s your cup of tea), decaffeinated green-tea drinkers (if that’s your cup of tea), short people (shout out to Randy Newman), and so on. But we can’t ensure that our algorithm is fair to ANY of these groups with regard to all of the results that are important. It’s just not possible. Fairness is an ideal we can strive toward, but never fully achieve. Did we really have to invent computers and algorithms, and curate huge data sets to figure this out? We knew this already!
Our machine learning models WILL be biased. What’s a machine learning practitioner to do?
Cover your ass! I’m being simultaneously facetious, cynical, and sincere here. Firstly, if you’re covering your ass then presumably you understand that any model you build is going to have some sort of bias. You’re no longer deluding yourself that a simple calibration of your model is going to guarantee equitable outcomes in the real world. That’s a necessary first step. You also understand that algorithmic bias is a liability for you. The best way to limit your liability is to limit the harm that your machine learning application can cause.
Harm reduction is possible. You can make good choices about the types of bias that you attempt to detect and mitigate in your AI models. You can also adopt best practices for transparency, accountability, and stakeholder feedback, in order to mitigate harm in your application, regardless of AI model bias. Your application will then reflect your own biases regarding which social inequalities are important. Although that’s yet another form of bias in your application, it’s probably much better than doing nothing. Also, the road to hell is paved with good intentions. To properly cover your ass, you’ll have to do some soul searching.
I find it necessary to think about applications in terms of power dynamics: which stakeholders are empowered by an application? Who will end up with less power? Are power dynamics made more equal or less equal by the application? When groups have unequal power, the groups with less power usually end up getting screwed over, one way or another.
If your application changes power dynamics to make them less equal, it’s going to cause harm.
Sometimes power dynamics are crystal clear. For example, there’s no question that the Chinese government uses AI models for surveillance in ways that violate the human rights of the Chinese people (especially minorities such as the Uyghurs). But sometimes power dynamics are murky. Does a home security camera empower law abiding citizens over criminals? Yes. Does it empower wealthier people over poor people? Yes, it does that too. Does it empower white people over people of color? Yes, that also happens. What can you do to make sure your application creates a more level playing field for all stakeholders?
It’s important to remember if you’re part of a product team or engineering team, that one of the stakeholders is you. And that you have enormous power — by the ways that we shape the application — to make decisions about which groups and stakeholders are valued and which power dynamics are considered important. This is one of the very unequal power dynamics at play. When we centralize a decision making progress that impacts a lot of people by relegating it to an AI model designed by a few people, we may be creating or exacerbating an imbalance of power.
So if you’re a machine learning practitioner looking to cover your ass, so to speak, you must ask some difficult questions about your application: Is it favoring certain people based on race, gender, etc.? Is it enabling a business to take advantage of its employees or its customers? Is it enabling a government to oppress people? And you also have to ask: Is it better for you via algorithm design to set priorities for a large number of people, or is it better for the decision making to be more distributed?
It’s also important to keep in mind that as a machine learning practitioner, your skills are in high demand. You can decide to go work elsewhere. An AI model that helps farmers optimize greenhouse conditions to grow better tomatoes is much less likely to be racist than a model that makes hiring or lending decisions. How will you feel if your company becomes mired in criticism over bias in AI? How will it look on your resume? If you’re working on an application that impacts the lives of many people and can have disparate impacts on different groups, and you’re not sure that you can sufficiently reduce harm, it might be a very good idea to cover your ass as a professional — by choosing to work on a more benign application.