Weapons of Math Destruction
Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy by Cathy O’Neil
🚀 The Book in 3 Sentences
WMDs can be characterised by three features: opacity, scale, and damage.
Big Data models that use algorithms to make decisions. WMDs ignore nuance and often rely on flawed data.
We need to bring fairness and accountability to the age of data. Maths deserves better than WMDs and democracy does too.
🎨 Impressions
This is a great book, really exploring the impact algorghtims have on our lives and exposing how unfair they often are.
There are many compelling case studies and examples that go to show how WMDs can be so dangerous as a result of applying big data technologies to everyday life. Examples include:
Algorithms used by judges to make sentencing decisions based on recidivism rates.
Algorithms that filter out job candidates from minimum-wage jobs.
Micro-targeting algorithms used in politics that allow campaigns to send tailored messages to individual voters.
Algorithms to assess teacher performance based on student standardized test scores.
Algorithms that use demographics data to determine online ads, judge our creditworthiness, etc.
How I Discovered It
Cathy O'Neil was featured in the also eye-opening Social Dilemma show on Netflix. Her views sounded interesting and she is a well credited expert in the field.
Who Should Read It?
If you think that big data is not impacting your life, then this book will likely change your mind. They are becoming more and more intertwined with our lives and they play a significant part in decisions on high impact topics such as democracy and justice.
✍️ My Top 3 Quotes
Amazon and Netflix can plunk their paying customers into little buckets and optimise them all they want. But the same algorithm cannot delivery justice or democracy.
Big Data processes codify the past. They do not invent the future.
In a system in which cheating is the norm, following the rules amounts to a handicap.
📒 Summary + Notes
What is a Weapon of Math Destruction: Opacity; Scale; Damage
define their own reality and use it to justify their results
self-perpetuating, highly destructive and very common
it never learns whether it was right
many poisonous assumptions are camouflaged by math, go largely untested and unquestioned
they are opaque and unaccountable and operate at a scale to sort, target and 'optimise' millions of people
they cause a downward spiral
These models powered by algorithms, slam doors in the face of million of people, often for the flimsiest of reasons and offer no appeal. Their are unfair.
You cannot appeal to a WMD. They do not listen. Nor do they bend. They are deaf not only to charm, threats are cajoling but also to logic - even when there is good reason to question the data that feeds their conclusions.
The human victims of WMD's, we'll see time and again are held to far higher standards of evidence than the algorithms themselves.
The people running the WMDs don't dwell on those errors. Their feedback is money, which is also their incentive. Their systems are engineered to gobble up more data and fine-tine their analytics so that more money will pour in.
They tend to punish the poor. The privileged time and gain are processed more by people, the masses by machines
People have deliberately yielded formulas to impress rather than clarify.
Proxies are easier to manipulate than the complicated reality they represent. As people game the system, the proxy loses effectiveness. Cheaters wind up as false positives. And sometimes there can be worthy goals behind them, but every ranking system can be gamed. And when that happens, it creates new and different feedback loops and a host of unintended consequences.
More data is easy to believe as better data - but this is not often the case (due to the pernicious feedback loop that spawns more data)
WMDs favour efficiency. By their very nature, they feed on data that can be measured and counted. But fairness is squishy and hard to quantify. It is a concept. And computers, for all their advances in language and logic, still struggle mightily with concepts. Fairness cannot be calculated into WMDs so what happens is an industrial production of unfairness. We need to consider whether we as a society are willing to sacrifice a bit of efficiency in the interest of fairness.
Mathematical models can sift through data to locate people who are likely to face great challenges, whether from crime, poverty or education. It's up to society whether to use that intelligence to reject and punish them - or to reach out to them with the resources they need. We can use the scale and efficiency that make WMDs so pernicious in order to help people. It all depends on what objective we use.
Once people recognise WMDs and understand their statistical flaws, they'll demand fairer evaluations. However, if the goal of testing is to find someone to blame, and to intimidate workers then a WMD spews out meaningless results actually gets a A+.
Conclusions:
The quiet and personal nature of targeting through big data can keep society's winners from seeing how the very same models are destroying lives, sometimes just a few blocks away.
We need to think about how we assign blame in modern life and how models exacerbate this cycle.
Human decision making, while often flawed, has one chief virtue. It can evolve. As human beings learn and adapt, we change and so do our processes. Automated systems by contrast stay stuck in time until engineers dive in to change them.
Big data codifies the past. They do not invent the future. Doing that requires moral imagination, and that is something only humans can provide. We have to explicitly embed better values into our algorithms, creating big data models that follow our ethical lead. This will sometimes mean putting fairness ahead of profits.
Amazon and Netflix can plunk their paying customers into little buckets and optimise them all they want. But the same algorithm cannot delivery justice or democracy.
We can also equip these powerful engines with steering wheels and brakes.
Cathy's hope is that WMDs will be remembered like the deadly coal mines of a century ago, as relics of the early days of this new revolution, before we learned how to bring fairness and accountability to the age of data. Maths deserves better than WMDs and democracy does too.
