Transplant surgeon Peter Stock probably never expected to face a crowd of two hundred people arguing about whether or not a computer system could help save lives. When Stock finished his training in the 1980s, the idea of using computers to guide life-and-death decisions was the stuff of science fiction. But two decades later, when Stock found himself in a St. Louis conference room discussing organ allocation in the United States, the question facing him was not whether computers would make these decisions—it was how.
As chair of the US Kidney Transplantation Commit- tee, Stock knew how much the possibility of a kidney transplant meant to every person with damaged kidneys. Kidneys filter toxins from the human bloodstream, and kidney failure can kill someone within weeks or even days. Roughly one in seven Americans, 37 million people, have some form of chronic kidney disease. More than 100,000 people are on the kidney transplant waiting list each year, but only some 21,000 kidneys are available for transplant. The waiting list gets longer each year.
In 1984, the US Congress established the Organ Procurement and Transplantation Network to find a way to direct donated kidneys to people who need them most. At first, doctors sent kidneys to the nearest person in need. Then, as kidney transplants became safer and more routine, groups of patients, doctors, and politicians called for a fairer system. By the early 2000s, statisticians had developed an automated transplant prioritization system called LYFT that predicted how much longer a per- son could live if they received a donor kidney (LYFT an acronym for “life years from transplant”). But not every- one agreed with this system. Elderly patients alleged that LYFT put them at a disadvantage by prioritizing younger patients. Communities of color and those facing poverty argued that it prioritized wealthier, more privileged patients. Everyone wanted to know how LYFT actually worked.
As chair of the Organ Procurement and Transplantation Network’s Kidney Transplantation Committee, Stock knew he needed to come to the debate equipped with both data and compassion. Stock’s committee opened the meeting in St. Louis by presenting a series of evaluations of the LYFT system. The committee hoped these independent system analyses could quell people’s concerns and inform a search for something better. The evaluations Stock’s group presented—audits of the LYFT system—sparked a series of analyses that identified disparities across patient populations, measuring differences based on age, gender, geography and race. This work informed decisions that would help bring greater equity to the use of this complex system for years to come.
Decision-making systems like LYFT represent a key component of what we now call artificial intelligence (AI). From the algorithms that direct lifesaving treatment to the software that filters our emails for spam, AI systems are now powering—and changing—countless aspects of our lives. Wherever AI systems are being used, people ask the same questions that Stock’s committee did in 2009: Does it work? Is it fair? How will it affect my well-being, my safety, my livelihood, and my community?
To have informed debates about AI, society needs reliable information to help infer what goes on behind the screen—evidence that experts gather through a process called an AI audit. Such audits have now become common and sometimes even legally required in fields ranging from employment law to social media. Experienced AI auditors, like the authors of this book, are excited by the potential for audits to help leaders and the public know when to use (or not use) AI systems, improve public trust in AI, and guide improvements over time.
For most people, AI audits are new territory. Maybe an AI company has approached you with a system it is selling, or someone has complained about a product your own company has deployed. You might work at a nonprofit, be a journalist, or be a concerned member of the public who has heard about a proposed AI system and wants to under- stand and assess it. You may be an expert in your field, but AI may be new to you; you may never have commissioned or had to interpret an AI audit. This book is for you.
Rigged Reservations
While AI has captured the public imagination in the twenty-first century, the story of the first audits of computers started half a century earlier, amid debates over airline reservations.
Before the 1970s, reserving an airline ticket required hours of phone calls to book every passenger’s seat. To buy a ticket, passengers would visit a travel agency, where an agent had to call an airline booking center to learn if a seat was available. Travel agents regularly spent hours on the phone booking travelers’ tickets. At the other end of the line, airline staff would reserve seats by inserting push pins into a “seat board” and writing passenger information in a paper ledger. Computers changed everything about airline booking.
In 1960, the release of the groundbreaking SABRE system(Semi-Automated Business Research Environment) computerized airline reservations in the United States for the first time (figure 1). Now travel agents could use a SABRE computer terminal to check for flights with available seats and book a plane ticket in just a few minutes.
But along with these gains in efficiency, new problems emerged. Some airlines noticed the SABRE system was behaving strangely. Republic Airlines alleged that its flights mysteriously disappeared from search results. Braniff International Airways alleged that passengers who booked with SABRE were then unbooked and switched to American Airlines flights. Continental Airlines claimed to have lost up to $55 million a day in business because its flights were absent from search results. The first booking suggestions on SABRE’s screen were frequently on American Air- lines flights—the same airline that owned and operated the SABRE reservation system for the entire industry.
Republic, Braniff, and other American Airlines competitors did not know how the SABRE system worked, so they conducted an early AI audit. They compiled evidence of its biased behavior and pushed the US Civil Aeronautics Board to launch an investigation in 1983.8 But rather than defending the system as fair, when then-president of American Airlines Robert L. Crandall testified before the US Congress, he instead declared that SABRE favored American Airlines flights intentionally! Crandall said, “The preferential display of our flights, and the corresponding
increase in our market share, is the competitive raison d’etre for having created the system in the first place.” He wondered why anyone would expect American Airlines not to build a system that favored itself.
Later authors identified this as a more general perspective on computers and automation and named it after the aggrieved CEO, christening it “Crandall’s Complaint.” Stated in more general terms it is the objection: Why bother to build an expensive AI system if you can’t bias it in your favor? The implication of this perspective is: With- out regulation, wouldn’t we expect all systems to be biased in someone’s favor?
Crandall’s justification did not go over well. In response, the Civil Aeronautics Board banned systems like SABRE from preferentially treating specific airline carriers and even gave the problem a name: display bias. The board ruled that the flight display algorithms—the set of steps performed by the SABRE system—must be made public. Any interested person could mail the company a written re- quest and receive the exact criteria used to order SABRE’s flight search results. This marked the first time in history that a regulation explicitly referenced an algorithm.
A version of SABRE is still in use today, providing the booking system for over four hundred airlines. As we will see in this book, contemporary life is heavily influenced by digital technologies like SABRE that help us find information, apply for jobs, rent apartments, choose videos to watch, select skincare products, and so on. Increasingly, AI systems are also playing a role in public agencies; in social services, healthcare, housing, and the courts, AI systems are making decisions that can change people’s lives for better or for worse.
Unlike in the 1980s, however, people today cannot mail a request to the company and receive an explanation of the deciding details of these systems. The Civil Aeronautics Board no longer exists, and its display bias regulations have been repealed. And even if there were such regulations today in other industries (there are not), the development of more capable AI means that the crucial internal logic powering our search engines, job sites, social media platforms, chatbots, and social services would not fit in an envelope.
What can be done to identify, prevent, and respond to misbehaving AI systems? This book will introduce and explain AI auditing, a valuable approach to answering such questions.
Targeted Discrimination
In 2022, several decades after SABRE, the US Department of Justice (DOJ) found itself dealing with another case of alleged computer bias. The case involved the social media giant Meta (formerly Facebook), whose site at that time served nearly two billion users. Most people are familiar with Facebook as users of its social media platform, but its parent company Meta also operates an AI system that enables advertisers to place ads uniquely targeted to specific kinds of people that it thinks might be interested in those ads. Would-be advertisers can log in to a web page and purchase ads, uploading an ad image and selecting the type of audience they want it delivered to using categories like “animal lover” or “interested in beauty pageants.”
A few years earlier, in 2016, journalists at the non- profit media outlet ProPublica had published a bombshell investigation: Facebook’s interface allowed advertisers to target—and exclude—audiences of different “ethnic affinities,” such as “African American,” “Asian American,” and “Hispanic” (figure 2). This was true even for ads related to housing or employment, a startling revelation since US federal law explicitly prohibits racial discrimination in housing and employment advertisements. The Pro- Publica investigation kicked off a multiyear saga including litigation brought by Facebook users against the company, investigations by multiple state governments, and eventually a lawsuit filed by the DOJ.
Throughout the nearly seven years between the initial story and when the DOJ secured a settlement in 2022, Facebook repeatedly insisted the issue had been addressed, or that the allegations were flawed. But every time journalists and academic researchers audited the ad system, they
found new evidence demonstrating that the platform enabled discrimination in areas like housing, credit, and employment which are covered by civil rights laws.
The case finally ended with a ruling that Facebook would pay the maximum penalty under the Fair Housing Act (or Title VIII of the Civil Rights Act of 1968)—a pal- try sum of just over $115,000.14 But the settlement also required Facebook to remove the functionality that led to the issue and to undergo routine, independent audits to proactively ensure that illegal ad targeting was not occur- ring on the platform.
AI auditing is a powerful method for understanding AI systems wherever they affect people’s lives. Since SABRE, AI audits have helped competitors ensure a fair market- place, informed important public dialogue about lifesaving treatments, and alerted democratic institutions to AI systems that have crossed legal or ethical lines.
* * *
In all three cases discussed in this chapter, leaders and the public sought clarity about an opaque system and conducted research to study the problem. Sometimes the investigation was started by doctors and patient groups (as with kidney allocation), sometimes by competitors and the government (as in the SABRE case), and sometimes by journalists (as in the Facebook ads example). While the SABRE incident involves a firm deliberately placing a thumb on the scale of its computer systems, it is not uncommon for algorithm developers to be unaware of difficulties or the correct solution. In every case, audits revealed the real-world impacts of the AI system and prompted the search for improvements.
Excerpted from Auditing AI by the Marquand House Collective, published by MIT Press, ©2026 by the Marquand House Collective. All Rights Reserved.
