Digital Library

Auditing AI ( etc.)(Z-Library)

,

Share E-Book

Auditing AI ( etc.)(Z-Library)

Author

,

Rating 3.5 AI initial rating

Log in to rate

AI
Language English

How tech companies, journalists, and policymakers can prevent AI decision-making from going wrong.

Format PDF
Size 2.5 MB
(First 20 pages)

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

Page 1
(This page has no text content)
Page 2
Auditing AI
Page 3
The MIT Press Essential Knowledge Series A complete list of books in this series can be found online at https://mitpress.mit.edu/books/series/mit-press-essential-knowledge-series.
Page 4
Auditing AI The Marquand House Collective: Marc Aidinoff Lena Armstrong Esha Bhandari Ellery Roberts Biddle Motahhare Eslami Karrie Karahalios J. Nathan Matias Danaé Metaxa Alondra Nelson Christian Sandvig Kristen Vaccaro The MIT Press | Cambridge, Massachusetts | London, England
Page 5
The MIT Press Massachusetts Institute of Technology 77 Massachusetts Avenue Cambridge, MA 02139 mitpress.mit.edu © 2026 Massachusetts Institute of Technology This work is subject to a Creative Commons CC BY-NC-ND license. Subject to such license, all rights are reserved. The MIT Press would like to thank the anonymous peer reviewers who provided comments on drafts of this book. The generous work of academic experts is essential for establishing the authority and quality of our publications. We acknowledge with gratitude the contributions of these otherwise uncredited readers. This book was set in Chaparral Pro by New Best-set Typesetters Ltd. Library of Congress Cataloging-in-Publication Data is available. ISBN: 978-0-262-05172-9 10 9 8 7 6 5 4 3 2 1 EU Authorised Representative: Easy Access System Europe, Mustamäe tee 50, 10621 Tallinn, Estonia | Email: gpsr.requests@easproject.com d_r0
Page 6
To Blake L., the standard poodle
Page 7
Contents Series Foreword 1 Introduction 2 What Is AI Auditing? 3 The Steps of an AI Audit 4 Interpreting Audit Results 5 After the Audit 6 A Healthy Audit Ecosystem Acknowledgments Glossary Notes Further Reading Index
Page 8
Series Foreword The MIT Press Essential Knowledge series offers accessible, concise, beautifully produced pocket-size books on topics of current interest. Written by leading thinkers, the books in this series deliver expert overviews of subjects that range from the cultural and the historical to the scientific and the technical. In today’s era of instant information gratification, we have ready access to opinions, rationalizations, and superficial descriptions. Much harder to come by is the foundational knowledge that informs a principled understanding of the world. Essential Knowledge books fill that need. Synthesizing specialized subject matter for nonspecialists and engaging critical topics through fundamentals, each of these compact volumes offers readers a point of access to complex ideas.
Page 9
1 Introduction Transplant surgeon Peter Stock probably never expected to face a crowd of two hundred people arguing about whether or not a computer system could help save lives. When Stock finished his training in the 1980s, the idea of using computers to guide life-and-death decisions was the stuff of science fiction. But two decades later, when Stock found himself in a St. Louis conference room discussing organ allocation in the United States, the question facing him was not whether computers would make these decisions —it was how. As chair of the US Kidney Transplantation Committee, Stock knew how much the possibility of a kidney transplant meant to every person with damaged kidneys. Kidneys filter toxins from the human bloodstream, and kidney failure can kill someone within weeks or even days. Roughly one in seven Americans, 37 million people, have some form of chronic kidney disease.1 More than 100,000 people are on the kidney transplant waiting list each year, but only some 21,000 kidneys are available for transplant.2 The waiting list gets longer each year.3 In 1984, the US Congress established the Organ Procurement and Transplantation Network to find a way to direct donated kidneys to people who need them most. At first, doctors sent kidneys to the nearest person in need. Then, as kidney transplants became safer and more routine, groups of patients, doctors, and politicians called for a fairer system. By the early 2000s, statisticians had developed an automated transplant prioritization system called LYFT that predicted how much longer a person could live if they received a donor kidney (LYFT an acronym for “life years from transplant”). But not everyone agreed with this system. Elderly patients alleged that LYFT put them at a disadvantage by prioritizing younger patients.4 Communities of color and those facing poverty argued that it prioritized wealthier, more privileged patients. Everyone wanted to know how LYFT actually worked.5
Page 10
As chair of the Organ Procurement and Transplantation Network’s Kidney Transplantation Committee, Stock knew he needed to come to the debate equipped with both data and compassion. Stock’s committee opened the meeting in St. Louis by presenting a series of evaluations of the LYFT system. The committee hoped these independent system analyses could quell people’s concerns and inform a search for something better.6 The evaluations Stock’s group presented—audits of the LYFT system—sparked a series of analyses that identified disparities across patient populations, measuring differences based on age, gender, geography and race. This work informed decisions that would help bring greater equity to the use of this complex system for years to come. Decision-making systems like LYFT represent a key component of what we now call artificial intelligence (AI). From the algorithms that direct lifesaving treatment to the software that filters our emails for spam, AI systems are now powering—and changing—countless aspects of our lives. Wherever AI systems are being used, people ask the same questions that Stock’s committee did in 2009: Does it work? Is it fair? How will it affect my well-being, my safety, my livelihood, and my community? To have informed debates about AI, society needs reliable information to help infer what goes on behind the screen—evidence that experts gather through a process called an AI audit. Such audits have now become common and sometimes even legally required in fields ranging from employment law to social media. Experienced AI auditors, like the authors of this book, are excited by the potential for audits to help leaders and the public know when to use (or not use) AI systems, improve public trust in AI, and guide improvements over time. For most people, AI audits are new territory. Maybe an AI company has approached you with a system it is selling, or someone has complained about a product your own company has deployed. You might work at a nonprofit, be a journalist, or be a concerned member of the public who has heard about a proposed AI system and wants to understand and assess it. You may be an expert in your field, but AI may be new to you; you may never have commissioned or had to interpret an AI audit. This book is for you. Rigged Reservations
Page 11
While AI has captured the public imagination in the twenty-first century, the story of the first audits of computers started half a century earlier, amid debates over airline reservations. Before the 1970s, reserving an airline ticket required hours of phone calls to book every passenger’s seat. To buy a ticket, passengers would visit a travel agency, where an agent had to call an airline booking center to learn if a seat was available. Travel agents regularly spent hours on the phone booking travelers’ tickets. At the other end of the line, airline staff would reserve seats by inserting push pins into a “seat board” and writing passenger information in a paper ledger. Computers changed everything about airline booking. In 1960, the release of the groundbreaking SABRE system (Semi-Automated Business Research Environment) computerized airline reservations in the United States for the first time (figure 1). Now travel agents could use a SABRE computer terminal to check for flights with available seats and book a plane ticket in just a few minutes. But along with these gains in efficiency, new problems emerged. Some airlines noticed the SABRE system was behaving strangely. Republic Airlines alleged that its flights mysteriously disappeared from search results. Braniff International Airways alleged that passengers who booked with SABRE were then unbooked and switched to American Airlines flights. Continental Airlines claimed to have lost up to $55 million a day in business because its flights were absent from search results. The first booking suggestions on SABRE’s screen were frequently on American Airlines flights—the same airline that owned and operated the SABRE reservation system for the entire industry.7 Republic, Braniff, and other American Airlines competitors did not know how the SABRE system worked, so they conducted an early AI audit. They compiled evidence of its biased behavior and pushed the US Civil Aeronautics Board to launch an investigation in 1983.8 But rather than defending the system as fair, when then-president of American Airlines Robert L. Crandall testified before the US Congress, he instead declared that SABRE favored American Airlines flights intentionally! Crandall said, “The preferential display of our flights, and the corresponding increase in our market share, is the competitive raison d’etre for having created the system
Page 12
in the first place.”9 He wondered why anyone would expect American Airlines not to build a system that favored itself.
Page 13
Figure 1 This 1961 advertisement highlights the partnership between IBM and American Airlines to produce a computerized airline reservation system,
Page 14
a milestone in the history of computing. American’s competitors found the order of its search results suspicious, claiming IBM’s system unfairly favored American’s flights. Later authors identified this as a more general perspective on computers and automation and named it after the aggrieved CEO, christening it “Crandall’s Complaint.” Stated in more general terms it is the objection: Why bother to build an expensive AI system if you can’t bias it in your favor? The implication of this perspective is: Without regulation, wouldn’t we expect all systems to be biased in someone’s favor?10 Crandall’s justification did not go over well. In response, the Civil Aeronautics Board banned systems like SABRE from preferentially treating specific airline carriers and even gave the problem a name: display bias. The board ruled that the flight display algorithms—the set of steps performed by the SABRE system—must be made public. Any interested person could mail the company a written request and receive the exact criteria used to order SABRE’s flight search results. This marked the first time in history that a regulation explicitly referenced an algorithm. A version of SABRE is still in use today, providing the booking system for over four hundred airlines. As we will see in this book, contemporary life is heavily influenced by digital technologies like SABRE that help us find information, apply for jobs, rent apartments, choose videos to watch, select skincare products, and so on. Increasingly, AI systems are also playing a role in public agencies; in social services, healthcare, housing, and the courts, AI systems are making decisions that can change people’s lives for better or for worse. The CEO wondered why anyone would expect American Airlines not to build a system that favored itself. Unlike in the 1980s, however, people today cannot mail a request to the company and receive an explanation of the deciding details of these systems. The Civil Aeronautics Board no longer exists, and its display bias regulations have been repealed.11 And even if there were such regulations today in other industries (there are not), the development of more capable AI means that the crucial internal logic powering our search engines, job sites,
Page 15
social media platforms, chatbots, and social services would not fit in an envelope. What can be done to identify, prevent, and respond to misbehaving AI systems? This book will introduce and explain AI auditing, a valuable approach to answering such questions. Targeted Discrimination In 2022, several decades after SABRE, the US Department of Justice (DOJ) found itself dealing with another case of alleged computer bias. The case involved the social media giant Meta (formerly Facebook), whose site at that time served nearly two billion users.12 Most people are familiar with Facebook as users of its social media platform, but its parent company Meta also operates an AI system that enables advertisers to place ads uniquely targeted to specific kinds of people that it thinks might be interested in those ads. Would-be advertisers can log in to a web page and purchase ads, uploading an ad image and selecting the type of audience they want it delivered to using categories like “animal lover” or “interested in beauty pageants.” A few years earlier, in 2016, journalists at the nonprofit media outlet ProPublica had published a bombshell investigation: Facebook’s interface allowed advertisers to target—and exclude—audiences of different “ethnic affinities,” such as “African American,” “Asian American,” and “Hispanic” (figure 2).13 This was true even for ads related to housing or employment, a startling revelation since US federal law explicitly prohibits racial discrimination in housing and employment advertisements. The ProPublica investigation kicked off a multiyear saga including litigation brought by Facebook users against the company, investigations by multiple state governments, and eventually a lawsuit filed by the DOJ. Throughout the nearly seven years between the initial story and when the DOJ secured a settlement in 2022, Facebook repeatedly insisted the issue had been addressed, or that the allegations were flawed. But every time journalists and academic researchers audited the ad system, they found new evidence demonstrating that the platform enabled discrimination in areas like housing, credit, and employment which are covered by civil rights laws.
Page 16
Figure 2 The Facebook Ads Manager interface once included a feature allowing advertisers to exclude people from seeing housing and employment ads based on “multicultural affinity.” The case finally ended with a ruling that Facebook would pay the maximum penalty under the Fair Housing Act (or Title VIII of the Civil Rights Act of 1968)—a paltry sum of just over $115,000.14 But the settlement also required Facebook to remove the functionality that led to the issue and to undergo routine, independent audits to proactively ensure that illegal ad targeting was not occurring on the platform. AI auditing is a powerful method for understanding AI systems wherever they affect people’s lives. Since SABRE, AI audits have helped competitors ensure a fair marketplace, informed important public dialogue about lifesaving treatments, and alerted democratic institutions to AI systems that have crossed legal or ethical lines.
Page 17
In all three cases discussed in this chapter, leaders and the public sought clarity about an opaque system and conducted research to study the problem. Sometimes the investigation was started by doctors and patient groups (as with kidney allocation), sometimes by competitors and the government (as in the SABRE case), and sometimes by journalists (as in the Facebook ads example). While the SABRE incident involves a firm deliberately placing a thumb on the scale of its computer systems, it is not uncommon for algorithm developers to be unaware of difficulties or the correct solution. In every case, audits revealed the real-world impacts of the AI system and prompted the search for improvements.
Page 18
2 What Is AI Auditing? Before discussing AI audits any further, we must first define what we mean by “AI systems.” Algorithms—computational processes formalized in code, often but not always driven by data—are ubiquitous today at the operational core of AI systems. All automated systems in which algorithms are integrated can be termed AI systems. Although there are narrower definitions, we use this one because the accepted use of the term “AI” has grown impossibly broad. We use such systems in consumer settings (booking flights, targeting advertisements), educational ones (teaching and assessing students), and even in making life-or-death decisions such as prioritizing patients receiving medical care or helping judges make criminal sentencing decisions. Why AI? If it seems to you that every digital service these days brands itself as AI, you are not alone. Industry leaders promise that, through automation, AI will improve efficiency and productivity, enhance decision- making, and augment human capabilities. Over the last decade, technological developments have taken AI from a niche academic field to a widely applicable foundation that can be deployed for seemingly endless uses. For all their purported promise, however, many AI systems have significant—and often silent—flaws. Whether due to unintended biases or outright malicious intent, they are far from infallible. We define an AI audit as a rigorous and systematic evaluation of the core claims, expectations, or standards pertaining to an AI system.1 Those standards may be based on what the law requires. The Fair Housing Act, for instance, was the law that Facebook violated when it allowed advertisers to exclude people, based on race, from seeing apartment rental ads. In other cases, we expect a system’s standards to adhere to widely held social norms. Social media companies are notorious for inferring information about us based on our browsing habits, a practice that, for many users, violates norms and expectations surrounding privacy, even though it is not illegal. The deft auditor begins with the mindset of a skeptic, looking for evidence to prove
Page 19
that the system in question does not meet a given standard, including for accuracy, compliance with laws, and identification of potential risks. The above definition is intended to capture the gravitational center of AI audits, describing the common thread found in most (but not all) strong AI auditing work. In the next chapter, we will explore the steps of a classic AI audit in more detail and discuss variations on that form. The deft auditor begins with the mindset of a skeptic. Auditing: From Taxes and Landlords to AI Readers will likely be most familiar with use of the word “audit” in financial settings. In the United States, individuals and small businesses can hire accountants to audit their finances. The Internal Revenue Service (IRS)—a government agency—also carries out audits. The latter is governed by tax code and has a more formal structure, from clear goals to potential financial penalties. Audits (minus the “AI”) are rigorous and systematic evaluations of an entity’s or actor’s claims, expectations, or standards. They are used across the private sector to assess whether a company adheres to laws and regulations governing its industry. US government entities, from the Food and Drug Administration to the Environmental Protection Agency to the Securities and Exchange Commission, encourage and sometimes require companies to undergo third-party audits to assess the effects of their business practices on public health, consumer interests, and the environment. But other kinds of audits can affect the public interest and how private companies do business. In the early twentieth century, auditing emerged as a response to the problem of “redlining,” specifically the practice of landlords and lenders who excluded Black people from certain housing and neighborhoods. Communities affected by redlining decided they needed a way to convince others that this was a pervasive problem. They needed evidence that was compelling and easy to understand, so local groups of tenants and homebuyers concerned about racial discrimination in housing banded together to establish themselves as fair housing committees. The committees would send two people, a Black person and a White person, to ask a real estate agent or a landlord about apartments and houses
Page 20
on the market. They would do this test at about the same time, with both people recording how they were treated and what properties they were shown. If the landlord invited the White tester to tour an apartment but told the Black tester that no units were available, the committee would take note.2 Perhaps inspired by tax audits, they called each series of tests an audit study. After 1960s Civil Rights–era laws made housing discrimination illegal, these audit studies became a way to enforce antisegregation laws, as these audits were “designed to be so simple that even a lawyer or judge can understand them.”3 The basic structure of these tests was then adapted to other situations, like the job market. In another series of studies, a man and a woman tester with the same qualifications would apply for the same job at the same time.4 These studies demonstrated pervasive discrimination in hiring (see figure 3). The structure of these audit studies is also used in today’s AI audits. Social science and activist audit studies systematically provide different inputs into a system (e.g., the tester’s race) and examine the outputs (e.g., the landlord’s reply) to understand whether the system is performing well against a given standard (e.g., the Fair Housing Act’s nondiscrimination rules). After decades of these audits, investigators began sending documents, rather than trained agents, to apply for jobs or housing. In one memorable audit (titled, “Are Emily and Greg more employable than Lakisha and Jamal?”), auditors sent nearly 5,000 fictional resumes to over 1,300 job postings, eventually finding that resumes whose names suggested the applicant was White received 50 percent more interview callbacks than those whose names suggested the applicant was Black.5 These audits were used for two different purposes. Sometimes, the government, nonprofits, or academics used large-scale audits to systematically collect evidence to draw statistical conclusions about a social problem (i.e., the prevalence of housing discrimination). Other audits were conceived by prosecutors and tenants’ rights organizations to point the finger at specific landlords or hiring managers. In both cases, though, the best audits drew conclusions about patterns of behavior, not singular or specific decisions.
The above is a preview of the first 20 pages. Register to read the complete e-book.

Recommended for You

Loading recommended books...
Failed to load, please try again later

Tip the Site

Scan the WeChat Pay or Alipay code to tip. No login required.

WeChat Pay
Alipay
← Back to List