Algorithmic decision systems are now used to make or inform consequential decisions across employment, lending, housing, insurance, and criminal justice. When those decisions produce discriminatory outcomes, the technical analysis required to identify and quantify the discrimination is substantially more complex than the statistical analysis used in traditional disparate impact cases.
How Algorithmic Bias Arises
Algorithmic bias can arise at multiple points in the design and deployment of an AI decision system. The most commonly discussed source is biased training data: if the data used to train a model reflects historical patterns of discrimination, the model may learn to replicate those patterns. A hiring model trained on historical hiring decisions made by biased human reviewers may learn to favor candidates who resemble those previously hired, perpetuating the original bias.
Bias can also arise from feature selection. If a model uses features that are correlated with protected characteristics, such as zip code as a proxy for race or certain educational credentials as a proxy for socioeconomic background, the model may produce discriminatory outcomes even if the protected characteristic itself is not included as an input. This form of bias is sometimes called proxy discrimination.
A third source of bias is validation methodology. A model may perform well on aggregate accuracy metrics while performing significantly worse for specific demographic subgroups. If the model was validated only on aggregate metrics, the subgroup performance disparities may not have been identified before deployment.
Disparate Impact Analysis for Algorithmic Systems
Disparate impact analysis in algorithmic discrimination cases follows the same basic framework as traditional disparate impact analysis: the plaintiff must show that a facially neutral practice produces a statistically significant adverse effect on a protected class, and the defendant must then demonstrate that the practice is justified by business necessity. However, applying this framework to algorithmic systems requires addressing several technical complications.
The first complication is defining the relevant comparison group. In a traditional hiring discrimination case, the comparison is typically between the selection rate for the protected class and the selection rate for the majority group. For algorithmic systems, the comparison may need to account for the fact that the algorithm's inputs vary across applicants, and that the algorithm may treat similarly situated applicants differently based on features that are correlated with protected characteristics.
The second complication is the four-fifths rule. The EEOC's Uniform Guidelines on Employee Selection Procedures use a four-fifths rule as a rule of thumb for identifying adverse impact: if the selection rate for a protected group is less than four-fifths of the selection rate for the highest-selected group, adverse impact is presumed. This rule was designed for relatively simple selection procedures and may not translate directly to complex algorithmic systems where the "selection rate" is not a simple binary outcome.
Key Technical Questions in Algorithmic Bias Litigation
- What data was used to train the model, and does it reflect historical discrimination?
- What features does the model use, and are any of them proxies for protected characteristics?
- How was the model validated, and were subgroup performance metrics examined?
- What is the model's false positive and false negative rate for different demographic groups?
- Has the model been audited for bias, and by whom?
- What counterfactual analysis has been conducted to assess the effect of removing or modifying biased features?
Validation Methodology Critique
One of the most productive lines of technical analysis in algorithmic bias cases is critique of the validation methodology used by the defendant. Validation is the process by which a model developer assesses whether the model performs as intended. Inadequate validation is both a technical failure and a legal vulnerability: a model that was not adequately validated for discriminatory impact before deployment is harder to defend as a business necessity.
Technical experts can evaluate validation methodology on several dimensions. Was the validation dataset representative of the population to which the model would be applied? Were subgroup performance metrics calculated and reviewed? Were the validation metrics appropriate for the specific application, or did they measure aggregate performance in a way that could mask subgroup disparities? Was the validation conducted by the model developer or by an independent party?
In lending cases, the Equal Credit Opportunity Act and the Fair Housing Act impose specific requirements on the validation of credit scoring models. Technical experts in these cases must be familiar with the regulatory framework as well as the statistical methods used to assess model fairness.
Counterfactual and Causal Analysis
Traditional disparate impact analysis is correlational: it identifies statistical associations between protected characteristics and adverse outcomes. More sophisticated algorithmic bias analysis uses counterfactual and causal methods to assess whether the protected characteristic is causally related to the adverse outcome, even when it is not directly included in the model.
Counterfactual analysis asks: what would the model's output have been if the applicant's protected characteristic had been different, holding all other features constant? If changing the protected characteristic changes the model's output, this is evidence that the model is using the protected characteristic, either directly or through a proxy, in its decision.
Causal analysis uses techniques from causal inference to identify the pathways through which protected characteristics affect model outputs. This analysis is more complex than simple counterfactual analysis and requires careful attention to the causal structure of the features used by the model.
Expert Testimony in Algorithmic Bias Cases
Expert testimony in algorithmic bias cases typically requires expertise in both statistics and machine learning. A statistician without machine learning expertise may not be able to address the specific technical characteristics of the AI system at issue. A machine learning engineer without statistical expertise may not be able to conduct the rigorous disparate impact analysis that the legal framework requires.
Experts in these cases should be prepared to explain complex technical concepts to lay factfinders in accessible terms, while maintaining the technical rigor required to withstand cross-examination. The ability to translate between the technical language of machine learning and the legal language of discrimination law is a specific skill that not all technical experts possess.
AI Expert Witness Services provides algorithmic bias analysis and expert testimony for attorneys handling employment, lending, and consumer protection cases involving AI decision systems.
Algorithm Bias Expert Services