AI system failure and product liability: technical analysis for attorneys

As AI systems are deployed in safety-critical applications (autonomous vehicles, medical diagnosis, industrial control, financial risk management), the risk of system failure causing serious harm has grown. When failure results in injury or loss, technical analysis establishing causation, foreseeability, and the standard of care requires expertise spanning AI systems engineering and the specific application domain.

Categories of AI system failure

AI system failures can be categorized along several dimensions. The first is the failure mode: whether the failure is a performance failure (the system produced an incorrect output), a reliability failure (the system produced inconsistent outputs under similar conditions), a robustness failure (the system performed poorly under conditions outside its training distribution), or a safety failure (the system produced an output that caused harm).

The second dimension is the failure origin: whether the failure originated in the training data, the model architecture, the deployment environment, the human-machine interface, or some combination of these factors. Root cause analysis in AI system failure cases must trace the failure back to its origin in the system's design, development, or deployment.

The third dimension is foreseeability: whether the failure mode was known or reasonably foreseeable at the time of design and deployment. AI systems can exhibit documented failure modes, including distributional shift, adversarial inputs, and edge-case failures. A technical expert may analyze the available evidence concerning those risks, safeguards, testing, and deployment context; the court or factfinder determines any legal consequence.

Root cause analysis methodology

Root cause analysis for AI system failures follows a structured methodology that begins with characterizing the failure and works backward through the system to identify the contributing factors. The analysis typically begins with the system's outputs at the time of the failure: what did the system produce, and how did that output differ from what a correctly functioning system would have produced?

The next step is to identify the proximate cause of the incorrect output. For a machine learning model, this typically involves analyzing the model's input at the time of the failure, the model's internal state, and the relationship between the input and the output. This analysis may require access to the model's architecture, weights, and inference logs.

The final step is to trace the proximate cause back to its origin in the system's design or deployment. If the model produced an incorrect output because the input was outside its training distribution, the root cause may be in the training data selection or the deployment context. If the model produced an incorrect output because of a specific architectural limitation, the root cause may be in the model design.

Technical standards and the standard of care

The standard of care for AI system design and deployment is an evolving area of technical and legal analysis. Several standards bodies have published guidance on AI system development, including the National Institute of Standards and Technology's AI Risk Management Framework, the ISO/IEC 42001 standard for AI management systems, and industry-specific standards for AI in healthcare, automotive, and financial services.

Expert testimony on the standard of care in AI system failure cases typically requires the expert to identify the applicable standards, assess whether the defendant's practices met those standards, and explain how deviations from the standard of care contributed to the failure. This analysis requires both technical expertise in AI systems engineering and familiarity with the relevant standards and their application.

For cases involving AI in safety-critical applications, additional standards may apply. Autonomous vehicle AI systems are subject to standards from SAE International and the National Highway Traffic Safety Administration. Medical AI systems are subject to FDA guidance on software as a medical device. Industrial AI systems may be subject to IEC 61508 and related functional safety standards.

Autonomous vehicle cases

Autonomous-vehicle disputes are among the more developed settings in which software-assisted system failures have been litigated. They illustrate questions about vehicle design, software behavior, human operation, technical causation, and allocation of responsibility. They do not establish a uniform body of law for every AI product-liability matter.

Technical analysis in autonomous vehicle cases typically requires access to the vehicle's data recorder, the AI system's inference logs, and the sensor data from the time of the incident. This data can be used to reconstruct the AI system's decision-making process at the time of the incident and identify the specific failure mode that contributed to the collision.

Medical AI and healthcare liability

AI systems are increasingly used in clinical decision support, diagnostic imaging, and treatment planning. When these systems produce incorrect outputs that contribute to patient harm, the liability analysis involves both the technical analysis of the AI system failure and the medical malpractice framework.

Expert testimony in medical AI cases typically requires experts with both AI technical expertise and clinical domain expertise. The technical expert addresses the AI system's failure mode and its contribution to the incorrect clinical output. The clinical expert addresses the standard of care for the clinical decision and how the AI system's output affected the clinician's decision-making.