The “Black Box” Problem at 35,000 Feet and Beyond: Who is Liable When an AI Pilot Makes a Catastrophic Decision?

By Bhavya Rai (Student, Rajiv Gandhi National Law University)

The accident inquiry into Air Lanka Flight 602 was like nothing before or since. The black  boxes came through intact, indicating the aircraft was performing flawlessly and responding  perfectly to every command. The conversations on the voice recorder captured the hum of the  plane’s systems and the two pilots’ confusion, “Why’s it doing that?” Then, in the final  moments, the First Officer begged the plane, “Override! Override!” There was nobody to  countermand. The self-fly system had made the perfectly rational, albeit fatally erroneous,  decision. Its logic was trapped within the black box of an algorithmic crypt that surrendered its  secrets to none. This is not science fiction. While Airbus and Boeing are investing heavily in  high-end autonomy and the Indian GAGAN-based positioning system is slowly incorporating  AI, the age of AI as the primary decision-maker in the cockpit is already in sight. Yet our  laws are still stuck in the age of “mechanical breakdowns and human mistakes.”  

The problem at hand then is: When an “opaque AI decision” results in a catastrophic event, who  or what should be held liable?  

The Double Black Box: A Forensic Dead End 

The findings of aircraft investigations rely on the principle of causality. The “black box,” the  physical Flight Data Recorder (FDR), essentially informs us of what happened, namely the  movements of the control surfaces and the failure of the engines. However, the “black box” of  modern AI systems refers to the metaphorical black box, which consists of the impenetrable  neural network. Deep reinforcement learning, for example, relies on millions of weighted  connections to make decisions that even the programmers cannot easily identify. They are  probabilistic systems, not deterministic.  

This is a forensic dead-end. So, what we know after a crash is that our AI made a 90-degree  bank, then a low altitude. But how can we determine why? Was this a sensor misunderstanding,  a training data corruption, environmental interaction, or a hidden bug in the programming? This  is why, without knowing why, determining legal causation, a direct link between defect and  injury, is made impossible. Our most basic legal need to establish liability is thereby  undermined. 

The Collapsing Pillars of Traditional Liability

Current aviation liability rests on three pillars, all crumbling before autonomous AI:  

1. Product Liability (Against the Manufacturer):  

To sue the maker of the AI, you have to prove a “defect in design or manufacture.” How do you  evidence a defect in a system whose decision-making pathway is opaque? The performance of  the AI is emergent, not explicitly programmed. Its “defect” may be a gap in its training data - a scenario it never encountered over billions of simulations. Is that a manufacturing defect, or  an inherent limitation of machine learning? Courts have limited jurisprudential guidance on liability frameworks for opaque, self-learning AI systems.

2. Vicarious Liability of the Operator (Respondeat Superior):  

“Airlines can be held responsible for the wrongdoing of their employees.” Is an “AI” an  “employee” of sorts? Is an “algorithm” perpetrating a “wrongful act” akin to “intent” and/or  “negligence” implied within it? This 500-year-old legal doctrine presumes the human principal  and the human agent.  

3. Negligence of the Human “Monitor”:  

The current model assumes a human pilot monitors and can override the AI. This is a legal  fiction that will fail. In a crisis, an AI may act in milliseconds, based on data streams a human  cannot process. Holding a pilot liable for failing to comprehend and override an inexplicable AI decision is both unjust and a poor deterrent. It punishes the last human in the chain for a systemic failure, contrary to foundational negligence doctrines of foreseeability, reasonable standard of care, and meaningful control, which predicate liability on a defendant’s capacity to anticipate and prevent the harm.

Charting a new Legal Flight Path: Three Possible Models

We cannot wait for the first catastrophic test case. Regulators must proactively choose a liability  paradigm. Currently, the model still relies on a human pilot to monitor and override the AI.  This represents a legal fiction that is bound to fail. An AI may, in a matter of milliseconds,  make a decision based on data streams that a human will not be able to process. As discussed above, the assumption that a pilot can meaningfully comprehend and override an opaque AI decision is increasingly untenable, making traditional fault-based attribution to the pilot both doctrinally strained and normatively misplaced. Regulators must proactively choose a liability paradigm.  

Model A: The Enterprise Liability Framework (The “Ecosystem” Approach)  This approach renounces the idea of a single causal fault. It considers the AI flight system as a  joint enterprise, which involves the manufacturer, the software developer, the operator, and the 

certification authority. In case of an accident, all parties are jointly and severally liable to the  victims. Liability is strict and automatic. It is the enterprise, which is the guilty party, that must  find out internally the one to blame, and improve the systems. This model primarily aims at  victim compensation and at industry-wide sharing of safety incentives.  

Model B: The Mandatory Explainability Mandate (The “Transparency” Fix) This  concept puts Explainable AI (XAI) at the center of the condition for certification in a  nonnegotiable manner. According to the regulations, it would be a legal requirement for the  decision-making of any safety-critical AI to always be open and interpretable by human  investigators. If the “reasoning” of the AI “cannot be made available” after the incident, the  party responsible will be liable automatically and severely. Hence, Research & Development  would be rerouted away from relying solely on performance to audit and transparent systems.  This way, the traditional causal link between a defect and harm would be preserved.  

Model C: The No-Fault Compensation Fund (The “Social Insurance” Model) Taking inspiration from nuclear accident and vaccine injury compensation frameworks, this model eliminates fault-finding altogether. Under this approach, a compulsory industry-funded contribution scheme would be established, creating a dedicated compensation fund for victims of AI-related aviation accidents, irrespective of fault or causation. Contributions could be calibrated according to autonomous flight hours or system deployment metrics. Claims would be processed administratively rather than through adversarial judicial proceedings. This structure ensures swift compensation, avoids protracted litigation over opaque and technically complex causation disputes, and allows innovation to continue while distributing the distinctive risks of autonomous systems across the industry as a whole.

Among the three, the most practical near-term model appears to be a calibrated version of Model C, supplemented by regulatory oversight mechanisms that preserve safety incentives. Autonomous AI systems introduce epistemic uncertainty that traditional fault doctrines are ill-equipped to manage. A no-fault compensation structure recognises this reality while ensuring predictable and prompt victim compensation. To mitigate moral hazard, contribution rates could be risk-weighted, and regulatory penalties could apply independently for safety violations.

In contrast, Model A may overburden courts with contribution disputes, and Model B may outpace current technological feasibility. Model C, though imperfect, offers administrative clarity, compensatory efficiency, and adaptability in an evolving technological landscape.

In the transition toward AI-dominant aviation, the law’s first obligation is not to perfect doctrinal purity but to ensure compensation, maintain incentives for safety, and provide regulatory certainty. A structured social insurance model, combined with strong oversight, best reconciles these competing imperatives.

Conclusion: Program Liability into the Code

The issue is not purely technological or legal in nature; it has to do with the constitutional  dimension. What is involved is the right to remedy, the principles of justice, and the social  contract that surrounds transformative technologies. The “black box” problem is a legal ticking  time bomb.  

For the case of India, where the aviation industry is expanding so vigorously, and the space  program has ambitious aspirations, the stakes are extremely high. While we begin to 

incorporate AI into Gaganyaan control systems and new UDAN airplanes, the DGCA and ISRO need to take the lead in this legal development. The challenge we face is not whether we will  adopt an approach, but which approach we will embed into our legal systems.  

Consequently, a resolute regulatory decision, either enterprise liability, explicability mandates,  or a shared fund, is the determining factor for the future timeline. This cannot be regarded as a  simple technical modification, but rather as a critical decision that will either insert fairness  into the autonomous system's code or be content with an era where the loudest mistakes go  unpunished. It is imperative that we create this new legal structure now, during a peaceful and  clear-sky era, rather than in the noisy and chaotic aftermath of a tragedy that could have been  averted.