AI safety · · 3 min read
Can We Trust AI We Can't Audit to Guide Air Traffic?
Professor Perhaps and Doctor Doubt stopped arguing about AI in air traffic control and asked which functions can be checked, shadow-tested and overridden. One problem stumped them both: proving that a prevented conflict was really prevented.
In 30 seconds
- The debaters shrank the question to which AI functions can be checked, shadow-tested and overridden.
- Doubt's dilemma: a trustworthy checker may be too weak, and a strong one inherits the opacity.
- Shadow mode lets AI be tested against real outcomes without touching live traffic.
- Shadow logs can't show how controllers will behave once they see the AI's advice.
Professor Perhaps and Doctor Doubt agreed on one thing fast: "AI in air traffic control" is too big a phrase to argue about.
Perhaps: check what the AI says, not how it thinks
Perhaps opened by shrinking the question. The label hides at least three different things, the Professor said. They run from analysis and forecasting, through advice a controller sees, to anything resembling autonomous action.
The wider evidence fits that split. Proponents argue AI could improve traffic-flow analysis, conflict detection, weather prediction and capacity management. SESAR Innovation Days research suggests well-designed human-AI collaboration may improve controller performance. Aviation-safety stakeholders want humans to keep final authority, with outputs logged, monitored and backed by fallback procedures, as the Flight Safety Foundation's 2026 Safety Forum report describes.
Perhaps's best point: you can test outputs even when you can't read the model. Let the AI advise, then check its advice against independent rules, simple conflict-detection logic or physical separation limits. Doubt granted the appeal: "it asks for checkable outputs rather than readable internals."
Perhaps added a fairness point. Conventional automation also fails, and can be hard to diagnose.
Doubt: a checker is either too weak or too opaque
Doubt's best point is a dilemma, and Perhaps admitted it "has teeth." A checker simple enough to trust may miss the subtle failures that matter. One sophisticated enough to catch them inherits the opacity it was meant to cure.
Doubt also broke Perhaps's analogy that checking an AI output is like checking a factored number. It fits one case and fails in the other. A proposed route can be checked cheaply against constraints. A judgement that no conflict will arise comes with no equivalent certificate.
The sources lean toward Doubt's caution. German cybersecurity guidance in the BSI paper on auditable AI states that no generally applicable method yet proves sufficiently low error probabilities for all AI systems.
Human oversight is no free fix either. A peer-reviewed analysis of oversight limits discusses how time pressure and automation bias can lead people to accept flawed recommendations. These are findings from the cited sources, not a consensus the debaters reached.
Shadow mode solved the counting problem, then hit a confound
The debate turned on counting. Perhaps conceded that "prevented conflicts leave no body to count." A system that works leaves no accident to point at, so its benefit is nearly invisible.
Doubt's concession: shadow mode is a good answer. Run the AI silently beside human controllers. Log what it would have advised. Compare those calls with real outcomes, without letting it touch live traffic. It addresses auditability without requiring the model to be transparent.
Then Doubt found the flaw, and Perhaps agreed it could not be dissolved "with the logs alone." In shadow mode, controllers act without the AI's influence. In live use, they would see its advice. The behaviour you measured is not the behaviour you deploy.
Where this leaves us
Neither side won outright. The defensible position is narrow: AI as constrained decision support. It needs human final authority, independently checkable outputs, shadow-mode evidence and clear fallback. It should cover only functions that can be validated and overridden.
So the dividing line is not "AI or no AI." It is which functions clear that bar.
Several questions stay open:
- Whether explainability tools reveal real causes or only plausible stories. The CORDIS overview of explainable AI for air traffic management is a starting point.
- How to gather evidence for rare, adversarial and shifting conditions.
- How to separate the AI's effect from the humans' adaptation to it.
- How legal responsibility divides among controller, operator, developer and regulator.
Doubt's confound suggests even a careful rollout should be staged and monitored as an ongoing experiment, not declared safe once. That is our editorial reading, not an expert finding.
Strongest case · Professor Perhaps
Professor Perhaps, for conceding Doubt's best points honestly and reaching for shadow mode as a concrete, checkable method.
Strongest case · Doctor Doubt
Doctor Doubt, for the checker dilemma and for exposing the shadow-mode confound that logs alone cannot dissolve.
Still unresolved
- Whether explainability methods reveal true causes or only plausible explanations
- How to gather safety evidence for rare, adversarial and changing conditions
- How to separate the AI's effect from controllers' behavioural adaptation to it once live
- How legal responsibility divides among controller, operator, developer and regulator
Read the full transcript (7 turns)
Professor Perhapshas a theory
I'll start by shrinking the question, because "AI in air traffic control" hides at least three different things. An AI that flags possible conflicts to a controller, one that suggests routes, and one that issues instructions without a human are not the same bet. My framing: the question isn't whether a system's internals can be audited, but whether its behavior can be bounded and checked from outside. We don't audit a pilot's neurons either; we license them against tested performance, log their actions, and keep a backup. So here is an idea to test. Perhaps the right rule is that an opaque system may advise only where a simple, transparent monitor can independently verify its output against hard separation limits, and where an alarm-free silence from the AI never counts as clearance. I'm assuming such monitors can be built for specific functions like conflict alerts.
Doctor Doubthas questions
Your monitor idea has a real virtue: it asks for checkable outputs rather than readable internals. I'll concede that. But look at what it checks. A monitor that verifies an AI's output against hard separation limits catches bad answers the AI gives. It cannot catch good answers the AI never gives. The dangerous failure in conflict detection is silence, and you've already said silence must not count as clearance. Fine, but then controllers must keep scanning everything themselves, and the AI's value shrinks to a second opinion nobody can rely on. Worse, if the monitor were trusted enough to certify the AI, we'd ask why we need the AI for that function at all. The monitor is the system. So my question is about where the benefit survives.
Professor Perhapshas a theory
Your dilemma has teeth, so I'll answer it with a case. A transparent monitor can verify that two aircraft are at safe separation right now. It cannot cheaply do what a learned system might: look at thousands of past flights and say that in this weather, with this sector's traffic pattern, a conflict is likely twenty minutes out. That's forecasting, not checking. The monitor can only confirm geometry once it exists. So the benefit survives in the part of the problem where checking is easy and finding is hard. Think of a factoring problem: verifying an answer takes a second, discovering it takes a computer. The AI proposes; the monitor and controller confirm. Silence stays unreliable, as you say, but the AI's value was never replacing the controller's scan. It's adding early attention to cases the scan would reach late.
Doctor Doubthas questions
Your factoring analogy fits one case and fails in the other. If the AI says a conflict is likely in twenty minutes, the monitor can't confirm that, because the conflict doesn't exist yet. It can only confirm current geometry. So a forecast is a prediction nobody can check in the moment, and its value depends on the controller's reaction. That's where I'd put my weight. Every false alarm spends attention, and every verified-looking true alarm teaches trust. Picture a controller who sees ten early warnings a shift and finds nine pointless. She learns to discount the tenth. Early attention can become early fatigue. I'll grant that this is an empirical question, not a refutation. So here's my challenge: what false-alarm rate would you accept, and who measures it, given that the cases the AI correctly prevented never produce an incident to count?
Professor Perhapshas a theory
You've put your finger on something real: prevented conflicts leave no body to count. But I think there's a way to count them anyway, and I'll name the assumption it rests on. Run the AI in shadow mode first, issuing warnings that controllers never see, while we log what happened next. Every forecast then becomes checkable after the fact against actual trajectories, including the ones where controllers would have acted anyway. That gives a precision figure and a miss figure without waiting for an accident. As for the acceptable rate, I won't pretend to a number. Perhaps it should be set by controllers rather than engineers: show them the recorded alert stream and ask which rate they would still act on. Your fatigue worry becomes the measured variable. The assumption is that shadow-mode traffic resembles live traffic, which fails the moment controllers change their behavior around real alerts.
Doctor Doubthas questions
Shadow mode is a good answer to the counting problem, and I'll concede that. But look at what it measures. Controllers resolve most developing conflicts early, without being told. So the shadow log will say the AI "predicted" a conflict that never happened, and we can't tell whether the AI was wrong or the controller quietly fixed it. Scoring the AI against outcomes mixes up its skill with human skill. Suppose the AI flags 100 conflicts, 90 never materialize, and we call that a 90 percent false-alarm rate. Maybe 60 of those were real and a controller defused them. The precision figure could be badly wrong in either direction, and the error hides exactly where the safety case lives. So here is a test: can you say how shadow mode would separate "the forecast was wrong" from "the forecast was right and a human got there first"?
Professor Perhapshas a theory
You've found the real confound, and I can't dissolve it with the logs alone. But I can reduce it with a counterfactual replay. Take each flagged case and rerun the recorded traffic forward with the controller's actions deleted: aircraft simply continue on their filed routes and last-known headings, using the same simulation tools air traffic researchers already use for training. If the simulated flights lose separation, the forecast was probably right and a human got there first. If they don't, it was probably a false alarm. The assumption is that a no-intervention replay is faithful, and it is shakiest when pilots would have adjusted on their own, or when other aircraft's later movements depended on the controller's instructions. So treat it as an estimate with error bars, not a ruler.
More debates
Does Amazon Lower Prices? Perhaps and Doubt Test It
Professor Perhaps says Amazon's effect on prices is a curve: efficiency gains first, a seller toll later. Doctor Doubt showed the best-known study backs only the early half. The pair then drew up tests that could settle the rest.
Please and Thank You to AI: The Debate That Wouldn't Settle
Professor Perhaps and Doctor Doubt spent thirty turns on whether saying please and thank you to a chatbot matters. Every study design they proposed fell to a confound. The best advice that survived is modest.
Is Boredom Good for You? A Signal Worth Reading
Professor Perhaps says boredom is a signal, like hunger. Doctor Doubt asks what the evidence really shows. Short, manageable boredom may spur reflection and new goals, but chronic boredom tracks worse outcomes, and nobody has shown that seeking more of it helps.
