When NOT to Use AI: Novel Decisions and Automation Bias

AI systems are, at their core, pattern-matching tools trained on historical data. They excel when the problem they are given resembles problems in their training set. When the situation is genuinely new, they produce outputs that look confident but may be entirely wrong. Think of a satellite navigation system. On familiar roads it is fast, accurate, and useful. Ask it to route you through terrain it does not recognise and it will do so with exactly the same confident voice it uses on the motorway, right up until it guides you across a flooded field. Confident output is not the same as correct output. This matters most in two types of situation: novel crises and accountable decisions. Novel crises are, by definition, ones with no close historical analogue. A sudden geopolitical event, an unprecedented product failure, a rapidly evolving regulatory change: these are precisely the situations where leadership earns its keep, and precisely the situations where a pattern-matching system has the least to offer. An AI trained on past data cannot reason about genuinely new circumstances. It can only extrapolate from patterns in that data, and in a genuinely novel situation those patterns may be misleading. Accountable decisions are ones where a named individual must stand behind the outcome: a disciplinary hearing, a performance review that affects someone's career, a clinical judgement that determines a course of treatment. These involve contextual judgement, ethical weight, and personal accountability that cannot be delegated to a model. Under the EU AI Act, Regulation (EU) 2024/1689, both providers and deployers of high-risk AI systems carry legal obligations, and a deployer cannot discharge its human-oversight and correct-use duties by arguing that "the model decided". The same principle applies in regulated sectors more broadly. In financial services, healthcare, and legal practice, explainability requirements mean a decision-maker must be able to articulate the reasoning behind an outcome, and a black-box model that produces a recommendation without a traceable rationale can be a compliance failure independent of whether the recommendation was correct.