Glossary · 6 · Quality, risk and governance

Automation bias

Also known as: Over-reliance on automation, Automation complacency, Automation-induced complacency

German: Automatisierungsbias

In AI governance and human factors, automation bias is the tendency of people to accept machine-generated output with less scrutiny than they would apply to output from a person, and to discount contrary evidence once a system has proposed an answer. It shows up as errors of commission (following a wrong recommendation) and errors of omission (missing a problem the system did not flag). Because it degrades the quality of review rather than its existence, automation bias is the main mechanism by which a formally required human check becomes a rubber stamp. The EU AI Act names it explicitly: persons assigned to oversee a high-risk AI system must be enabled to remain aware of the possible tendency to automatically rely or over-rely on its output.

Intermediate
Technical writers
Technical marketers
Technical project managers
Developers

In one sentence

Automation bias explained: why people under-scrutinize AI output, and how to keep human review structural instead of ceremonial.

Example

A reviewer waved through an AI-drafted lockout procedure because the model sounded confident and the checklist only asked "reviewed? yes/no" — after the step was changed to require a cited source per safety instruction, the same reviewer caught two invented torque values.

How it applies

Treat review design as a control, not a formality: A human in the loop only adds safety if the reviewer has the time, information and authority to disagree. Sign-off fields that can be ticked without opening the source, deadlines that assume the draft is already correct, and dashboards that show only the model's answer all invite automation bias.
Make disagreement cheap and visible: Ask reviewers to record what they checked (source, clause, measurement) rather than that they checked. Require a second opinion where the cost of a wrong output is high, and log rejected suggestions — a review step that never rejects anything is a signal, not a success metric.
Technical writers: Verify facts against the source of truth before editing for style; fluent prose is the strongest trigger of over-trust, and hallucination is invisible to a stylistic read. For safety-relevant instructions, check values and sequences against engineering sources, not against the model's own explanation.
Technical marketers: Claims, figures and competitor comparisons produced with AI need a source per claim. Automation bias also works on your audience: confident AI-generated answers about your product can be believed without checking your documentation, which is one reason AI content disclosure and verifiable sourcing matter.
Technical project managers: Budget review effort explicitly when AI drafting shortens production time; otherwise the saved hours are taken out of scrutiny. Under the EU AI Act, deployers of high-risk systems must assign human oversight to natural persons with the necessary competence, training, authority and support (Article 26), and the Act's AI literacy duty in Article 4 covers staff who operate such systems. Naming the Act, a standard or this entry never establishes compliance by itself — only your implemented and documented measures can do that.
Developers: Design interfaces that surface uncertainty, provenance and alternatives instead of a single confident answer: retrieval citations, confidence or coverage signals, diffs, and "no answer" paths. In an agentic workflow, prefer confirmation steps that require a specific decision (approve this command, this file, this record) over blanket approvals, and keep guardrails that fail closed when evidence is missing.
Measure it: Automation bias is testable. Seed known-bad suggestions into review samples and count how many pass; track time-on-review, rejection rate and defect escape rate before and after an AI tool is introduced. Feed the results into evaluation (evals) and into your AI usage policy.
Keep accountability attached: Over-reliance does not transfer responsibility to the tool. Accountability (AI) stays with the named reviewer and the organization, and unexamined AI output is a common route into the accountability gap.

Automation bias vs. bias (AI)

Bias (AI) sits in the system: skewed training data, unrepresentative sampling or objectives that systematically disadvantage some cases or groups. Automation bias sits in the human: it is a cognitive disposition to trust the machine's output and to stop looking for counter-evidence. The two compound each other — a biased model paired with an under-scrutinizing reviewer produces errors that pass every formal checkpoint — but they need different remedies. Model bias is addressed with data work, testing and documentation such as a model card; automation bias is addressed with workflow design, training, interface choices and evidence requirements for reviewers. Human factors research also distinguishes automation bias (a decision bias) from automation complacency (reduced monitoring attention); the terms are often used interchangeably in governance texts, and the practical countermeasures overlap.

In one sentence

Example

How it applies

Automation bias vs. bias (AI)

External references