Johannes Himmelreich (Syracuse University)
Title: Humans Out of the Loop: Metacognitive limitations undermine human–AI oversight
Abstract: In AI governance, human oversight is a near-universal prescription. We argue against it. Human oversight over AI requires that humans a good at overseeing AI. But humans are bad at overseeing AI: In decision-support systems, humans misperceive cases that are difficult as easy. As a result, they follow AI advice less at precisely those moments when following AI advice would be most beneficial. We defend two claims. First, just as Odysseus bound himself to the mast, decision makers who recognize their metacognitive limitations should pre-commit to defer to AI. Second, otherwise---when decision makers fail to pre-commit to defer to AI---third parties are permitted to do so. When human oversight means having a "human in the loop," we argue that, in some situations, humans should be out of the loop.