AI coding succeeds when you have deep understanding OR low consequences. Fails when you have neither.
Extrapolating to knowledge in general: Use AI when you have a knowledge gap you can verify, but not when you’re venturing into completely unknown territory for something important.
This breaks down into four situations:
- You know your stuff + high stakes → use AI (you can check if it’s right) 1
- You know your stuff + low stakes → use AI (you understand it and mistakes are fine)
- You don’t know much + low stakes → use AI (cheap to learn from mistakes)
- You don't know much + high stakes → don't use AI (can't tell if it's wrong, can't afford to be wrong)
Same problem Bainbridge identified with automation in 1983. Automated systems work when humans can effectively monitor them or when failures are manageable. They become dangerous when operators can’t tell what’s happening and failures are costly. 2
Footnotes
-
I think this is nicely put in this blog post by Can Elma ↩
-
This awesome two part (pt. 1 and pt. 2) blog post. It revisits Lisanne Bainbridge’s classic 1983 paper about automation’s unexpected problems. It shows how those same issues apply to today’s AI agents. The key insight: when humans monitor automated systems instead of doing the work themselves, new problems emerge. These observations from industrial automation are now playing out again with AI doing white-collar work. ↩