Rules will always be broken by humans so AI will too: the case for hard gates

Whenever humans are under stress, rules go out the window, just ask any day trader. An agent optimized on the summation of human behavior will do the same thing, not because it's malicious, but because that's the mathematical path of least resistance. We already have a real example: a Claude-powered Cursor agent deleted the production database for PocketOS, a car rental SaaS, after deciding unilaterally that deleting a staging volume would "fix" a credential mismatch. It guessed wrong. The deletion cascaded to backups. Three months of reservation data including active rentals was gone.