Problem is, it's alert after the fact. And it doesn't actually stop.

Unless you have scripts in place to nuke everything (which you will have to develop yourself since aws doesn not supply them), you have to manually login, and try to shut things of, while bill goes chaching