Also, make sure that broken tests actually break the bots.

And add a test to make sure that keeps happening.