In one specific test we expect an executable to be able to run the tests
multiple times. For that case we don't want to hard exit ever.