Tails

#3 Updated by intrigeri 2015-10-03 15:39:26

Whether WhisperBack’s SMTP server is up is one of the services that we’ll setup monitoring for with high priority (https://tails.boum.org/blueprint/monitor_servers/#services). I’d rather not notify all developers when it’s down, which would be basically what happens if that service is down and we rely on it in our test suite. It’s bad enough that our test suite relies on so much of our infra already, let’s not add to it when possible.

A way to test the Tails ISO side of things would be to fire up a tiny/mock SMTP server behind a onion service as part of the test suite, live-patch WhisperBack config to use that onion service and its TLS certificate, and ensure the email is sent to that SMTP server. This would be the ideal solution but I doubt the benefits are worth the effort, especially as long as we’ve no canonical, tested and robust way to fire up and manage such temporary services in our test suite framework.

> Alternatively, if we want such a test, we could tag it @release-testing, and somehow make it so that scenarios tagged @release-testing are only run when the test suite is invoked with a --release-testing option (or @ … — —tag release-testing), intended to only be used when testing a release (this one could also ensure that --iso != --old-iso, etc.).

This sounds like a fine way to drop the part of the manual test that can be automated. I’m a bit wary of twisting the test suite semantics, though: we’ll see “all tests are green”, while one is only green once frontdesk have confirmed they’ve received the email and it was encrypted. I guess we can live with it, but I’d love it if anonym gave it a few more thinking minutes of his amazing brain.

#4 Updated by intrigeri 2015-12-05 10:06:24

Note that these tests most likely will need the doc Cucumber tag.