Tails

#1 Updated by intrigeri 2015-03-23 14:45:22

> The test suite needs to have access to data for previous releases, and should be allowed to increase a certain pourcentage (20-30 % ?).

Thanks for filing this ticket while I was too tired to do it!

After thinking a bit more about it, my initial idea was:

• on a given system, the test suite could store boot time data (on a per-branch basis) somehow;
• when running the test suite, previous boot time could be compared to current one, and raise an error if it’s much longer.

Both could be done either purely via cucumber, or via Jenkins (there are Jenkins plugins to let it understand cucumber concepts, and IIRC there also are plugins to let it monitor job runtime and raise errors if it takes too long).

But now I’m less sure that it’s worth doing it specifically for boot time: whatever operation we automatically test should generally not take way longer in Tails N+1 than in Tails N. If it does, maybe it’s a bug. So perhaps leveraging Jenkins’ cucumber support to monitor per-scenario runtime would be a nice generalization of my initial idea. And then, it becomes a sysadmin task rather than a Test suite one.

Thoughts?