Feature #6218
Run our custom programs test suite on interesting commits
100%
Description
For now the scope of this ticket is limited to our 3 Perl code bases (perl5lib, persistence-setup, iuk)… simply because our other custom programs (sadly) have no test suite.
One way to do this would be to run our custom programs’ test suite, using Vagrant, on our existing isobuilders (that already have Vagrant installed and some room for storing baseboxes) or isotesters (which would feel more logical, because “testers”) or both. What it takes is:
- at the root of each Git repo we want to test:
- add a suitable Vagrantfile that installs all build/test deps during provisioning
- add a Makefile/Rakefile/script that starts the Vagrant VM, runs the test suite in there, and tears down the VM
- for each Git repo we want to test:
- create a Jenkins job that runs the aforementioned script /
rake test
/make test
- create a Jenkins job that runs the aforementioned script /
Open questions:
- how to deal with inter-dependencies between these code bases? e.g. persistence-setup and iuk depend on perl5lib; our options:
- start by testing perl5lib that has no such problem, gain experience with the general idea during this first iteration, and think about this problem later iff. the general idea is validated
- install current perl5lib master in the VM with
dzil install
: this works iff. we don’t delegate to the Debian packaging anything that’s required to use perl5lib (currently we install MO files viadebian/rules
, but perhaps we can do without them here) - set up the APT source for our
devel
custom APT repo and treat perl5lib just like any other build dependency
- how to avoid creating yet another complex setup for managing baseboxes? our options:
- don’t keep any basebox persistently on the Jenkins “slaves” (sic :/) that run the tests, instead download & provision them every time; we can probably leverage our existing apt-cacher-ng to speed things up (see file:///usr/share/doc/apt-cacher-ng/html/howtos.html#ssluse to make caching of HTTPS downloads work). This option implies we use Vagrant baseboxes published by third-parties (otherwise we’re just moving the basebox management problem elsewhere) which is probably acceptable here; an official Debian Stretch Vagrant box is available, and we can dist-upgrade it during provisioning to whatever version of Debian we want to test on.
- use the baseboxes we already use & manage for building ISO images: this probably requires quite some refactoring to extract code from tails.git, and forces us to test only on whatever snapshot of Debian we currently use, while we would like to test on current Debian sid too.
- other ideas?
- how to store or compute build & test dependencies? We already store them in two different places (
dist.ini
on themaster
branch,debian/control
on thedebian
branch) so it would be nice to avoid adding a 3rd one. Likely the easiest way is to extract build deps from thedebian/control
file and install them with something likecontrolfile=$(mktemp) ; git show origin/debian:debian/control > $controlfile ; mk-build-deps -i -r --root-cmd sudo $controlfile ; rm $controlfile
- how does this relate to Feature #7036 and Feature #6220#note-11? Can we implement this ticket in a way that also makes us progress on that other front? At least having to deal with custom APT sources, APT snapshots, build deps, will teach us a thing or three about the issues we’ll face when we tackle that other project, but perhaps we can do better.
Ideally we would run these tests both on Debian sid (to detect breakage in advance and make our RM’s life easier), whatever snapshot of Debian we currently use, and an unfrozen version of the Debian release we use. But already running them on one of them (probably sid) would be a pretty good start!
Related issues
Related to Tails - Bug #17636: tails-iuk: test suite usefulness/expectations | Confirmed | ||
Has duplicate Tails - |
Duplicate | 2013-11-27 | |
Blocked by Tails - Feature #7036: Move custom software to our main Git repository | In Progress |
History
#1 Updated by intrigeri 2017-09-16 07:48:03
- Description updated
#14674 prompted me to think about this topic: if we had this, then I would automatically get actionable bug reports from Jenkins about test suite fragility, breakage caused by updates in Debian sid, and of course actual bugs in the software under testing. And then, hopefully I’ll be able to resolve these problems before anonym faces them.
#2 Updated by intrigeri 2017-09-16 07:55:11
anonym & bertagaz, please take a look at the plan outlined in the description, and shout if it feels like the wrong approach, if you have better ideas (be it for the big picture or specific implementation details).
#3 Updated by intrigeri 2017-09-16 08:12:06
- related to #14674 added
#4 Updated by intrigeri 2017-09-16 08:43:32
- Description updated
#5 Updated by anonym 2017-09-18 11:50:05
intrigeri wrote:
> anonym & bertagaz, please take a look at the plan outlined in the description, and shout if it feels like the wrong approach […]
I think it sounds great! I feel I’m too unfamiliar with how things work on Jenkins to comment about specifics. However, I’d love if it was integrated in our existing rake
setup some how so I can also use it locally, but this would probably only make sense to do if we also do Feature #6220 and solve it with rake
+ Vagrant.
#6 Updated by intrigeri 2019-09-19 06:23:08
- blocked by Feature #7036: Move custom software to our main Git repository added
#7 Updated by intrigeri 2019-12-22 22:57:01
- has duplicate
Feature #6442: Run Cucumber and Test::Spec tests at tails-iuk Debian package build time added
#8 Updated by intrigeri 2020-03-10 08:55:28
See Feature #7036 for more recent thoughts on this, that take into account the fact that these custom programs are moving to tails.git and are not Debian packages anymore.
#9 Updated by intrigeri 2020-03-28 09:39:31
Idea: run these test suites inside a running Tails, as part of our automated test suite.
#10 Updated by CyrilBrulebois 2020-04-17 23:43:56
- related to Bug #17636: tails-iuk: test suite usefulness/expectations added