Tails

With the new “single SquashFS diff” upgrade scheme (Feature #15281 and friends), during the release process, the RM has to build many IUKs on Jenkins:

• to ensure they match those the RM has built locally
• to publish them without having to upload gigabytes of data

For example, when we’ll release 4.20, we’ll be building more than 2 dozens IUKs.

The first iteration (Feature #15287) of the https://jenkins.tails.boum.org/job/build_IUKs/ job builds all the needed IUKs sequentially, on one single isobuilder, which takes about 15 minutes per IUK ⇒ more than 6 hours in my 4.20 example. That’s going to be pretty bad both wrt. time to remediation for security flaws and wrt. RMs’ nerves.

Now that we’ve learned about this problem, we should iterate and improve on this.

The obvious solution is to parallelize these builds.

Potential sources of inspiration:

• https://docs.openstack.org/infra/jenkins-job-builder/builders.html#builders.trigger-builds
• https://docs.openstack.org/infra/jenkins-job-builder/builders.html#builders.multijob
• https://docs.openstack.org/infra/jenkins-job-builder/builders.html#builders.copyartifact (if one main job triggers all the other builds, ideally it would join all their artifacts so they’re available from a single place, where bin/copy-iuks-to-rsync-server-and-verify can fetch them from)

Worst case, if implementing this on Jenkins is too feasible soon enough, we can document in the release process how to split the set of IUKs to build into several runs of build_IUKs. This would also require extra work on bin/copy-iuks-to-rsync-server-and-verify.