Bug #14924
reproducibly_build_Tails_ISO_stable Jenkins job always fails
100%
Description
We’re quite unlucky: build_Tails_ISO_stable is scheduled daily, and with Jenkins’ hashing system it happens to run at 07:15 so reproducibly_build_Tails_ISO_stable starts around 08:00. We update the Debian security APT snapshot at 7:41 so in practice, reproducibly_build_Tails_ISO_stable always gets a different snapshot and thus always produces a different ISO.
This prevents us from easily noticing when an update or branch merge broke reproducibility. And possibly worse, it teaches us to ignore email notifications from Jenkins (this has been going on for a week and nobody noticed or if they did, then they kept it to themselves for some reason).
Due to the way our Jenkins jobs are generated, we have little flexibility here: we can’t simply reschedule build_Tails_ISO_stable (and only this one) to run at a different time. And if we start playing with the H symbol we’ll be lucky if we don’t break devel or testing.
The simplest option we have seems to slightly reschedule the APT snapshot update. Now, of course this will likely break “reproducibly” (sic) jobs for other branches, but until we have a better solution, not breaking stable/testing/devel gets higher priority by far. While implementing this I’ll need to be careful not to break testing or devel, but at least I can easily reason about it, while I don’t know how the Jenkins hash thing works exactly.
Long term, the only sane option is probably to teach our build system how to use snapshots specified on the command line (instead of “latest”), and to teach Jenkins to pass the correct parameters when trying to rebuild a given ISO image. Whoever feels responsible for this, please file a ticket about it.
Subtasks
Related issues
|
Related to Tails - |
Resolved | 2017-10-22 | |
|
Related to Tails - |
Resolved | 2017-12-26 | |
|
Blocked by Tails - |
Resolved | 2017-11-06 | |
|
Blocked by Tails - |
Resolved | 2017-11-10 |
History
#1 Updated by intrigeri 2017-11-06 20:03:08
- Parent task set to
Feature #5630
#2 Updated by intrigeri 2017-11-06 20:04:32
- related to
Feature #12633: Lower the workload caused by reproducible builds Jenkins jobs added
#3 Updated by intrigeri 2017-11-07 11:58:58
- % Done changed from 0 to 50
- QA Check set to Ready for QA
Let’s assume a 70 min delay between the time when the 1st build resolves the “latest” snapshot pointer and the time when the 2nd build does (currently we’re around 52 minutes but let’s be on the safe side: the longer this delay, the more chances the problem this ticket is about happens). Let’s assume this pointer resolution happens 1st thing during a build to simplify (in practice it’s more ~2 minutes later, whatever).
Frozen branches:
- daily build of stable starts at 07:15 => the debian-security repo should not be updated between 07:15 and 08:25
- daily build of testing starts at 15:23 => the debian-security repo should not be updated between 15:23 and 16:33
Non-frozen branches:
- daily build of devel starts at 06:56 => no repo should be updated between 06:56 and 08:06
- daily build of feature/buster starts at 09:15 => no repo should be updated between 09:15 and 10:25
- daily build of feature/tor-nightly-master starts at 08:15 => no repo should be updated between 08:15 and 09:25
So ideally no repo should be updated between 06:56 and 10:25, and the debian-security repo should not be updated between 15:23 and 16:33. I’ve reworked the way we schedule these cronjobs so this is now guaranteed: all updates now start between 05:00 and 05:45, between 11:00 and 11:45, between 17:00 and 17:45, and between 23:00 and 23:45 (UTC).
The catch is that I’ve assumed the repo updates take 0 time, which is of course incorrect. But as long as no such update takes more than ~1 hour we should be good. Anyway, I think that this change will already improve things in the vast majority of cases. If practice proves me wrong I’ll come back to it.
If Jenkins ever changes its hashing algorithm that determines when “daily” jobs are run for a given branch, we’ll need to come back to it and possibly adjust automatic_refresh_delta_hours in tails::reprepro::snapshots::time_based.
I’ll now wait ~36 hours so that all common cases have been tested.
#4 Updated by intrigeri 2017-11-07 12:21:20
- Status changed from Confirmed to In Progress
Applied in changeset commit:f3fafe5ad83e73102c134e637b756c470d786230.
#5 Updated by intrigeri 2017-11-08 10:17:04
- blocked by
Bug #14923: devel branch FTBFS since torbrowser-launcher 0.2.8-4 was uploaded added
#6 Updated by intrigeri 2017-11-08 10:40:50
So, at least this seems to be fixed for the stable branch :) I’ll come back to it once the other ones don’t FTBFS anymore.
#7 Updated by intrigeri 2017-11-12 10:30:36
- blocked by
Bug #14946: Topic branches that lag behind their base branch don't build reproducibly with mergebasebranch build option added
#8 Updated by intrigeri 2017-11-12 10:32:24
- % Done changed from 50 to 60
Daily build of devel is OK too. For feature/tor-nightly-master I’ll wait for Bug #14946 to be fixed before fixing the FTBFS (=> saving storage space). We’ll see if this happens early enough (i.e. today) for me to have data here before the 3.3 release.
#9 Updated by intrigeri 2017-11-14 08:08:45
- Target version changed from Tails_3.3 to Tails_3.5
intrigeri wrote:
> Daily build of devel is OK too. For feature/tor-nightly-master I’ll wait for Bug #14946 to be fixed before fixing the FTBFS (=> saving storage space). We’ll see if this happens early enough (i.e. today) for me to have data here before the 3.3 release.
The merges happened too late for me to get data already, so postponing.
#10 Updated by intrigeri 2017-11-15 16:34:03
- Status changed from In Progress to Resolved
- % Done changed from 60 to 100
- QA Check changed from Ready for QA to Pass
Everything works as intended. For the testing branch, we’ll see during the next freeze.
#11 Updated by bertagaz 2017-12-26 11:10:38
- related to
Bug #15107: Some topic branches are never built reproducibly during Jenkins daily runs => add option to specify which APT snapshot serials to use during build added