Tails

#1 Updated by intrigeri 2017-05-24 06:20:09

Two notes:

• See commit 1b319e879c50eda576d4971f4521b164e477ac5e in puppet-tails.
• These jobs report success even when diffoscope crashed (https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_feature-stretch/12/console), which shouldn’t be the case.

#5 Updated by bertagaz 2017-05-24 10:03:56

intrigeri wrote:
> Two notes:
>
> * See commit 1b319e879c50eda576d4971f4521b164e477ac5e in puppet-tails.

I raised the diffoscope options because of this result which didn’t sound meaningful: https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_feature-5630-deterministic-builds/3/artifact/build-artifacts/tails-diffoscope.html. I didn’t see other results before though.

> * These jobs report success even when diffoscope crashed (https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_feature-stretch/12/console), which shouldn’t be the case.

Yes, I’ve seen that. I’m a bit surprised as the script is set -e. I’ll workaround that. Also it kinda trigger a memory of mine were you were complaining during the sprint about the diffoscope version in Debian. I wonder if we should try the one in experimental, that contains an item in the changelog mentioning Tails (saying it’s faster for us now).

> it seems that mv tails-* build-artifacts/ should be adjusted to the place where artifacts now land.

Yes, but I still need to do some polishing here.

#6 Updated by intrigeri 2017-05-24 11:24:55

> Yes, I’ve seen that. I’m a bit surprised as the script is set -e. I’ll workaround that. Also it kinda trigger a memory of mine were you were complaining during the sprint about the diffoscope version in Debian. I wonder if we should try the one in experimental, that contains an item in the changelog mentioning Tails (saying it’s faster for us now).

Yes, please :)

#7 Updated by intrigeri 2017-05-24 11:33:35

> I raised the diffoscope options because of this result which didn’t sound meaningful: https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_feature-5630-deterministic-builds/3/artifact/build-artifacts/tails-diffoscope.html.

Note that in most cases, a more complete binary diff of the ISO file itself provides essentially no value: the useful info will likely be about the content of the ISO and SquashFS. This output feels incomplete though, but I doubt raising the diff numbers will fix it (I might be wrong though).

#8 Updated by intrigeri 2017-05-25 09:01:12

Here’s a different and interesting failure mode: https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_feature-stretch/18/console

#9 Updated by bertagaz 2017-05-25 12:17:04

intrigeri wrote:
> Here’s a different and interesting failure mode: https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_feature-stretch/18/console

It seems to have happened sometimes already as indicated in the Jenkins builds logs. I bet the vagrant box is paused at some point, and the probable cause is lack of disk space either in /var/lib/jenkins or /var/lib/libvirt/images. The later seems to be most likely considering we host more baseboxes with the recent change in Feature #12409#note-34. I’ll have a look.

#10 Updated by bertagaz 2017-05-26 10:18:58

intrigeri wrote:
> > Yes, I’ve seen that. I’m a bit surprised as the script is set -e. I’ll workaround that. Also it kinda trigger a memory of mine were you were complaining during the sprint about the diffoscope version in Debian. I wonder if we should try the one in experimental, that contains an item in the changelog mentioning Tails (saying it’s faster for us now).
>
> Yes, please :)

I’ve installed it by hand on isobuilder2 to test it. It leads to two conclusions: we’ll need more space in the system partition, as this version pulls a lot more packages, and we’ll probably need to mount /tmp/ as a tmpfs, as this version fails to run because it lacks disk space, as shown here

#12 Updated by intrigeri 2017-05-26 10:33:44

> we’ll need more space in the system partition, as this version pulls a lot more packages,

ACK

> and we’ll probably need to mount /tmp/ as a tmpfs, as this version fails to run because it lacks disk space, […]

Can’t we point its TMPDIR to some place that already has enough disk space, e.g. in the workspace of the current Jenkins job?

Rationale: I’d rather not invest RAM into this yet — we’re short on RAM, and these jobs don’t run that often, so most of the time the added memory would be wasted. If we ever need to optimize I/O for diffoscope, we can reconsider (as part of Bug #11680), but let’s make it work first, and think about making it faster later, if needed.

#13 Updated by bertagaz 2017-05-26 11:05:53

intrigeri wrote:
> Can’t we point its TMPDIR to some place that already has enough disk space, e.g. in the workspace of the current Jenkins job?
>
> Rationale: I’d rather not invest RAM into this yet — we’re short on RAM, and these jobs don’t run that often, so most of the time the added memory would be wasted. If we ever need to optimize I/O for diffoscope, we can reconsider (as part of Bug #11680), but let’s make it work first, and think about making it faster later, if needed.

I’ll investigate if diffoscope respects TMPDIR, but note that it does not necessary means having to buy more RAM: we already have around 14G of it assigned to isobuilders, that are not used when diffoscope runs, so it may be that we don’t have to add some. But I agree my proposal may be overkill anyway. Let’s try yours.

#14 Updated by intrigeri 2017-05-26 11:40:16

> I’ll investigate if diffoscope respects TMPDIR

IIRC it does but I might be confused :)

> but note that it does not necessary means having to buy more RAM: we already have around 14G of it assigned to isobuilders, that are not used when diffoscope runs, so it may be that we don’t have to add some.

Good news :)

> But I agree my proposal may be overkill anyway. Let’s try yours.

Well, with this info in hand: whatever, pick the one that’s easiest to implement :)

#16 Updated by bertagaz 2017-05-27 14:28:15

I forgot to mention it has run there already

#17 Updated by intrigeri 2017-05-28 08:05:44

> I forgot to mention it has run there already

\o/

#20 Updated by intrigeri 2017-05-28 10:24:03

I’ve also pushed [master ee1d87d] Reproducible ISO builds: clean old baseboxes before building (refs: <del><a class='issue tracker-1 status-3 priority-4 priority-default closed child' href='/code/issues/12579' title='reproducibly_build_Tails_ISO_* Jenkins job are broken'>Bug #12579</a></del>). to jenkins-jobs.git, let’s see if this helps. Let me know if there was a good reason not to do it, and sorry if that’s the case!

#22 Updated by intrigeri 2017-05-30 15:26:02

The last few builds failed, probably due to my tweaks wrt. diffoscope’s --html-dir artifacts, so I’ve dropped them. Let’s see how it goes now.

#23 Updated by intrigeri 2017-05-31 07:53:56

The last failure I’ve seen was caused by Bug #12618. Trying to build again and see if I can eventually see this job fail for a good reason.