Feature #17262

Make the build of overlayfs-based IUKs reproducible

Added by intrigeri 2019-11-27 08:53:40 . Updated 2020-01-28 09:06:42 .

Status:
Resolved
Priority:
High
Assignee:
Category:
Target version:
Tails_4.2
Start date:
Due date:
% Done:

100%

Feature Branch:
iuk:feature/17262-reproducible-overlayfs-iuks
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Upgrader
Deliverable for:

Description

Files

test-iuk-reproducibility.sh (652 B) intrigeri, 2019-12-05 11:19:08

Subtasks

Related issues

Related to Tails - Feature #15287: Make it possible to reproducibly generate IUKs in CI Resolved 2018-02-05
Related to Tails - Feature #11974: Reproducible IUK builds Resolved 2016-11-20
Related to Tails - Feature #9373: Make tails-iuk support overlayfs Resolved
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed
Blocks Tails - Feature #15283: Implement the "one single SquashFS diff" scheme in Tails Upgrader Resolved

History

#1 Updated by intrigeri 2019-11-27 08:54:28

  • related to Feature #15287: Make it possible to reproducibly generate IUKs in CI added

#2 Updated by intrigeri 2019-11-27 08:54:35

#3 Updated by intrigeri 2019-12-01 11:20:49

  • Priority changed from Elevated to High

#4 Updated by intrigeri 2019-12-05 10:10:51

#5 Updated by intrigeri 2019-12-05 11:30:44

My initial best-case test passes: with the attached script I get 2 identical overlayfs-based IUKs on my sid system.

Next step: try to reproduce on a different machine. I’m not sure yet if I’ll do that via Feature #15287 or if I’ll first do a quick’n’dirty attempt in some more ad-hoc manner.

#6 Updated by intrigeri 2019-12-06 07:18:48

#7 Updated by intrigeri 2019-12-06 07:21:20

  • related to Feature #9373: Make tails-iuk support overlayfs added

#8 Updated by intrigeri 2019-12-06 19:03:29

OK, first findings via Feature #15287:

  • I can’t reproduce the 4.0 to 4.1 IUK we’ve published but it was built using mksquahfs 1:4.3-3.0tails4 so I won’t bother trying harder.
  • The mksquashfs version matters: even with aufs, I get a 4.0 to 4.1 IUK on sid that’s 22MB smaller than on a stretch isobuilder that has squashfs-tools 1:4.3-8.0tails1.
  • Using squashfs-tools 1:4.4-1 on my sid system, I can reproduce the 4.0 to 4.1 aufs IUK built on Jenkins.

Next step: try the same thing using overlayfs.

#9 Updated by intrigeri 2019-12-06 20:49:17

intrigeri wrote:
> Next step: try the same thing using overlayfs.

They differ:

  • According to diffoscope, the only difference is the size of a bunch of directories, which varies by 12 bytes (sometimes in one direction, sometimes in the other).
  • After mounting the embedded live/4.1.squashfs, getfattr --dump --match=- returns some xattr for the IUK I’ve built on my sid, while it returns “Operation not supported” for the IUK from Jenkins (Stretch).

I don’t know if these are 2 different problems or only one, but it seems easier to investigate the second first.

When building the SquashFS diff, the read-write branch (upper dir) is a tmpfs and the lower, read-only branch is the old SquashFS; they’re union’ed with overlayfs. So my understanding is that the problem can come from overlayfs, tmpfs, or how the play with each other. tmpfs in the Stretch kernel already has support for xattrs (getfattr --dump --match=- /run/initramfs/ on one of our Stretch isobuilders returns nothing, but does not complain).

So my current hypothesis is that the difference is caused by overlayfs changes between Linux 4.9 and 5.3, only the latter using xattr on tmpfs, and then when building an IUK on 4.9, no xattr is saved in the tmpfs, mksquashfs detects this and disables xattr support, and thus getfattr returns “Operation not supported”.

Next steps could be:

  • Huge hammer approach: would it break stuff to pass -no-xattrs to mksquashfs?
  • Big hammer approach: would it break stuff to remote the overlayfs xattrs from the tmpfs before we run mksquashfs on it? (Ideally, there are some non-overlayfs xattrs we would keep, e.g. on my sid, /usr/bin/ping, /usr/bin/gnome-keyring-daemon, and /usr/bin/mtr-packet have the security.capability xattr set.)
  • Can we disable xattr support for a specific tmpfs mount?
  • Can we specify some overlayfs module or mount options on sid, so it does not use xattrs and thus behaves in a way that’s more similar to Linux 4.9’s overlayfs?
  • Compare IUK built on sid with IUK built on Stretch with a newer kernel. Then, if a newer kernel solves the problem:
    • Can we ensure all relevant systems (isobuilders, RMs’ systems) have a recent enough kernel to behave consistently with newer kernels?
    • Is it worth moving IUK generation to a Vagrant box? That would give us the same kernel everywhere but it adds quite some complexity.

#10 Updated by intrigeri 2019-12-06 21:21:35

intrigeri wrote:
> * Huge hammer approach: would it break stuff to pass -no-xattrs to mksquashfs?

This option made the IUK reproducible between Stretch and sid!

Next sub-step in the hope it would validate this cheap workaround: combine 4.0’s filesystem.squashfs with this IUK’s 4.1.squashfs and see how the resulting overlayfs mountpoint compares to 4.1’s filesystem.squashfs.

#11 Updated by intrigeri 2019-12-07 14:08:18

  • Feature Branch set to iuk:feature/17262-reproducible-overlayfs-iuks

#12 Updated by intrigeri 2019-12-08 09:13:54

  • Status changed from In Progress to Needs Validation
  • Type of work changed from Test to Code

> intrigeri wrote:
>> * Huge hammer approach: would it break stuff to pass -no-xattrs to mksquashfs?

> This option made the IUK reproducible between Stretch and sid!

This “huge hammer approach” experiment was useful as it pointed me in the right direction, but after further investigation I’ve found that we should, and can, do better.

First of all, note that our filesystem.squashfs for 4.0 and 4.1 include some potentially useful xattrs:

file: old-squashfs/bin/ping
security.capability=0sAQAAAgAgAAAAAAAAAAAAAAAAAAA=

file: old-squashfs/usr/bin/gnome-keyring-daemon
security.capability=0sAQAAAgBAAAAAAAAAAAAAAAAAAAA=

file: old-squashfs/usr/lib/x86_64-linux-gnu/gstreamer1.0/gstreamer-1.0/gst-ptp-helper
security.capability=0sAQAAAgAUAAAAAAAAAAAAAAAAAAA=

… but as soon as I use aufs to combine any of these SquashFS with a tmpfs, these xattrs are lost in the aufs mount; same if I also add an extra SquashFS diff. So our baseline is: no xattrs are visible in a running Tails system, be it freshly installed or upgraded.

overlayfs behaves differently: with no incremental upgrade applied, or with incremental upgrades applied that don’t modify files with xattrs, overlayfs preserves “security.capability” xattrs. So far so good.

But if we generate the IUK with -no-xattrs and the IUK updates a file that had xattrs in filesystem.squashfs, then we lose these xattrs in the resulting overlayfs mount. In other words, in some cases, a freshly installed Tails behaves differently than an upgraded one. I’m concerned that the resulting behavior variation can be confusing and make other problems harder to debug, so from this point on I’ve looked into including relevant xattrs in our IUK’s SquashFS, just like they are present in filesystem.squashfs already.

To achieve this I did this:

  • Revert the previous huge hammer approach (passing -no-xattrs to mksquashfs).
  • Pass --xattrs to the rsync command used to generate the SquashFS diff, otherwise there’s no chance that xattrs on updated files are preserved.
  • Remove all overlayfs xattrs before compressing the SquashFS, because they make the read-write upper dir non-deterministic between my sid system and our Stretch isobuilders. I suspect it’s a function of the kernel version; I’d rather not force all our RMs and isobuilders to run the same kernel (and I’m not even sure if that would be sufficient anyway).
  • Patch squashfs-tools to make my sid system and our Jenkins (Stretch) isobuilders generate an identical IUK :

I have tested that the security.capability xattrs are preserved for the 3 aforementioned files, in an overlayfs mount that combines 4.0’s filesystem.squashfs with:

  • as the 2nd lower dir, a 4.1.squashfs diff generated with iuk.git from this branch, that artificially updates the content of /bin/ping (without changing its xattrs)
  • a tmpfs upper dir

So I think we’re good! Not only we still have reproducible IUKs, but on top of that we consistently preserve useful xattrs such as security.capability, which is an improvement over aufs :)

If xattrs ever cause trouble again, we could pass -no-xattrs to mksquashfs again, but to avoid the “diverging behavior between freshly installed and upgraded systems” problem, we would have to pass it both in tails.git’s MKSQUASHFS_OPTIONS (used to generate filesystem.squashfs) and in iuk.git (used to generate IUKs’ SquashFS diffs).

With this code, our Jenkins (Stretch) isobuilders produce the same 4.0 to 4.1 IUK as my sid system. Still, it would be great if the reviewer could try this on their system and confirm they get the same result too. To do so, follow these instructions:

First, enable this APT source: deb http://deb.tails.boum.org/ iukbuilder-stretch main

Then:

sudo apt update && \
sudo apt install attr bsdtar libarchive-tar-wrapper-perl libcarp-assert-more-perl libcarp-assert-perl libclass-xsaccessor-perl libdevice-cdio-perl libdpkg-perl libfilesys-df-perl libfile-which-perl libfunction-parameters-perl libgnupg-interface-perl libipc-system-simple-perl libmoo-perl libmoox-handlesvia-perl libmoox-late-perl libmoox-options-perl libnamespace-clean-perl libpath-tiny-perl libtry-tiny-perl libtypes-path-tiny-perl libyaml-libyaml-perl libyaml-perl nocache rsync squashfs-tools=1:4.4-1+0.tails1 && \
git clone https://git.tails.boum.org/puppet-tails && \
cd puppet-tails && \
./files/jenkins/slaves/isobuilders/wrap_tails_create_iuks \
--source-versions 4.0 \
--new-version 4.1 \
--source-date-epoch=1575321815 \
--iuk-commit=c1662d593c6f1f645e1d5b46166716de4e5f80d4 \
--perl5lib-commit=206321a6e4080725f3801763b057545efd3ed844 \
--debug \
--create-iuk-extra-args "--union-type overlayfs" && \
sha256sum Tails_amd64_4.0_to_4.1.iuk

The printed sha256sum should be f2d2fd8bf4a9c3838d0ed5c7c1ffb9e5de4a7c7a0b917219c0497ebca2862af4.

#13 Updated by intrigeri 2019-12-08 09:14:31

  • Assignee deleted (intrigeri)

#14 Updated by intrigeri 2019-12-08 09:16:20

  • Assignee set to segfault

@segfault, could you please review this? See the previous comment for background info and additional instructions for testing reproducibility.

AFAICT this should not impact aufs-based IUKs so it should be safe to merge into iuk.git’s master branch, even if we don’t ship overlayfs in 4.2.

#15 Updated by intrigeri 2019-12-08 09:45:14

  • blocks Feature #15283: Implement the "one single SquashFS diff" scheme in Tails Upgrader added

#16 Updated by segfault 2019-12-12 10:41:10

intrigeri wrote:
> Then:
>
> […]

The wrap_tails_create_iuks command fails with

+ torsocks /usr/bin/wget --no-verbose https://iso-history.tails.boum.org/tails-amd64-4.1/tails-amd64-4.1.iso
Username/Password Authentication Failed.

When I try to access https://iso-history.tails.boum.org in the browser, I also get asked for username and password. Do I have to store the credentials (which I don’t have) somewhere so that wget automatically uses them?

#17 Updated by segfault 2019-12-12 11:44:09

  • Status changed from Needs Validation to Resolved
  • Assignee deleted (segfault)

Code LGTM and the IUK I created had the same sha256sum as yours. Merged into iuk.git’s master.

#18 Updated by intrigeri 2019-12-12 11:46:03

> the IUK I created had the same sha256sum as yours

Woohoo!!! Thanks :)

#19 Updated by intrigeri 2020-01-28 09:06:42

  • % Done changed from 0 to 100