Make the build of overlayfs-based IUKs reproducible
Related to Tails -
Related to Tails -
Related to Tails -
|Blocks Tails - Feature #16209: Core work: Foundations Team||Confirmed|
Blocks Tails -
#5 Updated by intrigeri 2019-12-05 11:30:44
- File test-iuk-reproducibility.sh added
- Status changed from Confirmed to In Progress
My initial best-case test passes: with the attached script I get 2 identical overlayfs-based IUKs on my sid system.
Next step: try to reproduce on a different machine. I’m not sure yet if I’ll do that via
Feature #15287 or if I’ll first do a quick’n’dirty attempt in some more ad-hoc manner.
#8 Updated by intrigeri 2019-12-06 19:03:29
OK, first findings via
- I can’t reproduce the 4.0 to 4.1 IUK we’ve published but it was built using
mksquahfs1:4.3-3.0tails4 so I won’t bother trying harder.
mksquashfsversion matters: even with aufs, I get a 4.0 to 4.1 IUK on sid that’s 22MB smaller than on a stretch isobuilder that has
squashfs-tools1:4.4-1 on my sid system, I can reproduce the 4.0 to 4.1 aufs IUK built on Jenkins.
Next step: try the same thing using overlayfs.
#9 Updated by intrigeri 2019-12-06 20:49:17
> Next step: try the same thing using overlayfs.
- According to diffoscope, the only difference is the size of a bunch of directories, which varies by 12 bytes (sometimes in one direction, sometimes in the other).
- After mounting the embedded
getfattr --dump --match=-returns some xattr for the IUK I’ve built on my sid, while it returns “Operation not supported” for the IUK from Jenkins (Stretch).
I don’t know if these are 2 different problems or only one, but it seems easier to investigate the second first.
When building the SquashFS diff, the read-write branch (upper dir) is a tmpfs and the lower, read-only branch is the old SquashFS; they’re union’ed with overlayfs. So my understanding is that the problem can come from overlayfs, tmpfs, or how the play with each other. tmpfs in the Stretch kernel already has support for xattrs (
getfattr --dump --match=- /run/initramfs/ on one of our Stretch isobuilders returns nothing, but does not complain).
So my current hypothesis is that the difference is caused by overlayfs changes between Linux 4.9 and 5.3, only the latter using xattr on tmpfs, and then when building an IUK on 4.9, no xattr is saved in the tmpfs, mksquashfs detects this and disables xattr support, and thus
getfattr returns “Operation not supported”.
Next steps could be:
- Huge hammer approach: would it break stuff to pass
- Big hammer approach: would it break stuff to remote the overlayfs xattrs from the tmpfs before we run
mksquashfson it? (Ideally, there are some non-overlayfs xattrs we would keep, e.g. on my sid,
- Can we disable xattr support for a specific tmpfs mount?
- Can we specify some overlayfs module or mount options on sid, so it does not use xattrs and thus behaves in a way that’s more similar to Linux 4.9’s overlayfs?
- Compare IUK built on sid with IUK built on Stretch with a newer kernel. Then, if a newer kernel solves the problem:
- Can we ensure all relevant systems (isobuilders, RMs’ systems) have a recent enough kernel to behave consistently with newer kernels?
- Is it worth moving IUK generation to a Vagrant box? That would give us the same kernel everywhere but it adds quite some complexity.
#10 Updated by intrigeri 2019-12-06 21:21:35
> * Huge hammer approach: would it break stuff to pass
This option made the IUK reproducible between Stretch and sid!
Next sub-step in the hope it would validate this cheap workaround: combine 4.0’s
filesystem.squashfs with this IUK’s
4.1.squashfs and see how the resulting overlayfs mountpoint compares to 4.1’s
#12 Updated by intrigeri 2019-12-08 09:13:54
- Status changed from In Progress to Needs Validation
- Type of work changed from Test to Code
> intrigeri wrote:
>> * Huge hammer approach: would it break stuff to pass
> This option made the IUK reproducible between Stretch and sid!
This “huge hammer approach” experiment was useful as it pointed me in the right direction, but after further investigation I’ve found that we should, and can, do better.
First of all, note that our
filesystem.squashfs for 4.0 and 4.1 include some potentially useful xattrs:
file: old-squashfs/bin/ping security.capability=0sAQAAAgAgAAAAAAAAAAAAAAAAAAA= file: old-squashfs/usr/bin/gnome-keyring-daemon security.capability=0sAQAAAgBAAAAAAAAAAAAAAAAAAAA= file: old-squashfs/usr/lib/x86_64-linux-gnu/gstreamer1.0/gstreamer-1.0/gst-ptp-helper security.capability=0sAQAAAgAUAAAAAAAAAAAAAAAAAAA=
… but as soon as I use aufs to combine any of these SquashFS with a tmpfs, these xattrs are lost in the aufs mount; same if I also add an extra SquashFS diff. So our baseline is: no xattrs are visible in a running Tails system, be it freshly installed or upgraded.
overlayfs behaves differently: with no incremental upgrade applied, or with incremental upgrades applied that don’t modify files with xattrs, overlayfs preserves “security.capability” xattrs. So far so good.
But if we generate the IUK with
-no-xattrs and the IUK updates a file that had xattrs in
filesystem.squashfs, then we lose these xattrs in the resulting overlayfs mount. In other words, in some cases, a freshly installed Tails behaves differently than an upgraded one. I’m concerned that the resulting behavior variation can be confusing and make other problems harder to debug, so from this point on I’ve looked into including relevant xattrs in our IUK’s SquashFS, just like they are present in
To achieve this I did this:
- Revert the previous huge hammer approach (passing
--xattrsto the rsync command used to generate the SquashFS diff, otherwise there’s no chance that xattrs on updated files are preserved.
- Remove all overlayfs xattrs before compressing the SquashFS, because they make the read-write upper dir non-deterministic between my sid system and our Stretch isobuilders. I suspect it’s a function of the kernel version; I’d rather not force all our RMs and isobuilders to run the same kernel (and I’m not even sure if that would be sufficient anyway).
- Patch squashfs-tools to make my sid system and our Jenkins (Stretch) isobuilders generate an identical IUK :
- Uploaded 1:4.4-1+0.tails1 to our
- Upstream bug: https://github.com/plougher/squashfs-tools/issues/78
- Debian bug: https://bugs.debian.org/946378
- Uploaded 1:4.4-1+0.tails1 to our
I have tested that the
security.capability xattrs are preserved for the 3 aforementioned files, in an overlayfs mount that combines 4.0’s
- as the 2nd lower dir, a
4.1.squashfsdiff generated with iuk.git from this branch, that artificially updates the content of
/bin/ping(without changing its xattrs)
- a tmpfs upper dir
So I think we’re good! Not only we still have reproducible IUKs, but on top of that we consistently preserve useful xattrs such as
security.capability, which is an improvement over aufs :)
If xattrs ever cause trouble again, we could pass
mksquashfs again, but to avoid the “diverging behavior between freshly installed and upgraded systems” problem, we would have to pass it both in tails.git’s
MKSQUASHFS_OPTIONS (used to generate
filesystem.squashfs) and in iuk.git (used to generate IUKs’ SquashFS diffs).
With this code, our Jenkins (Stretch) isobuilders produce the same 4.0 to 4.1 IUK as my sid system. Still, it would be great if the reviewer could try this on their system and confirm they get the same result too. To do so, follow these instructions:
First, enable this APT source:
deb http://deb.tails.boum.org/ iukbuilder-stretch main
sudo apt update && \ sudo apt install attr bsdtar libarchive-tar-wrapper-perl libcarp-assert-more-perl libcarp-assert-perl libclass-xsaccessor-perl libdevice-cdio-perl libdpkg-perl libfilesys-df-perl libfile-which-perl libfunction-parameters-perl libgnupg-interface-perl libipc-system-simple-perl libmoo-perl libmoox-handlesvia-perl libmoox-late-perl libmoox-options-perl libnamespace-clean-perl libpath-tiny-perl libtry-tiny-perl libtypes-path-tiny-perl libyaml-libyaml-perl libyaml-perl nocache rsync squashfs-tools=1:4.4-1+0.tails1 && \ git clone https://git.tails.boum.org/puppet-tails && \ cd puppet-tails && \ ./files/jenkins/slaves/isobuilders/wrap_tails_create_iuks \ --source-versions 4.0 \ --new-version 4.1 \ --source-date-epoch=1575321815 \ --iuk-commit=c1662d593c6f1f645e1d5b46166716de4e5f80d4 \ --perl5lib-commit=206321a6e4080725f3801763b057545efd3ed844 \ --debug \ --create-iuk-extra-args "--union-type overlayfs" && \ sha256sum Tails_amd64_4.0_to_4.1.iuk
The printed sha256sum should be f2d2fd8bf4a9c3838d0ed5c7c1ffb9e5de4a7c7a0b917219c0497ebca2862af4.
#14 Updated by intrigeri 2019-12-08 09:16:20
- Assignee set to segfault
@segfault, could you please review this? See the previous comment for background info and additional instructions for testing reproducibility.
AFAICT this should not impact aufs-based IUKs so it should be safe to merge into iuk.git’s master branch, even if we don’t ship overlayfs in 4.2.
#16 Updated by segfault 2019-12-12 10:41:10
wrap_tails_create_iuks command fails with
+ torsocks /usr/bin/wget --no-verbose https://iso-history.tails.boum.org/tails-amd64-4.1/tails-amd64-4.1.iso Username/Password Authentication Failed.
When I try to access https://iso-history.tails.boum.org in the browser, I also get asked for username and password. Do I have to store the credentials (which I don’t have) somewhere so that wget automatically uses them?