Feature #9373

Make tails-iuk support overlayfs

Added by intrigeri 2015-05-11 20:05:56 . Updated 2020-01-28 09:05:31 .

Status:
Resolved
Priority:
High
Assignee:
Category:
Installation
Target version:
Tails_4.5
Start date:
Due date:
% Done:

100%

Feature Branch:
iuk:feature/9373-overlayfs
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Upgrader
Deliverable for:

Description

The IUK creation process and its test suite both rely on aufs now. Whenever we’ll be ready to use overlayfs instead everywhere else, these two pieces of code will need to support overlayfs too. A first step could be to make this configurable, support both, so that there’s less of a flag day.

Subtasks

Related issues

Related to Tails - Feature #6876: Have the incremental upgrade process use less RAM Resolved
Related to Tails - Feature #17262: Make the build of overlayfs-based IUKs reproducible Resolved
Blocked by Tails - Feature #8083: Fix automatic upgrades on Jessie Resolved 2014-10-12
Blocks Tails - Feature #12106: Adjust test suite to overlayfs Resolved 2017-01-02
Blocked by Tails - Feature #8473: Add support to live-boot to support multiple read-only lower layers with overlayfs Resolved 2014-12-21
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed
Blocked by Tails - Feature #17152: Port tails-iuk to a more lightweight set of dependencies Resolved

History

#1 Updated by intrigeri 2015-06-12 22:01:03

  • blocked by Feature #8083: Fix automatic upgrades on Jessie added

#2 Updated by intrigeri 2015-07-13 03:50:02

  • Target version set to Sustainability_M1

#3 Updated by intrigeri 2015-07-19 02:00:08

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10
  • Feature Branch set to iuk:feature/9373-overlayfs

Initial porting done, test suite passes.

Next step: try installing the resulting IUK on a Tails built with overlayfs support.

#4 Updated by sajolida 2015-09-07 10:44:22

  • Target version changed from Sustainability_M1 to 2016

#5 Updated by intrigeri 2016-08-20 10:52:41

  • Assignee deleted (intrigeri)
  • Target version deleted (2016)

Given we could do Feature #10298 without migrating to overlayfs, we removed this from our roadmap at the summit this year.

#6 Updated by intrigeri 2017-01-02 17:40:17

> Next step: try installing the resulting IUK on a Tails built with overlayfs support.

Now that we have automated tests for incremental upgrades, this will be easier; one “just” needs to:

  1. generate and upload an IUK that’s essentially the same as the one we use currently in the test suite, except it must be generated with overlayfs
  2. add UDFs for 1.0~testoverlay and 1.1~testoverlay
  3. s/1.0~test/1.0~testoverlay/ in the test suite
  4. s/1.1~test/1.1~testoverlay/ in the test suite
  5. run the test suite

#7 Updated by intrigeri 2017-03-11 12:33:40

Also, we’ll probably need to bump the IUK format version number, and ensure we don’t generate a Tails.module file that mixes aufs and overlayfs SquashFS diffs.

#8 Updated by Anonymous 2018-01-15 11:10:21

  • Assignee set to anonym

Looks like this is part of our roadmap. Assigning to anonym who also committed to work on the parent ticket Feature #8415.

#9 Updated by intrigeri 2018-08-18 08:52:22

  • related to Feature #6876: Have the incremental upgrade process use less RAM added

#10 Updated by intrigeri 2018-08-18 08:54:04

intrigeri wrote:
> Also, we’ll probably need to bump the IUK format version number, and ensure we don’t generate a Tails.module file that mixes aufs and overlayfs SquashFS diffs.

In order to avoid breaking automatic upgrades between Tails N and N+1 more often than needed, ideally we should implement Feature #6876 at the same time and release all this in Tails 4.0.

#11 Updated by intrigeri 2018-09-12 06:51:44

  • Assignee changed from anonym to intrigeri
  • Target version set to Tails_3.11

#12 Updated by intrigeri 2018-09-12 06:51:53

#13 Updated by intrigeri 2018-09-12 06:52:07

#14 Updated by intrigeri 2018-09-12 06:54:34

  • blocked by Feature #8473: Add support to live-boot to support multiple read-only lower layers with overlayfs added

#15 Updated by intrigeri 2018-11-05 14:45:46

  • Target version changed from Tails_3.11 to Tails_3.12

#16 Updated by intrigeri 2018-11-06 15:04:44

  • Target version changed from Tails_3.12 to Tails_3.13

#17 Updated by intrigeri 2018-12-10 09:40:41

  • blocked by deleted (Feature #15506: Core work 2018Q4: Foundations Team)

#18 Updated by intrigeri 2018-12-10 09:40:51

#19 Updated by intrigeri 2019-01-25 16:31:56

  • Target version changed from Tails_3.13 to 2019

#20 Updated by intrigeri 2019-02-06 14:06:35

  • blocked by deleted (Feature #15507: Core work 2019Q1: Foundations Team)

#21 Updated by intrigeri 2019-02-06 14:06:38

#22 Updated by intrigeri 2019-08-30 20:50:33

  • Status changed from In Progress to Confirmed

(Not much progress lately. I’ll work on this later this year or early 2020.)

#23 Updated by intrigeri 2019-11-23 10:15:49

  • blocked by Feature #17152: Port tails-iuk to a more lightweight set of dependencies added

#24 Updated by intrigeri 2019-11-23 14:45:08

  • Status changed from Confirmed to In Progress

Refreshed the branch, merged Feature #17152 into it, and brought back aufs support ⇒ the iuk.git test suite passes both with UNION_TYPE=aufs (which is the default) and UNION_TYPE=overlayfs.

Next step: have the tails.git automated test suite exercise upgrading with an overlayfs-based IUK.

#25 Updated by intrigeri 2019-11-23 17:02:57

intrigeri wrote:
> Next step: have the tails.git automated test suite exercise upgrading with an overlayfs-based IUK.

I’ve done everything I could do today on this front:

  • prepared Tails_amd64_1.0~testoverlayfs_to_1.1~testoverlayfs.iuk, that uses overlayfs, using tails-create-iuk + manual post-processing to make it closer to Tails_amd64_1.0~test_to_1.1~test.iuk
  • manually installed this IUK with tails-install-iuk, rebooted, confirmed the changes are applied as expected
  • uploaded this IUK to rsync.lizard
  • updated the test suite to use this overlayfs-based IUK
  • pushed UDFs for 1.0~testoverlayfs and 1.1~testoverlayfs to our master branch

Next steps:

  1. sign these 4 new UDFs
  2. run Scenario: Upgrading Tails with Tails Upgrader through an incremental upgrade

#26 Updated by intrigeri 2019-12-01 10:59:39

  • Target version changed from 2019 to Tails_4.5

The milestone for this is “March 2020” so the current goal is to have this ready in time for 4.5~rc1. We might manage to complete Feature #8415 earlier, we’ll see.

#27 Updated by intrigeri 2019-12-01 11:20:28

  • Priority changed from Elevated to High

#28 Updated by intrigeri 2019-12-05 11:18:42

Another next step: ensure the test suites (in both iuk.git and tails.git) exercise whiteouts:

  • for non-directory deletion (in theory, managed by overlayfs with a xattr to make it opaque)
  • for directory deletion (managed by overlayfs with character devices or xattr)

#29 Updated by intrigeri 2019-12-05 13:12:02

intrigeri wrote:
> Another next step: ensure the test suites (in both iuk.git and tails.git) exercise whiteouts:
>
> * for non-directory deletion (in theory, managed by overlayfs with a xattr to make it opaque)

Interestingly, in my tests I see whiteouts (character device with 0/0 device number) for deleted directories too. I could not find any such xattr usage in the overlayfs-based IUKs I’ve generated.

Anyway, this is exercised already.

  • in tails.git: Tails_amd64_1.0~testoverlayfs_to_1.1~testoverlayfs.iuk deletes /usr/share/common-licenses/BSD
  • in iuk.git via Scenario: install an IUK that should delete some files

> * for directory deletion (managed by overlayfs with character devices or xattr)

AFAICT we have no test for this yet. I should write one, at least in iuk.git.

#30 Updated by intrigeri 2019-12-05 13:14:52

intrigeri wrote:
> intrigeri wrote:
> > Another next step: ensure the test suites (in both iuk.git and tails.git) exercise whiteouts:
> >
> > * for non-directory deletion (in theory, managed by overlayfs with a xattr to make it opaque)
>
> Interestingly, in my tests I see whiteouts (character device with 0/0 device number) for deleted directories too. I could not find any such xattr usage in the overlayfs-based IUKs I’ve generated.
>
> Anyway, this is exercised already:
>
> * in tails.git: Tails_amd64_1.0~testoverlayfs_to_1.1~testoverlayfs.iuk deletes /usr/share/common-licenses/BSD

Confirmed → good enough.

> * in iuk.git via Scenario: install an IUK that should delete some files

Scratch that, this scenario only tests deletion of files in the system partition, not via the SquashFS diff. So while adding a test that deletes stuff via the SquashFS diff, I should test both directory and non-directory deletion.

#31 Updated by intrigeri 2019-12-05 17:18:09

Tails_amd64_1.0~testoverlayfs_to_1.1~testoverlayfs.iuk (just uploaded, not on the mirrors yet) now deletes a whole directory (recursively), on top of deleting a single regular file (which it did already). I’m adjusting the corresponding test in tails.git so it verifies that this new change is indeed applied upon upgrade.

So I’m back to next step: run Scenario: Upgrading Tails with Tails Upgrader through an incremental upgrade.

#32 Updated by intrigeri 2019-12-06 07:18:30

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (intrigeri)

intrigeri wrote:
> So I’m back to next step: run Scenario: Upgrading Tails with Tails Upgrader through an incremental upgrade.

It passes on my machine.

I’ve verified that the test suite in iuk.git still passes both with UNION_TYPE=aufs (which is the default) and UNION_TYPE=overlayfs. I’ve also verified that setting $UNION_TYPE works correctly (unloading both aufs and overlay kernel modules before running the test suite and verifying that only the required one was loaded after it has run).

So I think we’re good here!

Note to the reviewer: this branch includes Feature #17152, which anonym reviewed already, so you can skip that part and compare this topic branch with “current master + Feature #17152 merged in”.

#33 Updated by intrigeri 2019-12-06 07:21:20

  • related to Feature #17262: Make the build of overlayfs-based IUKs reproducible added

#34 Updated by intrigeri 2019-12-06 09:38:22

  • Status changed from Needs Validation to In Progress

Applied in changeset commit:tails|d79bb63efbf1e8d2cae054ba0638ff43d867657d.

#35 Updated by intrigeri 2019-12-06 11:45:46

  • Status changed from In Progress to Needs Validation

#36 Updated by segfault 2019-12-07 12:49:08

  • Assignee set to segfault

#37 Updated by segfault 2019-12-07 13:12:14

  • Status changed from Needs Validation to Resolved
  • Assignee deleted (segfault)

#38 Updated by intrigeri 2020-01-28 09:05:31

  • % Done changed from 0 to 100