Feature #8534

Merge incremental upgrades to allow endless upgrading

Added by anonym 2015-01-05 20:45:39 . Updated 2018-02-06 15:35:00 .

Status:
Rejected
Priority:
Low
Assignee:
Category:
Target version:
Start date:
2015-01-05
Due date:
% Done:

0%

Feature Branch:
Type of work:
Research
Blueprint:

Starter:
Affected tool:
Upgrader
Deliverable for:

Description

While investigating overlayfs (Feature #8415) an idea was proposed to work around its filesystem stacking limitation of two that also would have the added benefit of allowing a Tails installation to be incrementally upgraded essentially endlessly, instead of just 4-5 times as currently restricted by the Tails partition size.

The basic idea is to stop stacking the IUKs’ squashfs:es but instead merge the new ones into the old one. To keep down the space requirements (which translates into RAM requirements in our setting) while doing the upgrade we’d want to modify the old squashfs on a file-by-file basis (the CPU (and hence time) requirements would be increased a lot though due to the extensive re-compression needed). We’d have to deal with deleted/whiteout:ed files some how too, of course.

With this approach we’d be able to upgrade Tails endlessly given that the Tails system partition is big enough (around 1 GiB after a “Clone & Install” should go a long way). A further optimization to consider would be to merge the IUK’s squashfs into the filesystem.squashfs to eliminate incremental upgrade disk overhead completely (then only a few 100 MiB should be enough for a long time, i.e. we could significantly decrease the current size of the Tails system partition).

A serious issue is that we may not be able to use squashfs any more since mksquashfs seemingly can only append new files to an existing squashfs archive, but neither remove files nor modify existing ones. In addition to squashfs the only filesystem live-boot supports that have compression is JFFS2, which perhaps is an alternative. Other filesystems with compression support could be considered although it’d require us to add support for them to live-boot.

It should be noted that verifying the integrity of a Tails installation after installing incremental upgrades will become a lot harder after this.


Subtasks


History

#1 Updated by intrigeri 2015-01-06 10:23:52

  • Category deleted (178)
  • Affected tool set to Upgrader

#2 Updated by sajolida 2015-01-06 15:09:30

  • Subject changed from Merge incremental upgrades to allow enldess upgrading to Merge incremental upgrades to allow endless upgrading

#3 Updated by bertagaz 2015-01-06 18:19:33

While this feature could be interesting, I wonder how much we want users to endlessly upgrade their Tails with iuks.

We have sometimes discussed about the interest to know the last time everyone did an upgrade from an ISO (like when boum.org changes their SSL certificate for example), to reset everyone’s state on the same ground.

#4 Updated by anonym 2015-01-06 19:10:51

bertagaz wrote:
> While this feature could be interesting, I wonder how much we want users to endlessly upgrade their Tails with iuks.

The “endless upgrades” part is just the potential. We’d likely find reasons from time to time to force a fresh install, e.g. when upgrading to a new Debian release, or when there’s partition level changes that IUKs cannot handle.

> We have sometimes discussed about the interest to know the last time everyone did an upgrade from an ISO (like when boum.org changes their SSL certificate for example), to reset everyone’s state on the same ground.

It’d be trivial to record the Tails version of the original installation and which IUKs has been applied in a text file on the Tails system partition.

Note that the current situation we’re in is not very clear either. Given the approximate free space remaining after an installation, and the average IUK squashfs size, and the minimum extra free space we require (for safety reasons) we can expect about ~5 incremental upgrades to be possible after a fresh installation. However, users may do their fresh installation at any version, so we cannot ever assume that we have a situation where everyone’s state is the same. In fact, if we could merge the IUKs squashfs into the filesystem.squashfs, we’d end up with installations that are much more similar to each other, modulo timestamps and ordering of the files inside the squashfs (like I noted above, we likely cannot use squashfs for merging, but I’m keeping on using it for clarity).

#5 Updated by bertagaz 2015-01-06 20:23:39

Ok, thanks for the clarification, I’m pretty convinced now. :)

#6 Updated by anonym 2015-01-07 02:31:31

This whole idea might still be a pipe dream unless we can find a way to efficiently recompress the Tails filesystem while merging upgrades into it. That includes that the end result still has a nice compression ratio, and that more or less no extra disk space is required when doing the recompression. For the latter property I think we pretty much need a filesystem with transparent compression (like btrfs). Or would a tar archive do with the --append and --delete options, or does it write the updated archive as a completely new file? Without trashing the compression ratio? Then we’d have to add something like archivemount support to live-boot, so it can mount a filesystem.tar.xz or similar as a read-only lower layer, but that shouldn’t be very hard.

#7 Updated by sajolida 2015-01-07 09:44:58

Is it really required to compress the filesystem? I mean, we have 2.5GiB allocated for that Tails partition, so if we can do without compression, then it will be easier for the user to upgrade and it might be easier to compare cryptographically.

How much is an uncompressed Tails filesystem?

#8 Updated by intrigeri 2015-01-07 10:19:28

sajolida wrote:
> How much is an uncompressed Tails filesystem?

On the last nightly build of the stable branch, SquashFS compresses to 28.37% of the original filesystem size:

  • compressed = 894786.72 Kbytes
  • uncompressed = 3153629.26 Kbytes

(I’m ignoring the inode and directory tables size, which are neglictible compared to the data.)

I expect the ratio to be very similar for IUKs, so an uncompressed IUK would be ~900 MiB.

#9 Updated by sajolida 2015-01-08 11:57:58

That’s well compressed indeed!

So if we want to have the uncompressed original filesystem + uncompressed IUK + uncompressed destination filesystem while compiling the result, then we need ~7GB. That could maybe be split between RAM and flash somehow, but that’s a lot indeed.

Note that if we want to reduce that amount of data to fit temporarily on the 2.5G of the system partition we need at least 35% of compression. So that would be tight anyway.

But maybe I’m interpreting the whole thing wrong…

#10 Updated by tchou 2015-02-03 10:33:27

  • related to Feature #7499: Extend the upgrader to allow full (self) upgrade added

#11 Updated by tchou 2015-02-03 10:35:35

This ticket can be related to Feature #7499. Improvements on one of both could solve the same kind of use-cases behind this tickets.

#12 Updated by sajolida 2015-02-07 13:25:40

Actually, Feature #7499 would be much easier to implement that Feature #8534 and would result in the same practical improvements: being able to upgrade Tails from Tails endlessly. Still, Feature #7499 would require to download more data when forced to do a full upgrade but I would say that the usability and security improvements are the major issues here and are resolved either way.

#13 Updated by anonym 2016-02-16 14:44:03

  • related to deleted (Feature #7499: Extend the upgrader to allow full (self) upgrade)

#14 Updated by anonym 2016-02-16 14:44:11

#15 Updated by intrigeri 2018-02-03 15:40:57

anonym, I think this is not a blocker for Feature #11131 so should not be a subtask. We could even decide to reject it at some point during our sprint.

#16 Updated by intrigeri 2018-02-06 15:35:00

  • Status changed from Confirmed to Rejected

Same as Feature #11131.