Merge incremental upgrades to allow endless upgrading
While investigating overlayfs (
Feature #8415) an idea was proposed to work around its filesystem stacking limitation of two that also would have the added benefit of allowing a Tails installation to be incrementally upgraded essentially endlessly, instead of just 4-5 times as currently restricted by the
Tails partition size.
The basic idea is to stop stacking the IUKs’ squashfs:es but instead merge the new ones into the old one. To keep down the space requirements (which translates into RAM requirements in our setting) while doing the upgrade we’d want to modify the old squashfs on a file-by-file basis (the CPU (and hence time) requirements would be increased a lot though due to the extensive re-compression needed). We’d have to deal with deleted/whiteout:ed files some how too, of course.
With this approach we’d be able to upgrade Tails endlessly given that the
Tails system partition is big enough (around 1 GiB after a “Clone & Install” should go a long way). A further optimization to consider would be to merge the IUK’s squashfs into the
filesystem.squashfs to eliminate incremental upgrade disk overhead completely (then only a few 100 MiB should be enough for a long time, i.e. we could significantly decrease the current size of the
Tails system partition).
A serious issue is that we may not be able to use squashfs any more since
mksquashfs seemingly can only append new files to an existing squashfs archive, but neither remove files nor modify existing ones. In addition to squashfs the only filesystem live-boot supports that have compression is JFFS2, which perhaps is an alternative. Other filesystems with compression support could be considered although it’d require us to add support for them to live-boot.
It should be noted that verifying the integrity of a Tails installation after installing incremental upgrades will become a lot harder after this.
#3 Updated by bertagaz 2015-01-06 18:19:33
While this feature could be interesting, I wonder how much we want users to endlessly upgrade their Tails with iuks.
We have sometimes discussed about the interest to know the last time everyone did an upgrade from an ISO (like when boum.org changes their SSL certificate for example), to reset everyone’s state on the same ground.
#4 Updated by anonym 2015-01-06 19:10:51
> While this feature could be interesting, I wonder how much we want users to endlessly upgrade their Tails with iuks.
The “endless upgrades” part is just the potential. We’d likely find reasons from time to time to force a fresh install, e.g. when upgrading to a new Debian release, or when there’s partition level changes that IUKs cannot handle.
> We have sometimes discussed about the interest to know the last time everyone did an upgrade from an ISO (like when boum.org changes their SSL certificate for example), to reset everyone’s state on the same ground.
It’d be trivial to record the Tails version of the original installation and which IUKs has been applied in a text file on the Tails system partition.
Note that the current situation we’re in is not very clear either. Given the approximate free space remaining after an installation, and the average IUK squashfs size, and the minimum extra free space we require (for safety reasons) we can expect about ~5 incremental upgrades to be possible after a fresh installation. However, users may do their fresh installation at any version, so we cannot ever assume that we have a situation where everyone’s state is the same. In fact, if we could merge the IUKs squashfs into the
filesystem.squashfs, we’d end up with installations that are much more similar to each other, modulo timestamps and ordering of the files inside the squashfs (like I noted above, we likely cannot use squashfs for merging, but I’m keeping on using it for clarity).
#6 Updated by anonym 2015-01-07 02:31:31
This whole idea might still be a pipe dream unless we can find a way to efficiently recompress the Tails filesystem while merging upgrades into it. That includes that the end result still has a nice compression ratio, and that more or less no extra disk space is required when doing the recompression. For the latter property I think we pretty much need a filesystem with transparent compression (like btrfs). Or would a tar archive do with the
--delete options, or does it write the updated archive as a completely new file? Without trashing the compression ratio? Then we’d have to add something like
archivemount support to live-boot, so it can mount a
filesystem.tar.xz or similar as a read-only lower layer, but that shouldn’t be very hard.
#7 Updated by sajolida 2015-01-07 09:44:58
Is it really required to compress the filesystem? I mean, we have 2.5GiB allocated for that Tails partition, so if we can do without compression, then it will be easier for the user to upgrade and it might be easier to compare cryptographically.
How much is an uncompressed Tails filesystem?
#8 Updated by intrigeri 2015-01-07 10:19:28
> How much is an uncompressed Tails filesystem?
On the last nightly build of the stable branch, SquashFS compresses to 28.37% of the original filesystem size:
- compressed = 894786.72 Kbytes
- uncompressed = 3153629.26 Kbytes
(I’m ignoring the inode and directory tables size, which are neglictible compared to the data.)
I expect the ratio to be very similar for IUKs, so an uncompressed IUK would be ~900 MiB.
#9 Updated by sajolida 2015-01-08 11:57:58
That’s well compressed indeed!
So if we want to have the uncompressed original filesystem + uncompressed IUK + uncompressed destination filesystem while compiling the result, then we need ~7GB. That could maybe be split between RAM and flash somehow, but that’s a lot indeed.
Note that if we want to reduce that amount of data to fit temporarily on the 2.5G of the system partition we need at least 35% of compression. So that would be tight anyway.
But maybe I’m interpreting the whole thing wrong…
#12 Updated by sajolida 2015-02-07 13:25:40
Feature #7499 would be much easier to implement that Feature #8534 and would result in the same practical improvements: being able to upgrade Tails from Tails endlessly. Still, Feature #7499 would require to download more data when forced to do a full upgrade but I would say that the usability and security improvements are the major issues here and are resolved either way.