Feature #11345

Ship less unneeded cache/generated files in the ISO and IUK

Added by intrigeri 2016-04-13 06:25:07 . Updated 2019-08-31 16:01:13 .

Status:
Duplicate
Priority:
Low
Assignee:
Category:
Installation
Target version:
Start date:
2016-04-13
Due date:
% Done:

10%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Upgrader
Deliverable for:

Description

To test the most up-to-date ideas in this thread:

  1. Fork the tag for Tails version N, apply the 99-set_mtimes trick. Build an ISO.
  2. Do the same for Tails version N+1.
  3. Build the IUK from N to N+1 using these ISOs.
  4. Compare the size of the IUK you’ve built with the size of the IUK we’ve published from N to N+1.

Subtasks

Related issues

Related to Tails - Bug #11211: Don't include files from /usr/share/mime in IUKs if their only modification is mtime Duplicate 2016-03-09
Related to Tails - Feature #6425: Do not save some large files whose only modification is mtime, for smaller IUK Resolved 2013-11-16
Is duplicate of Tails - Feature #15290: Reduce IUK size Resolved 2016-04-13

History

#1 Updated by intrigeri 2017-06-05 16:47:52

  • Status changed from Confirmed to In Progress
  • Target version set to Tails_3.3
  • % Done changed from 0 to 10

intrigeri wrote:
> Things we might want to stop including in the SquashFS:
>
> * /var/cache/fontconfig/

We tried that during the reproducible builds development, seen weird test suite failures & hard to analyze performance impact, so decided to keep it in the ISO (and make it reproducible).

> * /var/cache/ldconfig/aux-cache

Now removed in config/chroot_local-hooks/99-zzzzzz_reproducible-builds-post-processing.

> More can probably be found by looking at the content of a recent IUK.

Still the case, so I’m leaving this open. I’ll take a quick look at the 3.0 → 3.1 and 3.1 → 3.2 IUKs.

#2 Updated by intrigeri 2017-09-27 09:23:37

  • Subject changed from Ship less unneeded cache files in the ISO and IUK to Ship less unneeded cache/generated files in the ISO and IUK
  • Priority changed from Normal to Low
  • Target version deleted (Tails_3.3)
  • Type of work changed from Research to Code

Looking at Tails_amd64_3.2~rc1_to_3.2.iuk and Tails_amd64_3.1_to_3.2.iuk, I’ve spotted (sizes are uncompressed data):

  • /var/cache/debconf/, 3.2M: we need it in the ISO/IUK but if the content does not change often, the config/chroot_local-hooks/99-set_mtimes trick could be used
  • /var/lib/dkms/, 1.2M: there’s source code and compiled modules in there, probably due to commit:bdcfaad52e2cf7cb01a7134a910477469a5e089c aka. “WIP” (sic); I doubt I really intended to submit that change as-is for merging, and I bet it can be reverted.
  • /var/lib/dpkg/info, 7.9M: the config/chroot_local-hooks/99-set_mtimes trick could be used

Everything else is either too small to be worth spending any time on, or needed in the IUK :)

My take on it is that there are a few nice low hanging fruits, but all that stuff probably compresses pretty well with SquashFS, so the cost/benefit of working further on this is pretty low. So I’ll keep this on my low-priority plate for a rainy day if I’m bored, but anyone should feel free to steal it from me :)

#3 Updated by intrigeri 2017-09-27 09:23:52

  • related to Bug #11211: Don't include files from /usr/share/mime in IUKs if their only modification is mtime added

#4 Updated by intrigeri 2017-09-27 09:24:02

  • related to Feature #6425: Do not save some large files whose only modification is mtime, for smaller IUK added

#5 Updated by anonym 2018-02-05 16:28:38

#6 Updated by intrigeri 2019-03-07 15:42:20

  • Assignee deleted (intrigeri)

#7 Updated by intrigeri 2019-08-31 15:47:59

intrigeri wrote:
> * /var/cache/debconf/, 3.2M: we need it in the ISO/IUK but if the content does not change often, the config/chroot_local-hooks/99-set_mtimes trick could be used

Yes. Most changes in there are very likely to change the size of the files, so even if the mtime is the same, rsync will consider them.

> * /var/lib/dkms/, 1.2M: there’s source code and compiled modules in there, probably due to commit:bdcfaad52e2cf7cb01a7134a910477469a5e089c aka. “WIP” (sic); I doubt I really intended to submit that change as-is for merging, and I bet it can be reverted.

Still true. And then we should remove the dkms bits from config/chroot_local-hooks/99-zzzzzz_reproducible-builds-post-processing.

> * /var/lib/dpkg/info, 7.9M: the config/chroot_local-hooks/99-set_mtimes trick could be used

Hmmm… I see some potential here for files whose contents changes without the size of the file changing, which will make rsync skip that file, and the upgraded system still have the old version. This could affect maintainer scripts, e.g. a one-byte typo fixed in a .postinst script. But in the context of Tails, this won’t matter in practice, as long as users don’t deinstall/upgrade packages that are already shipped in Tails (which our doc strongly recommends against). So yeah, let’s try it.

#8 Updated by intrigeri 2019-08-31 15:53:14

  • Description updated

#9 Updated by intrigeri 2019-08-31 16:01:13

  • Status changed from In Progress to Duplicate
  • Parent task deleted (Feature #15290)

#10 Updated by intrigeri 2019-08-31 16:01:35