Feature #6425

Do not save some large files whose only modification is mtime, for smaller IUK

Added by intrigeri 2013-11-16 13:58:50 . Updated 2014-03-19 07:34:18 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Tails_0.23
Start date:
2013-11-16
Due date:
% Done:

100%

Feature Branch:
feature/better-iso-cleanup
Type of work:
Code
Blueprint:

Starter:
0
Affected tool:
Upgrader
Deliverable for:

Description

E.g. /usr/lib/locale/locale-archive (104MB) doesn’t change between 0.22~rc1 and 0.22~rc2, but it still is shipped in the corresponding IUK.

winterfairy has built a 27% smaller IUK by passing -rlpgoD --checksum to rsync instead of --archive.
See “Smaller IUK packages” thread on tails-dev (November 2013).

Subtasks

Related issues

Related to Tails - Bug #11211: Don't include files from /usr/share/mime in IUKs if their only modification is mtime Duplicate 2016-03-09
Related to Tails - Feature #11345: Ship less unneeded cache/generated files in the ISO and IUK Duplicate 2016-04-13
Related to Tails - Feature #15290: Reduce IUK size Resolved 2016-04-13

History

#1 Updated by winterfairy 2013-11-20 04:16:51

The title is somewhat misleading, that command will completely skip copying files those only modification is the mtime. The current behaviour copies the whole files even if the actual data is the same, only the mtime changed.

#2 Updated by intrigeri 2013-11-20 08:13:43

  • Subject changed from Investigate not saving mtime for smaller IUK to Investigate not saving files whose only modification is mtime, for smaller IUK

#3 Updated by intrigeri 2013-11-27 09:01:22

#4 Updated by intrigeri 2014-01-10 12:41:21

  • Feature Branch set to feature/6425-smaller-iuks

Implemented in the iuk repository (feature/6425-smaller-iuks branch), tested.

The upgraded system has at least one weird behaviour: GDM’s background is the default Debian one, instead of the one we’ve configured; GDM logs say that gnome-settings-daemon failed to register in time.

So, probably some software compares some files’ mtime to decide whether to do this or that, and we’re breaking this logic with this clever trick somehow. I’m expecting more breakage is caused by such things, that I didn’t notice in my very quick test.

Unfortunately, our current automated test suite is not able to run on an arbitrary USB stick, so we cannot easily evaluate to what extent the resulting system is broken.

I’m unsure what the next steps should be: once it’s clear that the mtime metadata is part of what makes a given Debian system consistent and working well, IMO it would not be scalable nor robust to merely build a list of what files need to see their mtime preserved.

So perhaps we should instead build a list of files that make the IUKs much larger for no good reasons, due to their mtime changing, and only skip updating these ones? The aforementionned /usr/lib/locale/locale-archive might be a good candidate. Presumably, it would be enough to reset these files’ mtime to epoch at ISO build time => it’s the same in every Tails release => as long as size doesn’t change, rsync will skip them.

#5 Updated by intrigeri 2014-01-10 12:41:34

  • % Done changed from 0 to 10

#6 Updated by intrigeri 2014-03-05 13:38:21

Some large files whose mtime has changed, but content has not, between Tails 0.22 and an ISO built shortly before 0.23~rc1:

  • 16M var/lib/anthy/anthy.dic
  • 15M var/lib/anthy/mkworddic/anthy.wdic
  • 38M usr/share/ppd/hplip/HP/*.ppd
  • 104M usr/lib/locale/locale-archive

#7 Updated by intrigeri 2014-03-05 17:34:25

  • Assignee set to intrigeri
  • % Done changed from 10 to 30
  • Feature Branch changed from feature/6425-smaller-iuks to feature/better-iso-cleanup

#8 Updated by intrigeri 2014-03-05 17:42:43

Skipping the aforementioned files if their content has not changed saves 21MB (9%) on the 0.22 to 0.22.1 IUK.

#9 Updated by intrigeri 2014-03-05 19:10:40

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to anonym
  • Target version set to Tails_0.23
  • % Done changed from 30 to 50
  • QA Check set to Ready for QA

#10 Updated by intrigeri 2014-03-06 00:00:43

  • Subject changed from Investigate not saving files whose only modification is mtime, for smaller IUK to Do not save some large files whose only modification is mtime, for smaller IUK
  • Type of work changed from Research to Code

#11 Updated by anonym 2014-03-06 01:25:15

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 50 to 100
  • QA Check changed from Ready for QA to Pass

Merged!

#12 Updated by anonym 2014-03-19 07:34:18

  • Status changed from Fix committed to Resolved

#13 Updated by intrigeri 2016-03-09 16:33:56

  • related to Bug #11211: Don't include files from /usr/share/mime in IUKs if their only modification is mtime added

#14 Updated by intrigeri 2017-09-27 09:24:03

  • related to Feature #11345: Ship less unneeded cache/generated files in the ISO and IUK added

#15 Updated by intrigeri 2019-08-31 16:00:26