Bug #12032

The SquashFS creation is not deterministic

Added by intrigeri 2016-12-14 13:03:02 . Updated 2017-03-15 06:00:30 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Start date:
2016-12-14
Due date:
% Done:

100%

Feature Branch:
Type of work:
Research
Blueprint:

Starter:
Affected tool:
Deliverable for:
289

Description

This applies both to the SquashFS in the ISO and the one in IUKs. I’m told that -no-fragments should fix it, but:

  • this might make the compression rate worse;
  • one might need to pass a fixed value to -processors too.

Contact info of the people investigating this: lynxis@fe80.eu and hanno@hboeck.de.

Subtasks

Related issues

Related to Tails - Feature #11974: Reproducible IUK builds Resolved 2016-11-20
Blocked by Tails - Feature #11966: Reproducible website build Resolved 2016-11-19

History

#1 Updated by intrigeri 2016-12-14 13:03:39

  • Description updated

#2 Updated by intrigeri 2016-12-14 14:53:00

#4 Updated by intrigeri 2017-03-13 11:08:08

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10
  • Deliverable for set to 289

intrigeri wrote:
> Next step: test with latest patchset applied by lynxyis

uploaded squashfs-tools 4.3-3.0tails4 that has this patchset + our own not-merged-yet patches on top.

#5 Updated by intrigeri 2017-03-14 06:41:04

#6 Updated by intrigeri 2017-03-14 12:12:39

  • Type of work changed from Test to Research

According to diff -r --brief --no-dereference, there is no remaining difference in the content of the files stored in the SquashFS! :) Which means we are now in a much better position to work on this ticket.

Sadly, even with squashfs-tools 4.3-3.0tails4 the SquashFS is not reproducible yet, regardless of whether we pass -no-fragments. FTR the other options passed to mksquashfs are -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -no-progress -sort squashfs.sort.

Next steps:

  • report this to Alexander Couzens, asking if there’s another set of options we should try
  • ensure deterministic sorting of files in the SquashFS, by generating a squashfs.sort file in a deterministic manner and appending it to squashfs.sort (I have WIP about this locally)

#7 Updated by intrigeri 2017-03-14 12:16:02 

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 1048576
        compressed data, compressed metadata, compressed fragments, compressed xattrs
        duplicates are removed
Filesystem size 1189902.38 Kbytes (1162.01 Mbytes)
        30.32% of uncompressed filesystem size (3924527.33 Kbytes)
Inode table size 1108786 bytes (1082.80 Kbytes)
        21.20% of uncompressed inode table size (5230321 bytes)
Directory table size 1390142 bytes (1357.56 Kbytes)
        34.74% of uncompressed directory table size (4001427 bytes)
Xattr table size 78 bytes (0.08 Kbytes)
        97.50% of uncompressed xattr table size (80 bytes)

vs.

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 1048576
        compressed data, compressed metadata, compressed fragments, compressed xattrs
        duplicates are removed
Filesystem size 1189902.42 Kbytes (1162.01 Mbytes)
        30.32% of uncompressed filesystem size (3924527.33 Kbytes)
Inode table size 1109362 bytes (1083.36 Kbytes)
        21.21% of uncompressed inode table size (5230321 bytes)
Directory table size 1390062 bytes (1357.48 Kbytes)
        34.74% of uncompressed directory table size (4001427 bytes)
Xattr table size 78 bytes (0.08 Kbytes)
        97.50% of uncompressed xattr table size (80 bytes)

#8 Updated by intrigeri 2017-03-14 12:35:16

intrigeri wrote:
> According to diff -r --brief --no-dereference, there is no remaining difference in the content of the files stored in the SquashFS! :)

I’ve looked more closely and while this is true, the inode data differs: in particular, the mtime of files that come from config/chroot_local-includes/ varies; as a consequence, of course the mtime of their parent directory varies too. Also, the uid/gid of /etc/resolv.conf vary. I’ll file tickets about those.

#9 Updated by intrigeri 2017-03-15 06:00:30

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 10 to 100

I’ll report success back to the new squashfs-tools upstream.