Tails

#2 Updated by Anonymous 2017-03-15 13:38:13

give me the shaaa!

#3 Updated by lamby 2017-03-15 13:39:55

.oO ( Or maybe sorting a bunch of stuff via deterministic modes means that’s being less efficient with inodes or something… )

#4 Updated by intrigeri 2017-03-15 13:54:01

> give me the shaaa!

Compression details found in the build logs (both with 4 CPUs):

feature/5630-deterministic-builds:

Creating 4.0 filesystem on filesystem.squashfs, block size 1048576.

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 1048576
    compressed data, compressed metadata, compressed fragments, compressed xattrs
    duplicates are removed
Filesystem size 1188906.36 Kbytes (1161.04 Mbytes)
    30.30% of uncompressed filesystem size (3923494.69 Kbytes)
Inode table size 1108626 bytes (1082.64 Kbytes)
    21.20% of uncompressed inode table size (5230220 bytes)
Directory table size 1389566 bytes (1357.00 Kbytes)
    34.73% of uncompressed directory table size (4001042 bytes)
Xattr table size 78 bytes (0.08 Kbytes)
    97.50% of uncompressed xattr table size (80 bytes)
Number of duplicate files found 6878
Number of inodes 150113
Number of files 119805
Number of fragments 2228
Number of symbolic links  18054
Number of device nodes 6
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 12248

feature/stretch:

Creating 4.0 filesystem on filesystem.squashfs, block size 1048576.

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 1048576
    compressed data, compressed metadata, compressed fragments, compressed xattrs
    duplicates are removed
Filesystem size 1089689.83 Kbytes (1064.15 Mbytes)
    27.55% of uncompressed filesystem size (3955809.15 Kbytes)
Inode table size 1136786 bytes (1110.14 Kbytes)
    21.47% of uncompressed inode table size (5294329 bytes)
Directory table size 1406442 bytes (1373.48 Kbytes)
    34.58% of uncompressed directory table size (4067027 bytes)
Xattr table size 78 bytes (0.08 Kbytes)
    97.50% of uncompressed xattr table size (80 bytes)
Number of duplicate files found 6878
Number of inodes 152111
Number of files 121782
Number of fragments 2244
Number of symbolic links  18052
Number of device nodes 6
Number of fifo nodes 0
Number of socket nodes 12
Number of directories 12259

#5 Updated by intrigeri 2017-03-15 15:57:12

• 00. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K (baseline)
  • size: 1191M
  • build time (intri’s Jenkins): 29 minutes
  • build time (Tails’ Jenkins): 47 minutes
• 01. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -Xe
  • size: same as baseline => fail
• 02. -comp xz -Xbcj x86 -Xpreset 9
  • bugfix/12347-test-02
  • size: 1215M => fail
  • build time (intri’s Jenkins): 25 minutes
• if -Xpreset 9 is too slow, try 7 or 8 and adjust the next steps
• 03. -comp xz -Xbcj x86 -Xpreset 9 -Xe
• 04. -comp xz -Xbcj x86 -Xe
  • size: 1215M => fail
  • build time (intri’s Jenkins): 25 minutes
• 05. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -Xpreset 9
  • bugfix/12347-test-05
  • size: same as baseline
  • build time (intri’s Jenkins): 27 minutes
  • build time (Tails’ Jenkins): 47 minutes
  • no size improvement, not much faster than baseline => fail
• 06. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -Xpreset 9 -Xe
  • bugfix/12347-test-06
  • size: same as baseline
  • build time (intri’s Jenkins): 27 minutes
  • build time (Tails’ Jenkins): 48 minutes
  • no size improvement, not faster than baseline => fail
• 07. -comp xz -Xbcj x86 -b 8096K -Xdict-size 8096K
  • “-b block size not power of two or not between 4096 and 1Mbyte” => fail
• 08. -comp lzma -Xe -Xpreset 9
  • bugfix/12347-test-08
  • build time (intri’s Jenkins): 22 minutes
  • build time (Tails’ Jenkins): 39 minutes
  • size: 1208M
  • fails to boot => fail
  • a tiny bit bigger, but substantially faster to compress
• 09. -comp lzo -Xcompression-level 9
  • segmentation fault (kernel: traps: mksquashfs[27815] general protection ip:409091 sp:7f3a55923e30 error:0 in mksquashfs[400000+2c000]) => fail
• 10. -comp lz4 -Xhc
  • size: much bigger than baseline => fail
  • not supported in Debian’s Linux kernel => fail
• 11. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -no-exports
  • size: 1190M => small improvement
  • build time (intri’s Jenkins): 30 minutes
• 12. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -Xlc 0 -Xlp 2 -Xpb 2
  • size: 1201M => fail
  • build time (intri’s Jenkins): 31 minutes
• 13. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -read-queue 16M -write-queue 16M
  • invalid values => fail

#12 Updated by bertagaz 2017-08-14 20:37:42

intrigeri wrote:
> Either we think it’s part of the contract (and then it needs target version + deliverable for), or not (and then it needs neither).

I don’t think that’s really an issue now that most of RB are in stable.

There may have been a bump in size with reproducible builds, but so far it doesn’t seem to have had impact on anything, even our infra. If it had, it’s probably not that much (like a few douzens MB). So maybe let’s consider this as a fair cost?

#13 Updated by intrigeri 2017-08-19 16:37:26

> There may have been a bump in size with reproducible builds, but so far it doesn’t seem to have had impact on anything, even our infra.
> If it had, it’s probably not that much (like a few douzens MB).

Thankfully we have actual data so we don’t have to guess. When this ticket was raised the difference was 92MB, i.e. +8.3% on the ISO size.

The impact I’m worried about is the one on users: how many incremental updates one can apply in a row (soon obsolete thanks to Feature #12707, so it’s too late to bother now), download time for end-users (for the ISO, and possibly for incremental upgrades if the size of IUKs is affected as well).

Now, I think it would be OK to merely report this to the author of the squashfs-tools patch and close this ticket.