Bug #16637

squashfs-tools from Buster (1:4.3-12) is much slower than 1:4.3-8.0tails1

Added by intrigeri 2019-04-06 16:38:36 . Updated 2019-08-27 07:22:58 .

Status:
Resolved
Priority:
Elevated
Assignee:
segfault
Category:
Build system
Target version:
Start date:
Due date:
% Done:

100%

Feature Branch:
bugfix/16637-faster-mksquashfs
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Compressing the SquashFS on lizard jumped from 20 min to 38 min since we upgraded from 1:4.3-8.0tails1.


Subtasks


Related issues

Related to Tails - Bug #16457: Building feature/buster in RAM sometimes fails: mksquashfs reaped by OOM killer Resolved 2019-02-11
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by intrigeri 2019-04-06 16:39:05

  • related to Bug #16457: Building feature/buster in RAM sometimes fails: mksquashfs reaped by OOM killer added

#2 Updated by intrigeri 2019-04-06 16:40:29

@lamby, this one might be interesting for you.

#3 Updated by intrigeri 2019-04-06 16:40:54

#4 Updated by segfault 2019-05-17 14:44:53

  • Status changed from Confirmed to In Progress

Applied in changeset commit:tails|bd9331292cb4565a2960de7f0c7bb586b3b12484.

#5 Updated by intrigeri 2019-05-17 16:01:56

  • Status changed from In Progress to Confirmed

(That commit is unrelated, typo for Feature #16337)

#6 Updated by intrigeri 2019-08-08 08:34:39

  • Description updated

#7 Updated by intrigeri 2019-08-24 07:51:26

  • Assignee set to intrigeri
  • Priority changed from Normal to Elevated

I’m tentatively making this a release blocker because of the combination of two factors:

  • Having our CI builds take about 30% more time has a painful impact on developer experience: our feedback loop is way too long already and there are not so many low-hanging fruits that would save 20 minutes.
  • This problem seems to be https://bugs.debian.org/921146, which is fixed in sid, so it might be that this is cheap to fix. I’ll give it a try.

#8 Updated by intrigeri 2019-08-24 07:57:25

  • Feature Branch set to bugfix/16637-faster-mksquashfs

#9 Updated by intrigeri 2019-08-24 07:57:58

  • Status changed from Confirmed to Needs Validation

(Forcing reproducibility tests on CI.)

#10 Updated by intrigeri 2019-08-24 09:40:26

  • Status changed from Needs Validation to In Progress

Applied in changeset commit:tails|cdf270b4769b8d3910c6f01fb211c0ff06e92857.

#11 Updated by intrigeri 2019-08-24 09:42:17

  • Status changed from In Progress to Needs Validation

It works: on isobuilder1.sid, build goes down from ~50 min to ~40 min.

#12 Updated by intrigeri 2019-08-24 12:08:53

First build on lizard took 65min, that is only slightly more than stable, and much less than devel, so I’m calling this a success. I’m trying to sneak another thing on top of this branch before sending it to QA.

#13 Updated by intrigeri 2019-08-24 18:56:41

  • Assignee deleted (intrigeri)

Regarding commit:c448a322b7784b7b0fa428197a5cdc218753b75c, it worked fine so far on lizard, my local Jenkins, and my own laptop (both with “ram” , “noram”, “defaultcomp noram”, and “defaultcomp ram” build options).

But none of these use the internal apt-cacher-ng, and I’d rather not download GBs of data just to test this, so I’d appreciate if the reviewer used that one and tried the “ram defaultcomp” and “noram” build options (I’m assuming that it’ll give us enough coverage and one does not need to test each cell of the 2×2 matrix). Thanks in advance!

#14 Updated by segfault 2019-08-25 13:12:40

  • Assignee set to segfault

#15 Updated by segfault 2019-08-25 13:13:23

LGTM, will now test with apt-cacher-ng and “ram defaultcomp” and “noram” build options

#16 Updated by segfault 2019-08-26 23:32:43

Both builds were fine.

#17 Updated by segfault 2019-08-26 23:35:03

  • Status changed from Needs Validation to Resolved
  • % Done changed from 0 to 100

Applied in changeset commit:tails|6d5fc2a922a9bd9d0c6a2959007754b96105d083.

#18 Updated by intrigeri 2019-08-27 07:22:58

Yeepee! :)