Feature #16138

Ship a pre-compiled AppArmor binary cache

Added by intrigeri 2018-11-19 09:12:27 . Updated 2019-04-03 15:13:50 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Hardware support
Target version:
Tails_4.0
Start date:
2018-11-19
Due date:
% Done:

100%

Feature Branch:
feature/16138-pre-compiled-AppArmor-cache-buster
Type of work:
Code
Blueprint:

https://tails.boum.org/contribute/design/application_isolation/#pre-compiled-AppArmor-policy

Starter:
Affected tool:
Deliverable for:

Description

Context: decreasing Tails startup time significantly would help us achieve our “Make it easier to switch between a Tails contextual identity and another identity outside of Tails” strategic planning goal.

One of the main reasons why Tails takes so long to start, especially on systems with few and/or not super fast CPU cores, is that we compile the AppArmor policy during every single boot. This is kinda dumb given this compilation will produce the same result every time, which should make any engineer immediately suggest “well, let’s cache this, mayyyyybe?”.

Caching the resulting pre-compiled policy was not straightforward until Stretch, inclusive. But all the pieces we need to do that are finally in place in Buster so we can finally do that! :)

Subtasks

Related issues

Related to Tails - Bug #16393: feature/buster boots much slower than Tails 3.x Rejected 2019-01-28
Has duplicate Tails - Feature #10120: Consider shipping a pre-compiled AppArmor policy Duplicate 2015-08-28
Blocked by Tails - Bug #16390: devel branch FTBFS since linux-image-4.19.0-1-amd64 was removed from sid Resolved 2019-01-26

History

#1 Updated by intrigeri 2018-11-19 09:12:41

#2 Updated by intrigeri 2018-11-19 09:19:26

In theory one could implement this without waiting for Tails 4.0, by backporting src:apparmor from Buster to Stretch. As the maintainer of that package in Debian, I would recommend to keep this as a Tails-only backport and not upload it to stretch-backports, to avoid the cost of maintaining it plus its dependencies that are not satisfied in Stretch for 1 bonus year after we’ve lost interest in Stretch.

I’m interested in implementing this myself at some point but if someone else wants to do it earlier, by all means, be my guest: reassign this ticket to you before you start so we avoid duplicating work.

#3 Updated by intrigeri 2018-11-19 09:20:41

  • has duplicate Feature #10120: Consider shipping a pre-compiled AppArmor policy added

#4 Updated by intrigeri 2019-01-28 14:36:04

I’ll stop using the blueprint as I’m making progress on the implementation side, so next updates will live here.

I think we need a config/chroot_local-hooks/99-cache-AppArmor-policy hook that does:

apparmor_parser \
   --write-cache \
   --skip-kernel-load \
   --verbose \
   --add \
   /etc/apparmor.d

Also, I’ve thought about adding a scenario in our test suite, to verify that the cache from the ISO/IMG is still valid. But my current understanding is that it has to be valid: the only difference between build and runtime environments is the kernel version, and with a fixed features-file and the same parser, this should not affect the resulting binary cache at all.

#5 Updated by intrigeri 2019-01-28 14:55:49

Actually, I don’t think this requires AppArmor from Buster: running the aforementioned command line in Tails 3.11, after moving away the cache files generated in /etc/apparmor.d/cache/ at boot, generates the exact same binary cache files. The features file is the same at runtime as at build time, the features file is the same too, only the kernel version differs which should not matter with a fixed features file, so it seems that this could actually be implemented in Tails 3.x!

The only change in Buster is per-features-file (and possibly per-parser-version, if the parser changes the way it computes the hash of the features which is used as the name as the cache dir) cache directories under /var/cache/apparmor, but that should not be relevant for Tails.

#6 Updated by intrigeri 2019-01-28 15:36:03

  • blocked by Bug #16390: devel branch FTBFS since linux-image-4.19.0-1-amd64 was removed from sid added

#7 Updated by intrigeri 2019-01-28 15:37:24

  • Feature Branch set to feature/16138-pre-compiled-AppArmor-cache

#8 Updated by intrigeri 2019-01-28 23:34:01

  • Status changed from Confirmed to In Progress

Applied in changeset commit:tails|9acec481f724698e5ca8391cdbea9f259974a0cb.

#9 Updated by intrigeri 2019-01-28 23:35:22

  • % Done changed from 0 to 20
  • Feature Branch changed from feature/16138-pre-compiled-AppArmor-cache to feature/16138-pre-compiled-AppArmor-cache-buster

Methodology:

  • Measured value: time from pressing ENTER in the bootloader menu, until the Greeter is ready.
  • Booting:
    • On laptops: from the same (pretty fast) USB drive, which was installed from the USB image using GNOME Disks, and booted once before the benchmarks (to avoid the repartitioning process from tainting results)
    • In VM: from ISO, (pretty fast) NVMe underlying storage
  • The two branches were built simultaneously in order to get the same APT snapshots and thus comparable results.
  • The defaultcomp build option was used in all cases, for more real-world-like results.
  • The SquashFS sort file was not updated for 4.19.0-2 (used on the Stretch branches) nor for Buster (where it’s probably very wrong). If this inaccuracy taints the results significantly, it’ll probably be in favor of the baseline, i.e. without a pre-compiled AppArmor policy: it’ll slow down I/O and will make the relative impact of CPU-bound tasks on boot time smaller; and compiling the AppArmor policy is CPU-bound. So worst case, this inaccuracy will make the product of this ticket appear worse than it would be in actual Tails releases, i.e. conservative results.
  • My reasoning about the SquashFS sort file implies that slower USB drives would benefit less from the improvements brought by this ticket.
  • I only did each test once, assuming that the duration of a Tails offline boot with the same system on the same hardware does not vary too much across runs.

Results on Stretch:

  • feature/16390-linux-4.19.0-2+force-all-tests at commit:65f2805fe00cec7978723ff101ded91394d43bc4, i.e. what’s needed to make current devel branch build successfully
    • AppArmor enabled (baseline):
      • ThinkPad X200: 90 seconds
      • HP EliteBook 840 G1: 66 seconds
    • AppArmor disabled (apparmor=0):
      • ThinkPad X200: 70 seconds
      • HP EliteBook 840 G1: 50 seconds
  • feature/16138-pre-compiled-AppArmor-cache (only change on top of the 3.x baseline branch: pre-compiled AppArmor policy):
    • ThinkPad X200: 90 seconds
    • HP EliteBook 840 G1: 57 seconds

Conclusions for Stretch:

  • This implementation does not work at all on Stretch: the cache is regenerated. It could probably made to work but if it works out of the box on Buster as I expect, I’ll target that.
  • The AppArmor policy compilation accounts for ~20-25% of the “boot to Greeter” duration. That’s… a lot. This confirms that this ticket may be a very interesting low-hanging fruit.

Results on Buster:

  • feature/buster at commit:7f0fddc8138d7cebc208759b74ff6365b2a9d0a2 (baseline)
    • ThinkPad X200: 140 seconds
    • HP EliteBook 840 G1: 104 seconds
    • VM with fast underlying storage: 43 seconds
  • feature/16138-pre-compiled-AppArmor-cache-buster at commit:9acec481f724698e5ca8391cdbea9f259974a0cb (only change on top of feature/buster: pre-compiled AppArmor policy):
    • ThinkPad X200: 96 seconds
    • HP EliteBook 840 G1: 62 seconds
    • VM with fast underlying storage: 37 seconds

Conclusions for Buster:

  • This optimization makes the boot 14-40% faster on Buster. Sweet!
  • Current feature/buster boots much slower than current Tails 3.x. Even if the improvements brought by this ticket cancel it almost entirely, that seems to be a serious problem. I was looking to improve things here, not merely to avoid them getting worse! ⇒ This needs a dedicated ticket.

#10 Updated by intrigeri 2019-01-28 23:38:56

  • Target version set to Tails_4.0

Most of the work was done and the initial benchark results exceed my expectations, so let’s be bold and try to ship this in 4.0 :)

#11 Updated by intrigeri 2019-01-28 23:47:59

> This needs a dedicated ticket.

Bug #16393

Next steps:

  • If the test suite is good enough on feature/buster to give us useful info, check its results.
  • Update design doc.

#12 Updated by intrigeri 2019-01-28 23:48:11

  • related to Bug #16393: feature/buster boots much slower than Tails 3.x added

#13 Updated by intrigeri 2019-04-02 15:28:08

  • Priority changed from Normal to Low

#14 Updated by intrigeri 2019-04-03 11:10:11

Measuring boot time from pressing Enter in the syslinux menu to the desktop being ready (including desktop icons), after pressing Enter quickly in the Greeter:

  • 3.13.1
    • HP EliteBook 840G1: 71 seconds
    • ThinkPad X200: 122 seconds
  • feature/buster (after updating SquashFS sort file on Bug #16393)
    • HP EliteBook 840G1: 75 seconds (+6% compared to 3.13.1)
    • ThinkPad X200: 148 seconds (+21% compared to 3.13.1)
  • feature/16138-pre-compiled-AppArmor-cache-buster
    • HP EliteBook 840G1: 65 seconds (–8% compared to 3.13.1, –13% compared to feature/buster)
    • ThinkPad X200: 116 seconds (–5% compared to 3.13.1, –22% compared to feature/buster)

So this branch cancels Buster’s boot slowdown and improves things a little bit compared to 3.13.1.

#15 Updated by intrigeri 2019-04-03 12:50:09

  • Assignee changed from intrigeri to segfault
  • Priority changed from Low to Normal
  • QA Check set to Ready for QA

#16 Updated by intrigeri 2019-04-03 12:51:03

(Raised priority as I have found no easy other way to fix Bug #16393, so let’s at least take this improvement.)

#17 Updated by segfault 2019-04-03 14:23:45

  • Assignee deleted (segfault)
  • QA Check changed from Ready for QA to Pass

Code LGTM, tested that it boots and AppArmor still works. Merged into feature/buster.

#18 Updated by intrigeri 2019-04-03 15:13:28

#19 Updated by intrigeri 2019-04-03 15:13:50

  • Status changed from In Progress to Resolved
  • % Done changed from 20 to 100