Bug #16097

Memory erasure tests regression on the devel branch

Added by intrigeri 2018-11-05 11:09:29 . Updated 2019-01-30 11:49:44 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Target version:
Tails_3.12
Start date:
2018-11-05
Due date:
% Done:

100%

Feature Branch:
bugfix/16097-memory-erasure-on-shutdown
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

The memory erasure tests have been failing on the devel branch for a few weeks.

Significant changes between these commits:

  • Linux upgraded from 4.17.0-3 to 4.18.0-2 and accordingly, aufs4-standalone upgraded from 01543e47eae7653c7e9a35a7204301f8a0b3ca50 to bdda97c749604bb9ea3f19e0c1ffac9042e79f77 → I doubt that matters because our stable branch also includes this change and there, the tests pass
  • debian APT snapshot updated from 2018100901 to 2018101503 which includes at least systemd 237-3~bpo9+1 → 239-7~bpo9+1

Sadly, the build artifacts for these 2 ISO build jobs and most of the relevant APT snapshots have been GC’ed already. Still, comparing the .build-manifest for the last successful builds of stable and devel and filtering out those that were upgraded in Debian after 20181015, the only potential culprit I see is the systemd 237-3~bpo9+1 → 239-7~bpo9+1 upgrade, which happened on 20181009 but strictly after the 2018100901 debian APT snapshot.

Files

tails-amd64-bugfix_16097-memory-erasure-on-shutdown-3.12-20190113T2209Z-4c051c657b.buildlog.xz (220612 B) lamby, 2019-01-14 09:57:03

Subtasks

Related issues

Related to Tails - Bug #16100: Fix systemd CVE-2018-15687 Rejected 2018-11-05
Blocks Tails - Feature #15507: Core work 2019Q1: Foundations Team Resolved 2018-04-08
Blocks Tails - Bug #16312: Enabling persistence in Buster leads to issues at shutdown Resolved 2019-01-06
Blocks Tails - Bug #16352: Fix systemd vulnerabilities: CVE-2018-16864, CVE-2018-16865 and CVE-2018-16866 Resolved 2019-01-13

History

#1 Updated by intrigeri 2018-11-05 11:10:11

#2 Updated by intrigeri 2018-11-05 11:27:09

Interestingly, most of “system memory erasure on shutdown” passes: only the one about the aufs RW branch fails.

https://github.com/systemd/systemd/issues/8221 might be relevant for emergency shutdown, although it’s reported to happen with v237, which works fine for us.

Next steps:

  • look at the videos from Jenkins, maybe they’ll give some hints
  • reproduce locally, perhaps with nosplash and making systemd log more (and to the console)

#3 Updated by intrigeri 2018-11-05 14:14:58

  • related to Bug #16100: Fix systemd CVE-2018-15687 added

#4 Updated by intrigeri 2018-12-03 13:41:20

  • related to Bug #16184: Intermittent test failures on the devel branch: fails to login "Failed to fully start up daemon: Permission denied" added

#5 Updated by intrigeri 2018-12-03 15:48:20

  • Assignee set to intrigeri

#6 Updated by intrigeri 2018-12-30 12:29:19

#7 Updated by intrigeri 2018-12-30 12:29:27

  • blocked by deleted (Feature #15506: Core work 2018Q4: Foundations Team)

#8 Updated by intrigeri 2019-01-07 10:34:13

  • blocks Bug #16312: Enabling persistence in Buster leads to issues at shutdown added

#9 Updated by intrigeri 2019-01-10 19:29:31

  • Status changed from Confirmed to In Progress

/lib/systemd/system-shutdown/tails is run but I see no trace of returning to the initramfs, which nicely explains the failure of the exact tests that rely on this mechanism (and “FindFailed: can not find MemoryWipeCompleted.png”), while the tests that verify that memory is erased on unmount and when processes are killed work just fine. Manually running /bin/mount -o remount,exec /run before halt fixes that; and doing this trick in the test suite fixes “Scenario: Erasure of the aufs read-write branch on shutdown”. But /lib/systemd/system-shutdown/tails should do that itself so something is wrong.

#10 Updated by intrigeri 2019-01-10 19:43:03

  • related to deleted (Bug #16184: Intermittent test failures on the devel branch: fails to login "Failed to fully start up daemon: Permission denied")

#11 Updated by intrigeri 2019-01-10 20:00:09

  • % Done changed from 0 to 20
  • Feature Branch set to bugfix/16097-memory-erasure-on-shutdown
  • Type of work changed from Research to Code

#12 Updated by intrigeri 2019-01-10 21:46:11

My branch fixes the regular (“clean”) shutdown but emergency shutdown tests still fail locally. Will investigate further.

#13 Updated by intrigeri 2019-01-12 20:07:38

  • Assignee changed from intrigeri to lamby
  • % Done changed from 20 to 50
  • QA Check set to Ready for QA

Fixed! All memory erasure tests pass locally. Please review. A build with my current proposal was just started on Jenkins. FTR: our review guidelines.

Note that I’ve left my first approach in the Git history. Its commit message explains why things are broken and need fixing. I was tempted to rewrite history and merge this explanation into the commit message that implements the approach that works, but this time I figured it would be useful to have a trace of something I’ve tried and that does not work, not particularly for posterity but rather for whoever would be tempted to try that in the future.

Thanks in advance :)

#14 Updated by intrigeri 2019-01-13 07:48:33

https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_bugfix-16097-memory-erasure-on-shutdown/3/cucumberTestReport/ confirms the fix (compared to job 2 where emergency shutdown tests failed).

#15 Updated by intrigeri 2019-01-13 12:04:45

  • blocks Bug #16352: Fix systemd vulnerabilities: CVE-2018-16864, CVE-2018-16865 and CVE-2018-16866 added

#16 Updated by lamby 2019-01-14 10:06:56

Methodology:

  • Checked out bugfix/16097-memory-erasure-on-shutdown branch.
  • Built tails-amd64-bugfix_16097-memory-erasure-on-shutdown-3.12-20190113T2209Z-4c051c657b.iso (see attached tails-amd64-bugfix_16097-memory-erasure-on-shutdown-3.12-20190113T2209Z-4c051c657b.buildlog.xz.
  • Booted in QEMU:
  • Logged in
  • Shutdown from normal menu. Did not see any errors, delays or timeouts.
  • Booted again.
  • Shutdown from greeter. Did not see any errors, delays or timeouts.
  • Burnt to USB stick
  • Repeated above on X230.

#17 Updated by intrigeri 2019-01-14 10:43:30

  • Status changed from In Progress to Fix committed
  • % Done changed from 50 to 100

Applied in changeset commit:tails|7f101d16651989211b08d3832b44fa3a54d21f86.

#18 Updated by intrigeri 2019-01-14 10:45:02

  • Assignee deleted (intrigeri)

#19 Updated by anonym 2019-01-30 11:49:44

  • Status changed from Fix committed to Resolved