Tails

#1 Updated by intrigeri 2017-03-17 10:20:37

• related to Feature #12089: Enable the kernel page allocator poisoning added

#2 Updated by intrigeri 2017-03-17 10:20:45

• related to Feature #12107: Can PAX_MEMORY_SANITIZE replace memory erasure on shutdown? added

#9 Updated by intrigeri 2017-03-18 16:13:32

• related to Bug #11786: System often crashes during/after memory wipe since Linux 4.6 added

#10 Updated by anonym 2017-03-18 17:35:50

• Status changed from Confirmed to In Progress
• % Done changed from 0 to 10
• Type of work changed from Research to Discuss

From the description I’m extracting the two problems we have: On shutdown:

1. the kernel panics when kexec:ing: no memory is wiped, and the system is powered so the memory is kept in a perfect state unlike if the system had shut down; the memory will quickly start to decay without power. In other words, in this failure mode our memory wipe results in something worse than a normal shutdown. :S
2. the kernel kexec:ing works, but then the system fails to shut down: the userland memory is (probably) wiped, but not the memory reserved for the kernel.

In both cases the user is faced with a system that is still running, with weird stuff on the screen, and has to manually power off the computer.

Problem 1 is due to hardware compatibility, i.e. kexec is broken on some hardware. New kernel releases may fix it for some hardware, and may break it for others. We’ve seen it go both ways, and it’s safe to say that we neither have the expertise, time nor access to enough hardware to follow this closely and make sure each kernel version doesn’t regress. So it seems impossible that we can completely (= for all hardware) solve problem 1. Conclusion: it seems kexec-based memory wiping will always remain unreliable on some of our users’ hardware.

Problem 2 is more tractable for us. We have several ideas of how to proceed, e.g. someone could spend a day or two to try e.g. Bug #11786#note-60. We can probably solve it, but it will still take quite some time.

So, given that we will not be able to solve problem 1, is it even worth spending time on problem 2? At best we’ll only be able to make “guarantees” that memory wiping works on certain specific hardware that we list somewhere, that very serious “high risk” users might invest in, but users with $random hardware will be left out, and some of them will be hit by this annoying issue and the uncertainty of whether the wiping worked at all.

So we have three options:

• a) We live with problem 1 but fix problem 2.
• b) Same as a), but we also enable Feature #12090 (memory allocated to userspace is wiped when free():d)
• c) We drop our kexec-based memory wiping, enable Feature #12090 and rely on that as our only active memory wiping measure.

(b) is quite obviously better than (a) except that we cannot automatically test our memory wiping feature any more (Feature #12090 makes this test impossible). So, really, (b) or ©? More detailed:

• b): crappy UX and less safety on some hardware, but more safety when it actually works.
• c): more polished UX, and reliable though imperfect memory wiping

Another way to look at it (read “>” as “is better than”):

• Safety: (b) when it works > © > (b) when it fails
• UX: (b) when it works == © > (b) when it fails

So, yeah, it’s a nasty trade-off we have to decide.

#13 Updated by intrigeri 2017-03-19 22:04:29

• % Done changed from 10 to 0
• Type of work changed from Discuss to Research

To test tmpfs: start Tails, check the known pattern cannot be found, mount a tmpfs of decent size (no need to aim for the maximum), fill it with a known pattern, dump memory and check the known pattern can be found, umount the tmpfs, verify that the known pattern cannot be found anymore. If it still can, test shut down and/or drop caches.

To test disk cache:

• background:
  • start Tails
  • create+unlock+mount a ext4-on-LUKS filesystem
• test write buffers:
  • drop caches
  • check that the known pattern cannot be found in memory
  • fill the FS with a known pattern
  • dump memory to check the known pattern can be found
• test read cache:
  • drop caches
  • check that the known pattern cannot be found
  • read the entire filesystem
  • check that the known pattern can be found
• test umount (after each of previous tests):
  • umount the filesystem
  • check if the known pattern can be found
  • if it can, drop caches, check again, and if this works better: consider dropping caches during the shutdown process

#15 Updated by intrigeri 2017-03-20 16:52:41

• related to deleted (Feature #12107: Can PAX_MEMORY_SANITIZE replace memory erasure on shutdown?)

#18 Updated by intrigeri 2017-04-05 13:37:04

• Feature Branch set to bugfix/12354-drop-kexec-memory-wipe

#19 Updated by intrigeri 2017-04-05 15:48:49

• Assignee changed from intrigeri to anonym
• QA Check set to Ready for QA

I have written automated tests (that pass successfully and reliably on my system) that, I believe, answer the few questions that were raised when we discussed this. Next steps:

1. anonym reviews the branch, confirms that kernel poisoning is good enough for us, and merges it up to commit:0c8e076a5dac72cc6898ad9d3ce4156699ed2d06 into feature/stretch if happy;
2. I’m going to file a ticket about dropping the kexec-based implementation.

#20 Updated by intrigeri 2017-04-05 15:53:52

• related to deleted (Bug #11786: System often crashes during/after memory wipe since Linux 4.6)

#21 Updated by intrigeri 2017-04-05 15:53:59

• has duplicate Bug #11786: System often crashes during/after memory wipe since Linux 4.6 added

#23 Updated by intrigeri 2017-04-05 16:11:49

• Subject changed from What to do wrt. shutdown regressions on 3.0~betaN? to Fix shutdown and memory wipe regressions on 3.0~betaN
• Type of work changed from Research to Code

#25 Updated by intrigeri 2017-04-17 13:16:25

• Assignee changed from anonym to intrigeri
• QA Check changed from Ready for QA to Dev Needed

> IMHO the fs variable shouldn’t be human-readable text, but optional arguments (I recommend a Hash so a call could look like mount_USB_drive(some_disk, filesystem: 'ext4', luks_passphrase: 'asdf')) so it fits in more caller contexts.
> […]
> Here you can parameterize the size unit instead of fixing on MiB. The same applies to several other steps you created.

Agreed for both. I’ll do that when I resume work on this branch.

#27 Updated by intrigeri 2017-04-18 15:04:56

• Target version changed from Tails_3.0 to Tails_3.0~rc1

#29 Updated by intrigeri 2017-05-18 08:50:18

• related to Bug #12393: Emergency shutdown leaves the Desktop on for much longer in Stretch added

#30 Updated by intrigeri 2017-05-18 09:08:17

• Assignee changed from intrigeri to anonym
• QA Check changed from Dev Needed to Ready for QA

#31 Updated by intrigeri 2017-05-18 09:10:51

• related to Feature #5417: Improve user experience when wiping memory at shutdown added

#32 Updated by intrigeri 2017-05-18 09:57:55

• blocked by Feature #12554: Bump APT snapshots for 3.0~rc1 added

#33 Updated by anonym 2017-05-18 15:02:19

• Assignee changed from anonym to intrigeri
• QA Check changed from Ready for QA to Dev Needed

Awesome work! I found no blockers (so feel free to merge this yourself as long as fixing my remarks below are tracked somehow and fixed before Tails 3.0) and I was hard-pressed to find anything to comment on, but here goes:

In wiki/src/contribute/design/memory_erasure.mdwn:
> In order to protect against memory recovery such as cold boot attack,
> most of the system RAM is overwritten when Tails is being shutdown or when the
> boot medium is physically removed.

I guess we could add “and immediately after applications exit” or something to that effect? That is an improvement over the old kexec-based approach when treated in isolation.

> #### The big picture

Can we put the story about the old design in another sub section, like “##### Obsolete kexec-based approach”, and the current one in “##### Current memory poisoning-based approach”? I.e. just insert these two headers ( and possibly adapt the edges of the affected paragraphs so the text still makes sense). I’d just like to make the history lesson more optional. :) Is it really needed for understanding the current design?

> Different kinds of events trigger the memory erasure process. All lead
> to run the `tails-kexec` shutdown script.

There’s no tails-kexec any more.

> [[!tails_gitweb features/erase_memory.feature desc=“Automated tests”]]
> ensure that the most important parts of memory are erased this way.

Can we describe “the most important parts of memory” a bit better than this (not necessarily at this place in the design page, btw)? Or do we want the .feature file to enumerate what we think is “important”? I really think this design page should list any limitations of the memory poisoning approach.

In features/step_definitions/common_steps.rb:
> -def mount_USB_drive(disk, fs)
> +def mount_USB_drive(disk, fs_options = {})
> + fs_options[:encrypted] ||= false

Just a FYI: nil is treated as false when interpreted as a boolean, so this assignment is unnecessary. That said, I don’t really mind making the default more explicit, for readability.

In features/step_definitions/erase_memory.rb:
> kernel_mem_reserved_k = 64*1024 # Duplicated in /usr/share/initramfs-tools/scripts/init-premount/memory_wipe
> […]
> admin_mem_reserved_k = 128*1024 # Duplicated in /usr/share/initramfs-tools/scripts/init-premount/memory_wipe

These don’t duplicate anything any longer.

In config/chroot_local-includes/usr/share/initramfs-tools/hooks/shutdown:
> # systemd-shutdown itself
> mkdir -p $DESTDIR/lib/systemd
> copy_exec /lib/systemd/systemd-shutdown /shutdown
>
> # Our shutdown hook (run from inside the initramfs)
> mkdir -p $DESTDIR/lib/systemd/system-shutdown
> copy_file "regular file" \
> @ /usr/local/lib/initramfs-pre-shutdown-hook \@
> @ /lib/systemd/system-shutdown/tails@

To me it looks like you are following the common pattern of mkdir -p:ing the target directories for the subsequent copy actions, but these directories are not used as the target (in fact they are not used at all). I’m just double-checking that these mkdir calls still are relevant, and no residues from some Git history rewrite.

In features/emergency_shutdown.feature:
> Then I find very few patterns in the guest’s memory
> And Tails eventually shuts down

This means that we will always wait the full two minutes (the “Happy dumping!” timeout) for each of these scenarios: there’s currently four, so that’s almost 2*4 = 8 minutes of needless waiting. I’m tempted to suggest that we add a scenario dedicated to make sure that the shutdown happens without any of the memory erasure checks, and that we simply remove the Tails eventually shuts down step from these four scenarios. In fact, I think you should move these four scenario to erase_memory.feature as well — I know it can be argued both ways, but I slightly prefer what I suggest (so no strong opinion here, just ignore this move if you disagree).

#34 Updated by intrigeri 2017-05-18 15:35:19

• related to Bug #12560: Polish the new memory wiping implementation and design doc added

#35 Updated by intrigeri 2017-05-18 15:37:41

• Status changed from In Progress to Fix committed

Applied in changeset commit:5f588e526056ed6384659e7e04ef39edf0a93634.

#36 Updated by intrigeri 2017-05-18 15:39:42

• Assignee deleted (intrigeri)
• QA Check changed from Dev Needed to Pass

Follow-ups are tracked on Bug #12560. Thanks!

#37 Updated by intrigeri 2017-05-23 09:13:25

• Status changed from Fix committed to Resolved