Tails

#3 Updated by elouann 2016-09-13 02:37:32

intrigeri wrote:
> Can you please add some details to the “isn’t working anymore” part? Description of what happens exactly, what’s on screen, screenshots, would be useful.

We received a more detailed bug report:

Shutdown on Tails 2.5:
1. Screen shows “[ OK ] Started Modem Manager”, followed by “You can now remove…”, followed by “Starting new kernal”
2. Sceen fills with asterisk as memory is wiped, followed by “clean exit”
3. System turns off

Shutdown on Tails 2.6~rc1:
1. Screen shows “You can now remove…”, followed by “kexec_core: Starting new kernal”
2. Systems turns off
Issue occurs every time.

#4 Updated by intrigeri 2016-09-13 02:46:54

> We received a more detailed bug report:
[…]
> Shutdown on Tails 2.6~rc1:
> 1. Screen shows “You can now remove…”, followed by “kexec_core: Starting new kernal”
> 2. Systems turns off
> Issue occurs every time.

Hi elouann! This ticket is about memory wipe failing. In that report, the system turns off so memory wipe succeeded, so this seems off-topic here unless I’ve missed something ⇒ please report whatever other problem you wanted to report in a dedicated ticket. Thanks in advance! (If it’s about the lack of visual feedback (asterisks) on screen: presumably that’s caused by the upgrade to Linux 4.6, not much we can do about it, and having this feedback working or not has always been highly hardware-dependent; the best we can do is to ensure our doc/FAQ/known issues/whatever is good enough to avoid doc-reading-user confusion.)

#5 Updated by sonicsnail 2016-09-13 17:26:57

In my case, 2.5 was fine, but on 2.6~rc1 the screen shows “You can now remove…”, followed by “kexec_core: Starting new kernel”, and it just hangs there. It never progresses to the next screen filled with asterisks and the system never turns off, I have to manually hold the power button until it turns off. So I assume the memory isn’t getting wiped. This happens every time on 2.6~rc1.

intrigeri wrote:
> Also, is there any chance that you get the bug reporters to test another, experimental ISO image, if I provided one?

I could test an experimental ISO if needed.

#7 Updated by sonicsnail 2016-09-18 17:40:25

anonym wrote:
> Can you please test this ISO image: http://nightly.tails.boum.org/build_Tails_ISO_feature-10733-reverted/builds/lastSuccessfulBuild/archive/build-artifacts/
>
> If you could do it before Monday it might influence what we do for the final Tails 2.6 release.

I tested the ISO, booting and shutting down several times, with the same results every time. Here are the results I got:

1. Screen shows “You can now remove…”, followed by “kexec_core: Starting new kernel”.
2. Screen fills with asterisks.
3. Kernel panic (see output below). It hangs at the end of the output.
4. System does not turn off automatically. Need to hold down the power button until it turns off.

So it now progesses past “Starting new kernel” to the screen filled with asterisks, which looks like memory wipe is working again, but now there’s a kernel panic issue that prevents powering off.

Kernel panic output:

[    5.938190] usb 2-4: string descriptor 0 read error: -22
modprobe: module microcode not found in modules.dep
Starting Wiping the memory, press Control-C to abort earlier. Help: "/usr/bin/sdmem -h"
Wipe mode is insecure (one pass with 0x00)
********************************************************************************
[Edit: ...]
********************************************************************************Killed
/scripts/init-premount/memory_wipe: line 57: can't fork
/init: /scripts/init-premount/ORDER: line 5: can't fork
[    8.033091] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200
[    8.033091] 
[    8.033143] CPU: 1 PID: 1 Comm: init Tainted: G            E   4.6.0-0bpo.1-amd64 #1 Debian 4.6.4-1~bpo8+1
[    8.033192] Hardware name: [Edit: hardware redacted]
[    8.033224]  [Edit: hardware redacted]
[    8.033274]  [Edit: hardware redacted]
[    8.033323]  [Edit: hardware redacted]
[    8.033371] Call Trace:
[    8.033392]  [<ffffffff813124c5>] ? dump_stack+0x5c/0x77
[    8.033424]  [<ffffffff8117069f>] ? panic+0xdf/0x226
[    8.033455]  [<ffffffff8107f448>] ? do_exit+0xb68/0xba0
[    8.033486]  [<ffffffff8107f4f9>] ? do_group_exit+0x39/0xb0
[    8.033519]  [<ffffffff8107f580>] ? SyS_exit_group+0x10/0x10
[    8.033552]  [<ffffffff81003e28>] ? do_int80_syscall_32+0x58/0x160
[    8.033587]  [<ffffffff815caa66>] ? entry_INT80_compat+0x36/0x50
[    8.033626] Kernel Offset: disabled
[    8.033648] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200
[    8.033648] 

#8 Updated by anonym 2016-09-20 06:25:51

For the record, I’ve also observed this kernel panic on Tails with Linux 4.7 (KVM guest).

#11 Updated by emmapeel 2016-09-25 01:48:24

With a Thinkpad x201

Tails 2.5:
Laptop shuts down ok, with askterisks and everything

Tails 2.6rc~1 and Tails 2.6:
Laptop shuts down but no asterisks.

tails-i386-feature_10733-reverted-2.6-20160918T1709Z-7f88af1+testing@f8e35e3:
Laptop shuts down ok, with askterisks and everything

#15 Updated by Anonymous 2016-09-29 19:15:15

can confirm no asterisk just power off
in 2.5 everytime with asteriks then off with 2.6 no single time with asteriks just power off

#21 Updated by intrigeri 2016-10-02 12:02:17

anonym, emmapeel, elouann, goupille, mercedes508: did you see this bug with a system that has no more than 2GB of RAM?

#22 Updated by intrigeri 2016-10-02 12:24:00

Bug reporters & frontdesk: did you see this bug on systems that are are not “64-bit CPU + strictly more than 2GB of RAM”?

I’ve verified that busybox behaves sanely wrt. how we deal with background processes: in a busybox shell within our initramfs, (sleep 30) & (sleep 20) & (sleep 10) & wait returns after 30 seconds. I’ve also verified that killall sdmem works in that context.

I suspect that our “one instance of sdmem per 2 GiB of RAM” tweak causes this bug on 64-bit systems with strictly more than 2GB of RAM, where we uselessly create a risky situation with a bunch of sdmem process each trying to fill all memory, and rely on the OOM killer to behave properly… and for some reason we see that it sometimes tries to kill init. This does not explain the case when poweroff can’t fork though: when we reach this point, if everything had gone well all sdmem processes would have been killed so there’s no reason why there’s no memory left, so I really don’t understand what’s happening there.

Still, I propose that we use the “one instance of sdmem per 2 GiB of RAM” tweak on 32-bit systems only (Feature #8183 already drops it with commit:619c536a458177e7972ad6e0086a4174ee57a60d). I’m not sure it will fix this bug, but it makes sense to first eliminate some useless complexity, which can help us debug the problem further if it’s not fixed by that change. I’ll prepare a branch that does so.

#24 Updated by intrigeri 2016-10-02 12:29:25

intrigeri wrote:
> Still, I propose that we use the “one instance of sdmem per 2 GiB of RAM” tweak on 32-bit systems only (Feature #8183 already drops it with commit:619c536a458177e7972ad6e0086a4174ee57a60d). I’m not sure it will fix this bug, but it makes sense to first eliminate some useless complexity, which can help us debug the problem further if it’s not fixed by that change. I’ll prepare a branch that does so.

Done on the topic branch, let’s see what Jenkins thinks about it.

#25 Updated by intrigeri 2016-10-03 08:06:08

Interesting: with that change in, according to Jenkins only ~2.5-3GB is wiped. The video shows that sdmem prints “done” before we display our “Happy dumping” message. I wonder if that’s caused by our 32-bit userspace (sorry I’m quite clueless in such low-level stuff).

#26 Updated by anonym 2016-10-03 17:57:04

intrigeri wrote:
> Interesting: with that change in, according to Jenkins only ~2.5-3GB is wiped. The video shows that sdmem prints “done” before we display our “Happy dumping” message. I wonder if that’s caused by our 32-bit userspace (sorry I’m quite clueless in such low-level stuff).

I am very certain our 32-bit userspace is the issue. After all, a 32-bit program will not “learn” how to deal with 64-bit pointers just because the kernel knows it — in the compiled code, the size of pointers are 32-bit still, and no matter what you do you cannot fit a 64-bit pointer. 4 GiB of virtual memory per process is the hard limit.

The only way to save this KISS approach, I guess, is to ship a statically linked 64-bit sdmem in our initramfs… :)

#27 Updated by anonym 2016-10-03 17:59:48

intrigeri wrote:
> anonym, emmapeel, elouann, goupille, mercedes508: did you see this bug with a system that has no more than 2GB of RAM?

Note that our automated test suite defaults to VMs with exactly 2 GiB of RAM, and it frequently happens when running it. FWIW I also run other Tails VMs with that exact amount of RAM, with the same problem.

#28 Updated by intrigeri 2016-10-04 07:08:22

> Note that our automated test suite defaults to VMs with exactly 2 GiB of RAM, and it frequently happens when running it. FWIW I also run other Tails VMs with that exact amount of RAM, with the same problem.

OK, too bad. So my hypothesis was wrong and it’s not worth fixing the issue with my branch (that I’m going to delete).

#30 Updated by intrigeri 2016-10-09 20:13:32

FWIW, since the beginning of October:

• among 22 test suite runs on the stable branch (Linux 4.6), I see 5 such failures
• among 6 test suite runs on the devel branch (Linux 4.7), I see no such failure

I’ve just started a few runs on the devel branch to confirm whether the move to 4.7 has fixed this bug.

Meanwhile, frontdesk: please have bug reporters test recent builds from the devel branch.

#32 Updated by emmapeel 2016-10-10 16:09:24

So, I have tested several times each with a Thinkpadx201 and:

- Tails 2.6:
When clicking restart, the message about ‘now you can take your USB out’ appears, and after approx. 15 seconds of it, the laptop restarts (no stars! :S)

- tails-i386-devel-2.7-20161005T0955Z-bc6a783
No change with relation to 2.6
- Tails 2.5:
Similar to Tails 2.6, takes the same amount of time, but I get the screen full as asterisks before restarting, at some point inside the 15 seconds window (around 10 secs)

#34 Updated by intrigeri 2016-10-29 12:15:00

bertagaz wrote:
> I’ll give it a try. Hope I’ll find an easy way to include this kernel.

Any progress update / ETA? Do you need help?

#35 Updated by bertagaz 2016-10-29 13:24:10

intrigeri wrote:
> Any progress update / ETA? Do you need help?

Progress yes, more tests to analyze, dead line en of the week end best (specially now that merging the other branches just went out of my next moves).

#36 Updated by intrigeri 2016-10-29 14:05:27

> Progress yes, more tests to analyze,

What’s the branch? I see none with “11786” in its name.

#37 Updated by intrigeri 2016-11-02 07:55:11

[Moved to Bug #11885]

#38 Updated by intrigeri 2016-11-02 08:46:09

I guess that this might be relevant and is worth having a closer look at:

linux-latest (75) unstable; urgency=medium

  * From Linux 4.7, the iptables connection tracking system will no longer
    automatically load helper modules.  If your firewall configuration
    depends on connection tracking helpers, you should explicitly load the
    required modules.  For more information, see
    <https://home.regit.org/netfilter-en/secure-use-of-helpers/>.

 -- Ben Hutchings <ben@decadent.org.uk>  Sat, 29 Oct 2016 01:53:18 +0100

#40 Updated by intrigeri 2016-11-06 10:46:44

Will this be fixed by the branch for Bug #11885?

#41 Updated by bertagaz 2016-11-06 12:41:28

intrigeri wrote:
> Will this be fixed by the branch for Bug #11885?

I’m not sure yet, I’ve seen failures when trying at home to run a 4.7 kernel, but that was with the 4.7.2. I think this ticket needs to wait a bit more to see how the newer 4.7.8 one works regarding this. I also want to test it a bit on bare metal.

#42 Updated by bertagaz 2016-11-08 20:21:20

I’ve removed the outdated related branch that existed. Linux 4.7 is now in all main branches, so we’ll track this bug though their respective Jenkins runs. We can create a more meaningful and active branch for this ticket.

#43 Updated by intrigeri 2016-11-13 08:59:53

It happened on https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_test-11616-emergency-shutdown-applet-is-fragile/1/ that has Linux 4.7.

#44 Updated by intrigeri 2016-11-16 10:17:13

And a user just reported that he still sees the bug on Tails 2.7.

#46 Updated by sonicsnail 2016-11-27 21:54:59

I tested again with Tails 3.0~alpha1, which is on Linux 4.8.0. Now the machine successfully powers off, whereas on 2.6~rc1 through 2.7 it would hang. I presume the memory is successfully wiping too because there’s a delay between “You can now remove…” and powering off, but no asterisks are shown.

#49 Updated by intrigeri 2016-12-21 17:51:04

FWIW the workaround added in our automated test suite (commit:cd66f0b) doesn’t work on current feature/stretch apparently: all I see during memory wipe is a black screen (at least in features/usb_install.feature), so if there’s a kernel panic we don’t see the busybox error message. This blocks updating parts of the test suite for Stretch, e.g. large parts of the USB and persistence tests. Bare metal is not affected (at least one one MacBook Pro I tried this on).

I’m currently testing a workaround to this problem, that should apply to Tails 2.x and 3.x if it works.

#50 Updated by intrigeri 2016-12-21 18:39:29

intrigeri wrote:
> FWIW the workaround added in our automated test suite (commit:cd66f0b) doesn’t work on current feature/stretch apparently: all I see during memory wipe is a black screen (at least in features/usb_install.feature), so if there’s a kernel panic we don’t see the busybox error message.

Err, no: it’s “just” that it takes a long while before the message about memory erasure is displayed; I’ll file a separate ticket to track this, and meanwhile I’ll teach the test suite to be more patient.

#53 Updated by intrigeri 2016-12-22 16:10:48

intrigeri wrote:
> * wrt. what the “can’t fork” problem comes from:
> made the memory wipe script more verbose (commit:be9b18da8681cefd1a73de18dc44b15813f6281e) on all base branches; I want more data!

So it seems that memory wiping succeeds, but then attempting to run poweroff triggers the “can’t fork”, which kills the script. I’m trying something else on feature/stretch+11786-wait-before-poweroff and bugfix/11786-wait-before-poweroff.

#54 Updated by intrigeri 2016-12-22 17:26:14

intrigeri wrote:
> I’m trying something else on feature/stretch+11786-wait-before-poweroff and bugfix/11786-wait-before-poweroff.

Crap, even calling sleep triggers “can’t fork”: with busybox, even builtins are copies of the original busybox multi-call binary, so calling “sleep” is not an option; and “sleep” is not a builtin in dash nor bash anyway.

#60 Updated by intrigeri 2017-01-29 11:34:59

intrigeri wrote:
> One option (until we can rely on PaX’ memory sanitization or something) proposed by anonym would be to rewrite that section of the script in C and statically compiling it.

C, or anything else that can be statically compiled, e.g. Go or Rust.