Tails

#1 Updated by goupille 2019-12-06 18:46:48

just resent the second report (the ticket number is in the subject line)

#2 Updated by goupille 2019-12-06 18:52:56

I just resent you a third report (Apple Macbook air 4,1 mid 2011)

#3 Updated by goupille 2019-12-06 19:15:28

more reports :

macbook pro 2012 -> fowarded
macbook air 2012 -> Bug report: e5e2e75f8dccb894ec897184ccc295b4 (this user says that a macbook air 2013 is not affected)
macbook pro mid 2012 -> forwarded

then I stopped forwarding the reports (I can do that if needed) but it seems that a lot of people are affected…

#6 Updated by CyrilBrulebois 2019-12-07 23:25:09

Haven’t spotted anything obvious on the Linux kernel side at this point. And I couldn’t reproduce the issue with some other hardware (tails-installer’d 4.0 image upgraded to 4.1 after enabling persistence, or tails-installer’d 4.1 image).

I’m currently wondering whether some microcode update might be responsible for this apparently rather hardware-specific (even if affecting a wide range of models) issue. I couldn’t spot anything obvious in the Debian BTS though.

I think it might make sense to prepare unofficial images: one with the intel-microcode package downgraded to its earlier version (in Tails 4.0), and one with the linux-image-* package downgraded to its earlier version (in Tails 4.0). Possibly a third one with both packages downgraded, just in case.

Would submitters be happy to test such images? In which case, would they prefer ISO and/or IMG images?

#7 Updated by CyrilBrulebois 2019-12-08 16:24:13

I’ve pushed three branches to git:

1. hack/17320-revert-microcode
2. hack/17320-revert-linux
3. hack/17320-revert-both-linux-and-microcode

The relevant diffs of .packages files, compared to 4.1, are respectively:

1. with hack/17320-revert-microcode:

-intel-microcode    3.20191112.1~deb10u1
+intel-microcode    3.20190618.1
-libnss3:amd64  2:3.42.1-1+deb10u1
+libnss3:amd64  2:3.42.1-1+deb10u2

2. with hack/17320-revert-linux:

-libnss3:amd64  2:3.42.1-1+deb10u1
+libnss3:amd64  2:3.42.1-1+deb10u2
-linux-image-5.3.0-2-amd64  5.3.9-2
+linux-image-5.3.0-trunk-amd64  5.3.2-1~exp1

3. with hack/17320-revert-both-linux-and-microcode:

-intel-microcode    3.20191112.1~deb10u1
+intel-microcode    3.20190618.1
-libnss3:amd64  2:3.42.1-1+deb10u1
+libnss3:amd64  2:3.42.1-1+deb10u2
-linux-image-5.3.0-2-amd64      5.3.9-2
+linux-image-5.3.0-trunk-amd64  5.3.2-1~exp1

(I didn’t try to get rid of the libnss3 difference.)

All resulting .img files were tested by dropping them on USB devices with gnome-disks, and can boot properly on a Thinkpad X230. I didn’t try to run the test suite, and didn’t test much more than booting to the desktop. But at least they aren’t obviously broken, so I’d be happy to have test results from users affected by this MacBookPro booting issue.

I’m thinking about sharing the resulting files, signed with my ckbriseup.net@ key, on my company’s webserver and maybe via Torrent files if people are more comfortable with this way of grabbing files. Of course, the latter would likely result in fewer peers than is usual for official releases.

#8 Updated by pushbox 2019-12-08 16:42:34

I can test the images on macbook pro.
Using balenaEtcher, so both formats should work.
I’m okay with downloading them from web or torrent.

#9 Updated by CyrilBrulebois 2019-12-08 18:36:46

Here we go: https://tails.debamax.com/ has ISO+IMG images for all 3 proposed branches, with detached signatures from my personal Tails key.

If users would have enough time/bandwidth/patience, I’d be happy to have results of Tails 4.1 image deployed “from scratch” (to rule out possible troubles due to upgrading from 4.0), and with the 3 images available on my company’s webserver. I have no guarantee the issue is caused by the microcode update and/or the linux update, but those are my best guess at the moment…

Also please mention which MacBookPro you’re testing with (e.g. 4,1 or 8,1, etc.).

Thanks already!

#10 Updated by pushbox 2019-12-08 21:45:58

MacBookPro10,1
All 3 of the branches booting up without any problem.
ISO and IMG format.

The official 4.1 build is not booting, I had to recheck it after the 6 successes.

#11 Updated by Jambalaya 2019-12-10 06:54:36

I just registered after finding this thread and it being my savior for Tails 4.1 release. I do not have a MacBook Pro, I have an iMac (iMac11,3) that could no longer boot Tails with 4.1, no other changes. Got kernel panic screen and saw a lot of people on Reddit had the same issue. I found my way back here and this thread, tried the above and can report - after having nothing to lose - that the “Revert both Linux and microcode” img fixed my issue. The “Revert Linux” img produced the same kernel panic. Thanks for your test builds and hope this data helps.

#12 Updated by goupille 2019-12-10 15:43:43

@CyrilBrulebois, given the reports we’re receiving, it looks like the problem is fixed with hack/17320-revert-microcode and hack/17320-revert-both-linux-and-microcode. However hack/17320-revert-linux doesn’t fixe the issue. I’m forwarding you the reports

#13 Updated by geb 2019-12-11 12:23:18

Hi,

Just tested on a MacBook Pro (13 pouces, mi-2009) (as presented on https://checkcoverage.apple.com/) / Macbook pro 5.5 (as show in mac os / linux kernel panic trace).

- 4.1 generated the same kernel panic message other reported (have pictures if needed).

- tails-amd64-hack_17320-revert-both-linux-and-microcode-4.1.1-20191208T1518Z-84f1e5957a.img works

- tails-amd64-hack_17320-revert-linux-4.1.1-20191208T1328Z-1af7c412ed.img do not work
- tails-amd64-hack_17320-revert-microcode-4.1.1-20191208T1411Z-927a0f6f85.img works

If needed I could ask the owner of the computer to do more tests later.

#14 Updated by intrigeri 2019-12-11 16:23:32

Thanks everyone who’s been helping debug this problem!

FWIW here’s a semi-random hunch (disclaimer: I have some fever).

The Initramfs unpacking failed: invalid magic at start of compressed archive message, combined with the fact only Apple hardware seems to be affected (as opposed to specific CPU models that may be installed on MacBooks and on PCs) makes me wonder if the microcode update is directly the culprit, or if it broke stuff indirectly.

For example, in 4.0, initrd.img is 33367476 bytes large, which is less than 32 MiB; while in 4.1, it is 33817076 bytes large, which is greater than 32 MiB. I find this suspicious and I would not be surprised if Apple’s EFI firmware behaved in a different way than others.

I can’t easily build/download @CyrilBrulebois’s test images today but I would be very curious to know if the test images that fix the problem bring back the initramfs size under 32 MiB :) If they do, it would be interesting to test images that have the updated microcodes but an initramfs smaller than 32 MiB.

Also, it would be nice if affected users could try adding the dis_ucode_ldr boot option, which disables the microcode loader.

#17 Updated by CyrilBrulebois 2019-12-12 16:07:05

Thanks everyone for your input indeed!

I had some troubles reconciling some findings, namely the absence of obvious reports on the Debian BTS regarding possible regressions on all Mac hardware after the intel-microcode update, and even if an upload to unstable happened yesterday mentioning some rollback, there wasn’t anything obvious regarding our rather global problem.

Looking back at the size of our initramfs in test images:

-r--r--r-- 1 root root 33353972 Dec  8 15:09 tails-amd64-hack_17320-revert-both-linux-and-microcode-4.1.1-20191208T1518Z-84f1e5957a.iso.loop/live/initrd.img
-r--r--r-- 1 root root 33362904 Dec  8 15:09 tails-amd64-hack_17320-revert-microcode-4.1.1-20191208T1411Z-927a0f6f85.iso.loop/live/initrd.img
-r--r--r-- 1 root root 33797596 Dec  8 14:26 tails-amd64-hack_17320-revert-linux-4.1.1-20191208T1328Z-1af7c412ed.iso.loop/live/initrd.img

With 32 MiB = 33554432 meaning both images with a microcode revert are below this limit, and the one with just the linux revert is above the limit. This seems to neatly match most of our findings, except for the very first report by @pushbox who mentioned the third image booted fine as well. I’m going to concentrate on the other reports for the time being.

While wondering what we could strip from the 4.1 image, without touching anything else, we figured removing all network drivers from the initramfs would make a huge difference, possibly without having high impacts on general use cases.

I’ve therefore pushed a new branch called hack/17320-remove-net-drivers which implements that by removing them from the generated initramfs, also lowering the maximal allowed size from 35 MiB to 32 MiB (this check was initially intended to make sure compression happens), with a final commit adding the +x bit on the initramfs hook after I verified failing to do so triggers a build failure because the check is triggered. The resulting initramfs is way smaller than 32 MiB:

-r--r--r-- 1 root root 29197812 Dec 12 15:29 tails-amd64-hack_17320-remove-net-drivers-4.1.1-20191212T1503Z-094dbcce39.iso.loop/live/initrd.img

and I’d be happy to have confirmations from everyone able to test that this image (containing the same versions of intel-microcode and linux as 4.1) boots fine, which would likely confirm the size of the initramfs is the matter, rather than the actual versions/contents of those packages.

Direct links (add .sig for the detached signature):

• https://tails.debamax.com/tails-amd64-hack_17320-remove-net-drivers-4.1.1-20191212T1503Z-094dbcce39.iso
• https://tails.debamax.com/tails-amd64-hack_17320-remove-net-drivers-4.1.1-20191212T1503Z-094dbcce39.img

#18 Updated by intrigeri 2019-12-12 16:18:19

Yeah! :)))

> Direct links (add .sig for the detached signature):

> * https://tails.debamax.com/tails-amd64-hack_17320-remove-net-drivers-4.1.1-20191212T1503Z-094dbcce39.iso
> * https://tails.debamax.com/tails-amd64-hack_17320-remove-net-drivers-4.1.1-20191212T1503Z-094dbcce39.img

Dear testers, please use the .img, not the .iso. Thanks!

(Rationale: the ISO image is known to not boot on MacBooks, for mostly unrelated reasons, and when it fails the error message is very similar to the one we’re seeing here, so testing reports about the ISO won’t provide useful info but can instead muddy the water.)

#19 Updated by CyrilBrulebois 2019-12-12 16:23:24

Thanks for mentioning that. I wasn’t sure at first whether the ISO→IMG process might have anything to do with the issue, and some users mentioned tails-installer, that’s why I published ISO+IMG for all test images, so that everyone could pick their favorite process.

#20 Updated by intrigeri 2019-12-12 16:42:04

> Thanks for mentioning that. I wasn’t sure at first whether the ISO→IMG process might have anything to do with the issue, and some users mentioned tails-installer, that’s why I published ISO+IMG for all test images, so that everyone could pick their favorite process.

Indeed, you’re right, ISO + Tails Installer should work. But ISO installed “ala dd/cat” (e.g. with GNOME Disks) will not.

#21 Updated by CyrilBrulebois 2019-12-12 22:32:49

I’ve got an offline report that iMac12,2 boots successfully with my latest image (shrinking initramfs without downgrading components), hardware that was not booting with Tails 4.1.

#22 Updated by cheesyvampire 2019-12-13 02:57:08

I tried using the .IMG with balenaEtcher and the 17320-remove-net-drivers-4.1.1 .IMG and it booted without the previous kernel panic that I was suffering. It’s a MacBookPro8_1 per everymac.com.

Gracias!

#24 Updated by goupille 2019-12-13 21:25:22

tails-amd64-hack_17320-remove-net-drivers-4.1.1-20191212T1503Z-094dbcce39.img
is booting fine on these (they were alll affected by this problem with tails 4.1):

MacBook 13-inch (Late 2009)
MacBook Pro 10.1 (mid 2012)
Macbook Air 4,1 11-inch (mid 2011)

#26 Updated by abZLQrnoXoj 2019-12-14 05:45:32

The remove-net-drivers version boots fine on MacBookPro8,1 and MacBookPro9,1

#31 Updated by goupille 2019-12-14 21:29:50

CyrilBrulebois wrote:
> Thanks, @goupille, that’s very encouraging.
>

it is also fixing the issue on a MacBook Pro 10,2 (late 2012)

> I’m contemplating shipping a variant of this bugfix as 4.1.1, seeing if removing drivers we blacklist anyway would be sufficient. Otherwise, double-checking we don’t regress with my latest published image, on various non-Mac laptops would be great. I’ll be trying to report back on this front during the week-end.

I think that would be great

#32 Updated by geb 2019-12-16 10:34:21

CyrilBrulebois wrote:
> Thanks, @goupille, that’s very encouraging.
>
> I’m contemplating shipping a variant of this bugfix as 4.1.1

+1 on this (especially considering that it may prevent people who did not already upgrade to need to manually upgrade). Happy to perform tests (or more precisely ask somebody to do, who would only be available this week) if that helps.

#36 Updated by CyrilBrulebois 2020-01-03 05:53:45

For the avoidance of doubt/surprise if users were to stumble upon this bug report in the future: I’ve removed the test images to the staging area hosted by my company (https://tails.debamax.com), following the 4.1.1 release; I’ve also mentioned this in its new README.txt (as I might re-use this space for other Tails purposes later on).