Bug #16818

Impossible to start on MacBookPro 8,1

Added by sajolida 2019-06-17 17:33:34 . Updated 2019-07-09 11:59:24 .

Status:
Resolved
Priority:
Elevated
Assignee:
intrigeri
Category:
Hardware support
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
bugfix/16818-buster-vagrant-box
Type of work:
Research
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

That’s with bc022ef71e, the Tails Mac I have with me, and the same USB stick (and copy of Tails) that I’ve used for all my other tests.

Behavior is a bit strange:

  • At first I had a hard time making the USB stick appear in the Mac startup disks.
  • I finally got it there, both on its own and by plugging a Tails 3.14 first and replacing it.
  • Both times I got a kernel panic right after the boot loader menu. See attachment.

The main error message in the attachment is: “Initramfs unpacking failed: junk in compressed archive”.


Files


Subtasks


Related issues

Related to Tails - Bug #16755: Call for testing: feature/buster (June 2019 edition) Resolved 2019-06-18
Blocks Tails - Feature #8393: Replace sound-juicer with Goobox (aka. GNOME CD Player) Confirmed 2014-12-05
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by intrigeri 2019-06-18 10:18:10

  • Description updated
  • Category set to Hardware support
  • Type of work changed from Code to Research

@sajolida, a few questions:

  • This machine is booting in UEFI mode, right? (grey background in the boot loader, no nice Tails custom syslinux theme)
  • Just to make sure we can reasonably eliminate a faulty USB stick or corrupted data, can you please retry this with another USB stick on which you’ve installed from scratch a brand new feature/buster?

Meanwhile, I’m going to briefly research the error message to see if there’s another explanation.

#2 Updated by intrigeri 2019-06-18 10:23:44

Next step for me: compare the size of the initrd in 3.14 and feature/buster.

@sajolida, can you also confirm that 3.14 boots fine on this machine?

#3 Updated by intrigeri 2019-06-18 10:30:28

  • related to Bug #16755: Call for testing: feature/buster (June 2019 edition) added

#4 Updated by intrigeri 2019-06-18 11:35:34

initrd.img is 34730980 bytes (34M) on current feature/buster vs. 32292792 bytes (31M) on 3.14. This might explain “Initramfs unpacking failed: junk in compressed archive”, e.g. in case the interaction between syslinux and the Apple-style EFI firmware has trouble loading this much data in memory. It could be that there’s a 32M limit somewhere. I’ve compared their content and there’s no huge difference. We could try:

  • trimming some useless stuff, e.g. cryptsetup and friends, ntfs-3g, kernel modules that are useless in the initrd (sound, net, scsi, infiniband)
  • making our aufs.ko smaller (it’s 20MB uncompressed!)

Then, once we have a feature/buster -based branch with an initramfs that’s not bigger than 3.14’s, we could ask sajolida to retry. Let’s do all this once the faulty USB stick hypothesis has been fully invalidated.

Of course, this does not explain “At first I had a hard time making the USB stick appear in the Mac startup disks”, which I suspect is an entirely different bug (I’ve seen this happen occasionally on various Macs including this one and could never find a way to reliably reproduce it).

#5 Updated by intrigeri 2019-06-18 11:36:33

  • Priority changed from Normal to Elevated

#6 Updated by sajolida 2019-06-19 09:38:25

  • blocks Feature #8393: Replace sound-juicer with Goobox (aka. GNOME CD Player) added

#7 Updated by sajolida 2019-06-19 10:31:32

Different USB, same kernel panic. After only a quick look, many of the error messages are similar. The “Kernel Offset” is different.
Tell me if I should inspect it more in depth.

And yes, 3.14 starts fine. It’s the first time I see something like this on this machine.

#8 Updated by intrigeri 2019-06-19 10:46:38

  • Subject changed from Impossible to start on Mac to Impossible to start on MacBookPro 8,1

#9 Updated by intrigeri 2019-06-19 10:48:04

  • Status changed from New to Confirmed

Thank you.

The “faulty USB stick” hypothesis has been fully invalidated ⇒ time to try & fix this. Bug #16818#note-4 has my ideas.

#10 Updated by intrigeri 2019-07-05 11:57:45

  • Assignee set to intrigeri

So:

  • the aufs.ko we build ourselves from upstream sources is 20MB and running strings on it yields 290564 lines, including strings I would not expect there, such as scsi (35 times) and sndbuf_policy (34 times); file says that it is compiled with debug_info and not stripped; gzip’ing it with default settings gives me a 5.8MB file
  • the aufs.ko compiled on my sid by dkms is 572KB and running strings on it yields 3319 lines; not stripped, but no debug info; gzip’ing it → 147KB

So it looks like we could save a few MB on the initramfs size (therefore making boot faster for every user, and shortening the time between the bootloader and the plymouth splash, on top of hopefully fixing the problem this ticket is about), by removing this debug_info. I’ll look into this option before starting to remove kernel modules.

#11 Updated by intrigeri 2019-07-05 12:05:01

  • Feature Branch set to bugfix/16818-smaller-aufs-module

#12 Updated by intrigeri 2019-07-05 12:05:18

  • Status changed from Confirmed to In Progress

Applied in changeset commit:tails|4e7504c7e51bf30b019147df7ee06daef2ade7c6.

#13 Updated by intrigeri 2019-07-05 15:49:50

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to sajolida

initrd.img is down to 29MB on this branch => merged into feature/buster so I can ask sajolida to test it (my branch is based on stable so Jenkins won’t build it for Buster).

@sajolida, can you please try booting a feature/buster USB image dated from tomorrow (earliest) on this MacBookPro 8,1 and tell me if it helps?

#14 Updated by intrigeri 2019-07-05 15:57:18

And to be clear, regardless of sajolida’s feedback, I’ll request a proper review and merge into stable once I’ve seen Jenkins results.

#15 Updated by intrigeri 2019-07-05 20:31:27

intrigeri wrote:
> initrd.img is down to 29MB on this branch

… and to 31MB (31858656 bytes) on feature/buster with this branch merged, which is smaller than the initramfs we shipped in 3.14.2, so if initramfs size is the problem, then this should definitely fix it, and I’ll want to update the limit we set in config/binary_local-hooks/05-check_initramfs-size. But if my branch does not fix this bug, then we’ll know we have a different problem and I will want to try feature/buster with Stretch’s syslinux.

#16 Updated by intrigeri 2019-07-05 20:48:53

@sajolida, if current feature/buster does not fix this, I’m still interested in the answer to this question: “This machine is booting in UEFI mode, right? (grey background in the boot loader, no nice Tails custom syslinux theme)”

#17 Updated by intrigeri 2019-07-05 20:50:42

@alant, please try booting current feature/buster on the MacBookPro you have access to.

#18 Updated by intrigeri 2019-07-06 08:06:39

  • Assignee deleted (sajolida)
  • Target version changed from Tails_4.0 to Tails_3.15

intrigeri wrote:
> And to be clear, regardless of sajolida’s feedback, I’ll request a proper review and merge into stable once I’ve seen Jenkins results.

Jenkins is happy enough (https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_bugfix-16818-smaller-aufs-module/, https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-buster/) so please review and merge into stable; but don’t mark this ticket as “Fix Committed” until we have sajolida’s or alant’s feedback. segfault, anonym, @hefee: anyone?

#19 Updated by segfault 2019-07-07 11:36:45

  • Assignee set to segfault

#20 Updated by segfault 2019-07-07 11:40:53

  • Status changed from Needs Validation to In Progress

Applied in changeset commit:tails|f636011767eedd0567598689a8b392b3cac03821.

#21 Updated by segfault 2019-07-07 11:44:21

  • Assignee changed from segfault to sajolida

Merged into stable -> devel -> feature/buster. Assigning to sajolida to provide feedback.

#22 Updated by sajolida 2019-07-08 16:58:48

> @sajolida, if current feature/buster does not fix this, I’m still interested in the answer to this question: “This machine is booting in UEFI mode, right? (grey background in the boot loader, no nice Tails custom syslinux theme)”

Correct!

#23 Updated by sajolida 2019-07-09 06:45:38

  • Assignee changed from sajolida to intrigeri

#25 Updated by intrigeri 2019-07-09 08:19:13

  • Target version changed from Tails_3.15 to Tails_3.16

#26 Updated by intrigeri 2019-07-09 08:39:14

  • Feature Branch deleted (bugfix/16818-smaller-aufs-module)

#27 Updated by sajolida 2019-07-09 08:43:32

The same in public:

I’m coming back with bad news: it got worse :(

Here is what I did:

  • Install 3.14.2 on a USB stick
    • Starts fine on the Mac.
  • Install 5712f23dbe from June 19 on another USB stick
    • As reported initially, it shows up in the Mac startup disks, then displays the UEFI mode bootloader (gray background) and the kernel panic
  • Install 2007a18fa5 from July 8 on another USB stick
    • This time it doesn’t even appear in the Mac startup disks.
    • The same USB stick starts fine on a X200.

I’ve used systematically my most reliable technique, which is to get to the startup disks with no USB stick plugged in (only the internal hard disk appears) and then only plug the USB stick. If I unplug the USB stick it disappears and I can plug another one (I have only 1 USB port). Both 3.14.2 and 5712f23dbe appear and disappear fine from the startup disks, with 2007a18fa5 the USB stick itself blinks but doesn’t appear in the list.

Sorry!

NB: You changed the target version from 4.0 to 3.15 while this bug is only affecting feature/buster, but you probably have a good reason for that :)

#28 Updated by intrigeri 2019-07-09 09:13:57

  • Target version changed from Tails_3.16 to Tails_4.0

> I’m coming back with bad news: it got worse :(

Ouch!

> * Install 5712f23dbe from June 19 on another USB stick
> As reported initially, it shows up in the Mac startup disks, then displays the UEFI mode bootloader (gray background) and the kernel panic
> * Install 2007a18fa5 from July 8 on another USB stick
> This time it doesn’t even appear in the Mac startup disks.
> The same USB stick starts fine on a X200.

I’ve looked at the changes between these 2 images:

  • on our side (5712f23dbe..2007a18fa5):
    • no change to partitioning or MBR -related stuff
    • only update between 3.14 and 3.15 APT suites: Thunderbird
  • on Debian side: no syslinux update

So I really don’t understand how behavior can have changed for you :/

You initially reported “At first I had a hard time making the USB stick appear in the Mac startup disks”, about an earlier version (bc022ef71e, June 8), which suggests that there’s some amount of non-determinism around all this, and makes me wonder why an intermediary version (5712f23dbe, June 19) works better than that one. So I’d like to know how many times you did both tests (the successful one and the failing one) and whether you did them in exactly the same way: all kinds of small details can matter here, for example “was the USB stick already started on another computer before?”, “did you reboot or power off?”, “was that ‘another USB stick’ the same one in both cases?”, etc.

Also, it would be useful to know whether a feature/buster USB stick cloned with Tails Installer displays the same behavior or not: there’s a small chance that the bootloader bits on the resulting USB stick are 100% Buster while the ones you’ve tested are a mix between Stretch and Buster (see below for technical details if curious). This would help tell if the problem comes from the ISO to USB image conversion process.

In any case, it seems clear that you’ve never had any issue with 3.x but you see trouble with feature/buster, so let’s try some things that may help us understand where the problem comes from:

  • Buster with Stretch’s syslinux
  • Buster built in a Buster Vagrant box: I’m concerned that since commit:6e097195f13e44a4664051cad79f32dc2f134c9f, auto/scripts/create-usb-image-from-iso uses the syslinux binary from the (Buster) chroot, but it’s not run with chroot, so there’s a chance that it uses files from the Vagrant box (e.g. the MBR or whatnot), which might create inconsistencies and trouble

> NB: You changed the target version from 4.0 to 3.15 while this bug is only affecting feature/buster, but you probably have a good reason for that :)

That was because I wanted bugfix/16818-smaller-aufs-module into 3.15 which is now done ⇒ reverting.

#29 Updated by intrigeri 2019-07-09 09:37:16

  • Feature Branch set to bugfix/16818-buster-vagrant-box

#30 Updated by sajolida 2019-07-09 11:34:52

  • Status changed from In Progress to Resolved

OMG! You rock! It’s working now :)

This morning I missed the fact that I had USB images for bc022ef71e and 5712f23dbe but an ISO image for 2007a18fa5. D’oh!

Now I downloaded an USB image for 7523d0b2a0 and it starts fine :)))

Also, in my previous tests I managed to get rid of the non-determinism by always getting to the startup disks menu before plugging the USB stick.

#31 Updated by intrigeri 2019-07-09 11:57:39

#32 Updated by intrigeri 2019-07-09 11:59:24

I’ve moved my WIP to Bug #16868.