Tails

#1 Updated by intrigeri 2015-04-04 15:32:49

Note that I’ve tried removing irqbalance, as it seemed to be a workaround for people affected by similar problems, but it didn’t help.

Next steps are to try and reproduce in various settings, e.g.:

• starting the same ISO with QEMU, outside of the test suite environment and of libvirt
• starting the same ISO with libvirt, outside of the test suite environment
• changing the QEMU command line a bit
• allocating HugePages statically
• disabling automatic allocation of AnonHugePages
• more ideas welcome!

#2 Updated by intrigeri 2015-04-09 13:07:57

Next step is probably to retry with linux (3.16.7-ckt9-1) first.

#3 Updated by intrigeri 2015-04-17 13:01:00

With linux 3.16.7-ckt9-2, I see basically the same:

[  231.044003] INFO: rcu_sched self-detected stall on CPU { 2}  (t=5250 jiffies g=3872 c=3871 q=50)
[  256.064003] BUG: soft lockup - CPU#1 stuck for 22s! [qemu-system-x86:1221]
[  256.080003] BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:2:328]
[  284.064002] BUG: soft lockup - CPU#1 stuck for 22s! [qemu-system-x86:1221]
[  284.080003] BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:2:328]
[  294.056003] INFO: rcu_sched self-detected stall on CPU { 2}  (t=21003 jiffies g=3872 c=3871 q=50)
[  320.064002] BUG: soft lockup - CPU#1 stuck for 23s! [qemu-system-x86:1221]
[  320.080003] BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:2:328]
[  348.064003] BUG: soft lockup - CPU#1 stuck for 23s! [qemu-system-x86:1221]
[  348.080003] BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:2:328]
[  357.068003] INFO: rcu_sched self-detected stall on CPU { 2}  (t=36756 jiffies g=3872 c=3871 q=50)
[  384.064002] BUG: soft lockup - CPU#1 stuck for 22s! [qemu-system-x86:1221]
[  384.080003] BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:2:328]
[  412.064002] BUG: soft lockup - CPU#1 stuck for 22s! [qemu-system-x86:1221]
[  412.080003] BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:2:328]
[  420.080003] INFO: rcu_sched self-detected stall on CPU { 2}  (t=52509 jiffies g=3872 c=3871 q=50)
[  448.064004] BUG: soft lockup - CPU#1 stuck for 23s! [qemu-system-x86:1221]
[  448.080003] BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:2:328]
[  476.064006] BUG: soft lockup - CPU#1 stuck for 22s! [qemu-system-x86:1221]
[  476.080004] BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:2:328]

#4 Updated by intrigeri 2015-04-17 13:20:58

With Linux 3.19.3-1~exp1:

[   76.076003] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:19]
[  104.076004] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:19]
[  132.076004] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:0:19]
[  160.076003] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:0:19]
[  188.076003] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:0:19]
[  216.076003] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:0:19]
[  244.076003] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:0:19]
[  272.076003] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:19]
[  300.076003] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:19]
[  328.076003] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:19]
[  356.076003] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:19]
[  384.076003] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:19]
[  412.076003] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:19]
[  440.076004] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:19]
[  468.076004] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:19]

#6 Updated by intrigeri 2015-04-17 16:36:09

Reported on the Debian BTS: https://bugs.debian.org/782774

#7 Updated by hybridwipe 2015-07-07 21:38:33

How is the first kvm instance being started (i.e., what QEMU command line), and what image/iso is it running? I tried booting a testing image to boot tails 1.3.2 (to minimize changes..), but kvm wouldn’t even load inside kvm instance 1.

#8 Updated by intrigeri 2015-07-08 01:42:52

> How is the first kvm instance being started (i.e., what QEMU command line), and what image/iso is it running? I tried booting a testing image to boot tails 1.3.2 (to minimize changes..), but kvm wouldn’t even load inside kvm instance 1.

We’re using libvirt, both on the host and (via the test suite) in the level 1 guest.
I suggest taking this question to our support channels.

#9 Updated by intrigeri 2015-07-08 11:05:40

Reproduced with today’s Jessie kernel (3.16.7-ckt11-1).

#10 Updated by bertagaz 2015-09-19 09:40:09

Reproduced with jessie-backport kernel (4.1.0-0.bpo.2 : 4.1.6-1~bpo8+1 (2015-09-09)).

Tried to boot the L1 VM with nmi_watchdog=0, also fixed the CPUs frequency on the host as I read it may be due to frequency scaling, but none of that worked so far.

I’ve tried to set the L1 VM cpu in host-passthrough mode, and then saw an error I haven’t seen before:

[  231.456005] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 1, t=36762 jiffies, g=432, c=431, q=92)
[  231.456005] rcu_sched kthread starved for 36762 jiffies!
[  240.168087] INFO: task jbd2/dm-0-8:139 blocked for more than 120 seconds.
[  240.169158]       Tainted: G             L  4.1.0-0.bpo.2-amd64 #1
[  240.170057] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  256.076002] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:3:329]

#14 Updated by bertagaz 2015-09-25 03:30:54

Note that all this tests were done with a qemu from Wheezy on the host, which is a bit old (1.1.2). Meanwhile there have been changes in qemu related to APIC and such. There’s a wheezy-backports version available which is 2.1, so a next step could be to upgrade to it on the host and give some more testings.

#16 Updated by intrigeri 2015-11-01 02:12:58

bertagaz wrote:
> Note that all this tests were done with a qemu from Wheezy on the host, which is a bit old (1.1.2). Meanwhile there have been changes in qemu related to APIC and such. There’s a wheezy-backports version available which is 2.1, so a next step could be to upgrade to it on the host and give some more testings.

Good idea.

I suggest we hold on, and block on Feature #9264 though: if we’re going to stop doing nested virtualization soonish, then it’s not worth investing lots of time into fixing it now (fixing it 6 months ago would have been different). What do you think?

#19 Updated by intrigeri 2015-12-06 15:22:24

I think we should now do this, in that order, and stopping on success:

1. upgrade the host (lizard) to Jessie: we have to do it soonish anyway, and it’ll give us the QEMU upgrade bertagaz wanted to test :)
2. reproduce the bug on another isotesterN if not done yet
3. test with Linux 4.3 on isotesterN
4. make sure Feature #9264 saves the day

#22 Updated by kytv 2015-12-07 12:44:51

intrigeri wrote:

> # test with Linux 4.3 on isotesterN

I need to try 4.3 (and 4.4-rc) but to add a datapoint which may be helpful even though my hardware is wildly different.

With 4.2.x my kernel reliably OOPSes during the UEFI Tails-Installer test. I reverted to a self-compiled 4.1.13 in the meantime.

#23 Updated by kytv 2015-12-07 14:34:25

kytv wrote:
> intrigeri wrote:
>
> > # test with Linux 4.3 on isotesterN
>
> I need to try 4.3 (and 4.4-rc) but to add a datapoint which may be helpful even though my hardware is wildly different.
>

The oops can be reproduced with 4.4.0-rc4, but the screen is still responsive, unlike with 4.2.x.

Perhaps isoteesterX has better luck with it than I do.

#25 Updated by intrigeri 2016-02-29 15:26:43

FWIW:

• I’ve set cpu mode=host-passthrough on all isotesters, as suggested by the OpenStack doc when doing nested KVM.
• Since commit:8544f14485289653b8dde9a29dc496f79e768866 (introduced on feature/jessie, part of Tails 2.0+) the test suite is run with cpu mode='host-model'.

It might be that one of these changes fixes the problem this ticket is about => worth retrying with Linux 3.16 from Jessie, and Linux 4.4 from sid, maybe. I might play with this one of these days when bored.

Also, wrt. the QEMU backport for Wheezy: it seems to be unmaintained and lacks a big pile of security fixes, so I don’t think it’s an option.

After the conclusions I’ve reached on Feature #11011, I think we’re back to needing to fix this problem before Wheezy’s Linux 3.2 is not maintained anymore, that is before the end of wheezy-lts, since we can’t count on a 2nd testing machine to magically solve this problem for us => bertagaz, can you set an appropriate target version?

#27 Updated by anonym 2016-03-01 10:40:44

I think I’ve stumbled into this issue too now: Feature #6354#note-16.

intrigeri wrote:
> Another option would be to try a newer kernel on the host.

On my level-0 host I am running Debian Unstable, with kernel version 4.3.0-1-amd64 (and I’ve tried with the same backported kernels in the other two VMs), so I don’t think this will help. :/

#28 Updated by intrigeri 2016-03-01 18:31:57

> I think I’ve stumbled into this issue too now: Feature #6354#note-16.

> intrigeri wrote:
>> Another option would be to try a newer kernel on the host.

> On my level-0 host I am running Debian Unstable, with kernel version 4.3.0-1-amd64 (and I’ve tried with the same backported kernels in the other two VMs), so I don’t think this will help. :/

I’d like to confirm this is really the same problem we’re seeing, first, so: is it fixed for you if the level-1 VM (where Vagrant will be run) runs Wheezy’s kernel + Jessie’s userspace?

#29 Updated by anonym 2016-03-01 22:58:49

intrigeri wrote:
> > On my level-0 host I am running Debian Unstable, with kernel version 4.3.0-1-amd64 (and I’ve tried with the same backported kernels in the other two VMs), so I don’t think this will help. :/
>
> I’d like to confirm this is really the same problem we’re seeing, first, so: is it fixed for you if the level-1 VM (where Vagrant will be run) runs Wheezy’s kernel + Jessie’s userspace?

As mentioned on #tails-dev, this doesn’t fix the problem. Specifically I was running:

• level 0: 4.4.0-1-amd64 (Sid kernel/user space)
• level 1: 3.2.0-4-amd64 (Wheezy kernel, Jessie user space)
• level 2: 3.16.0-4-amd64 (Jessie kernel/user space)

I tried twice, and the first time even level-0 crashed! Second time I got this in level-2:

kernel:[ 1092.132037] BUG: soft lockup - CPU#3 stuck for 22s! [apt-get:25441] 

which actually isn’t exactly the same as I reported in Feature #6354#note-16. I couldn’t see what happened in level-1 since it crashed, and the journal was lost.

I wonder if it matters that level-0 runs kernel 4.4.0. I guess lizard runs the Jessie (or Wheezy?) kernel?

#30 Updated by intrigeri 2016-03-02 10:43:27

> I wonder if it matters that level-0 runs kernel 4.4.0. I guess lizard runs the Jessie (or Wheezy?) kernel?

lizard level-0 host runs 3.16 from wheezy-backports.

#31 Updated by anonym 2016-03-02 21:54:34

I do not have the issue when both level-0 and level-1 are running Debian Sid, see Feature #6354#note-18. So if my kernel issue is related somehow to the kernel issue this ticket is about, perhaps everything will be good once Debian Stretch is out, and lizard and the isobuildes migrate to it.

An interesting experiment that we could do how would be installing the 4.4.0-1 kernel from Jessie backports on both lizard and the isotesters (and isobuilders?). What do you think? Perhaps the user land also matters.

#32 Updated by intrigeri 2016-03-12 10:25:58

anonym wrote:
> I do not have the issue when both level-0 and level-1 are running Debian Sid, see Feature #6354#note-18. So if my kernel issue is related somehow to the kernel issue this ticket is about, perhaps everything will be good once Debian Stretch is out, and lizard and the isobuildes migrate to it.
>
> An interesting experiment that we could do how would be installing the 4.4.0-1 kernel from Jessie backports on both lizard and the isotesters (and isobuilders?). What do you think? Perhaps the user land also matters.

Tried it, but sadly we can’t do that as long as we’re running Wheezy on the host system: “linux-image-4.3.0-0.bpo.1-amd64 : Breaks: udev (< 208-8~) but 175-7.2 is to be installed”. So we’re back to blocking on Feature #11178, as previously.

#34 Updated by intrigeri 2016-03-31 15:45:41

After upgrading the level-0 host to Jessie, this combination (quite expectedly) still fails:

• level-0: Jessie userspace + kernel
• level-1: Jessie userspace, 4.4 kernel from jessie-backports
• level-2: Jessie userspace + kernel

But this works:

• level-0: Jessie userspace, 4.4 kernel from jessie-backports
• level-1: Jessie userspace, 4.4 kernel from jessie-backports
• level-2: Jessie userspace + kernel

Thanks anonym for the lead!