Tails

#1 Updated by intrigeri 2017-01-16 11:08:47

I could try reproducing this if it helps.

Note that switched to nec-xhci for a reason so let’s make sure we don’t break stuff it repaired, while repairing the problem you saw.

#3 Updated by anonym 2017-01-16 13:14:15

Uh… completely ignore the previous comment.

intrigeri wrote:
> I could try reproducing this if it helps.

That’d be great! Running the Booting Tails from a USB drive without a persistent partition and creating one scenario should be enough.

#5 Updated by intrigeri 2017-01-16 13:47:33

I guess that next step is to identify if this is the result of a change in Tails, in Debian (guest), or in Debian (host).

#6 Updated by anonym 2017-01-16 13:48:19

Now when I reproduced, the VM remained fully functional, but the persistence setup still freezes.

This is what I get in the journal:

amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: sd 8:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
amnesia kernel: sd 8:0:0:0: [sda] tag#0 CDB: Write(10) 2a 00 00 66 49 86 00 08 00 00
amnesia kernel: blk_update_request: I/O error, dev sda, sector 6703494
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1573254, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1573255, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1573256, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1573257, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1573258, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1573259, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1573260, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1573261, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1573262, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1573263, lost async page write
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: INFO: task mkfs.ext4:7244 blocked for more than 120 seconds.
amnesia kernel:       Tainted: G           OE   4.8.0-0.bpo.2-amd64 #1
amnesia kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
amnesia kernel: mkfs.ext4       D ffffa13dcc726000     0  7244   6078 0x20020000
amnesia kernel:  ffffa13dcc433080 ffffa13dff0e0080 0000000000000102 000000000000000e
amnesia kernel:  ffffa13dd5f44000 ffffa13dd5f43d08 7fffffffffffffff ffffffff8f9f0060
amnesia kernel:  ffffa13dd5f43d88 0007ffffffffffff ffffffff8f9ef7d1 0000000000000000
amnesia kernel: Call Trace:
amnesia kernel:  [<ffffffff8f9f0060>] ? bit_wait_timeout+0xa0/0xa0
amnesia kernel:  [<ffffffff8f9ef7d1>] ? schedule+0x31/0x80
amnesia kernel:  [<ffffffff8f9f2d8c>] ? schedule_timeout+0x21c/0x3c0
amnesia kernel:  [<ffffffff8f59010d>] ? pagevec_lookup_tag+0x1d/0x30
amnesia kernel:  [<ffffffff8f9f0060>] ? bit_wait_timeout+0xa0/0xa0
amnesia kernel:  [<ffffffff8f9eefe4>] ? io_schedule_timeout+0xb4/0x130
amnesia kernel:  [<ffffffff8f4bfdc6>] ? prepare_to_wait+0x56/0x80
amnesia kernel:  [<ffffffff8f9f0077>] ? bit_wait_io+0x17/0x60
amnesia kernel:  [<ffffffff8f9efb5c>] ? __wait_on_bit+0x5c/0x90
amnesia kernel:  [<ffffffff8f57fe1e>] ? find_get_pages_tag+0x15e/0x300
amnesia kernel:  [<ffffffff8f57ea94>] ? wait_on_page_bit+0xc4/0xe0
amnesia kernel:  [<ffffffff8f4c0130>] ? autoremove_wake_function+0x40/0x40
amnesia kernel:  [<ffffffff8f57eb87>] ? __filemap_fdatawait_range+0xd7/0x150
amnesia kernel:  [<ffffffff8f57ec0f>] ? filemap_fdatawait_range+0xf/0x30
amnesia kernel:  [<ffffffff8f581799>] ? filemap_write_and_wait_range+0x49/0x70
amnesia kernel:  [<ffffffff8f63f2b6>] ? blkdev_fsync+0x16/0x40
amnesia kernel:  [<ffffffff8f6383e8>] ? do_fsync+0x38/0x60
amnesia kernel:  [<ffffffff8f63865c>] ? SyS_fsync+0xc/0x10
amnesia kernel:  [<ffffffff8f403cf8>] ? do_int80_syscall_32+0x58/0x160
amnesia kernel:  [<ffffffff8f9f5473>] ? entry_INT80_compat+0x33/0x40
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: sd 8:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
amnesia kernel: sd 8:0:0:0: [sda] tag#0 CDB: Write(10) 2a 00 00 66 51 86 00 08 00 00
amnesia kernel: blk_update_request: I/O error, dev sda, sector 6705542
amnesia kernel: buffer_io_error: 2038 callbacks suppressed
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1575302, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1575303, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1575304, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1575305, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1575306, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1575307, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1575308, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1575309, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1575310, lost async page write
amnesia kernel: Buffer I/O error on dev dm-0, logical block 1575311, lost async page write
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
amnesia kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd

#7 Updated by anonym 2017-01-16 13:49:46

intrigeri wrote:
> I guess that next step is to identify if this is the result of a change in Tails, in Debian (guest), or in Debian (host).

I think “Debian (host)” because I can reproduce with Tails 2.9.1, but I did not have this issue when working on that release ~1 month ago.

#11 Updated by anonym 2017-01-17 11:20:00

intrigeri wrote:
> > Downgrading to qemu-system-x86 1:2.7+dfsg-3+b1 fixes the issue for me. Can you please confirm?
>
> Confirmed.
>
> FWIW hw/usb/hcd-xhci.c has changed quite a bit in v2.7.0..v2.8.0.

I saw that too (“135 insertions(+), 103 deletions(-)” for the record).

> It’s tempting to try reverting these changes and see what happens.

Sure, but it seems like a lot of work. I’ll give git-bisect a try.

If I fail, I suspect we’ll have to live with this issue for a while, so what about patching the domain with the fix above (i.e. switching USB controller) if we detect qemu-system-x86 > 2.7.0? And make it part of Feature #11739 (or similar) to find a proper solution.

#12 Updated by intrigeri 2017-01-17 11:26:11

> If I fail, I suspect we’ll have to live with this issue for a while, so what about patching the domain with the fix above (i.e. switching USB controller) if we detect qemu-system-x86 > 2.7.0?

Yes, if it doesn’t reintroduce the bugs we fixed by moving to nec-xhci.

#13 Updated by anonym 2017-01-17 14:07:47

anonym wrote:
> intrigeri wrote:
> > It’s tempting to try reverting these changes and see what happens.
>
> Sure, but it seems like a lot of work. I’ll give git-bisect a try.

Reverting the 10 commits made to hw/usb/hcd-xhci.c indeed fixes the issue for me! So, now I am bisecting (which will take a while…).

intrigeri wrote:
> > If I fail, I suspect we’ll have to live with this issue for a while, so what about patching the domain with the fix above (i.e. switching USB controller) if we detect qemu-system-x86 > 2.7.0?
>
> Yes, if it doesn’t reintroduce the bugs we fixed by moving to nec-xhci.

According to Bug #11588 it was introduced to fix USB-related issues on Jenkins, no mention of problems in anyone’s local setup. Let’s put an EOL of this workaround by reverting it in the branch for Feature #11739.

#15 Updated by intrigeri 2017-01-17 21:50:09

> As for now, we still need the workaround until Debian ships a fixed package by applying my patch or something similar, or by waiting for a fix from upstream. I’m wondering if it’s worth adding the workaround, or if we should hope for a speedy reaction from the Debian QEMU team. I think I’ll wait a few days and see if anything happens.

Sounds great! The few affected people should be able to apply your patch, or to downgrade QEMU, so there’s no big rush.

#16 Updated by anonym 2017-01-18 07:47:18

anonym wrote:
> I’ve also sent the patch to the QEMU devel mailing list, but it hasn’t appeared in their archive yet.

• http://lists.nongnu.org/archive/html/qemu-devel/2017-01/msg03433.html
• http://lists.nongnu.org/archive/html/qemu-devel/2017-01/msg03434.html

#18 Updated by intrigeri 2017-03-02 08:54:23

What’s the status of the patches sent upstream? (Maybe time to ping them?) It would be really nice if this was fixed in time for Stretch, otherwise we’ll waste quite some time dealing with the fallout.

#22 Updated by intrigeri 2017-04-16 13:27:04

Chances are that Stretch gets QEMU 2.8.1, so if our fix goes in 2.8.2 it might go into Stretch as well.

#24 Updated by intrigeri 2017-04-20 06:03:30

> Target version changed from Tails_2.12 to Tails_3.0

IMO our last chance to get this fixed in Stretch (and save developers & sysadmins some pain in the next 2 years) requires pinging upstream really soon. Do you need help writing this email? Feel free to reassign to me for 3.0~rc1 and I’ll nag them, I don’t find it hard to do personally.

#25 Updated by anonym 2017-04-20 09:40:32

intrigeri wrote:
> > Target version changed from Tails_2.12 to Tails_3.0
>
> IMO our last chance to get this fixed in Stretch (and save developers & sysadmins some pain in the next 2 years) requires pinging upstream really soon. Do you need help writing this email? Feel free to reassign to me for 3.0~rc1 and I’ll nag them, I don’t find it hard to do personally.

Sorry, I know I’m bad at pinging! :/ I have tried twice now to create a simple reproducer to post there as a sort of ping, but failed. Now I sent a ping to the list, plus Cc:ed the committer who introduced this potential regression, asking if my patch makes sense.

#32 Updated by intrigeri 2017-09-25 07:26:19

The exact same change that you’ve submitted upstream was independently applies in commit 99f9aeba5d461f79c9ce73f968ba0feb77fc1f5a. Let’s say it’s good news..

Next steps:

1. test this to confirm the bug is fixed: I could not do that as our test suite fails earlier with QEMU 2.10 (Call to virDomainAttachDeviceFlags failed: internal error: unable to execute QEMU command 'device_add': Property 'usb-storage.drive' can't find value 'drive-usb-disk0' (Guestfs::Error)), I’ll report this in a dedicated ticket that’ll block this one
2. propose the Debian maintainers to backport the fix to Stretch