Tails

#1 Updated by goupille 2019-12-07 15:17:58

Bug report: 935ae832daa20a6ea1f44bdfa5b8c00d

#5 Updated by intrigeri 2019-12-12 08:30:35

This seems to be caused by a failure to load the firmware:

Dec 05 00:14:02 amnesia kernel: rtw_pci 0000:04:00.0: enabling device (0000 -> 0003)
Dec 05 00:14:02 amnesia kernel: rtw_pci 0000:04:00.0: firmware: failed to load rtw88/rtw8822b_fw.bin (-2)
Dec 05 00:14:02 amnesia kernel: firmware_class: See https://wiki.debian.org/Firmware for information about missing firmware
Dec 05 00:14:02 amnesia kernel: rtw_pci 0000:04:00.0: Direct firmware load for rtw88/rtw8822b_fw.bin failed with error -2
Dec 05 00:14:02 amnesia kernel: rtw_pci 0000:04:00.0: failed to request firmware
Dec 05 00:14:02 amnesia kernel: rtw_pci 0000:04:00.0: failed to load firmware
Dec 05 00:14:02 amnesia kernel: rtw_pci 0000:04:00.0: failed to setup chip efuse info
Dec 05 00:14:02 amnesia kernel: rtw_pci 0000:04:00.0: failed to setup chip information
Dec 05 00:14:02 amnesia kernel: rtw_pci: probe of 0000:04:00.0 failed with error -22

Note that the user for wb://935ae832daa20a6ea1f44bdfa5b8c00d had MAC spoofing disabled.

#8 Updated by goupille 2019-12-23 13:34:59

intrigeri wrote:

>
> I’m happy to treat this as FT work if it affects lots of Tails users.

there were 3 reports during the first week after tails 4.1 and probably more since then

#13 Updated by CyrilBrulebois 2020-03-01 16:56:19

Checked, and there’s no upload yet.

How acceptable would it be to dump the needed files into our tree and put them in place through a local hook until the updated package reaches the archive? It’s been several months already. :/

#14 Updated by intrigeri 2020-03-02 07:36:27

Hi @CyrilBrulebois,

I’m glad you’re caring!

> How acceptable would it be to dump the needed files into our tree and put them in place through a local hook until the updated package reaches the archive? It’s been several months already. :/

First, I’ll share information about how we’re handling this sort of hardware support issues in general, and the reason why. I do not mean it’s the best and only way to approach this, nor that it’s set in stone. My goal is to provide food for the collective & individual thought processes that we go through when picking a task to work on.

To put this into perspective, wrt. support for specific hardware devices, with my FT “frontdesk” hat on I see many issues reported every month. If they’re tracked upstream already, in general we don’t do anything ourselves besides documenting workarounds (if any is available). We could surely do something about some of these individual issues but the current capacity of the FT does not allow us to work on all of them, and it’s often hard to tell which ones matter more than others. So in general I try to focus our efforts on problems that we know affect a larger subset of our users (e.g. currently: the network adapter support regressions that seemingly were introduced in 4.1.1; the “Hot topics on our help desk” section of our monthly report is often useful to identify such higher-impact problems).

Now, for the case at hand, the workaround seems pretty cheap to implement, so even if the benefit is small, the cost/benefit could be worth it. I would not actively look for FT resources to put into this, but if that’s something you’d like to fix, please go ahead! In any case, let’s not spend more time debating “is it worth it?” than it would take to fix the bug :)

Implementation-wise, if possible I’d rather not add binary files to Git: it would bloat the Git repo forever and might have annoying licensing implications. Maybe we could use a build hook to wget/curl them from upstream + verify them against a hard-coded checksum? (That’s essentially what we do for Tor Browser; that’s what a couple packages we install do under the hood, e.g. libdvdcss & firmware-b43*, so it would not be a first.)

Cheers!

#18 Updated by CyrilBrulebois 2020-03-04 03:53:18

Except direct wget/curl don’t get any DNS resolution; stealing the “let’s set all HTTP/HTTPS proxy variables” dance from 10-tbb doesn’t work either since HTTPS connections are denied. See Tor Browser where we need http:// for the same reason…

@intrigeri:

Do we have a somewhat generic HTTP-not-redirecting-to-HTTPS service we could use for such tiny hacks, instead of relying on individual developer’s machines? (ab)using torbrowser-archive seems wrong, but maybe maintaining a duplicate service for very infrequent hacks isn’t much better because of the extra sysadmin time?

(BTW, cachewebsite is wow!)

#20 Updated by intrigeri 2020-03-05 07:59:11

Hi @CyrilBrulebois,

> To make it explicit: my interest in this particular topic was prompted by the fact it indeed seemed like a regression introduced by the “hot fix” for Mac machines, and it seemed to affect several chipsets.

FWIW (not that it matters much anymore), I doubt that’s the case: this very issue was created a week before that hot fix was released.
In our January report, help desk said that the issues possibly introduced in 4.1.1 were: Bug #17388, Bug #17430, Bug #17418.

> intrigeri wrote:
>> Implementation-wise, if possible I’d rather not add binary files to Git: it would bloat the Git repo forever and might have annoying licensing implications. Maybe we could use a build hook to wget/curl them from upstream + verify them against a hard-coded checksum? (That’s essentially what we do for Tor Browser; that’s what a couple packages we install do under the hood, e.g. libdvdcss & firmware-b43*, so it would not be a first.)
>
> ACK. I was definitely fishing for ideas, and adding binary stuff to the git repository made me twitch when I wrote it, but I thought you’d get the general idea. :)

> Except direct wget/curl don’t get any DNS resolution; stealing the “let’s set all HTTP/HTTPS proxy variables” dance from 10-tbb doesn’t work either since HTTPS connections are denied. See Tor Browser where we need http:// for the same reason…

Ah, right!

> Do we have a somewhat generic HTTP-not-redirecting-to-HTTPS service we could use for such tiny hacks, instead of relying on individual developer’s machines? (ab)using torbrowser-archive seems wrong, but maybe maintaining a duplicate service for very infrequent hacks isn’t much better because of the extra sysadmin time?

We also have {,tagged.snapshots,time-based.snapshots}.deb.tails.boum.org that don’t redirect to HTTPS for the same reason. If we end up (ab)using one of these things, I slightly prefer torbrowser-archive because there’s some minimal versioning thanks to git-annex.

Other options I can think of:

• bypass apt-cacher-ng
  • pro: hopefully very cheap to implement, which is good for a temporary hack
  • drawback: breaks the offline build option; I’m not aware of anyone using it these days and I have no idea if it still works, so IMO it would be acceptable to do that as a temporary hack, and see if anyone complains
• add linux-firmware.git as a Git submodule
  • pro: consistency, as it’s one of our most common ways to track & pull third-party code/files during the build
  • cons: I dunno, maybe it’s huge?

These 3 options seem acceptable to me.

#21 Updated by CyrilBrulebois 2020-03-05 17:51:53

Hey @intrigeri,

Thanks for your thoughts. At the bottom, I’m wondering about a possibly more permanent/generic solution for such workarounds; it likely exceeds the scope of this particular issue, but it would be great if I, and we, had a generic way to implement them without entering the “nasty territory”. (For what it’s worth, I’m not tracking the extra time spent on this topic, I’ve capped this issue at 2 hours.)

intrigeri wrote:
> FWIW (not that it matters much anymore), I doubt that’s the case: this very issue was created a week before that hot fix was released. In our January report, help desk said that the issues possibly introduced in 4.1.1 were: Bug #17388, Bug #17430, Bug #17418.

It’s entirely possible I mixed things up a little, and the vague feeling I had was “woops, maybe the hotfix broke many chipsets, let’s look at that one I’m seeing on our plate”.

> We also have {,tagged.snapshots,time-based.snapshots}.deb.tails.boum.org that don’t redirect to HTTPS for the same reason. If we end up (ab)using one of these things, I slightly prefer torbrowser-archive because there’s some minimal versioning thanks to git-annex.

Yeah, I thought about the deb.tails.boum.org one but that didn’t seem a good fit… I thought about torbrowser-archive but given it only had version numbers directly under the root directory, adding some workarounds/ top-level directory seemed a little sad.

> Other options I can think of:
>
> * bypass apt-cacher-ng
> pro: hopefully very cheap to implement, which is good for a temporary hack
> drawback: breaks the offline build option; I’m not aware of anyone using it these days and I have no idea if it still works, so IMO it would be acceptable to do that as a temporary hack, and see if anyone complains

I’m skipping this proposal because I’m tempted to build on the second one.

> * add linux-firmware.git as a Git submodule
> pro: consistency, as it’s one of our most common ways to track & pull third-party code/files during the build
> cons: I dunno, maybe it’s huge?
>
> These 3 options seem acceptable to me.

It seems a bit heavy for this specific case, seeing 350+ MB worth of .git.

But I’m wondering whether having some tails-workarounds git submodule would be suitable for the random workarounds we might need over time? This would mean being able to drop anything at once without breaking the history in the main repository, if there were any issues (e.g. mainly we don’t like tracking binary contents of the main repo, or maybe more annoyingly redistributability disputes, or whatever). That could also be used to beta-test changes involving updated or new Debian packages that we wouldn’t need to upload to deb.tails.boum.org, and that we wouldn’t have to self-host (with all the keyring fun that involves), the latter being what I ended up doing when testing the feasibility of 4.1.1. And that wouldn’t involve more sysadmin-side setup (e.g. no need to duplicate what we do for torbrowser-archive).

I can propose that plan on tails-foundations@ / tails-dev@ to avoid having you spend more time on this/burden you with making a call.

.oO( thinking about it a little more, not sure whether I have the appropriate rights to create a new repository at the moment anyway… I’ve never needed to do that yet… )

#22 Updated by intrigeri 2020-03-06 07:43:52

Hi,

> But I’m wondering whether having some tails-workarounds git submodule would be suitable for the random workarounds we might need over time? This would mean being able to drop anything at once without breaking the history in the main repository, if there were any issues (e.g. mainly we don’t like tracking binary contents of the main repo, or maybe more annoyingly redistributability disputes, or whatever).

Sounds good to me!

> That could also be used to beta-test changes involving updated or new Debian packages that we wouldn’t need to upload to deb.tails.boum.org, and that we wouldn’t have to self-host (with all the keyring fun that involves), the latter being what I ended up doing when testing the feasibility of 4.1.1.

I don’t understand this part. More specifically, I don’t understand what is the limitation of our custom APT repo (deb.t.b.o) that makes workarounds desirable or even needed. Could you please clarify?

> .oO( thinking about it a little more, not sure whether I have the appropriate rights to create a new repository at the moment anyway… I’ve never needed to do that yet… )

Indeed, currently only sysadmins can create new repos: https://tails.boum.org/contribute/git/#creating-a-new-repository
I’ve created this to unblock your progress:

git clone tails@git.tails.boum.org:tails-workarounds

#24 Updated by CyrilBrulebois 2020-03-07 22:55:52

I’d be happy to get a review on the bugfix/17323-fix-missing-rtw88-firmware branch.

Maybe intrigeri or segfault would have a chance before 4.4?

I’ll need to concentrate on merging Tor Browser soon anyway, so if for some reason we miss the window for 4.4, this can likely wait until 4.5 anyway.

#28 Updated by intrigeri 2020-03-08 07:49:53

Hi!

(1st part is for goupille, 2nd part is for kibi)

@goupille,

> Ideally I’d like 1+ affected user to confirm it fixes the problem, but given the timing I’m not going to block on that.

I’ve asked one such user (wb://935ae832daa20a6ea1f44bdfa5b8c00d) to try this branch.

goupille wrote:
> there were 3 reports during the first week after tails 4.1 and probably more since then

I’ve found one single report (935ae832daa20a6ea1f44bdfa5b8c00d) that got classified as such (at least, in a way that’s visible to me).
Could you please point me to these other reports? Unless perhaps 935ae832daa20a6ea1f44bdfa5b8c00d was the only one with contact info, in which case: forget it!

@CyrilBrulebois,

>> I’d be happy to get a review on the bugfix/17323-fix-missing-rtw88-firmware branch.
> I’ll give it a shot this morning.

Code review passes. Great job! I’m now building an image to verify that the firmware indeed lands where I would expect.

Just one comment: I’d rather see “WARNING: Firmware for $file found, maybe this hook could be dropped” be a fatal build error, to avoid shipping outdated firmware once Debian includes and they get updated there. I’m concerned that this warning will be hard to spot in the middle of a 15k-lines build log, that has 30+ other warnings which are all false positives. If you agree, I can make this trivial change myself before merging.

In passing, I’ve verified that we don’t need rtw8822c_wow_fw.bin: it’s only needed to resume a suspended system using Wake-on-LAN over Wi-Fi, which I don’t expect Tails users will ever try :)

#31 Updated by goupille 2020-03-09 14:39:12

intrigeri wrote:

> > Ideally I’d like 1+ affected user to confirm it fixes the problem, but given the timing I’m not going to block on that.
>
> I’ve asked one such user (wb://935ae832daa20a6ea1f44bdfa5b8c00d) to try this branch.
>
> goupille wrote:
> > there were 3 reports during the first week after tails 4.1 and probably more since then
>
> I’ve found one single report (935ae832daa20a6ea1f44bdfa5b8c00d) that got classified as such (at least, in a way that’s visible to me).
> Could you please point me to these other reports? Unless perhaps 935ae832daa20a6ea1f44bdfa5b8c00d was the only one with contact info, in which case: forget it!

they weren’t wp reports, I just resent them your email to the other user (i’ve cc’ed you so you should have their contact as well now)

#32 Updated by intrigeri 2020-03-09 14:51:23

Hi!

> I just resent them your email to the other user

Thank you. I see 2 such messages presumably from you today.

> (i’ve cc’ed you so you should have their contact as well now)

That part did not work: I see the Cc header to me, but no To header.
I suspect you used x-resend-cc, which may not work in the way you would expect (at least, it does not work the way I thought): https://0xacab.org/schleuder/schleuder/issues/345

#33 Updated by goupille 2020-03-09 15:08:25

intrigeri wrote:

> That part did not work: I see the Cc header to me, but no To header.
> I suspect you used x-resend-cc, which may not work in the way you would expect (at least, it does not work the way I thought): https://0xacab.org/schleuder/schleuder/issues/345

one of them was encrypted for all the recipients, I assumed at least this one should have worked…
well, I’m not sure what is the difference between x-resend and x-resend-cc, then, I give it a try every now and then and still end up resending everything manually…

any way, you should have received those emails again with the users contact :)

#35 Updated by intrigeri 2020-03-15 17:59:32

I’ve emailed the two users for which I have contact info, asking them to test with Tails 4.4.

#36 Updated by intrigeri 2020-03-15 18:12:20

> I’ve emailed the two users for which I have contact info, asking them to test with Tails 4.4.

One of these 2 messages was rejected due to an over-quota mailbox.