Tails

#1 Updated by sajolida 2019-10-02 22:13:17

I got the same failure again today when coming back from lunch.

#7 Updated by anonym 2019-10-07 09:05:52

intrigeri wrote:
> I’m not surprised this happens: lots of our codebase was written before we started supporting suspend-to-RAM and was never adapted to support such situations.

And it was certainly the case with this feature.

> I would even go further: our current MAC spoofing implementation is essentially designed for systems with a fixed set of hardware; I don’t recall if we took hotplugging new devices into account in the design phase, but for example Bug #6550 shows that the initial implementation was not fully ready for this.

Hotplugging was part of the design, but the initial implementation was indeed buggy since that case got much less testing and attention (see e.g. commit:d4fd3edab513d5777aec435f25ae57404c020ce9).

However, then there is the udev vs NM race made apparent in tails-unblock-network — clearly that workaround (starting NM last) has never been applied to hotplugged devices. So, indeed, this feature probably only worked properly for hardware present before logging in. :/

> Brainstorming possible solutions:
>
> * Letting NetworkManager handle the whole MAC spoofing dance itself (Feature #11293) would almost certainly fix that: it’s not going to race against itself.

This would be so nice!

> * We could stop NetworkManager on resume and then start it again once we’re sure that all network adapters have been put back online and their MAC address spoofed.

Wow, eek, this is getting pretty hairy!

> Personally, I’m getting more and more convinced that we should look into Feature #11293 again: there’s a strong trend towards OS vendors enabling MAC randomization by default in more and more cases, NetworkManager is following this trend, and now could be a good time to benefit from it and get rid of our own, in-house implementation.

100% agreed!

#8 Updated by segfault 2019-10-07 09:38:39

> Personally, I’m getting more and more convinced that we should look into Feature #11293 again: there’s a strong trend towards OS vendors enabling MAC randomization by default in more and more cases, NetworkManager is following this trend, and now could be a good time to benefit from it and get rid of our own, in-house implementation.

I also agree, this sounds like the best solution.

#9 Updated by anonym 2019-10-07 10:16:23

anonym wrote:
> However, then there is the udev vs NM race made apparent in tails-unblock-network — clearly that workaround (starting NM last) has never been applied to hotplugged devices. So, indeed, this feature probably only worked properly for hardware present before logging in. :/

This made me uneasy and contradicted my memories of how udev vs NM interacts, so I made an experiment by leveraging our “MAC address spoofing is successful” scenario (our automated test suite is damn practical for setting up manual testing scenarios :D ):

1. I injected the “I pause” step as breakpoints before and after the “I hotplug a network device” step
2. I started the test and waited until the first “I pause”
3. I took control of the VM added a long sleep to tails-spoof-mac
4. I unpaused the test and waited to the next “I pause”
5. I checked the journal in the VM

According to the journal, NM has only been made aware of the existence of the new device (“new Ethernet device”) at this point, and has not changed its state. That happens as soon as I kill tails-soof-mac though. So my conclusion is that there is no race between udev hooks and NM here: NM waits until udev hooks finish.

So, regarding the comments and order of things in tails-unblock-network, perhaps we were hit by some bug that confused us to think this race was the problem? Or perhaps the problem was actually having NM running before we unblock the network? Any way, in conclusion it actually does look like hotplugging works exactly as it should.

So it seems sajolida’s bug is caused by something else. I wanted to run udevadm monitor during a suspend-resume cycle to see what is reported for the device but it doesn’t seem supported by libvirt (or my configuration) so I gave up, and I lack any such hardware to try for real. intrigeri or sajolida, can you please record what udevadm monitor reports during a full suspend-resume cycle when the spoof failure occurs? Our hook only triggers on the add action but perhaps something else is emitted when resuming from sleep? I suspect it could be change, and in that case this ticket is fixed by triggering on ACTION=="add|change" instead.

#11 Updated by intrigeri 2019-10-09 15:19:39

Hi,

anonym wrote: > According to the journal, NM has only been made aware of the existence of the new device ("new Ethernet device") at this point, and has *not* changed its state. That happens as soon as I kill tails-soof-mac@ though. So my conclusion is that there is no race between udev hooks and NM here: NM waits until udev hooks finish.

I did not fully review the test procedure (should I?) but I’m inclined to blindly trust your conclusions.

> So, regarding the comments and order of things in tails-unblock-network, perhaps we were hit by some bug that confused us to think this race was the problem? Or perhaps the problem was actually having NM running before we unblock the network?

Or, when we implemented this initially, NM did not wait for udev hooks to complete. That was years ago :)

> Any way, in conclusion it actually does look like hotplugging works exactly as it should.

Great! I’m very happy I was wrong. I’ve optimistically fixed the ticket title accordingly, hoping that you merely forgot.

> intrigeri or sajolida, can you please record what udevadm monitor@ reports during a full suspend-resume cycle when the spoof failure occurs?

I’ve dug through my hardware storage and found a USB Wi-Fi adapter I had totally forgotten about ⇒ I’ll try to give you this info.

#12 Updated by intrigeri 2019-10-09 15:49:21

Hi again @anonym,

> anonym wrote:
>> intrigeri or sajolida, can you please record what udevadm monitor reports during a full suspend-resume cycle when the spoof failure occurs?

I can’t reproduce the spoof failure with the hardware I have handy: I’ve tried 5 times (sometimes with the sleep hardware button, sometimes by closing the lid, sometimes by pressing the suspend button in GNOME Shell’s system menu); Wi-Fi came back up every time, each time with a shiny new MAC address. Which, I guess, at least confirms that external network adapters are subject to the MAC spoofing dance upon resume, while in my experience (Bug #17128#note-3) at least some internal network adapters are immune to it.

> I’ve dug through my hardware storage and found a USB Wi-Fi adapter I had totally forgotten about

FTR this USB Wi-Fi adapter is a super old Belkin “Wireless G USB Network Adapter”, “Model no. F5D7050eaE”.

> ⇒ I’ll try to give you this info.

It’s full of private info so I’ll send it to you privately… FWIW given I could not reproduce.

⇒ I think it would be useful if sajolida sent you this info (+ possibly the output of journalctl) when the spoof failure occurs.

#13 Updated by anonym 2019-10-10 08:56:20

intrigeri wrote:
> Hi,
>
> anonym wrote: > > According to the journal, NM has only been made aware of the existence of the new device ("new Ethernet device") at this point, and has *not* changed its state. That happens as soon as I kill tails-soof-mac@ though. So my conclusion is that there is no race between udev hooks and NM here: NM waits until udev hooks finish.
>
> I did not fully review the test procedure (should I?) but I’m inclined to blindly trust your conclusions.

No, I’m pretty confident in my procedure. I documented my steps “just in case”.

> > So, regarding the comments and order of things in tails-unblock-network, perhaps we were hit by some bug that confused us to think this race was the problem? Or perhaps the problem was actually having NM running before we unblock the network?
>
> Or, when we implemented this initially, NM did not wait for udev hooks to complete. That was years ago :)

I guess, but above I said that that hypothesis “contradicted my memories of how udev vs NM interacts”: I have a distinct memory of doing essentially the test I did above years ago when developing this feature.

> > intrigeri or sajolida, can you please record what udevadm monitor@ reports during a full suspend-resume cycle when the spoof failure occurs?
>
> I’ve dug through my hardware storage and found a USB Wi-Fi adapter I had totally forgotten about ⇒ I’ll try to give you this info.

So hotplugged devices are removed and then added, there’s no change, so my hunch was wrong, damn.

Comparing your log vs the excerpt from sajolida’s we can see that NM has done the same state changes (activated -> unmanaged and bound -> done) and the device should be unmanaged. But then for sajolida macchanger fails with “Could not change MAC: interface up or insufficient permissions: Device or resource busy”.

I’m trying to figure out why it is busy. In sajolida’s case we can see nm-dispatcher running scripts for “wlan0 down” between spoof-mac’s “Trying to spoof” and the failure, which is suspect, because in intri’s case that happens way earlier. I looked through the NM dispatcher scripts, and nothing in them uses the device on down, but I guess NM keeps some sort of handle on the device until all scripts are done, explaining why it’s “busy”.

So my best approximation of a potential solution would be to make spoof-mac wait until the device is done with it’s NM disatcher scripts (e.g. in first NM dispatch script we write a state file, and we remove it on the last NM dispatch script, then spoof-mac will wait until this file does not exist), or we just make spoof-mac re-try macchanger on failure for up to 20 seconds or whatever.

Thoughts?

#14 Updated by intrigeri 2019-10-11 06:48:09

> I’m trying to figure out why it is busy. In sajolida’s case we can see nm-dispatcher running scripts for “wlan0 down” between spoof-mac’s “Trying to spoof” and the failure, which is suspect, because in intri’s case that happens way earlier. I looked through the NM dispatcher scripts, and nothing in them uses the device on down, but I guess NM keeps some sort of handle on the device until all scripts are done, explaining why it’s “busy”.

Yeah, it could be, for example, that NM is asking wpasupplicant to disconnect cleanly from the Wi-Fi AP (assuming there’s such a thing as a clean disconnection); and for some reason, on my test system this happens (and completes) earlier than on sajolida’s.

So my hunch was probably totally wrong when I dropped this part of the process from the list of potential culprits :)

> So my best approximation of a potential solution would be to make spoof-mac wait until the device is done with it’s NM disatcher scripts (e.g. in first NM dispatch script we write a state file, and we remove it on the last NM dispatch script, then spoof-mac will wait until this file does not exist),

(Here I’m in hunch/feelings land, and did not bother checking if there’s any rationality attached to my current thoughts.) I’m really not enthusiastic at the idea of having to carry & maintain state in this subsystem. I can see that it would work in most cases (starting with: on the developer’s machine) but could be subtly racy in ways that are too hard to reproduce for us having much hope of debugging issues and gaining confidence in tentative fixes.

> or we just make spoof-mac re-try macchanger on failure for up to 20 seconds or whatever.

Yeah! While perhaps less “clean” conceptually, to me, retrying a few times like this seems much simpler to implement — and more importantly, to reason about — than the state file idea.
IMO it makes sense to try this cheap workaround and see if it fixes the problem. And if it does not, then I think I’d rather investigate the NM MAC randomization way before putting much more efforts into our current implementation.