Bug #12364

Race condition at greeter startup

Added by Anonymous 2017-03-18 10:46:47 . Updated 2020-04-15 06:01:44 .

Status:
Rejected
Priority:
Normal
Assignee:
alant
Category:
Target version:
Start date:
2017-05-16
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Welcome Screen
Deliverable for:

Description

Here are some screenshots and a journal from when this happened.


Files


Subtasks


Related issues

Related to Tails - Bug #12249: Greeter's Formats popover may be cut Resolved 2017-02-17
Related to Tails - Bug #12228: Greeter's headerbar is sometimes not themed right Confirmed 2017-02-14
Related to Tails - Bug #12241: New Greeter's additional options window sometimes opens in a weird place In Progress 2017-02-15
Related to Tails - Bug #12548: The Greeter sometimes does not display the GNOME Shell top bar Rejected 2017-05-16
Related to Tails - Bug #12386: Greeter sometimes fails to set Admin password on Stretch Duplicate 2017-03-19

History

#1 Updated by Anonymous 2017-03-18 10:48:51

So the window opens in a weird location after clicking “+”.
Then I enter the password and click “Add”.
Then the option window closes and the option seems set, but it’s marked as “off”.
When i reclick a second time on the option I get a window bubble with the password and the password confirmation. I click enter and then it is set to on.

#2 Updated by Anonymous 2017-03-18 10:50:57

And here is the journal from this boot.

#3 Updated by intrigeri 2017-03-18 11:15:37

  • Priority changed from Normal to Elevated
  • Target version set to Tails_3.0

#4 Updated by alant 2017-03-18 16:09:59

#5 Updated by alant 2017-03-18 17:40:06

  • Status changed from Confirmed to In Progress
  • QA Check set to Info Needed

I don’t have enough info to understand nor to reproduce this issue. In addition, it seems that patching the loglevel and restart the greeter prevents the reproduction of the issue. I’m building an ISO with a greeter that honor debug kernel command line parameter, to gather enough information to understand that bug.

#6 Updated by intrigeri 2017-03-19 08:57:43

  • related to Bug #12228: Greeter's headerbar is sometimes not themed right added

#7 Updated by intrigeri 2017-03-19 08:58:05

  • Subject changed from Greeter in Tails 3.0 feature-stretch sometimes fails to set Admin password to Greeter sometimes fails to set Admin password on Stretch

#8 Updated by intrigeri 2017-03-19 08:58:44

  • related to Bug #12241: New Greeter's additional options window sometimes opens in a weird place added

#9 Updated by intrigeri 2017-03-19 08:59:55

Alan thinks that Bug #12228, Bug #12241 and this ticket share the same root cause, which might be the Greeter starting before some other services it depends on, in some cases.

#10 Updated by intrigeri 2017-03-19 09:00:55

  • blocked by Feature #12373: Greeter should honor "debug" kernel command line parameter added

#11 Updated by intrigeri 2017-03-19 09:01:40

  • Priority changed from Normal to Elevated

#12 Updated by alant 2017-03-19 12:43:03

I tested with the Tails branch that honor debug kernel commandline parameter for greeter logging (see Feature #12373) and systemd-analyse. I cannot easily reproduce the password issue, but there are reproducible cases with some USB sticks in which the greeter interface is rendered a wierd way: the headerbar has square angle (Bug #12228), and the popover may be cut (Bug #12243). All known reports of the password issue have this wierd interface issue, so I assume they are somehow linked.

It appears that all the time the interface is rendered wierd, polkit.service and accounts-daemon.service are not reported to be fully started by systemd-analyse.

Buggy:

OK:

#13 Updated by alant 2017-03-19 13:05:58

This may be realted to this issue https://github.com/coreos/bugs/issues/462

#14 Updated by alant 2017-03-19 14:37:38

alant wrote:
> This may be realted to this issue https://github.com/coreos/bugs/issues/462

It reads: “it appears to be a race between a daemon-reload and bus messages. It looks like there is a brief period during the reload that the bus matchers are uninstalled and that’s when a message could sneak in and get dropped on the floor. We still have to verify this theory though.”

But this is about systemd 225, and is apparently solved.

#15 Updated by alant 2017-03-19 14:44:59

  • QA Check deleted (Info Needed)

I built an ISO with gdm.service requiring and starting after polkit.service:

--- /dev/null
+++ b/config/chroot_local-includes/etc/systemd/system/gdm.service.d/require-polkit.conf
@@ -0,0 +1,3 @@
+[Unit]
+Requires=polkit.service
+After=polkit.service

With that ISO, I was able to reproduce Bug #12228 with polkit.service and accounts-daemon.service correctly reported as started!

#16 Updated by alant 2017-03-19 15:06:35

gdm.service requiring and starting after polkit.service fixes the polkit/account-daemon to properly reported as started, but not Bug #12228. I wasn’t able to reproduce the password issue yet though.

#17 Updated by alant 2017-03-19 16:32:34

  • related to deleted (Bug #12228: Greeter's headerbar is sometimes not themed right)

#18 Updated by alant 2017-03-19 16:32:38

  • related to deleted (Bug #12241: New Greeter's additional options window sometimes opens in a weird place)

#19 Updated by alant 2017-03-19 16:32:41

  • blocks deleted (Feature #12373: Greeter should honor "debug" kernel command line parameter)

#20 Updated by alant 2017-03-19 16:33:14

#21 Updated by alant 2017-03-19 16:33:42

  • related to Bug #12228: Greeter's headerbar is sometimes not themed right added

#22 Updated by alant 2017-03-19 16:34:18

  • related to Bug #12241: New Greeter's additional options window sometimes opens in a weird place added

#23 Updated by alant 2017-03-19 17:56:30

I added sleep 5 in the wrapper that launches the greeter and can’t reproduce any of the wierd behaviours with the ISO including that. That seems to prove there is a race condition somewhere.

#24 Updated by Anonymous 2017-03-19 19:49:30

Could not reproduce this issue anymore with the new ISOs you sent to me.

#25 Updated by alant 2017-03-19 19:53:19

  • related to deleted (Bug #12228: Greeter's headerbar is sometimes not themed right)

#26 Updated by alant 2017-03-19 19:53:30

  • related to deleted (Bug #12241: New Greeter's additional options window sometimes opens in a weird place)

#27 Updated by alant 2017-03-19 19:55:57

  • Subject changed from Greeter sometimes fails to set Admin password on Stretch to Race condition at greeter startup

Let’s use this ticket to track the race condition issue. I created a Bug #12386 for the original issue.

#28 Updated by alant 2017-03-20 16:31:16

I’m wondering if it could be that gnome-shell, which provides (among other services) the window manager is not fully responsive when the greeter window is shown. I failed to find a way to wait for the window manager to be ready before showing the window though.

#29 Updated by alant 2017-03-20 18:57:26

I reproduced some of the symptoms of this bug:

  • there was no gnome-shell running
  • the shutdown button didn’t work (uses org.gnome.SessionManager over DBus)

The system had very high system load when starting the buggy Tails VM.

#30 Updated by intrigeri 2017-03-21 08:19:33

python3[3909]: gtk_window_set_transient_for: assertion 'window != parent' failed might explain Bug #12241, and might be a symptom of our GTK application starting while the GNOME session is not fully ready yet. Do we see this error messages during boots that don’t expose any of these bugs?

#31 Updated by intrigeri 2017-03-21 08:23:05

FWIW, to help investigate this, I would need the entire Journal of a boot that 1. exposes these bugs; 2. was done with the debug option passed on the kernel command line.

#32 Updated by intrigeri 2017-04-18 15:16:03

  • Target version changed from Tails_3.0 to Tails_3.0~rc1

#33 Updated by intrigeri 2017-04-19 09:25:15

All the following testing reports are in a libvirt/kvm VM, booting from DVD without persistence, and most of the time on a pretty loaded host system.

I can reproduce the minor UI issues (Bug #12228 + Bug #12249) consistently in my test VM, even when polkit and accounts-daemon completed their startup before gdm started.

Also, pretty often the desktop has a black background in the GDM session. Last time I’ve seen that happen, the systemd-analyze plot graph suggests that accounts-service is not fully started (its bar remains red), but both systemctl status and journalctl -u accounts-service disagree. When this happens, I see no Registered Authentication Agent for unix-session:c1 (system bus name :1.27 [/usr/bin/gnome-shell], […] in the logs for the GDM gnome-shell, contrary to what I see on my sid GNOME system. This suggests that gnome-shell wasn’t started at all, and in the logs I see nothing about GDM’s gnome-shell, while amnesia’s gnome-shell logs a few errors. And indeed, I see no GNOME Shell running for Debian-gdm:

Debian-+  3758  0.0  0.2 201092  5620 tty1     Ssl+ 08:12   0:00  |   \_ /usr/lib/gdm3/gdm-x-session gnome-session --autostart /usr/share/gdm/greeter/autostart
Debian-+  3760  0.3  1.9 340232 38468 tty1     Sl+  08:12   0:06  |       \_ /usr/lib/xorg/Xorg vt1 -displayfd 3 -auth /run/user/114/gdm/Xauthority -background none -noreset -keeptty -verbose 3
Debian-+  3832  0.0  0.6 546608 12524 tty1     Sl+  08:12   0:00  |       \_ /usr/lib/gnome-session/gnome-session-binary --autostart /usr/share/gdm/greeter/autostart
Debian-+  3888  0.0  1.4 1009792 28264 tty1    Sl+  08:12   0:00  |           \_ /usr/lib/gnome-settings-daemon/gnome-settings-daemon
Debian-+  3731  0.0  0.3  64980  6456 ?        Ss   08:12   0:00 /lib/systemd/systemd --user
Debian-+  3732  0.0  0.0 164740  1928 ?        S    08:12   0:00  \_ (sd-pam)
Debian-+  3830  0.0  0.1  45088  3816 ?        Ss   08:12   0:00  \_ /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation
Debian-+  3855  0.0  0.2 348780  5816 ?        Ssl  08:12   0:00  \_ /usr/lib/at-spi2-core/at-spi-bus-launcher
Debian-+  3860  0.0  0.1  44988  3572 ?        S    08:12   0:00  |   \_ /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
Debian-+  3862  0.0  0.3 220344  6904 ?        Sl   08:12   0:00  \_ /usr/lib/at-spi2-core/at-spi2-registryd --use-gnome-session
Debian-+  4270  0.0  0.2 187320  4824 ?        Sl   08:12   0:00  \_ /usr/lib/dconf/dconf-service
Debian-+  3892  0.0  0.3 176332  7120 ?        Ss   08:12   0:00 /usr/bin/spice-vdagent
Debian-+  3957  0.0  0.5 626964 10832 ?        Sl   08:12   0:01 /usr/bin/pulseaudio --start --log-target=syslog

/usr/lib/gnome-session/gnome-session-binary --autostart /usr/share/gdm/greeter/autostart is the thing that’s supposed to start GNOME Shell, so I’ve looked at our custom GDM and GNOME Shell initialization settings.

Note that /usr/share/applications/org.gnome.Shell.desktop has a few diffences compared to our own /usr/share/gdm/greeter/applications/gdm-shell-tails.desktop:

  • it has NoDisplay=true
  • it has X-GNOME-Autostart-Phase=DisplayServer
  • it has X-GNOME-Provides=panel;windowmanager;
  • it has X-GNOME-AutoRestart=false while ours has X-GNOME-AutoRestart=true

Also, config/gdm-tails.json says "isLocked": true while GDM’s js/ui/sessionMode.js only sets this for the GDM lock screen.

I see no justification in our Git wrt. these settings, that were all introduced in a commit whose message is “Use a GNOME shell session”. So I’m very much tempted to align all those files with the regular GNOME Shell / GDM ones. I might try this myself if Alan doesn’t provide an ETA earlier than May 15.

Also, FWIW /usr/share/gnome-session/sessions/gdm-tails.session has RequiredComponents=gdm-shell-tails;tails-greeter;gnome-settings-daemon;, while /usr/share/gnome-session/sessions/gnome.session has RequiredComponents=org.gnome.Shell;gnome-settings-daemon;. There’s a tiny chance that ordering counts (but I didn’t check the GSM code closely enough), so I suggest we move g-s-d before tails-greeter in that file.

But I still haven’t seen Bug #12386 happen, and I don’t see Bug #12241 anymore.

#34 Updated by alant 2017-05-11 15:39:06

  • Assignee changed from alant to intrigeri

intrigeri wrote:
> Also, pretty often the desktop has a black background in the GDM session. Last time I’ve seen that happen, the systemd-analyze plot graph suggests that accounts-service is not fully started (its bar remains red), but both systemctl status and journalctl -u accounts-service disagree. When this happens, I see no Registered Authentication Agent for unix-session:c1 (system bus name :1.27 [/usr/bin/gnome-shell], […] in the logs for the GDM gnome-shell, contrary to what I see on my sid GNOME system. This suggests that gnome-shell wasn’t started at all, and in the logs I see nothing about GDM’s gnome-shell, while amnesia’s gnome-shell logs a few errors. And indeed, I see no GNOME Shell running for Debian-gdm:
>
That confirms what I’ve seen too.

> /usr/lib/gnome-session/gnome-session-binary --autostart /usr/share/gdm/greeter/autostart is the thing that’s supposed to start GNOME Shell, so I’ve looked at our custom GDM and GNOME Shell initialization settings.
>
> Note that /usr/share/applications/org.gnome.Shell.desktop has a few diffences compared to our own /usr/share/gdm/greeter/applications/gdm-shell-tails.desktop:

I think the delta has been added to gnome-shell after the fork of the file.

> * it has NoDisplay=true

No idea why, but it shouldn’t harm.

> * it has X-GNOME-Autostart-Phase=DisplayServer

That may fix the issue. It’s to start the shell earlier.

> * it has X-GNOME-Provides=panel;windowmanager;

That shouldn’t harm.

> * it has X-GNOME-AutoRestart=false while ours has X-GNOME-AutoRestart=true
>
I don’t know about this one, but it shouldn’t harm either.

> Also, config/gdm-tails.json says "isLocked": true while GDM’s js/ui/sessionMode.js only sets this for the GDM lock screen.
>
This one should be kept as far as I know. The greeter has a isGreeter mode, and as we don’t use it we want the isLocked to put gnome-shell in a mode that restricts items in its menu (e.g. control of the network, calendar, …)

> I see no justification in our Git wrt. these settings, that were all introduced in a commit whose message is “Use a GNOME shell session”. So I’m very much tempted to align all those files with the regular GNOME Shell / GDM ones. I might try this myself if Alan doesn’t provide an ETA earlier than May 15.

Please do, keeping isLocked. I won’t be able to handle that before the deadline.

> Also, FWIW /usr/share/gnome-session/sessions/gdm-tails.session has RequiredComponents=gdm-shell-tails;tails-greeter;gnome-settings-daemon;, while /usr/share/gnome-session/sessions/gnome.session has RequiredComponents=org.gnome.Shell;gnome-settings-daemon;. There’s a tiny chance that ordering counts (but I didn’t check the GSM code closely enough), so I suggest we move g-s-d before tails-greeter in that file.
>
It wouldn’t harm.

#35 Updated by intrigeri 2017-05-16 13:33:25

  • blocks Bug #12249: Greeter's Formats popover may be cut added

#36 Updated by intrigeri 2017-05-16 13:35:25

  • blocks Bug #12228: Greeter's headerbar is sometimes not themed right added

#37 Updated by intrigeri 2017-05-16 13:36:53

  • blocks Bug #12241: New Greeter's additional options window sometimes opens in a weird place added

#38 Updated by intrigeri 2017-05-16 13:37:43

  • blocks Bug #12548: The Greeter sometimes does not display the GNOME Shell top bar added

#39 Updated by intrigeri 2017-05-16 13:37:49

  • blocks Bug #12386: Greeter sometimes fails to set Admin password on Stretch added

#40 Updated by intrigeri 2017-05-16 13:38:38

  • Priority changed from Normal to Elevated

#41 Updated by intrigeri 2017-05-16 13:49:00

>> Note that /usr/share/applications/org.gnome.Shell.desktop has a few diffences compared to our own /usr/share/gdm/greeter/applications/gdm-shell-tails.desktop:

> I think the delta has been added to gnome-shell after the fork of the file.

So we need a process to keep our fork up-to-date. Can you please set one up?

>> Also, config/gdm-tails.json says "isLocked": true while GDM’s js/ui/sessionMode.js only sets this for the GDM lock screen.

> This one should be kept as far as I know. The greeter has a isGreeter mode, and as we don’t use it we want the isLocked to put gnome-shell in a mode that restricts items in its menu (e.g. control of the network, calendar, …)

OK, but while I’m at it: why don’t we use the isGreeter mode?

> Please do, keeping isLocked.

OK, I’ll prepare a branch this week, that applies all the suggested changes you agree with, and we’ll see if it helps :)

If it doesn’t, then I would like to reassign to you: will you have time to work on this ticket by the end of May?

#42 Updated by intrigeri 2017-05-16 14:40:55

  • Feature Branch set to bugfix/12364-greeter-races, greeter:bugfix/12364-greeter-races

#43 Updated by intrigeri 2017-05-16 14:59:14

intrigeri wrote:
> >> Note that /usr/share/applications/org.gnome.Shell.desktop has a few diffences compared to our own /usr/share/gdm/greeter/applications/gdm-shell-tails.desktop:
>
> > I think the delta has been added to gnome-shell after the fork of the file.
>
> So we need a process to keep our fork up-to-date. Can you please set one up?

Moved this to Bug #12551 as it is pretty much off-topic here.

#44 Updated by intrigeri 2017-05-16 17:57:44

Sadly, my tweaks to the desktop file were not enough to fix the problem. I’ll try removing the set-cursor kludge, just in case (the xinput errors made me wonder), and if that doesn’t work either I’ll send this back to Alan’s plate.

#45 Updated by intrigeri 2017-05-16 18:03:29

  • related to Bug #12553: Make gdm-shell-tails.desktop more similar to the one shipped in Debian's gnome-shell package added

#46 Updated by intrigeri 2017-05-16 18:05:16

  • Feature Branch deleted (bugfix/12364-greeter-races, greeter:bugfix/12364-greeter-races)

#47 Updated by intrigeri 2017-05-18 15:15:38

  • Assignee changed from intrigeri to alant
  • Target version changed from Tails_3.0~rc1 to Tails_3.0

#48 Updated by anonym 2017-05-18 15:37:41

  • Status changed from In Progress to Fix committed
  • % Done changed from 0 to 100

Applied in changeset commit:299183475b1ef93566b792a3c9f4d8d479679227.

#49 Updated by intrigeri 2017-05-20 07:23:21

  • Status changed from Fix committed to Confirmed
  • % Done changed from 100 to 0

#50 Updated by alant 2017-06-04 06:56:56

> > This one should be kept as far as I know. The greeter has a isGreeter mode, and as we don’t use it we want the isLocked to put gnome-shell in a mode that restricts items in its menu (e.g. control of the network, calendar, …)
>
> OK, but while I’m at it: why don’t we use the isGreeter mode?
>
Because isGreeter mode would display the stock GNOME Shell greeter instead of Tails-greeter, which is obviously not what we want.

#51 Updated by intrigeri 2017-06-04 10:03:31

I’ve tried hard to reproduce these bugs by overloading my host system’s CPU (stress --cpu 4) and/or throttling the Tails VM’s I/O (blkiotune), both when booting from a virtual DVD and a virtual USB stick (the latter with a persistent volume). Given Bug #12548 I’ve also tried with/without a virtual USB tablet input device, in case it matters. I did not experience any of these issues, so I’m at a loss here. I’m starting to wonder if the root cause of the bug was already fixed in Debian, and picked up in Tails via the APT snapshots bump we did on May 18 (Bug #12548 was reported earlier). So I’d be very curious to know if anyone has seen any such problem again since 3.0~rc1.

#52 Updated by intrigeri 2017-06-04 14:03:20

  • Target version deleted (Tails_3.0)

So, anyone who can somewhat reliably reproduce any of these bugs, please:

  • get used to add the debug option in the boot loader;
  • send a WhisperBack bug report every time you hit any of these problems, clearly stating which one(s), and provide a contact email address;
  • ideally, send the boot.svg file generated by systemd-analyze plot > boot.svg to alan@boum.org and intrigeri@boum.org, referring to the bug report you’ll have sent already.

Once we have enough data, I’d like to sit down with Alan and .

Meanwhile, I’m removing the 3.0 target version as I really don’t see how we can realistically aim to reproduce and fix these bugs in the next 5 days.

#53 Updated by intrigeri 2017-06-04 14:10:52

  • blocked by deleted (Bug #12249: Greeter's Formats popover may be cut)

#54 Updated by intrigeri 2017-06-04 14:11:11

  • blocked by deleted (Bug #12228: Greeter's headerbar is sometimes not themed right)

#55 Updated by intrigeri 2017-06-04 14:11:13

  • blocked by deleted (Bug #12241: New Greeter's additional options window sometimes opens in a weird place)

#56 Updated by intrigeri 2017-06-04 14:11:15

  • blocked by deleted (Bug #12386: Greeter sometimes fails to set Admin password on Stretch)

#57 Updated by intrigeri 2017-06-04 14:11:18

  • blocked by deleted (Bug #12548: The Greeter sometimes does not display the GNOME Shell top bar)

#58 Updated by intrigeri 2017-06-04 14:11:20

  • related to deleted (Bug #12553: Make gdm-shell-tails.desktop more similar to the one shipped in Debian's gnome-shell package)

#59 Updated by intrigeri 2017-06-04 14:11:35

#60 Updated by intrigeri 2017-06-04 14:11:46

  • related to Bug #12249: Greeter's Formats popover may be cut added

#61 Updated by intrigeri 2017-06-04 14:11:50

  • related to Bug #12228: Greeter's headerbar is sometimes not themed right added

#62 Updated by intrigeri 2017-06-04 14:11:54

  • related to Bug #12241: New Greeter's additional options window sometimes opens in a weird place added

#63 Updated by intrigeri 2017-06-04 14:12:01

  • related to Bug #12548: The Greeter sometimes does not display the GNOME Shell top bar added

#64 Updated by intrigeri 2017-06-04 14:12:06

  • related to Bug #12386: Greeter sometimes fails to set Admin password on Stretch added

#65 Updated by alant 2018-01-28 14:51:16

  • Priority changed from Elevated to Normal

I haven’t seen that happen nor heard of it since we released the greeter. I suggest we reject this ticket if we don’t get a way to reproduce the issue within the next 3 months.

#66 Updated by Anonymous 2018-08-17 09:45:19

  • Status changed from Confirmed to Rejected

Closing as no news since 7 months and no known complaints.

#67 Updated by intrigeri 2020-04-15 06:01:44

  • Affected tool changed from Greeter to Welcome Screen