Feature #14585

Investigate Dogtail's long-term viability

Added by intrigeri 2017-09-02 08:12:18 . Updated 2019-01-09 16:46:15 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Test suite
Target version:
Tails_3.9.1
Start date:
2017-09-02
Due date:
% Done:

100%

Feature Branch:
Type of work:
Communicate
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Part of the “Make test suite survive tech changes” roadmap item.

We have multiple concerns about the long-term viability of using Dogtail:

* intrigeri heard a GUADEC speaker mention that dogtail does not work on Wayland. If that’s indeed the case, then we should look into this problem before we keep porting tons of tests to dogtail: it would be a waste of time to do so only to have to port them to another technology again a year later. See https://gitlab.com/dogtail/dogtail/issues/3.

* The package is orphaned in Debian and was of course not updated to the latest upstream release.

* Fedora (and Redhat in general?) now uses openQA, which only supports image-based testing.

* Dogtail development has slowed down considerably since we adopted it. Here’s the a number of commits per year breakdown:

2011: 4
2012: 52
2013: 25
2014: 27
2015: 13
2016: 41
2017: 9

Subtasks

Related issues

Related to Tails - Bug #12191: Dogtail's showingOnly option is not working sometimes Confirmed 2017-01-30
Blocks Tails - Feature #12185: Update Debian package to run Dogtail using Python 3 Resolved 2017-01-28
Blocks Tails - Bug #15953: Make our test suite survive changes in the surrounding environment Resolved 2018-09-14

History

#1 Updated by intrigeri 2017-09-14 17:24:55

  • blocks Feature #12185: Update Debian package to run Dogtail using Python 3 added

#2 Updated by anonym 2017-09-15 16:14:07

  • Type of work changed from Code to Communicate

Fuck. I’ll try to get a clarification about dogtail’s status from the upstream (it seems they’ve moved from Fedora’s infra to GitLab), and what the plans for Wayland are. Until then I’ll suspend all my Dogtail work. :_(

#3 Updated by anonym 2017-09-22 01:34:52

I’ve sent an email to its mailing list asking for clarification. We’ll see.

Also, Dogtail 0.9.10 was released this May, which inspires some confidence!

#4 Updated by anonym 2017-10-02 15:40:17

  • Subject changed from Make our dogtail-based automated tests work on Wayland to Investigate Dogtail's long-term viability
  • Description updated

#5 Updated by anonym 2017-10-02 15:41:51

  • blocks Feature #13240: Core work 2017Q4: Test suite maintenance added

#6 Updated by anonym 2017-10-02 15:42:19

  • Status changed from Confirmed to In Progress
  • Priority changed from Normal to Elevated
  • Target version set to Tails_3.3
  • % Done changed from 0 to 10

#7 Updated by intrigeri 2017-11-06 15:43:22

  • Target version changed from Tails_3.3 to Tails_3.5

#8 Updated by intrigeri 2017-12-07 12:44:18

  • Target version deleted (Tails_3.5)

#9 Updated by intrigeri 2018-01-01 16:59:14

  • blocked by deleted (Feature #13240: Core work 2017Q4: Test suite maintenance)

#10 Updated by intrigeri 2018-01-01 16:59:30

  • Target version set to 2019

#11 Updated by Anonymous 2018-05-28 12:46:43

anonym wrote:
> I’ve sent an email to its mailing list asking for clarification. We’ll see.

I can’t find your email on the list archives. It looks like they’ve ceased to exist since 10 months already. I wanted to see if there were any replies to this.

#12 Updated by intrigeri 2018-05-28 15:22:01

  • Description updated

#13 Updated by intrigeri 2018-07-07 20:53:16

> * intrigeri heard a GUADEC speaker mention that dogtail does not work on Wayland.

I see that dogtail uses pyatspi to generate mouse and keyboard events so I’ve followed the suggestions on https://www.freedesktop.org/wiki/Accessibility/AT-SPI2/ and did that:

export GTK_MODULES=gail:atk-bridge
export OOO_FORCE_DESKTOP=gnome
export GNOME_ACCESSIBILITY=1
export QT_ACCESSIBILITY=1

and gsettings set org.gnome.desktop.interface toolkit-accessibility true which is a pre-requisite anyway.

I’ve tried on my own GNOME + Wayland session (sid), that has XWayland, and while dogtail can definitely query info about GUIs, I could not manage to make it interact with gedit (that seems to run as a native Wayland app): Dogtail thinks it has successfully clicked around but nothing visible happens in the app. So indeed, dogtail is currently broken for native Wayland apps.

import dogtail
import dogtail.tree
dogtail.tree.root.application('gedit').window('Untitled Document 1 - gedit').button('Save').click()

Now, if I start gedit with GDK_BACKEND=x11 gedit (XWayland) then things work again as in the good old days… without the security benefits of Wayland. But that gives us one potential option: we could use the x11 backend in the test suite for the apps we want to test with Dogtail, but not in a real Tails. The main problem with this option is of course that we might miss some Wayland-specific issues. I would have been concerned with that a couple years ago but it’s been a year or so since the last time I noticed such a problem on my own system: things have been fixed.

#14 Updated by intrigeri 2018-07-07 20:55:39

  • Description updated

#15 Updated by Anonymous 2018-08-17 09:55:39

  • related to Bug #12191: Dogtail's showingOnly option is not working sometimes added

#16 Updated by intrigeri 2018-08-28 05:33:14

  • Assignee changed from anonym to intrigeri
  • Target version changed from 2019 to Tails_3.10.1

We should make a decision wrt. dogtail before the work to port the test suite to Buster resumes. I’ll handle it.

#17 Updated by nodens 2018-08-30 15:07:50

#18 Updated by intrigeri 2018-09-14 07:56:05

  • Description updated

#19 Updated by intrigeri 2018-09-14 08:00:09

  • blocks Bug #15953: Make our test suite survive changes in the surrounding environment added

#20 Updated by intrigeri 2018-09-14 08:32:54

#21 Updated by intrigeri 2018-09-18 16:24:01

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 10 to 100

tl;dr: let’s invest as little as we can into Dogtail but for now, we can use it.

It’s hard to tell about long-term but so far:

  • It does not seem unreasonable to keep using Dogtail and even to port specific tests to it when it gives us substantial benefits. But let’s not port tests to Dogtail just for fun (anyway, as I’ve learned the hard way when writing the VeraCrypt tests, Dogtail does not deliver the expected robustness, and sometimes Sikuli works much better).
  • A workaround is available for the lack of Wayland support (which is really not a problem in Dogtail itself, but rather missing a11y APIs in Wayland).

Now, we should start contributing to the maintenance of the package in Debian, so we get Python 3 and Unicode support; I’ll report this on Feature #12185.

#22 Updated by intrigeri 2018-10-01 13:29:38

  • Target version changed from Tails_3.10.1 to Tails_3.9.1

#23 Updated by intrigeri 2019-01-09 16:46:16

intrigeri wrote:
> * A workaround is available for the lack of Wayland support (which is really not a problem in Dogtail itself, but rather missing a11y APIs in Wayland).

Another workaround: gnome-ponytail-daemon is a helper daemon intended for dogtail on Wayland.