Tails

#1 Updated by anonym 2016-02-14 19:02:10

• Subject changed from Improve automated GUI testing robustness "GUI aware" technologies to Improve automated GUI testing robustness using "GUI aware" technologies

#2 Updated by anonym 2016-02-14 19:02:49

• Blueprint set to https://tails.boum.org/blueprint/automated_builds_and_tests/testing/

#3 Updated by intrigeri 2016-02-14 21:15:31

• Feature Branch set to test/10721-a11y-technologies

#4 Updated by anonym 2016-04-06 07:30:23

• % Done changed from 0 to 30
• Type of work changed from Research to Discuss

I’ve force pushed a re-written version of the test/10721-a11y-technologies branch. Currently I’ve only migrated all application starting (via the Applications menu) and encryption.feature to Dogtail, and the results according to

git diff --stat origin/devel... -- features/*.feature features/step_definitions/ features/images/

is pretty promising: 37 images killed, and we netted 31 lines less than before. Woo! However, looking at the step definitions of encryption.feature, it actually grew 23 lines (net). Most of the loss is explained by killing all the crazy code we had for working around Sikuli not handling the Applications menu navigation very well, and who knows, maybe some of it has to be revived for the Dogtail version — we’ll see.

Any way, I expect that when Sikuli works well, it will require less code than Dogtail (“a picture is worth a kLOC?” :D), but the migration will hopefully be worth it given the eliminated maintenance of these images, and increased robustness, despite this.

BTW, I will have to upstream one new feature to dogtail (see config/chroot_local-patches/python-dogtail_searchShowingOnly.diff) but IMHO it is a brilliant feature that I expect most users will want to enable by default, so I don’t think upstreaming it should be a problem.

So I think this concludes the research part, with the conclusion being that Dogtail seems like a great addition to the toolbox we have in our test suite, but not a complete replacement of Sikuli still. I’m unsure what the next step is, given that converting the rest of the test suite is a pretty significant piece of work. I suggest we briefly discuss this during our next CI meeting.

#7 Updated by anonym 2016-04-15 08:41:25

• Target version set to Tails_2.4
• % Done changed from 30 to 40
• Type of work changed from Discuss to Contributors documentation

Here are the next steps, as discussed during April’s CI meeting:

1. Update design docs, write minimal guidelines/tips-and-tricks (e.g. use sniff/accerciser/ipython to explore Dogtail’s PoV) from test writers, and explain/recommend when to use which.
2. Merge this quickly, so new stuff can change paradigm towards Dogtail when more suitable.
3. Only migrate stuff to Dogtail as we encounter issues, or part of “refactoring”? I.e. no big migration project. However, when working on the Debian migration (currently feature/stretch) we should try to use Dogtail first when an image needs updating, and we do so in devel so we start benefiting from it on current releases as well.

#8 Updated by anonym 2016-04-26 13:54:14

• Status changed from Confirmed to In Progress
• Assignee changed from anonym to intrigeri
• % Done changed from 40 to 50
• QA Check set to Ready for QA
• Type of work changed from Contributors documentation to Code

anonym wrote:
> Here are the next steps, as discussed during April’s CI meeting:
>
> 1. Update design docs, write minimal guidelines/tips-and-tricks (e.g. use sniff/accerciser/ipython to explore Dogtail’s PoV) from test writers, and explain/recommend when to use which.

Done in commit:4491b80.

> 2. Merge this quickly, so new stuff can change paradigm towards Dogtail when more suitable.

Next step! What do you think, intri?

> 3. Only migrate stuff to Dogtail as we encounter issues, or part of “refactoring”? I.e. no big migration project. However, when working on the Debian migration (currently feature/stretch) we should try to use Dogtail first when an image needs updating, and we do so in devel so we start benefiting from it on current releases as well.

As the Test Automaton Grand General, I’ll keep this in mind when commanding my minions.

#10 Updated by intrigeri 2016-04-29 02:07:41

• Assignee changed from intrigeri to anonym
• QA Check changed from Ready for QA to Dev Needed

intrigeri wrote:
> I’m now going to do a full test suite run […]

Done (on an ISO built from the branch). I see a lot of scenarios that rely on /^I start "([^"]+)" via the GNOME "([^"]+)" applications menu$/ fail, e.g. all those that start Pidgin, and:

  Scenario: Install packages using Synaptic                                                                        # features/apt.feature:23
    When I start Synaptic                                                                                          # features/step_definitions/apt.rb:52
      Command failed: /usr/bin/python '/tmp/tmp.6yLZBfPfbF'
      error code: 1
      stderr: .
      <false> is not true. (ExecutionFailedInVM)
      ./features/support/helpers/vm_helper.rb:441:in `rescue in execute_successfully'
      ./features/support/helpers/vm_helper.rb:438:in `execute_successfully'
      ./features/support/helpers/dogtail.rb:87:in `run'
      ./features/support/helpers/dogtail.rb:203:in `block (2 levels) in <class:Node>'
      ./features/step_definitions/common_steps.rb:644:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/common_steps.rb:643:in `each'
      ./features/step_definitions/common_steps.rb:643:in `/^I start "([^"]+)" via the GNOME "([^"]+)" applications menu$/'
      ./features/step_definitions/apt.rb:53:in `/^I start Synaptic$/'
      features/apt.feature:24:in `When I start Synaptic'


  Scenario: A warning will be displayed if Electrum is not persistent              # features/electrum.feature:9
    Given I have started Tails from DVD and logged in and the network is connected # features/step_definitions/snapshots.rb:208
    When I start Electrum through the GNOME menu                                   # features/step_definitions/electrum.rb:1
      Command failed: /usr/bin/python '/tmp/tmp.ll1lGWeKyd'
      error code: 1
      stderr: .
      <false> is not true. (ExecutionFailedInVM)
      ./features/support/helpers/vm_helper.rb:441:in `rescue in execute_successfully'
      ./features/support/helpers/vm_helper.rb:438:in `execute_successfully'
      ./features/support/helpers/dogtail.rb:87:in `run'
      ./features/support/helpers/dogtail.rb:203:in `block (2 levels) in <class:Node>'
      ./features/step_definitions/common_steps.rb:644:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/common_steps.rb:643:in `each'
      ./features/step_definitions/common_steps.rb:643:in `/^I start "([^"]+)" via the GNOME "([^"]+)" applications menu$/'
      ./features/step_definitions/electrum.rb:2:in `/^I start Electrum through the GNOME menu$/'
      features/electrum.feature:11:in `When I start Electrum through the GNOME menu'


  Scenario: Deleting a Tails persistent partition                                                                                       # features/persistence.feature:46
    Given I have started Tails without network from a USB drive with a persistent partition and stopped at Tails Greeter's login screen # features/step_definitions/snapshots.rb:208
    And I log in to a new session                                                                                                       # features/step_definitions/common_steps.rb:280
    Then Tails is running from USB drive "__internal"                                                                                   # features/step_definitions/usb.rb:374
    And the boot device has safe access rights                                                                                          # features/step_definitions/usb.rb:396
    And persistence is disabled                                                                                                         # features/step_definitions/usb.rb:346
    But a Tails persistence partition exists on USB drive "__internal"                                                                  # features/step_definitions/usb.rb:271
    And all notifications have disappeared                                                                                              # features/step_definitions/common_steps.rb:409
    When I delete the persistent partition                                                                                              # features/step_definitions/usb.rb:570
      Command failed: /usr/bin/python '/tmp/tmp.NOurVspCjK'
      error code: 1
      stderr: .
      <false> is not true. (ExecutionFailedInVM)
      ./features/support/helpers/vm_helper.rb:441:in `rescue in execute_successfully'
      ./features/support/helpers/vm_helper.rb:438:in `execute_successfully'
      ./features/support/helpers/dogtail.rb:87:in `run'
      ./features/support/helpers/dogtail.rb:203:in `block (2 levels) in <class:Node>'
      ./features/step_definitions/common_steps.rb:644:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/common_steps.rb:643:in `each'
      ./features/step_definitions/common_steps.rb:643:in `/^I start "([^"]+)" via the GNOME "([^"]+)" applications menu$/'
      ./features/step_definitions/usb.rb:571:in `/^I delete the persistent partition$/'
      features/persistence.feature:54:in `When I delete the persistent partition'


  Scenario: Fetching OpenPGP keys using Seahorse should work and be done over Tor. # features/torified_gnupg.feature:21
    When I fetch the "10CC5BC7" OpenPGP key using Seahorse                         # features/step_definitions/torified_gnupg.rb:154
      Command failed: /usr/bin/python '/tmp/tmp.zbDeJjhLOg'
      error code: 1
      stderr: .
      <false> is not true. (ExecutionFailedInVM)
      ./features/support/helpers/vm_helper.rb:441:in `rescue in execute_successfully'
      ./features/support/helpers/vm_helper.rb:438:in `execute_successfully'
      ./features/support/helpers/dogtail.rb:87:in `run'
      ./features/support/helpers/dogtail.rb:203:in `block (2 levels) in <class:Node>'
      ./features/step_definitions/common_steps.rb:644:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/common_steps.rb:643:in `each'
      ./features/step_definitions/common_steps.rb:643:in `/^I start "([^"]+)" via the GNOME "([^"]+)" applications menu$/'
      ./features/step_definitions/torified_gnupg.rb:23:in `start_or_restart_seahorse'
      ./features/step_definitions/torified_gnupg.rb:92:in `/^I start Seahorse( via the OpenPGP Applet)?$/'
      ./features/step_definitions/torified_gnupg.rb:155:in `/^I fetch the "([^"]+)" OpenPGP key using Seahorse( via the OpenPGP Applet)?$/'
      features/torified_gnupg.feature:22:in `When I fetch the "10CC5BC7" OpenPGP key using Seahorse'

Any clue what’s happening?

#11 Updated by anonym 2016-05-04 13:36:53

• Assignee changed from anonym to intrigeri
• % Done changed from 50 to 60
• QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:
> >> 2. Merge this quickly, so new stuff can change paradigm towards Dogtail when more suitable.
>
> > Next step! What do you think, intri?
>
> Great job! It’s lovely to see this many images go away :)
>
> Code review done, only a few details and minor comments:
>
> * decrypt_verify_helper and friends: I find the function signature obscure when it’s used. How about using named arguments (with whatever tricks Ruby requires to have them)?

Done in commit:ccff595.

> * “This was we can support stuff like” ← I can’t parse this sentence.

“This way we …”. Fixed now!

> * Can’t we do python3 instead of python2? (As you can guess, I’m mostly looking ahead to future pain points here.)

Debian ships Dogtail 0.9.0, which is Python2. Dogtail 0.9.9 adds Python3 support, so we’ll have to wait until it is available for the Debian version Tails is based on.

> * self.args_to_s might benefit from a comment about its args (I think I got it right by reading the code, but still).

Done in commit:d490bbd.

> * Thanks for the comment at the top of def children because oh my, I would have a hard time improving this function if I ever had to.

You are welcome! :) I know it all got a bit crazy complexity-wise, but I failed to find anything simpler that still would be convenient enough for us.

> * I expected that features/support/helpers/dogtail.rb would be much bigger. I’m glad it’s not! :)

Recently I started toying with the idea of extending the remote shell to, essentially, run a python shell loaded with dogtail that we can interact with, so objects persist and can be referenced later. That should simplify the this glue code a lot, but, well, later.

> Regarding how it fares in practice, I see two recent failures on Jenkins that make me feel a bit uneasy:
>
> * https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10721-a11y-technologies/21/cucumberTestReport/encryption-and-verification-using-gnupg/symmetric-encryption-and-decryption-using-openpgp-applet/ ← this one is quite obscure; can we get more debugging info from the Python script / from dogtail?

Interesting. So I looked at the debug.log and found this for a bit more context:

[...]
calling as amnesia: echo '#!/usr/bin/python
from dogtail import tree
from dogtail.config import config
config.searchShowingOnly = True
application = tree.root.application('"'"'gedit'"'"')
application.button('"'"'New'"'"').click()' >> '/tmp/tmp.kg0NHEhCeZ'
call returned: [0, "", ""]
calling as amnesia: /usr/bin/python '/tmp/tmp.kg0NHEhCeZ'
call returned: [0, "Creating logfile at /tmp/dogtail-amnesia/logs/tmp.kg0NHEhCeZ_20160425-195535_debug ...\nDogtail: warning: application may be hanging\nClicking on [push button | New]\nMouse button 1 click at (207,70)\n", ""]
calling as root: rm -f '/tmp/tmp.kg0NHEhCeZ'
call returned: [0, "", ""]
calling as amnesia: mktemp
call returned: [0, "/tmp/tmp.qHGZBlkVKv\n", ""]
calling as root: rm -f '/tmp/tmp.qHGZBlkVKv'
call returned: [0, "", ""]
calling as amnesia: echo '#!/usr/bin/python
from dogtail import tree
from dogtail.config import config
config.searchShowingOnly = True
application = tree.root.application('"'"'gedit'"'"')
application.child(roleName='"'"'text'"'"').click(button=3)' >> '/tmp/tmp.qHGZBlkVKv'
call returned: [0, "", ""]
calling as amnesia: /usr/bin/python '/tmp/tmp.qHGZBlkVKv'
call returned: [0, "Creating logfile at /tmp/dogtail-amnesia/logs/tmp.qHGZBlkVKv_20160425-195556_debug ...\nDogtail: warning: application may be hanging\nClicking on [text | ]\nMouse button 3 click at (511,399)\n", ""]
calling as root: rm -f '/tmp/tmp.qHGZBlkVKv'
call returned: [0, "", ""]
calling as amnesia: mktemp
call returned: [0, "/tmp/tmp.dBE66obICt\n", ""]
calling as root: rm -f '/tmp/tmp.dBE66obICt'
call returned: [0, "", ""]
calling as amnesia: echo '#!/usr/bin/python
from dogtail import tree
from dogtail.config import config
config.searchShowingOnly = True
application = tree.root.application('"'"'gedit'"'"')
application.menuItem('"'"'Paste'"'"').click()' >> '/tmp/tmp.dBE66obICt'
call returned: [0, "", ""]
calling as amnesia: /usr/bin/python '/tmp/tmp.dBE66obICt'
call returned: [1, "Creating logfile at /tmp/dogtail-amnesia/logs/tmp.dBE66obICt_20160425-195618_debug ...\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 3)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 4)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 5)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 6)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 7)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 8)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 9)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 10)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 11)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 12)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 13)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 14)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 15)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 16)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 17)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 18)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 19)\nTraceback (most recent call last):\n  File \"/tmp/tmp.dBE66obICt\", line 6, in <module>\n    application.menuItem('Paste').click()\n  File \"/usr/lib/python2.7/dist-packages/dogtail/tree.py\", line 1008, in menuItem\n    return self.findChild(predicate.IsAMenuItemNamed(menuItemName=menuItemName), recursive, showingOnly=showingOnly)\n  File \"/usr/lib/python2.7/dist-packages/dogtail/tree.py\", line 897, in findChild\n    raise SearchError(describeSearch(self, pred, recursive, debugName))\nSearchError: descendent of [application | gedit]: \"Paste\" menuitem\n\n", ""]
calling as root: rm -f '/tmp/tmp.dBE66obICt'
call returned: [0, "", ""]
    And I symmetrically encrypt the message with password "asdf"      # features/step_definitions/encryption.rb:143
      Command failed: /usr/bin/python '/tmp/tmp.dBE66obICt'
[...]

The interesting parts are the occurrences of “Dogtail: warning: application may be hanging”. I see them nowhere else in the debug.log, and it seems that those two actions (first left-clicking Gedit’s “New” button, then right-clicking the text area to get the context menu) fail, which then make the third action (find the “Paste” entry in the context menu) fail.

This is a bit worrying — it seems like a random source of robustness issues. Luckily I haven’t seen it any of the other 10 debug.log:s I’ve looked at. I feel tempted to add a check for this error and make us abort with a more easily identifiable message, but perhaps it is not always fatal so that is unnecessary. I’ll keep an eye out for this.

> * https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10721-a11y-technologies/15/cucumberTestReport/encryption-and-verification-using-gnupg/symmetric-encryption-and-decryption-using-tails-openpgp-applet/:
> “WARNING : AT-SPI: Error in GetItems, sender=org.freedesktop.DBus, error=The name :1.0 was not provided by any .service files.” ← ?!

I see it all over the place in the debug.log so I don’t think it is fatal. Looking a bit earlier the issue becomes clearer; fixed in commit:c2db06a.

The causes for the errors you reported in Feature #10721#note-10 are fixed in commits commit:52069cb and commit:fc77772. I’m really surprised how I didn’t notice this during my tests. I cannot possibly have run the full test suite, which I am sorry for.

Massive thanks for your review!

#12 Updated by intrigeri 2016-05-05 03:39:55

• Assignee changed from intrigeri to anonym
• QA Check changed from Ready for QA to Info Needed

> intrigeri wrote:
>> * decrypt_verify_helper and friends: I find the function signature obscure when it’s used. How about using named arguments (with whatever tricks Ruby requires to have them)?

> Done in commit:ccff595.

Cool. Just curious: why is the Hash approach more suitable here than Ruby 2.x’s keyword arguments ?

>> * Can’t we do python3 instead of python2? (As you can guess, I’m mostly looking ahead to future pain points here.)

> Debian ships Dogtail 0.9.0, which is Python2. Dogtail 0.9.9 adds Python3 support, so we’ll have to wait until it is available for the Debian version Tails is based on.

OK, sounds good enough!

>> https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10721-a11y-technologies/21/cucumberTestReport/encryption-and-verification-using-gnupg/symmetric-encryption-and-decryption-using-openpgp-applet/ ← this one is quite obscure; can we get more debugging info from the Python script / from dogtail?

> Interesting. So I looked at the debug.log and found this for a bit more context: […]

> The interesting parts are the occurrences of “Dogtail: warning: application may be hanging”. I see them nowhere else in the debug.log, and it seems that those two actions (first left-clicking Gedit’s “New” button, then right-clicking the text area to get the context menu) fail, which then make the third action (find the “Paste” entry in the context menu) fail.

> This is a bit worrying — it seems like a random source of robustness issues. Luckily I haven’t seen it any of the other 10 debug.log:s I’ve looked at. I feel tempted to add a check for this error and make us abort with a more easily identifiable message, but perhaps it is not always fatal so that is unnecessary.

Hmmm. On the one hand, I’m tempted to say that this specific test scenario was fragile anyway, so we’re not worse with dogtail. OTOH, I’m concerned that this class of problems may introduce new robustness issues elsewhere, which feels wrong given we’re working on this precisely as a mean to improve robustness.

> I’ll keep an eye out for this.

Note that it happened again already: https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10721-a11y-technologies/29/cucumberTestReport/encryption-and-verification-using-gnupg/symmetric-encryption-and-decryption-using-openpgp-applet/ — which makes me seriously wonder if the failure frequency is worrying enough to be a merge blocker. What do you think?

I’m now doing another full test suite run on that branch, but still I’m reassigning to you so you can comment on the above two questions.

#13 Updated by intrigeri 2016-05-05 11:27:15

• % Done changed from 60 to 70

After a bunch of retries, I got the full testing passing, modulo the Icedove test bug I just reported. Congrats!

#16 Updated by anonym 2016-05-10 08:59:50

• % Done changed from 70 to 80
• QA Check changed from Info Needed to Dev Needed

> Let’s discuss this at the CI meeting.

We decided to revert all dogtail stuff for encryption.feature due to the “application may be hanging” bug, and then switftly merge this. Then dogtail will be available as a tool for other tests we need to write now, and we can continue investigating the “Dogtail: warning: application may be hanging” bug. Any new tests using dogtail will be tested extra to see if we can spot the bug in other contexts.

#17 Updated by anonym 2016-05-11 03:29:33

• Status changed from In Progress to Fix committed
• Assignee deleted (anonym)
• % Done changed from 80 to 100
• QA Check changed from Dev Needed to Pass

anonym wrote:
> > Let’s discuss this at the CI meeting.
>
> We decided to revert all dogtail stuff for encryption.feature

Done in commit:339fced5d1da8c28434a4360eb7acd8860c0e9d3.

> due to the “application may be hanging” bug,

I filed Bug #11409 to track it.

> and then switftly merge this.

Merged!

#18 Updated by anonym 2016-06-08 01:24:56

• Status changed from Fix committed to Resolved