Bug #17029

Test suite: findfailed_hook makes Sikuli use fuzzy images even when fuzzy image matching is disabled

Added by intrigeri 2019-09-06 09:07:32 . Updated 2019-09-12 16:14:12 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Test suite
Target version:
Start date:
Due date:
% Done:

100%

Feature Branch:
https://salsa.debian.org/tails-team/tails/merge_requests/37#note_108671
Type of work:
Code
Blueprint:

Starter:
Affected tool:
OpenPGP Applet
Deliverable for:

Description

For example, Sikuli clicks on “Untitled document 1” tab while looking for OpenPGP Applet’s GpgAppletDecryptVerify.png.


Files


Subtasks


Related issues

Related to Tails - Bug #17030: Missing "Tor Browser" shortcut in the Places menu Resolved
Has duplicate Tails - Bug #17006: Test suite: clicking on the OpenPGP applet icon is fragile Duplicate
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by intrigeri 2019-09-06 09:20:09

This happened 4 times in the last 10 runs on Jenkins (devel) branch ⇒ FT.

#2 Updated by intrigeri 2019-09-06 09:20:33

#3 Updated by intrigeri 2019-09-06 09:23:02

  • Priority changed from Normal to Elevated

#4 Updated by intrigeri 2019-09-06 10:16:24

Actually, Sikuli is too fast: on the video, I see the mouse cursor moving towards the “Untitled document 1” tab before OpenPGP Applet had time to open its menu. Now, I think there’s a bug in our code. Fuzzy matching is used, that’s why Sikuli clicks on the (vaguely similar) tab instead of waiting any further:

03:19:20.989537354: Sikuli: calling wait("GpgAppletDecryptVerify.png")...
03:19:45.108976627: Found fuzzy candidate picture for GpgAppletDecryptVerify.png with similarity 0.4
03:19:45.109665405: FindFailedHookFailure was raised, re-running the failing Sikuli method.
03:19:45.172519400: Sikuli: calling click(M[158,82 176x22]@S(S(0)[0,0 1024x768]) S:0.42 C:246,93 [62 msec])...
03:19:45.902044690: [log] CLICK on L(246,93)@S(0)[0,0 1024x768]

It’s not the first time I notice a fuzzy image being used while I don’t expect it to be.
I don’t see “Using fuzzy candidate picture for #{picture}” in the debug log, which confirms that SIKULI_FUZZY_IMAGE_MATCHING is not true. But FindFailedHookFailure is invoked and on 2nd try, Sikuli is happy to match the fuzzy tab header (I assume “S:0.42” means similarity = 0.42). It looks exactly as if us retrying in findfailed_hook with lower similarity expectations reset the MinSimilarity setting for the following Sikuli action.

I see the same kind of behavior elsewhere, e.g.

03:26:45.644231746: Sikuli: calling wait("Gtk3PrintButton.png")...
03:27:02.733362618: Found fuzzy candidate picture for Gtk3PrintButton.png with similarity 0.6
03:27:02.733999556: FindFailedHookFailure was raised, re-running the failing Sikuli method.
03:27:02.803860495: Sikuli: calling click(M[569,103 81x36]@S(S(0)[0,0 1024x768]) S:0.69 C:609,121 [70 msec])...
03:27:03.527528786: [log] CLICK on L(609,121)@S(0)[0,0 1024x768]

Without diving into the rjb craziness, what we could do is:

  • Call pattern.similar(0.9) at the end of each [0.80, 0.70, 0.60, 0.50, 0.40].each do |similarity| iteration, in the hope we revert the side effect.
  • Skip [0.80, 0.70, 0.60, 0.50, 0.40].each do |similarity| entirely unless fuzzy image matching is enabled. For sure, this will make some of our work more difficult (one will have to run tests locally and enable fuzzy image matching) but IMO it’s worth it, in order to fix weird behavior that breaks the test suite in a way that affects every developer, in many runs.

#5 Updated by intrigeri 2019-09-06 10:17:27

  • Subject changed from Test suite clicks on "Untitled document 1" tab while looking for OpenPGP Applet's GpgAppletDecryptVerify.png to Test suite: findfailed_hook makes Sikuli use fuzzy images even when fuzzy image matching is disabled
  • Description updated

#6 Updated by intrigeri 2019-09-06 10:42:20

  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri
  • Feature Branch set to test/17029+17027

#7 Updated by intrigeri 2019-09-06 10:48:51

  • Feature Branch changed from test/17029+17027 to test/17029+17027+force-all-tests

#8 Updated by intrigeri 2019-09-06 12:55:20

  • has duplicate Bug #17006: Test suite: clicking on the OpenPGP applet icon is fragile added

#9 Updated by intrigeri 2019-09-06 18:16:05

  • related to Bug #17030: Missing "Tor Browser" shortcut in the Places menu added

#10 Updated by intrigeri 2019-09-06 18:20:22

  • Feature Branch changed from test/17029+17027+force-all-tests to test/17029+17027+17028+16931+force-all-tests

#11 Updated by intrigeri 2019-09-06 18:26:55

Note that this test suite bug make us miss an actual, user-impacting, bug: Bug #17030.

#12 Updated by intrigeri 2019-09-08 15:50:31

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (intrigeri)
  • Feature Branch changed from test/17029+17027+17028+16931+force-all-tests to https://salsa.debian.org/tails-team/tails/merge_requests/37#note_108671

#13 Updated by intrigeri 2019-09-08 15:51:10

  • Target version set to Tails_4.0

#14 Updated by segfault 2019-09-12 16:14:12

  • Status changed from Needs Validation to Resolved
  • % Done changed from 0 to 100

Applied in changeset commit:tails|e889519458da6235b299e4077a03d88a4fbdea7e.