Bug #10222

Pidgin tests with certificates are fragile

Added by kytv 2015-09-19 09:23:07 . Updated 2015-09-22 14:40:45 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Test suite
Target version:
Start date:
2015-09-19
Due date:
% Done:

100%

Feature Branch:
kytv:test/10222-pidgin-certificates
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Instant Messaging
Deliverable for:

Description

Setting the TailsToaster VM to using multple CPU cores (as per Feature #6729) caused the Pidgin tests to fail frequently when attempting to add certificates. The problem was that the windows we look for aren’t in focus. Unfortunately, just using vm.focus_window won’t be enough to solve the problem because sometimes focusing the window will sometimes fail (Bug #9330 — in progress).

Note that this isn’t just a theoretical problem; similar failures can also be seen on isotester1, such as:

FindFailed: can not find PidginCertificateImportFailed.png on the screen.
Line ?, in File ? (RuntimeError)
features/pidgin.feature:82:in `Then I cannot add a certificate from the
"/home/amnesia/.gnupg" directory to Pidgin'

I created the following temporary feature to test this.

@product
Feature: Chatting anonymously using Pidgin
  As a Tails user
  when I chat using Pidgin
  I should be able to use OTR
  And I should be able to persist my Pidgin configuration
  And AppArmor should prevent Pidgin from doing dangerous things
  And all Internet traffic should flow only through Tor

  Background:
    Given a computer
    And I start Tails from DVD with network unplugged and I login
    Then Pidgin has the expected accounts configured with random nicknames
    And I start Pidgin through the GNOME menu
    And I see Pidgin's account manager window
    And I close Pidgin's account manager window
    And I save the state so the background can be restored next scenario

  Scenario: Adding a certificate to Pidgin
    Then I can add a certificate from the "/home/amnesia" directory to Pidgin

  Scenario: Failing to add a certificate to Pidgin
    Then I cannot add a certificate from the "/home/amnesia/.gnupg" directory to Pidgin
    When I close Pidgin's certificate import failure dialog
    And I close Pidgin's certificate manager

I repeated the scenarios until there were 74 in total. On isotester1 I had the following results when setting the TailsToaster VM to use 3 CPUs:

Failing Scenarios:
cucumber features/certtest.feature:19 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:46 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:145 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:163 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:184 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:190 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:199 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:256 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:289 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:316 # Scenario: Adding a certificate to Pidgin

74 scenarios (10 failed, 64 passed)
666 steps (10 failed, 4 skipped, 652 passed)
20m39.316s

On my T-430 the results were even more grim.

Failing Scenarios:
cucumber features/certtest.feature:37 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:46 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:73 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:82 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:91 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:121 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:127 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:130 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:136 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:139 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:148 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:163 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:172 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:184 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:190 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:199 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:217 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:238 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:244 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:265 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:271 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:280 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:283 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:289 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:292 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:307 # Scenario: Adding a certificate to Pidgin
cucumber features/certtest.feature:328 # Scenario: Failing to add a certificate to Pidgin
cucumber features/certtest.feature:343 # Scenario: Adding a certificate to Pidgin

74 scenarios (28 failed, 46 passed)
666 steps (28 failed, 20 skipped, 618 passed)
17m12.320s

With code that I have in my workspace the robustness improves tremendously. I have yet to see it fail (but I’m trying!)


Subtasks


Related issues

Related to Tails - Feature #5330: Test suite: identify and document race conditions Resolved 2015-01-19

History

#1 Updated by kytv 2015-09-19 11:58:04

Locally, with my changes applied:

296 scenarios (296 passed)
2664 steps (2664 passed)
60m21.445s

It’s still running on isotester1.

#2 Updated by kytv 2015-09-19 12:20:06

kytv wrote:
> Locally, with my changes applied:
>
> […]
>
> It’s still running on isotester1.

…and it just finished.

296 scenarios (296 passed)
2664 steps (2664 passed)
85m42.758s

#3 Updated by kytv 2015-09-19 17:38:35

  • Assignee changed from kytv to anonym
  • QA Check set to Ready for QA
  • Feature Branch set to kytv:test/10222-pidgin-certificates

#4 Updated by kytv 2015-09-20 08:30:45

Applied in changeset commit:f92bc6e0c0e3daec59bc8e27757b88fe5a01d161.

#5 Updated by anonym 2015-09-20 08:30:46

  • Status changed from In Progress to Fix committed
  • % Done changed from 40 to 100

Applied in changeset commit:430078e885c3098c9db9cd4cf7caad39e1eb1b2a.

#6 Updated by anonym 2015-09-20 08:31:20

  • Assignee deleted (anonym)
  • Target version changed from Tails_1.7 to Tails_1.6
  • QA Check changed from Ready for QA to Pass

#7 Updated by kytv 2015-09-20 15:36:34

  • related to Feature #5330: Test suite: identify and document race conditions added

#8 Updated by bertagaz 2015-09-22 14:40:45

  • Status changed from Fix committed to Resolved
  • QA Check deleted (Pass)