Feature #8539

Make the test suite robust enough to be run as part of a CI setup

Added by intrigeri 2015-01-06 12:49:51 . Updated 2015-09-22 15:12:27 .

Status:
Resolved
Priority:
High
Assignee:
Category:
Test suite
Target version:
Tails_1.6
Start date:
2014-10-05
Due date:
2015-03-06
% Done:

100%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Subtasks

Feature #5330: Test suite: identify and document race conditions Resolved

100
Bug #8059: Windows Camouflage automated tests sometimes failed due to differently ordered icons in the notification area Resolved

100
Bug #8140: Running applications via the GNOME menu is fragile Resolved

100
Bug #7233: How to deal with tests expected to fail? Resolved

100
Bug #8697: Step "I update APT using apt-get" succeeds even if apt-get update failed Resolved

100
Bug #8698: Step "I open the address "([^"]*)" in the Tor Browser" is fragile Resolved

100
Bug #8707: The automated test suite does not clean up when it's finished Resolved

100
Bug #8714: Step 'Tor is ready' fails randomly Resolved

100
Bug #8875: Make running applications from the GNOME menu more robust, again Resolved

100
Bug #8876: "the I2P router console is ready" step can break due to too small timeout Resolved

100
Bug #8941: The remote shell server is racing against Tails Greeter Resolved

100
Bug #8009: The Unsafe Browser tests sometimes fail due to too fast keyboard interaction Resolved

100
Bug #8950: Test suite fails if features/misc_files/* or parent directories have strict permissions Resolved

100
Bug #8928: "unclutter" can cause spurious errors with some configurations Resolved

100
Bug #9072: Pidgin IRC tests often fail due to OFTC Tor blocking Resolved

100
Bug #8907: storage_helper.rb does not check available disk space Resolved

100
Bug #9095: Seahorse tests lack robustness Resolved

100
Bug #9139: Test suite stalled, lots of virt-viewer defunct processes Resolved

100
Bug #9140: ISO selection in test suite "Upgrade from ISO" is fragile Rejected

0
Bug #9317: Pidgin IRC test can fail with buddy list window over conversation one Resolved

100
Bug #9329: fillram sometimes doesn't complete Resolved

100
Bug #9330: Focusing windows during the test suite is fragile Resolved

100
Bug #9375: Pidgin XMPP test fails with 'No handlers could be found for logger "jabberbot"' Resolved

100
Feature #9425: Enable Spice with an absolute pointing device in the test suite guest Resolved

100
Bug #9478: How to deal with transient network errors in the test suite? Resolved

100
Feature #9515: Improve test suite robustness vs transient network errors Resolved

100
Bug #6092: Tails Installer sometimes hangs late in the installation process Resolved

100
Bug #9633: waitAny is failing to find images that are on the screen Resolved

100
Bug #10116: "Watching a WebM video" test is fragile Resolved

100
Bug #10219: Escape regexp used to match nick in CTCP replies Resolved

100

Related issues

Blocks Tails - Feature #5288: Run the test suite automatically on autobuilt ISOs Resolved 2014-01-03 2015-10-15

History

#1 Updated by intrigeri 2015-01-06 12:50:12

  • blocks #8538 added

#2 Updated by intrigeri 2015-01-06 12:50:41

  • blocks Feature #5288: Run the test suite automatically on autobuilt ISOs added

#3 Updated by anonym 2015-01-09 13:15:19

  • Target version changed from Hardening_M1 to Tails_1.5

#4 Updated by anonym 2015-01-10 17:42:52

  • Target version changed from Tails_1.5 to Tails_1.4

#5 Updated by anonym 2015-03-31 19:50:54

  • Assignee set to anonym

#6 Updated by kytv 2015-04-26 06:23:07

  • related to Bug #8686: Sometimes notification-daemon aborts, causing desktop notifications to not be displayed added

#7 Updated by kytv 2015-04-26 06:24:20

  • related to deleted (Bug #8686: Sometimes notification-daemon aborts, causing desktop notifications to not be displayed)

#8 Updated by intrigeri 2015-05-09 02:54:26

  • Target version changed from Tails_1.4 to Tails_1.4.1

Postponing. IIRC we didn’t really expect to complete all of the sub-tasks during this cycle, right?

Also, it’s become the fallback parent ticket for all kinds of test suite robustness bugs. I think we’ll need to find a better tool at some point to track this class of problems, but I think we can go on like that for a couple more cycles.

#9 Updated by anonym 2015-06-02 14:11:36

  • blocked by Feature #9515: Improve test suite robustness vs transient network errors added

#10 Updated by intrigeri 2015-06-04 21:24:26

  • blocks deleted (Feature #9515: Improve test suite robustness vs transient network errors)

#11 Updated by intrigeri 2015-06-29 00:47:22

  • Target version changed from Tails_1.4.1 to Tails_1.5

Postponing again.

#12 Updated by intrigeri 2015-07-19 07:34:57

  • Status changed from Confirmed to In Progress

#13 Updated by intrigeri 2015-07-19 07:36:11

  • Target version changed from Tails_1.5 to Tails_1.6

(The only remaining subtask is scheduled for 1.6.)

#14 Updated by intrigeri 2015-09-22 02:13:27

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)

intrigeri wrote:
> Also, it’s become the fallback parent ticket for all kinds of test suite robustness bugs. I think we’ll need to find a better tool at some point to track this class of problems, but I think we can go on like that for a couple more cycles.

I think it’s time to kill this ticket. Whatever is too fragile in Jenkins should now be filed with priority = high (at least for a couple cycles IMO), marked as blocking #8668, with an assignee and a sensible target version. If we later realize we need a way to list all such issues, we’ll see (I suspect that “Blocks” will be a more appropriate relationship than parenthood).

#15 Updated by bertagaz 2015-09-22 15:12:27

  • Status changed from Fix committed to Resolved

Killing this ticket then. Let’s use other means to go on with this.