Feature #5288

Run the test suite automatically on autobuilt ISOs

Added by Tails 2013-07-18 07:38:37 . Updated 2018-04-08 19:27:46 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Tails_2.3
Start date:
2014-01-03
Due date:
2015-10-15
% Done:

96%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
0
Affected tool:
Deliverable for:
267

Description

Running the test suite automatically on the ISO built by autobuild would improve the quality of our development branches and overall robustness.

Subtasks

Feature #6564: Deploy a platform for automated testing Resolved

90
Feature #6565: Run the test suite automatically on Git push Duplicate

0
Feature #7631: Get a server able to run our automated test suite Resolved

100
Feature #8667: Lead a discussion to specify what automatically build ISOs needs to be tested, when and how Resolved

100
Feature #9486: Support running multiple instances of the test suite in parallel Resolved

100
Feature #9597: Set up a way to share artifacts between jobs Resolved

100
Bug #9645: Create at least a second VM for testing ISO images Resolved

100
Feature #10117: Design how to run our test suite in Jenkins Resolved

100
Feature #10118: Write library code that maps Jenkins jobs from building to testing Resolved

100
Feature #10119: Adapt the job generation script to also output corresponding test jobs Resolved

100
Feature #10158: Mount /dev/vdc in /var/lib/jenkins on isotester1 Resolved

100
Bug #10229: ISO testing jobs seem to lack a timeout Resolved

100
Feature #10287: Set up limited email notification on automatic test failure for the initial deployment Resolved

100
Feature #10296: Reevaluate how the email notifications for failed automatic tests scale Resolved

100
Feature #10382: Implement the specified notification system for test suite failures on Jenkins Resolved

100
Bug #10396: Sort out overallocated storage situation on isotesterN.lizard Resolved

100
Bug #10409: Reintroduce the old_iso logic for release branches Resolved intrigeri

100
Feature #10445: Pass --artifacts-base-uri in Jenkins test suite wrapper Resolved

100
Feature #10507: Document how to get the statistics we need out of Jenkins Resolved

100
Bug #10601: isotesterN:s are sometimes put offline and never back online Needs Validation zen

10
Bug #10717: Concerning amount of test suite runs aborted on Jenkins due to timeout Rejected

10
Bug #10725: Test jobs sometimes hang at cucumber report parsing step Rejected

0
Feature #10863: Collect & analyze false positives in Jenkins test suite runs (late December 2015 to late January 2016) Resolved

100
Bug #10993: Define and bootstrap a process to collect & analyze false positives in Jenkins test suite runs Resolved

100

Related issues

Related to Tails - Feature #6090: Automated builds Resolved 2013-07-26 2015-02-28
Blocked by Tails - Bug #7060: Fix automated test suite's "I run ..." step Resolved 2014-04-10
Blocked by Tails - Feature #8539: Make the test suite robust enough to be run as part of a CI setup Resolved 2014-10-05 2015-03-06

History

#1 Updated by intrigeri 2013-07-20 02:10:24

  • Category set to Continuous Integration
  • Starter set to No

#2 Updated by bertagaz 2013-07-23 02:03:07

  • Assignee set to bertagaz

#3 Updated by intrigeri 2013-08-07 12:48:55

  • Subject changed from run the test suite on autobuild ISOs to run the test suite on autobuilt ISOs

#4 Updated by intrigeri 2014-01-03 08:47:44

  • Subject changed from run the test suite on autobuilt ISOs to Run the test suite on autobuilt ISOs

#5 Updated by intrigeri 2014-01-03 08:58:55

  • Subject changed from Run the test suite on autobuilt ISOs to Run the test suite automatically on autobuilt ISOs

#6 Updated by intrigeri 2014-08-02 16:38:25

#7 Updated by intrigeri 2014-08-02 16:43:49

  • blocked by Bug #6092: Tails Installer sometimes hangs late in the installation process added

#8 Updated by intrigeri 2014-08-02 16:44:06

  • blocks Bug #7060: Fix automated test suite's "I run ..." step added

#9 Updated by intrigeri 2014-08-08 21:14:33

  • blocked by deleted (Bug #7060: Fix automated test suite's "I run ..." step)

#10 Updated by intrigeri 2014-08-08 21:14:47

  • blocked by Bug #7060: Fix automated test suite's "I run ..." step added

#11 Updated by intrigeri 2014-10-05 11:20:05

  • blocked by Bug #8009: The Unsafe Browser tests sometimes fail due to too fast keyboard interaction added

#12 Updated by intrigeri 2014-10-11 03:02:19

  • blocked by Bug #8059: Windows Camouflage automated tests sometimes failed due to differently ordered icons in the notification area added

#13 Updated by intrigeri 2014-10-16 03:11:10

  • blocked by Bug #8140: Running applications via the GNOME menu is fragile added

#14 Updated by intrigeri 2014-10-16 03:14:45

  • blocked by Feature #5330: Test suite: identify and document race conditions added

#15 Updated by intrigeri 2015-01-06 12:50:41

  • blocked by Feature #8539: Make the test suite robust enough to be run as part of a CI setup added

#16 Updated by intrigeri 2015-01-06 12:51:16

  • blocks deleted (Bug #6092: Tails Installer sometimes hangs late in the installation process)

#17 Updated by intrigeri 2015-01-06 12:52:16

  • blocks deleted (Bug #8009: The Unsafe Browser tests sometimes fail due to too fast keyboard interaction)

#18 Updated by intrigeri 2015-01-06 12:52:44

  • blocks deleted (Bug #8059: Windows Camouflage automated tests sometimes failed due to differently ordered icons in the notification area)

#19 Updated by intrigeri 2015-01-06 12:53:08

  • blocks deleted (Bug #8140: Running applications via the GNOME menu is fragile)

#20 Updated by intrigeri 2015-01-06 12:53:33

  • blocks deleted (Feature #5330: Test suite: identify and document race conditions)

#21 Updated by bertagaz 2015-01-10 06:46:52

  • Target version changed from Hardening_M1 to Tails_1.6

#22 Updated by intrigeri 2015-05-28 15:02:27

  • blocks #8668 added

#23 Updated by intrigeri 2015-06-11 14:00:12

  • blocks Feature #9264: Consider buying more server hardware to run our automated test suite added

#25 Updated by intrigeri 2015-08-26 05:54:06

  • Deliverable for set to 267

#30 Updated by bertagaz 2015-09-23 01:26:36

  • Target version changed from Tails_1.6 to Tails_1.7

#31 Updated by sajolida 2015-10-01 07:25:07

Note that this is due on October 15 which is actually before Tails 1.7.

#32 Updated by bertagaz 2015-11-03 02:27:11

  • Target version changed from Tails_1.7 to Tails_1.8

Postponing hopefully for the last time.

#33 Updated by intrigeri 2015-11-06 07:33:22

  • blocked by deleted (Feature #9264: Consider buying more server hardware to run our automated test suite)

#34 Updated by bertagaz 2015-12-15 03:39:20

  • Target version changed from Tails_1.8 to Tails_2.0

Postponing

#35 Updated by intrigeri 2015-12-19 11:02:02

  • Target version changed from Tails_2.0 to Tails_2.3

Please set a target version that means “all subtasks are fixed”. That won’t be 2.0 so postponing quite a bit already.

#36 Updated by intrigeri 2016-01-04 12:31:12

  • Status changed from Confirmed to In Progress

#37 Updated by intrigeri 2016-01-06 14:20:44

  • blocked by Bug #10601: isotesterN:s are sometimes put offline and never back online added

#38 Updated by intrigeri 2016-01-06 14:21:33

(Bug #10601 should be a subtask, but Redmine doesn’t want to hear about it.)

#39 Updated by intrigeri 2016-02-08 12:51:00

  • blocks deleted (Bug #10601: isotesterN:s are sometimes put offline and never back online)

#40 Updated by intrigeri 2016-04-06 12:51:10

  • Status changed from In Progress to Resolved
  • Assignee deleted (bertagaz)

All subtasks have been resolved.

#41 Updated by sajolida 2016-04-16 02:13:42

High five!