Bug #11092

Deal with November 2016 false positive scenarios

Added by bertagaz 2016-02-09 11:07:16 . Updated 2016-12-20 11:03:27 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Tails 2.10
Start date:
2016-11-05
Due date:
2016-12-05
% Done:

100%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
0
Affected tool:
Deliverable for:

Description

As defined in Bug #10993, check is there are still some of them and tag them as fragile if any (see Bug #10288)

Subtasks

Related issues

Blocks Tails - Bug #11892: Sometimes the remote shell doesn't start because of missing initial Space when modifying the kernel cmdline In Progress 2016-10-31
Blocks Tails - Bug #11409: Deal with the 'Dogtail: warning: application may be hanging' bug Resolved 2016-05-11

History

#1 Updated by bertagaz 2016-02-09 11:14:27

  • Target version set to Tails_2.9.1

#2 Updated by intrigeri 2016-02-09 18:39:29

  • Subject changed from Deal with November false positive scenarios to Deal with November 2016 false positive scenarios

#3 Updated by intrigeri 2016-12-05 20:11:10

  • blocked by Bug #11409: Deal with the 'Dogtail: warning: application may be hanging' bug added

#4 Updated by intrigeri 2016-12-05 20:23:55

  • blocks Bug #11892: Sometimes the remote shell doesn't start because of missing initial Space when modifying the kernel cmdline added

#5 Updated by anonym 2016-12-14 20:11:23

  • Target version changed from Tails_2.9.1 to Tails 2.10

#6 Updated by intrigeri 2016-12-18 10:00:36

  • Assignee changed from bertagaz to intrigeri

#7 Updated by intrigeri 2016-12-19 14:21:03

  • blocks deleted (Bug #11409: Deal with the 'Dogtail: warning: application may be hanging' bug)

#8 Updated by intrigeri 2016-12-19 14:21:11

  • blocks Bug #11409: Deal with the 'Dogtail: warning: application may be hanging' bug added

#9 Updated by intrigeri 2016-12-19 15:58:23

  • % Done changed from 0 to 40

Initial triaging for stable done, 2 failures left to investigate further.

#10 Updated by intrigeri 2016-12-19 17:31:51

  • % Done changed from 40 to 80

Initial triaging for devel done, a few failures left to investigate further.

#11 Updated by intrigeri 2016-12-19 19:20:17

  • % Done changed from 80 to 90

Finished triaging and creating tickets. Left to do: mark tests as fragile & send a summary that will help re-prioritizing test suite robustness work.

#12 Updated by intrigeri 2016-12-20 11:03:27

  • Status changed from Confirmed to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 90 to 100

This is about October + November.

  • Parts of the test suite (and sometimes large parts of it) have been broken a few times during weeks, which made the analysis very long and painful, not to mention that it decreases the value (for developers) of running tests. IMO it’s totally worth it to prioritize fixing temporary test suite breakage very high: we’ll have to fix them anyway, and the more we wait, the more work we’ll have to do to triage false positives, and the more value we lose from our test suite. Example problems: Bug #11960, Bug #11911, Bug #11906, Bug #11901, Bug #11874
  • Recurring issues:
    • the usual ones: Bug #11872, Bug #11816, Bug #11786; these are most likely bugs in Tails rather than in the test suite; some of them might be fixed for free in Stretch, we’ll see. I guess the Foundations Team should add them to their list of high-prio bugs to fix or workaround in some way.
    • Bug #11890 => marked as fragile
    • Bug #12045, not sure if it’s a bug in the test suite or in Tails Installer
    • Bug #11508 => raised priority
  • Tickets I had to create: Bug #12040, Bug #12041, Bug #12042, Bug #12043, Bug #12044, Bug #12045, Bug #12046 and Bug #12047. That’s a lot, but I doubt that they really are all new problems; likely we were lazy and didn’t create them during the last shifts.