Bug #11093

Deal with December 2016 false positive scenarios

Added by bertagaz 2016-02-09 11:08:14 . Updated 2017-03-05 16:20:24 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Start date:
2016-12-05
Due date:
2017-01-05
% Done:

100%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
0
Affected tool:
Deliverable for:

Description

As defined in Bug #10993, check is there are still some of them and tag them as fragile if any (see Bug #10288).

Also, this is our last ticket of this kind, so part of it will be to decide what we’re going to do afterwards.


Subtasks


History

#1 Updated by bertagaz 2016-02-09 11:10:29

  • Target version set to 284

#2 Updated by intrigeri 2016-02-09 18:36:35

  • Subject changed from Deal with December false positive scenarios to Deal with December 2016 false positive scenarios
  • Deliverable for set to 270

#3 Updated by intrigeri 2016-02-09 18:39:48

  • Deliverable for deleted (270)

#4 Updated by intrigeri 2016-10-31 08:47:07

  • Description updated

#5 Updated by anonym 2016-11-25 10:57:06

  • Target version changed from 284 to Tails 2.10

#6 Updated by anonym 2017-01-11 19:08:11

  • Assignee changed from anonym to intrigeri
  • % Done changed from 0 to 50
  • QA Check changed from Dev Needed to Info Needed

Done!

New tickets:

  • Bug #12131: Step ‘I double-click the Report an Error launcher on the desktop’ sometimes fails
  • Bug #12132: Step ‘I shutdown Tails and wait for the computer to power off’ sometimes fails by rebooting instead

Summary (including number of occurrences):

  • 1 Bug #11786 => As per Bug #10776, I’ll backport the fix we have in feature/stretch soon
  • 9 Bug #11816 => Bug in Tails, cannot do much in the test suite
  • 3 Bug #11872 => Bug in Tails, cannot do much in the test suite
  • 2 Bug #12009 => Fixed after these occurrences, so hopefully we can forget about it
  • 2 Bug #12041 => bug in Tails, cannot do much in the test suite. Well, I just posted an ugly solution to the ticket.
  • 1 Bug #12044 => cannot be marked @fragile due to disabling too many tests
  • 1 Bug #12045 => cannot be marked @fragile due to disabling too many tests
  • 1 Bug #12046
  • 2 Bug #12131
  • 1 Bug #12131 => cannot be marked @fragile due to disabling too many tests

As for @fragile tagging, I’m not sure what criteria we have these days. What do you think? Bug #12046 and Bug #12131 could be tagged @fragile if we want, but none of the others.

#7 Updated by intrigeri 2017-01-16 17:13:19

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Info Needed to Dev Needed

> Done!

Great job!

> * 2 Bug #12009 => Fixed after these occurrences, so hopefully we can forget about it

I don’t understand what Bug #12009 has to do with the test suite. Care to clarify?

> As for @fragile tagging, I’m not sure what criteria we have these days. What do you think? Bug #12046 and Bug #12131 could be tagged @fragile if we want, but none of the others.

Wrt. Bug #12046: on the test suite side, we both have seen it during these triaging sessions, so in theory it should be marked as fragile. But I’d like to see it primarily handled as a bug in Icedove, as I doubt it’s specific to our test suite. The test suite can be a great way to gather debugging info that could be useful to report the bug upstream, and I doubt we’ll see the bug often enough to do so, if we disable the test everywhere but on a dedicated branch. So I say it’s your call: if you think you can handle this (as a follow-up on the Icedove work, and/or with Foundations Team hat), the make the test gather debugging info and leave it enabled; if your plate is too full already, mark it as fragile for now.

Wrt. Bug #12131: I’ve seen it a lot on Stretch ISO images, and am actually happy to see it’s not a regression. Seems like a clear candidate for @fragile to me.

#8 Updated by anonym 2017-01-24 20:48:49

  • Target version changed from Tails 2.10 to Tails_2.11

#9 Updated by anonym 2017-02-20 15:02:04

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Dev Needed to Info Needed

intrigeri wrote:
> > * 2 Bug #12009 => Fixed after these occurrences, so hopefully we can forget about it
>
> I don’t understand what Bug #12009 has to do with the test suite. Care to clarify?

Err, I must have skipped reading the full ticket subject, and thought this was about the iso*builders* being very unreliable.

> > As for @fragile tagging, I’m not sure what criteria we have these days. What do you think? Bug #12046 and Bug #12131 could be tagged @fragile if we want, but none of the others.
>
> Wrt. Bug #12046: on the test suite side, we both have seen it during these triaging sessions, so in theory it should be marked as fragile. But I’d like to see it primarily handled as a bug in Icedove, as I doubt it’s specific to our test suite. The test suite can be a great way to gather debugging info that could be useful to report the bug upstream, and I doubt we’ll see the bug often enough to do so, if we disable the test everywhere but on a dedicated branch. So I say it’s your call: if you think you can handle this (as a follow-up on the Icedove work, and/or with Foundations Team hat), the make the test gather debugging info and leave it enabled; if your plate is too full already, mark it as fragile for now.

I have tried to follow the debugging of this problem on Debian BTS and it definitely seem like something more time-consuming than I can commit to. OTOH it’s not clear to me what sort of debugging help you have in mind. Care to clarify (if you have some general idea — I don’t want you to waste time looking into this particular issue)? <— Info Needed

> Wrt. Bug #12131: I’ve seen it a lot on Stretch ISO images, and am actually happy to see it’s not a regression. Seems like a clear candidate for @fragile to me.

I went for a fix instead. I mean, I’ve let this issue pollute our results for months now, so what’s another few days/weeks…

#10 Updated by intrigeri 2017-03-02 08:14:24

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Info Needed to Dev Needed

>> Wrt. Bug #12046: […]

> I have tried to follow the debugging of this problem on Debian BTS and it definitely seem like something more time-consuming than I can commit to. OTOH it’s not clear to me what sort of debugging help you have in mind. Care to clarify (if you have some general idea — I don’t want you to waste time looking into this particular issue)? <— Info Needed

I had nothing specific steps in mind, except: whatever could help upstream / Debian fix the root cause. Identifying what debugging is required for it would be part of the work that neither you nor I are willing to do, so let’s forget it.

>> Wrt. Bug #12131: I’ve seen it a lot on Stretch ISO images, and am actually happy to see it’s not a regression. Seems like a clear candidate for @fragile to me.

> I went for a fix instead. I mean, I’ve let this issue pollute our results for months now, so what’s another few days/weeks…

\o/

Reassigning to you for the “Also, this is our last ticket of this kind, so part of it will be to decide what we’re going to do afterwards” part of this ticket. IIRC we’ve discussed this during the last CI meeting, but I don’t remember the conclusions (except “it’s useful and we should keep doing it”), and I dunno if these conclusions have been implemented.

#11 Updated by anonym 2017-03-05 16:20:24

  • Status changed from Confirmed to Resolved
  • Assignee deleted (anonym)
  • % Done changed from 50 to 100
  • QA Check changed from Dev Needed to Pass

intrigeri wrote:
> Reassigning to you for the “Also, this is our last ticket of this kind, so part of it will be to decide what we’re going to do afterwards” part of this ticket. IIRC we’ve discussed this during the last CI meeting, but I don’t remember the conclusions (except “it’s useful and we should keep doing it”), and I dunno if these conclusions have been implemented.

During the March CI meeting we decided to keep doing this, and tickets have been created: Feature #12284 - Feature #12295.