Bug #12683
Move DAVE manual test suite out of the Tails release process
0%
Description
In Feature #11799 we’ve added DAVE tests to https://tails.boum.org/contribute/release_process/test/. That’s a great start! But, there’s a but:
- Most of these tests are orthogonal to the Tails release process, so it’s suboptimal to have on the critical path to a Tails release. The fact that some of these tests seem too unclear, at least for a few of us, doesn’t help (
Bug #12005). - The goal of
Feature #11799was “to detect problems with new Firefox versions early enough”, but the testing instructions merely require “a Firefox-based browser”; so if the tests are performed in the ISO under testing, i.e. with an oldish Firefox ESR, then this testing process is not addressing the initial goal; and if the tests are performed elsewhere, with a recent Firefox, then we’re not verifying that DAVE works well in the upcoming version of Tails, although that would be a useful additional goal of running these tests during the Tails release process.
So I propose:
- Keep the simplest test in the Tails manual test suite: download, pause/resume, appears in the downloads list.
- Move all other tests to a dedicated DAVE test suite, that shall be run (probably by the DAVE maintainer, but suggestions are welcome):
- at DAVE release time;
- whenever a new Firefox is released.
- Consider automating these tests before we get bored and sloppy and start secretly skipping the “whenever a new Firefox is released” tests ;)
But I might have missed some reasons why things currently are done like this, so please correct my assumptions/reasoning if needed :)
Subtasks
Related issues
Related to Tails - |
Resolved | 2016-11-30 | |
Related to Tails - |
Resolved | 2017-06-12 |
History
#1 Updated by intrigeri 2017-06-12 08:02:58
- related to
Bug #12005: Problems with DAVE manual tests added
#2 Updated by intrigeri 2017-09-07 08:23:35
- Priority changed from Normal to Elevated
- Target version changed from Tails_3.2 to Tails_3.3
I suggest you ignore DAVE until the 3.2 release and then put it on your list of top priorities for the 3.3 cycle.
#3 Updated by intrigeri 2017-11-06 16:04:54
- Target version changed from Tails_3.3 to Tails_3.5
Let’s wait for DAVE v2 first.
#4 Updated by intrigeri 2017-11-18 10:53:43
- Affected tool changed from Download and Verification Extension to Verification Extension
#5 Updated by anonym 2017-12-14 14:31:36
- Assignee changed from anonym to sajolida
#6 Updated by sajolida 2018-01-15 16:08:02
- Parent task set to
Bug #12328
#7 Updated by sajolida 2018-01-17 18:03:02
- Parent task changed from
Bug #12328toFeature #15188
intrigeri wrote:
> In Feature #11799 we’ve added DAVE tests to https://tails.boum.org/contribute/release_process/test/. That’s a great start! But, there’s a but:
>
> * Most of these tests are orthogonal to the Tails release process, so it’s suboptimal to have on the critical path to a Tails release. The fact that some of these tests seem too unclear, at least for a few of us, doesn’t help (Bug #12005).
> * The goal of Feature #11799 was “to detect problems with new Firefox versions early enough”, but the testing instructions merely require “a Firefox-based browser”; so if the tests are performed in the ISO under testing, i.e. with an oldish Firefox ESR, then this testing process is not addressing the initial goal; and if the tests are performed elsewhere, with a recent Firefox, then we’re not verifying that DAVE works well in the upcoming version of Tails, although that would be a useful additional goal of running these tests during the Tails release process.
>
> So I propose:
>
> # Keep the simplest test in the Tails manual test suite: download, pause/resume, appears in the downloads list.
> # Move all other tests to a dedicated DAVE test suite, that shall be run (probably by the DAVE maintainer, but suggestions are welcome):
> ## at DAVE release time;
> ## whenever a new Firefox is released.
> # Consider automating these tests before we get bored and sloppy and start secretly skipping the “whenever a new Firefox is released” tests ;)
>
> But I might have missed some reasons why things currently are done like this, so please correct my assumptions/reasoning if needed :)
#8 Updated by sajolida 2018-01-17 18:27:46
- Status changed from Confirmed to Resolved
- Assignee deleted (
sajolida) - Parent task changed from
Feature #15188toBug #12328
The DAVE manual test suite was moved out of the Tails release process with 238cb3e008.
See you on Feature #15188!
#9 Updated by sajolida 2018-01-17 18:27:59
- related to
Feature #15188: Write manual tests for Tails Verification added