Bug #17636
tails-iuk: test suite usefulness/expectations
Start date:
Due date:
% Done:
0%
Description
We’ve had a little chat lately about the tails-iuk test suite, I’m quoting parts of the discussion between intrigeri and me (as usual, I’m asking questions, intrigeri has answers):
>> The way I see it, running this test suite is primarily useful:
>> - When a developer works on the code base, to iterate quickly before
>> pushing to CI (I do mostly TDD on our Perl code bases; this
>> requires a short feedback loop).
>>
>> Given I work on sid, CI would help me spot regressions
>> that happen in a Tails-like environment, but not on my system.
>>
>> And for other people, what you're saying is very true: for example,
>> if *you* can't reliably run the test suite, you cannot work on this
>> code base with confidence. That's a problem.
>>
>> - When a reviewer looks at a submitted branch.
>> - When a RM wants to build confidence in the state of the code
>> they're going to publish.
>>
>> I think server-side CI, run in a Tails-like environment, would be
>> sufficient. That's what we're doing for other dev work, where we
>> don't expect reviewers and RMs to *also* run the CI locally.
>> Fair enough?
>
> In this specific case, what would a Tails-like environment mean? A
> Buster+ system?
Ideally it would be run inside a running Tails system, possibly as
part of our test suite. And if that's cheaper, IMO it's good enough to
set the bar at the same level as Debian sbuild/autopkgtest, i.e. "a
clean system running the same Debian release as current Tails, with
the needed dependencies installed".
I'd rather not run that server-side CI in an environment that runs
a newer Debian than Tails: I've already seen cases when the code that
worked on my sid would break in Debian stable.
> Maybe with a hint somewhere based on /etc/os-release or lsb_release
> warning developers/RMs when they run it on an older system? In this
> case, if ID=debian and VERSION_ID is << 10?
Sounds good to me!
Subtasks
Related issues
Related to Tails - Feature #16209: Core work: Foundations Team | Confirmed | ||
Related to Tails - Feature #6218: Run our custom programs test suite on interesting commits | Confirmed | 2013-08-07 |
History
#1 Updated by CyrilBrulebois 2020-04-17 23:42:16
- related to Feature #16209: Core work: Foundations Team added
#2 Updated by CyrilBrulebois 2020-04-17 23:43:45
Even if it’s a bit dated, Feature #6218 is related.
#3 Updated by CyrilBrulebois 2020-04-17 23:43:56
- related to Feature #6218: Run our custom programs test suite on interesting commits added