Bug #17636

tails-iuk: test suite usefulness/expectations

Added by CyrilBrulebois 2020-04-17 23:41:06 . Updated 2020-04-17 23:43:45 .

Status:
Confirmed
Priority:
Normal
Assignee:
Category:
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

We’ve had a little chat lately about the tails-iuk test suite, I’m quoting parts of the discussion between intrigeri and me (as usual, I’m asking questions, intrigeri has answers):

>> The way I see it, running this test suite is primarily useful:
>>  - When a developer works on the code base, to iterate quickly before
>>    pushing to CI (I do mostly TDD on our Perl code bases; this
>>    requires a short feedback loop).
>>
>>    Given I work on sid, CI would help me spot regressions
>>    that happen in a Tails-like environment, but not on my system.
>>
>>    And for other people, what you're saying is very true: for example,
>>    if *you* can't reliably run the test suite, you cannot work on this
>>    code base with confidence. That's a problem.
>>
>>  - When a reviewer looks at a submitted branch.
>>  - When a RM wants to build confidence in the state of the code
>>    they're going to publish.
>>
>>    I think server-side CI, run in a Tails-like environment, would be
>>    sufficient. That's what we're doing for other dev work, where we
>>    don't expect reviewers and RMs to *also* run the CI locally.
>>    Fair enough?
>
> In this specific case, what would a Tails-like environment mean? A
> Buster+ system?

Ideally it would be run inside a running Tails system, possibly as
part of our test suite. And if that's cheaper, IMO it's good enough to
set the bar at the same level as Debian sbuild/autopkgtest, i.e. "a
clean system running the same Debian release as current Tails, with
the needed dependencies installed".

I'd rather not run that server-side CI in an environment that runs
a newer Debian than Tails: I've already seen cases when the code that
worked on my sid would break in Debian stable.

> Maybe with a hint somewhere based on /etc/os-release or lsb_release
> warning developers/RMs when they run it on an older system? In this
> case, if ID=debian and VERSION_ID is << 10?

Sounds good to me!

Subtasks


Related issues

Related to Tails - Feature #16209: Core work: Foundations Team Confirmed
Related to Tails - Feature #6218: Run our custom programs test suite on interesting commits Confirmed 2013-08-07

History

#1 Updated by CyrilBrulebois 2020-04-17 23:42:16

#2 Updated by CyrilBrulebois 2020-04-17 23:43:45

Even if it’s a bit dated, Feature #6218 is related.

#3 Updated by CyrilBrulebois 2020-04-17 23:43:56

  • related to Feature #6218: Run our custom programs test suite on interesting commits added