tails-iuk: test suite usefulness/expectations
We’ve had a little chat lately about the tails-iuk test suite, I’m quoting parts of the discussion between intrigeri and me (as usual, I’m asking questions, intrigeri has answers):
>> The way I see it, running this test suite is primarily useful: >> - When a developer works on the code base, to iterate quickly before >> pushing to CI (I do mostly TDD on our Perl code bases; this >> requires a short feedback loop). >> >> Given I work on sid, CI would help me spot regressions >> that happen in a Tails-like environment, but not on my system. >> >> And for other people, what you're saying is very true: for example, >> if *you* can't reliably run the test suite, you cannot work on this >> code base with confidence. That's a problem. >> >> - When a reviewer looks at a submitted branch. >> - When a RM wants to build confidence in the state of the code >> they're going to publish. >> >> I think server-side CI, run in a Tails-like environment, would be >> sufficient. That's what we're doing for other dev work, where we >> don't expect reviewers and RMs to *also* run the CI locally. >> Fair enough? > > In this specific case, what would a Tails-like environment mean? A > Buster+ system? Ideally it would be run inside a running Tails system, possibly as part of our test suite. And if that's cheaper, IMO it's good enough to set the bar at the same level as Debian sbuild/autopkgtest, i.e. "a clean system running the same Debian release as current Tails, with the needed dependencies installed". I'd rather not run that server-side CI in an environment that runs a newer Debian than Tails: I've already seen cases when the code that worked on my sid would break in Debian stable. > Maybe with a hint somewhere based on /etc/os-release or lsb_release > warning developers/RMs when they run it on an older system? In this > case, if ID=debian and VERSION_ID is << 10? Sounds good to me!