Tails

#9 Updated by intrigeri 2019-04-07 08:47:49

A discussion with @segfault convinced me that at least some of the components that currently live in their own Git repo, and that we include as custom Debian packages in the images we build, should instead live in our main tails.git repo. For example, there’s very little value in having Tails Greeter in its own repo, while it does significantly increase friction and overhead when working on its codebase.

#12 Updated by intrigeri 2019-04-28 08:12:12

segfault wrote on Feature #16678:
> We maintain some Debian packages which are solely used in Tails and cannot reasonably used outside of Tails. IMO, maintaining these as Debian packages makes our lives unnecessarily harder, because we have to release, build, and upload a new Debian package every time we change something in these packages.
>
> We also share code between the main repo and some packages, which makes it hard to find usages/implementations.
>
> The two packages I currently have in mind are tails-greeter and tails-persistence-setup.
>
> I propose that we integrate them into the main git repo by copying the files that are installed by the package into config/chroot_local_includes.

Fully agreed in principle for these 2 repos. Potential difficulties:

• tails-persistence-setup might require special care to keep its own test suite working
• PO/POT files management: currently these are translated under their own Transifex resources; I don’t know if/how we can preserve existing translations across this migration

I think WhisperBack and Tails Installer would be good candidates for this first batch too.

Some other candidates (tails-iuk, tails-perl5lib) might be a bit less simple since IIRC RMs need to install the resulting packages on their system, or similar. Let’s do some easier ones first :)

Next step: try this for one repo and see how it goes wrt. POT/PO files.

#15 Updated by segfault 2019-07-21 18:59:24

I would like to start with the greeter, and then move pythonlib to config/chroot_local_includes.

#23 Updated by intrigeri 2019-10-14 13:26:37

intrigeri wrote:
> Some other candidates (tails-iuk, tails-perl5lib) might be a bit less simple since IIRC RMs need to install the resulting packages on their system, or similar.

I thought about this again while working in these 2 repositories yesterday and remembering how painful it is to validate such work in the context of a real Tails system, anticipating on how my work on Feature #15281 would be easier if we integrated these code bases in a different manner.

So I’ve looked into this problem a bit and our 3 Perl codebases (perl5lib, iuk, persistence) are in similar situations:

• The main added complexity in the upstream and Debian build systems comes from the localization stuff, which we would simply get rid of if we moved these files to tails.git: we already have all the bits in place to manage Perl + Locale::gettext there. So we lose nothing on this front, yeah.
• Regarding automated tests run by developers, reviewers, RMs, and/or some kind of CI, this is the most unclear part for me at the moment. It depends on when/where we want to run these tests.
  • Dist::Zilla generates some static analysis tests automatically: code & POD formatting. Perhaps we can drop a dist.ini file at the root of tails.git and configure/tweak things so that we don’t have to reinvent this wheel or to switch to a totally different system.
  • On top of the static code analysis mentioned above, 2 of these 3 codebases have quite extensive test suites (rspec and cucumber -style). We currently run them whenever we prepare new releases of these codebases. Some of the tests need to do privileged operations which might be tricky in container environments (thinking of GitLab CI). They might occasionally leave the system in a dirty state when something goes really wrong. This set of constraints looks suspiciously like the Tails test suite’s ones so dedicated tester VMs like we have on Jenkins could fit the bill. But ideally we want to run these tests on the same version of Debian as the one which the target Tails system is based on (we currently get part of that via pbuilder/sbuild, for the tests that can run in there); as it happens, currently our isotesters run Stretch but we target Buster; this has happened in the past and will happen again. This seems to call for a Vagrant-like system to run these test suites in a suitable environment.
  • For the first iteration, IMO we should only aim to provide means for developers, reviewers, and RMs to run these tests locally, sometimes with different versions of the dependencies than what ends up in Tails, just like they’ve been doing so far. Running this in CI can come later.
• Regarding RMs and dependency tracking, there are two things:
  • In our release process we simply point the PERL5LIB environment variable to wherever the RM has cloned this repo, so if we move these files to tails.git, we can get make this a constant; great.
  • RMs run code that lives in the tails-iuk package, which currently depends on the tails-perl5lib package, so they get the dependencies they need via APT, because we currently maintain the list of these dependencies in debian/control. If we move perl5lib to tails.git, then we need a different way for RMs to know which dependencies they need to install. The option I’m thinking of at the moment is to list these dependencies in a new config/chroot_local-packageslists/tails-perl5lib.list file. Compared to the current debian/control and dist.ini dependency management:
    • We lose the ability to declare versioned dependencies, but I’m willing to take the risk.
    • We lose the ability to declare dependencies that are only needed to run the tests. I can think of various solutions already, depending on where/when we want to run the tests, so I’m not too worried about this.

While working on Feature #15281, if I get too annoyed by the current state of things, I might give a try to the cheapest possible way to do this migration, and we’ll see how it goes.

#24 Updated by segfault 2019-10-15 13:45:15

intrigeri wrote:
> On top of the static code analysis mentioned above, 2 of these 3 codebases have quite extensive test suites (rspec and cucumber -style). We currently run them whenever we prepare new releases of these codebases. Some of the tests need to do privileged operations which might be tricky in container environments (thinking of GitLab CI).

GitLab CI also supports VM executors. VirtualBox is supported natively [1], and using libvirt is possible via the custom executor [2] (but I never used the latter, so I don’t know how well that works).

[1] https://docs.gitlab.com/runner/executors/virtualbox.html
[2] https://docs.gitlab.com/runner/executors/custom_examples/libvirt.html

> While working on Feature #15281, if I get too annoyed by the current state of things, I might give a try to the cheapest possible way to do this migration, and we’ll see how it goes.

Cool!