Feature #16935

Move tailslib to main repo

Added by segfault 2019-08-04 18:57:56 . Updated 2019-11-08 18:00:37 .

Status:
Resolved
Priority:
Normal
Assignee:
intrigeri
Category:
Target version:
Tails_4.1
Start date:
Due date:
% Done:

0%

Feature Branch:
feature/16935-move-tailslib-to-main-repo+force-all-tests, puppet-tails:feature/16935-move-tailslib-to-main-repo
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Subtasks

Related issues

Blocks Tails - Feature #16664: Simplify "Tor has bootstrapped" status check Resolved
Blocks Tails - Feature #17048: Make the tails::pythonlib Puppet class track the stable branch of tails.git Resolved
Blocks Tails - Feature #13284: Core work: Sysadmin (Adapt our infrastructure) Confirmed 2017-06-30

History

#1 Updated by intrigeri 2019-08-04 19:11:01

If this is about pythonlib.git: it has other consumers than tails.git. I can think of one (our Jenkins jobs generator). I don’t know by heart if there are more.

#2 Updated by intrigeri 2019-08-04 19:11:34

#3 Updated by segfault 2019-08-04 19:20:53

  • Parent task deleted (Feature #7036)
  • Feature Branch set to feature/16935-move-tailslib-to-main-repo+force-all-tests

This seems to work. A few notes though:

  • The feature branch is based on feature/16912-move-greeter-to-main-git-repo+force-all-tests
  • Moving tailslib to config/chroot_local_includes/usr/local/lib/python3/ caused the build to fail with this error:
P: Begin building root filesystem image...
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  squashfs-tools
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 125 kB of archives.
After this operation, 305 kB of additional disk space will be used.
Get:1 http://deb.tails.boum.org stable/main amd64 squashfs-tools amd64 1:4.3-3.0tails4 [125 kB]
Traceback (most recent call last):
  File "/usr/local/sbin/tails-additional-software", line 16, in <module>
    from tailslib import LIVE_USERNAME
ImportError: No module named 'tailslib'

I moved it to config/chroot_local_includes/usr/lib/python3/ instead, which works fine. The cause of the error is probably that /usr/local/lib/python3/dist-packages is a non-standard directory for Python packages, and config/chroot_local-includes/etc/environment, where we add this directory to the PYTHONPATH, wasn’t sourced at the point where tails-additional-software is executed (but I don’t know why tails-additional-software is executed during build at all). We had issues with the non-standard Python directory in the past (Bug #15732) and I don’t really see why we prefer it over /usr/lib/python3. I’d suggest we just move all our Python packages to /usr/lib/python3/dist-packages.

  • I don’t know where the jenkins.py and git.py modules are being used. Maybe those should be extracted into a separate repository or, if they are only being used in one repo, be moved to that repo.

#4 Updated by segfault 2019-08-04 19:21:48

  • Status changed from Confirmed to In Progress

Applied in changeset commit:tails|31e7bee18caa205c04df396c208ffe877217e55e.

#5 Updated by segfault 2019-08-04 19:23:01

#6 Updated by segfault 2019-08-04 19:25:47

intrigeri wrote:
> If this is about pythonlib.git: it has other consumers than tails.git. I can think of one (our Jenkins jobs generator).

Yeah, I thought so, because of the jenkins.py and git.py. So those modules should probably be moved there.

> I don’t know by heart if there are more.

The greeter also uses it, that’s why this branch is based on the one for Feature #16912. I couldn’t find any other repos which use it.

#7 Updated by intrigeri 2019-09-01 14:41:35

>> pythonlib.git: it has other consumers than tails.git. I can think of one (our Jenkins jobs generator).

> Yeah, I thought so, because of the jenkins.py and git.py. So those modules should probably be moved there.

I think that moving these 2 modules elsewhere than to tails.git would cause too much trouble:

  • git.py is used by tails.git:bin/delete-merged-git-branches so we can’t simply move it elsewhere. We could move bin/delete-merged-git-branches to that same place but it makes the RM’s jobs more complex and error-prone, which I’d rather avoid for destructive operations.
  • The main class in jenkins.py inherits from the one from git.py, so having these 2 modules in different repos would make some operations needlessly complex. So if we move git.py to tails.git, IMO we should move jenkins.py there as well.

> I don’t know by heart if there are more.

I’ve checked in our Puppet codebase and the only user of tailslib in there is our Jenkins jobs generator. It could use the library from its own clone of tails.git just like it’s currently using it from a clone of pythonlib.git.

⇒ IMO, the way to go is to move git.py and jenkins.py to tails.git, along with the rest of pythonlib.

(Another option would be to keep these two files in the pythonlib repo and leave it as a submodule of tails.git, but then we lose some of the benefits of what this ticket is about, which would be a bit sad.)

What do you think?

#8 Updated by segfault 2019-09-12 16:38:44

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (segfault)

> ⇒ IMO, the way to go is to move git.py and jenkins.py to tails.git, along with the rest of pythonlib.

Great! Then this is ready for review now.

#9 Updated by intrigeri 2019-09-12 16:50:44

  • Assignee set to intrigeri

#10 Updated by intrigeri 2019-09-12 16:51:06

  • Target version set to Tails_4.0

#11 Updated by intrigeri 2019-09-13 05:00:46

Code review passes. FTR I’ve verified that the imported code is identical to the one in the submodule (cd812663fc47ca2c93257ae8968917282c419089).

I’ll now build + run the test suite (locally, without waiting for our shared Jenkins to be back up).

Regarding how to coordinate the merge in tails.git with our Puppet stuff, my plan is:

  1. Merge this branch into devel.
  2. Adjust Puppet code to clone tails.git’s devel branch and use the library from there.
  3. File a ticket to make that Puppet code instead use the stable branch, once 4.0 is out (I initially thought we would use the master branch there, but it’s often awkward when we have to backport code changes to master for similar reasons).

#12 Updated by intrigeri 2019-09-13 05:01:11

  • blocks Feature #16664: Simplify "Tor has bootstrapped" status check added

#13 Updated by intrigeri 2019-09-13 05:34:06

  • Status changed from Needs Validation to In Progress

Applied in changeset commit:tails|aeb2f6292259589bb035920cb80a25b1a08f16ec.

#14 Updated by intrigeri 2019-09-13 05:39:42

  • Status changed from In Progress to Needs Validation
  • Feature Branch changed from feature/16935-move-tailslib-to-main-repo+force-all-tests to feature/16935-move-tailslib-to-main-repo+force-all-tests, puppet-tails:feature/16935-move-tailslib-to-main-repo

#15 Updated by intrigeri 2019-09-13 05:57:55

  • blocks Feature #17048: Make the tails::pythonlib Puppet class track the stable branch of tails.git added

#16 Updated by intrigeri 2019-09-13 05:59:55

@segfault, I’ve pushed commit:aeb2f6292259589bb035920cb80a25b1a08f16ec on top. Can you please review it?

@zen, could you please review https://git.tails.boum.org/puppet-tails/log/?h=feature/16935-move-tailslib-to-main-repo? Things to consider:

  • Do not merge it: I need to coordinate the merge with another one (in tails.git) which is not ready yet.
  • Context: our pythonlib Git repo is being deprecated so our infra code that uses libraries that lived there now needs to instead use these libraries from tails.git.
  • I have a doubt about environment => "PYTHONPATH='${tails::pythonlib::pythonpath}'", ← not sure I can access the $pythonpath variable from tails::pythonlib here; worst case, I’ll move it to a class parameter and that will work (see e.g. how we reuse tails::borgbackup::borgpassphrase in tails::borgbackup::{fs,lv}).
  • For now I’m tracking the devel branch of tails.git, since that’s where segfault’s branch will be merged into initially. As part of the Tails 4.0 release process, devel will be merged into stable, and then we should use the stable branch of tails.git in tails::pythonlib → tracked by Feature #17048.

Thanks in advance!

#17 Updated by intrigeri 2019-09-13 12:03:17

  • Assignee changed from intrigeri to segfault

@segfault, the full test suite passed locally (except Bug #17007) so feel free to merge into devel if you’re happy with my extra commit, then reassign for review to zen for the sysadmin part. Thanks!

#18 Updated by segfault 2019-09-15 15:19:51

  • Status changed from Needs Validation to Resolved
  • % Done changed from 0 to 100

Applied in changeset commit:tails|dfb9f49dea723262a9d1722cec4969eea9fda447.

#19 Updated by segfault 2019-09-15 15:20:52

  • Status changed from Resolved to Needs Validation
  • Assignee changed from segfault to zen
  • % Done changed from 100 to 0

#20 Updated by intrigeri 2019-09-19 06:09:23

  • blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#21 Updated by intrigeri 2019-10-21 11:46:09

  • Target version changed from Tails_4.0 to Tails_4.1

#22 Updated by intrigeri 2019-10-31 10:51:50

zen, test: you should receive two notifications about this (one because you're assigned to this ticket, one because of the \ mention.

#23 Updated by zen 2019-11-02 21:06:03

  • Status changed from Needs Validation to In Progress
  • Assignee changed from zen to intrigeri

intrigeri wrote:
> @zen, could you please review https://git.tails.boum.org/puppet-tails/log/?h=feature/16935-move-tailslib-to-main-repo? Things to consider:

Looks good, one minor (maybe negligible) comment: `tails.git` makes me think of a bare repository, maybe call it just `git`?

All the rest looks fine.

#24 Updated by zen 2019-11-02 21:07:30

zen wrote:
> Looks good, one minor (maybe negligible) comment: `tails.git` makes me think of a bare repository, maybe call it just `git`?

oops, i meant “just tails”.

#25 Updated by intrigeri 2019-11-08 18:00:37

  • Status changed from In Progress to Resolved

> Looks good, one minor (maybe negligible) comment: `tails.git` makes me think of a bare repository, maybe call it just `git`?
> oops, i meant “just tails”.

Good idea, done!

I’ve merged this branch, deployed, added a brown paper bag fix on top.
The Jenkins jobs generator still works ⇒ closing.