Tails

#1 Updated by sajolida 2015-11-27 04:41:25

• Target version changed from 246 to Tails_2.0

#2 Updated by intrigeri 2015-12-17 05:41:38

• Assignee set to intrigeri

Either I’ll manage to do it by the end of the year, or it’ll be postponed to the last call for upgrades from Wheezy, I guess.

#3 Updated by intrigeri 2015-12-22 14:45:57

• Target version changed from Tails_2.0 to Tails_2.6

#4 Updated by intrigeri 2016-08-17 00:49:53

• Target version changed from Tails_2.6 to Tails_2.7

Let’s see if we manage to handle that in September. If we don’t, we’ll need to reschedule this to some later sysadmin sprint.

#5 Updated by intrigeri 2016-11-05 13:52:44

• Target version changed from Tails_2.7 to 284

Best case I’ll work on this around December 17-20.

#6 Updated by anonym 2016-11-25 10:57:11

• Target version changed from 284 to Tails 2.10

#7 Updated by intrigeri 2016-12-18 12:45:35

• Target version changed from Tails 2.10 to Tails_2.11

I had to take over a number of more urgent sysadmin tasks, so this’ll have to wait.

#8 Updated by intrigeri 2017-02-27 09:55:46

• Target version changed from Tails_2.11 to Tails_3.2

I don’t see myself working on this before 3.0 is out, and probably not during summer.

#9 Updated by intrigeri 2017-06-01 14:39:49

• Target version changed from Tails_3.2 to Tails_3.3

#10 Updated by intrigeri 2017-06-05 13:37:53

• Target version changed from Tails_3.3 to 2018

#11 Updated by intrigeri 2017-09-02 12:09:48

• Subject changed from Upgrade to Gitolite v3 to Upgrade to Gitolite v3 on puppet-git.lizard

#12 Updated by intrigeri 2017-09-02 12:10:10

• copied to Bug #14587: Remove Gitolite from buse added

#13 Updated by intrigeri 2017-09-02 12:10:35

• blocks Feature #13529: Upgrade puppet-git.lizard to Stretch added

#14 Updated by intrigeri 2017-09-02 12:12:26

• blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#15 Updated by intrigeri 2017-09-02 12:25:45

• Description updated

#16 Updated by intrigeri 2017-09-02 13:01:13

• blocked by deleted (Feature #13529: Upgrade puppet-git.lizard to Stretch)

#18 Updated by intrigeri 2017-12-18 16:49:49

• Assignee changed from intrigeri to groente

#20 Updated by groente 2018-11-01 16:43:41

• Assignee changed from groente to intrigeri
• QA Check set to Info Needed

a short local test has shown that dummy-versions of our hooks still work after updating to gitolite3.

we’ll have to decide whether we want gitolite3 to run as user gitolite3 (debian default) or keep it running as user gitolite (with the advantage of not having to replace the username in our entire setup). the same goes for the basedir, debian default is /var/lib/gitolite3, but this will break various references to /var/lib/gitolite in our setup.

i’m tempted to stick to debian defaults.

what we could do as a dirty hack is to replace the uid/gid/homedir/shell attributes of the existing gitolite user with the gitolite3 values, so we’ll have two identical users and the old gitolite@puppet-git will keep working. furthermore, /var/lib/gitolite can be replaced by a symlink to /var/lib/gitolite3.

what needs to be done is add a ~160GB disk to puppet-git to store the backups of all the repositories during the migation.

after that, it’s a matter of:

- backing up all the repositories and hooks

- dpkg —purge gitolite

- rm -rf /var/lib/gitolite

- install gitolite3

- putting the repositories and hooks back in place

- running
gitolite compile
gitolite setup —hooks-only
gitolite trigger POST_COMPILE
- give the gitolite-admin repo a push.

then optionally we add a hacky fake user gitolite and a symlink, or we adapt all the puppet-references to /var/lib/gitolite and gitolite@puppet-git.

i’d quite like your opinion on this before working things out further.

TODO: i still need to test git-annex.

#21 Updated by intrigeri 2018-11-01 17:38:22

• Assignee changed from intrigeri to groente
• QA Check changed from Info Needed to Dev Needed

> we’ll have to decide whether we want gitolite3 to run as user gitolite3 (debian default) or keep it running as user gitolite (with the advantage of not having to replace the username in our entire setup). the same goes for the basedir, debian default is /var/lib/gitolite3, but this will break various references to /var/lib/gitolite in our setup.

> i’m tempted to stick to debian defaults.

I’d love to fully switch to the defaults too but our canonical Git repo got migrated a couple months ago there so there’s already quite a few people with gitolite hard-coded in many Git remote URLs that we cannot update for them. So:

> what we could do as a dirty hack is to replace the uid/gid/homedir/shell attributes of the existing gitolite user with the gitolite3 values, so we’ll have two identical users and the old gitolite@puppet-git will keep working. furthermore, /var/lib/gitolite can be replaced by a symlink to /var/lib/gitolite3.

… yes, this seems to be the cheapest way to get backwards compat with existing Git remote URLs.

> what needs to be done is add a ~160GB disk to puppet-git to store the backups of all the repositories during the migation.

> after that, it’s a matter of:

+ timing this right: we can chat on XMPP to figure out when would be a good time for such a disruptive operation. Then you can ask moire & sajolida (who are running our end of year fundraiser at the moment and need a working web site) if the proposed dates work for them.

I see nothing about Puppet in there. What about tails::gitolite and the gitolite Puppet module it uses?

> TODO: i still need to test git-annex.

Ah, the ADC thing. This one has quite some potential to be loads of fun :P

#22 Updated by intrigeri 2019-02-24 11:23:11

• Target version changed from 2018 to Tails_3.15

2018 is over but it’ll will definitely be done by end of 2019Q2 so setting target version to the first release after that.

#23 Updated by CyrilBrulebois 2019-07-10 10:33:57

• Target version changed from Tails_3.15 to Tails_3.16

#24 Updated by groente 2019-08-20 11:31:09

• Status changed from Confirmed to Needs Validation
• Assignee changed from groente to Sysadmins
• % Done changed from 0 to 50

As far as I can tell everything works again with the fresh new gitolite3, can you confirm and review the changes i made to puppet-lizard-manifests and puppet-tails today?

#26 Updated by intrigeri 2019-08-21 10:22:46

• Assignee changed from Sysadmins to intrigeri

I’m on it!

#27 Updated by intrigeri 2019-08-21 10:43:27

• Status changed from Needs Validation to In Progress
• Assignee changed from intrigeri to groente

Hi @groente!

> As far as I can tell everything works again with the fresh new gitolite3, can you confirm and review the changes i made to puppet-lizard-manifests and puppet-tails today?

From a user PoV, it works nicely so far:

• Regular Git interaction with tails.git works. Ditto for a git-remote-crypt repo.
• I’ve fixed the “trigger CI jobs on Git push” thing by adjusting jenkins-jobs.git.
• The Salsa mirror of tails.git seems to be correctly updated.
• Pushing new data with git-annex works.

From a sysadmin PoV, I did a code review and it looks great in general; I only have a few rather minor comments/questions/suggestions:

• I’m a bit surprised to see no change in the gitolite-admin repo. Was our config 100% ready for Gitolite 3? This would be good news :)
• /usr/local/lib/gitolite needs to be cleaned up, no?
• What’s /var/lib/gitolitee?
• In x-systemd.requires-mounts-for=/var/lib/gitolite I think I’d rather see gitolite3 instead of gitolite instead of relying on the fact systemd will follow symlinks (I’m assuming you’ve tested this and it does, but still :)
• Regarding commit9592a3bc1f97fc14320dd71bf3f300c753a60851 and b9afa28905da507ec4c69e09b8e0619077ebf199 in the manifests repo: in general, having to use the Exec resource is a code smell; instead, I think we should s/gitolite/gitolite3/ in hieradata/node/puppet-git.lizard.yaml and drop this from the manifests: this class parameter was introduced to allow us to nicely support this very need. Makes sense?
• About commit 6a8542bbae605ae4c7ee179fa2165e1c33212c65 in the manifests repo: I’d rather see this in tails::gitolite, with the uid/gid as class parameters. Rationale: anyone who wants to replicate this part of our setup will need this gitolite user, because we have it hard-coded in too many other places, some that we could update (see Hiera), some that we can’t easily migrate (contributors’ habits / scripts / config). And it would avoid me having to duplicate this code in jenkins.sib’s node definition :) What do you think?

#28 Updated by groente 2019-08-21 15:17:03

• Status changed from In Progress to Needs Validation
• Assignee changed from groente to Sysadmins
• % Done changed from 50 to 60

> From a sysadmin PoV, I did a code review and it looks great in general; I only have a few rather minor comments/questions/suggestions:
>
> * I’m a bit surprised to see no change in the gitolite-admin repo. Was our config 100% ready for Gitolite 3? This would be good news :)

Seems so, yes.

> * /usr/local/lib/gitolite needs to be cleaned up, no?

Done!

> * What’s /var/lib/gitolitee?

A typo :)

> * In x-systemd.requires-mounts-for=/var/lib/gitolite I think I’d rather see gitolite3 instead of gitolite instead of relying on the fact systemd will follow symlinks (I’m assuming you’ve tested this and it does, but still :)

Ha, nice catch, fixed!

> * Regarding commit9592a3bc1f97fc14320dd71bf3f300c753a60851 and b9afa28905da507ec4c69e09b8e0619077ebf199 in the manifests repo: in general, having to use the Exec resource is a code smell; instead, I think we should s/gitolite/gitolite3/ in hieradata/node/puppet-git.lizard.yaml and drop this from the manifests: this class parameter was introduced to allow us to nicely support this very need. Makes sense?

Done!

> * About commit 6a8542bbae605ae4c7ee179fa2165e1c33212c65 in the manifests repo: I’d rather see this in tails::gitolite, with the uid/gid as class parameters. Rationale: anyone who wants to replicate this part of our setup will need this gitolite user, because we have it hard-coded in too many other places, some that we could update (see Hiera), some that we can’t easily migrate (contributors’ habits / scripts / config). And it would avoid me having to duplicate this code in jenkins.sib’s node definition :) What do you think?

I moved the backward compatibility features from puppet-lizard-manifests to puppet-gitolite, using external facts to determine the uid and gid, so it should work generically on any system now.

#29 Updated by intrigeri 2019-08-22 06:54:18

• Assignee changed from Sysadmins to intrigeri

@groente, very nice! I’m glad you wrote new facts :)

Code review passes. I’ll now apply this to my local Jenkins and see what happens.

#30 Updated by intrigeri 2019-08-22 07:16:40

• Status changed from Needs Validation to In Progress
• Assignee changed from intrigeri to groente

Hi again @groente!

Hmmm, looks like the backwards compat code in the gitolite class has some issues:

• It fails if the gitolite3 user does not exist yet, i.e. if the package has not been installed yet, which is the case when one tries to apply this class on a system that was running gitolite v2 previously. This class of problems is typical when one incrementally develops & applies manifests: it’s very easy to introduce code that assumes a previous version of the same code has been applied already. The only way to avoid this is what the cool kids do, i.e. test code on clean systems. My local Jenkins is not exactly a brand new, clean system, but in this case it’s enough to expose this problem :)
• It assumes $gituser is set to the non-default gitolite3 value.
• It assumes $gituser was previously set to the non-default gitolite value.
• Accordingly, it hard-codes /var/lib/gitolite and /var/lib/gitolite3 while the latter should be $path and the former depends on the settings of the previous gitolite v2 installation.

To fix this, I think we would need something like this:

• new class parameters: $oldgituser = 'git' and $oldpath = "/var/lib/${oldgituser}"
• when backwards compat is enabled, check that $gituser and $oldgituser are different and fail if they’re not; ditto for the paths
• retrieving the uid and gid depends on $gituser so I believe it can’t be a fact; I guess a function would work?
• move the backwards compat code to a dedicated class with a relationship that ensures it’s applied after the new package (in our case: gitolite3) is installed

So frankly, at this point I’m not sure anymore that it’s worth the effort to add this backward compat support to the gitolite module: making it as generic as the gitolite module is still requires quite some work. If you’re happy to do it as a learning experience or something, great! Else, feel free to move it to tails::gitolite, where we can:

• assume lots of things that we can’t in the gitolite module
• decide that manual steps are OK for this one-time migration, e.g. I should manually run apt install gitolite3 and then the code will take care of the following steps

#31 Updated by CyrilBrulebois 2019-09-05 00:05:29

• Target version changed from Tails_3.16 to Tails_3.17

#32 Updated by intrigeri 2019-09-12 14:25:11

• Target version changed from Tails_3.17 to Tails_4.0

#33 Updated by intrigeri 2019-10-21 11:46:09

• Target version changed from Tails_4.0 to Tails_4.1

#34 Updated by CyrilBrulebois 2019-12-04 11:31:15

• Target version changed from Tails_4.1 to Tails_4.2

#35 Updated by groente 2020-01-04 20:04:12

• Status changed from In Progress to Needs Validation
• Assignee changed from groente to intrigeri
• Priority changed from Elevated to Low
• % Done changed from 60 to 90

honestly, i feel i’ve spent enough time on this. if you insist the puppet-gitolite module needs further cleanup, be my guest, but at this point i’d rather just consider the migration done and leave things as is. mind if i close the ticket or do you want to take it over?

#36 Updated by CyrilBrulebois 2020-01-07 18:00:37

• Target version changed from Tails_4.2 to Tails_4.3

#37 Updated by intrigeri 2020-01-09 12:08:30

• Status changed from Needs Validation to Resolved
• Assignee deleted (intrigeri)

OK, let’s move on :)