Tails

#1 Updated by intrigeri 2017-09-02 12:10:11

• copied from Bug #10093: Upgrade to Gitolite v3 on puppet-git.lizard added

#2 Updated by intrigeri 2017-09-02 12:11:06

• blocks Feature #13528: Upgrade buse to Stretch added

#3 Updated by intrigeri 2017-09-02 12:12:13

• blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#4 Updated by intrigeri 2017-09-02 12:26:09

• Description updated

#5 Updated by intrigeri 2017-09-02 12:29:28

• Description updated

#6 Updated by intrigeri 2017-09-02 13:01:06

• blocked by deleted (Feature #13528: Upgrade buse to Stretch)

#7 Updated by intrigeri 2017-12-18 16:49:41

• Assignee changed from intrigeri to groente

#8 Updated by intrigeri 2018-02-10 10:28:40

• Description updated

#9 Updated by groente 2018-09-28 15:49:34

• Status changed from Confirmed to In Progress
• Assignee changed from groente to bertagaz
• QA Check set to Info Needed

After going through the gitolite setup on buse, I came to the following conclusions:

- the only repository that is still in use on buse’s gitolite is tails.git

- the sole reason we have gitolite on buse is for integration with redmine, which uses redmine’s gitolite plugin
- all the hooks in the tails.git repo do little more than sending an http call to redmine

Then I noticed: “Redmine natively supports integration with different SCM-tools”

And this made me wonder whether we’re better off getting rid of gitolite alltogether, uninstalling the redmine gitolite plugin, and using the native support for git integration instead.

Considering that “Redmine requires a repository which is bare and local to enable browsing through Redmine”, we would have to:

- git clone —mirror the tails.git into a local directory on buse
- tell redmine where to find this repo
- set up either:
- a cronjob that cd’s to the repo and runs git remote update
OR
- allow ssh access from git.t.b.o and add a hook to git.t.b.o:tails.git that runs git push —mirror to buse

And that would be all.

Now, all this hinges on my assumption that we don’t actually use the extra features that the gitolite plugin provides over the native git support. See http://redmine-git-hosting.io/features/ for an overview.

Can you confirm whether this assumption is correct and tell me what you think about this plan? Thank you!

#11 Updated by groente 2018-10-14 00:15:28

• % Done changed from 0 to 40

I tested by creating a local copy of buse without the gitolite plugin, simply cloning tails.git and adding that as a repository to redmine and I’m happy that it ticks all the boxes.

There is one important detail that should be added to the cronjob/hook, namely a call to https://labs.riseup.net/code/sys/fetch_changesets?key=<KEY> to let redmine parse the latest changes in git.

The next steps I think are now:

* 1. warn all the old projects that their gitolite will be remove
* 2. cd /srv; git clone —mirror https://git-tails.immerda.ch/tails
* 3. add a user named tails to buse
* 4. chown -R tails:tails /srv/tails.git
* 5. add the appropriate ssh key to tails’ authorized_keys
* 6. remove the Xitolite repository from the Tails project
* 7. remove Gitolite from the repositories settings in redmine and add Git instead
* 8. add the git repository /srv/tails.git to the Tails project and set and set a WS API key
* 9. call https://labs.riseup.net/code/sys/fetch_changesets?key=KEY to process the repository (probably needs to be done multiple times because of timeouts)
* 10. adapt the hook on git.t.b.o. to push changes to tails.git to tails@buse:/srv/tails.git and, once done, call https://labs.riseup.net/code/sys/fetch_changesets?key=KEY
* 11. remove the Redmine Git Hosting plugin from redmine
* 12. uninstall gitolite and clean up the junk

Steps 6-10 need to be coordinated and done fast to avoid changes in git not coming through in redmine.

Does this make sense to you?

#13 Updated by groente 2018-10-16 11:26:09

• Assignee changed from bertagaz to groente
• % Done changed from 40 to 50
• QA Check changed from Info Needed to Dev Needed
• Feature Branch set to bugfix/14587-gitolite-on-buse

#15 Updated by groente 2018-10-16 14:28:20

• Assignee changed from groente to intrigeri
• QA Check changed from Dev Needed to Ready for QA

Can you review the branch I made? My aim was to cover:

- the creation of a user tails_git on buse

- the creation of a mirror of the tails.git repo

- the removal of gitolite dependencies and sudo rights

- giving gitolite on puppet-git ssh access to push changes
- adapting the hook on puppet-git to indeed push the changes and call the redmine API

This should cover steps 1—5 and 10.

Deploying this should be immediately followed by executing steps 6—9, and then 11 and 12 can be done in a more relaxed manner.

#16 Updated by intrigeri 2018-10-16 16:05:49

• Feature Branch changed from bugfix/14587-gitolite-on-buse to puppet-tails:bugfix/14587-gitolite-on-buse

#17 Updated by intrigeri 2018-10-16 16:34:41

• Assignee changed from intrigeri to groente
• QA Check changed from Ready for QA to Dev Needed

> Can you review the branch I made?

Sure! General comment: you’ve made huge progress at Puppet \o/

> - the creation of a user tails_git on buse
> - the creation of a mirror of the tails.git repo
> - the removal of gitolite dependencies and sudo rights
> - giving gitolite on puppet-git ssh access to push changes

> This should cover steps 1—5 and 10.

• Please align your => as per Puppet style guide. I think puppet-lint --fix will do it for you :)
• Please clean up the resources you’ve removed (File[/srv/gitosis/repositories], sudo::conf, etc.). I see no “removal of gitolite dependencies and sudo rights”, merely “stop setting them up but if they’re there already, do nothing”.
• Applying this on labs will put the system in a different state than applying this on a clean new system: user['www-data'] will be configured differently. If you want to set that user’s config to some different state (e.g. the Debian defaults), please handle the transition.
• I don’t understand why you removed the ruby-* Redmine dependencies. Are they all only needed for the Gitolite plugin?
• I think Vcsrepo[/srv/repositories/tails.git] needs a dependency on File[/srv/repositories] (I don’t think there’s an auto-dep for this, but maybe I’m wrong :)
• Will ensure => mirror ever update the repo? If yes, I’m concerned that it causes a race condition vs. the push notification this repo will get, which is supposed to ping Redmine, and if the Puppet agent wins, there’s a chance that some changes are not fed into Redmine. I think we need to ensure that only one actor updates this repo (the hook from puppet-git.lizard).
• Too much white space before the arrow in user => 'tails_git'. puppet-lint --fix should fix it.
• commit ddfe3f50faf0ddec16fba11397575334b0a4347d would work on a clean new system, but it won’t transition puppet-git to the new remote.

> - adapting the hook on puppet-git to indeed push the changes and call the redmine API

Somehow I’m not a big fan of Gitolite pinging Redmine directly, which forces us to include Redmine-specific stuff in tails::gitolite. I would prefer Gitolite merely pushing --mirror and having /srv/repositories/tails.git/hooks/post-update on labs ping Redmine itself. Also, this would give us nicer separation of concerns & credentials. I can live with the current implementation but it feels a little bit wrong: I mean, if we’re happy with having Redmine updates be triggered remotely, why do we need a local mirror? If we’re going this far, we can as well have Redmine pull from a remote, already existing repo at step 8, no?

> Deploying this should be immediately followed by executing steps 6—9, and then 11 and 12 can be done in a more relaxed manner.

Step 9 could be implemented on this branch right now, no? This would leave only steps 6-8 for the “immediately” part, which feels better.

#18 Updated by groente 2018-10-16 18:24:21

• Assignee changed from groente to intrigeri
• QA Check changed from Dev Needed to Info Needed

> * Please clean up the resources you’ve removed (File[/srv/gitosis/repositories], sudo::conf, etc.). I see no “removal of gitolite dependencies and sudo rights”, merely “stop setting them up but if they’re there already, do nothing”.

Ah, the removal of the files i was planning to do manually in step 12, it seemed odd to me to add puppet code for the sole sake of removing a whole lot of stuff that was never in puppet in the first place and will not be there whenever we deploy this class on a clean system.

As for the sudo rights, I thought config_file_replace => false in our base tails class implied that sudoers.d would be purged and filled with whatever is defined in puppet at every run, no?

> * Applying this on labs will put the system in a different state than applying this on a clean new system: user['www-data'] will be configured differently. If you want to set that user’s config to some different state (e.g. the Debian defaults), please handle the transition.

How exactly do you mean? As far as I can tell there’s nothing to transition even if www-data is set back to Debian defaults?

> * I don’t understand why you removed the ruby-* Redmine dependencies. Are they all only needed for the Gitolite plugin?

That’s what I assumed based on the comment that was in puppet, and the general assumption that the debian redmine package would already imply all the necessary dependencies. I’ll check on my local copy to see if things break when I remove these packages.

> * I think Vcsrepo[/srv/repositories/tails.git] needs a dependency on File[/srv/repositories] (I don’t think there’s an auto-dep for this, but maybe I’m wrong :)

Okay!

> * Will ensure => mirror ever update the repo? If yes, I’m concerned that it causes a race condition vs. the push notification this repo will get, which is supposed to ping Redmine, and if the Puppet agent wins, there’s a chance that some changes are not fed into Redmine. I think we need to ensure that only one actor updates this repo (the hook from puppet-git.lizard).

As far as I can tell it will not update the repo, if I interpret the vcsrepo source correctly, it only runs git clone —mirror.

> * commit ddfe3f50faf0ddec16fba11397575334b0a4347d would work on a clean new system, but it won’t transition puppet-git to the new remote.

Care to elaborate on that?

> > - adapting the hook on puppet-git to indeed push the changes and call the redmine API
>
> Somehow I’m not a big fan of Gitolite pinging Redmine directly, which forces us to include Redmine-specific stuff in tails::gitolite. I would prefer Gitolite merely pushing --mirror and having /srv/repositories/tails.git/hooks/post-update on labs ping Redmine itself. Also, this would give us nicer separation of concerns & credentials. I can live with the current implementation but it feels a little bit wrong: I mean, if we’re happy with having Redmine updates be triggered remotely, why do we need a local mirror? If we’re going this far, we can as well have Redmine pull from a remote, already existing repo at step 8, no?

We need a local mirror because redmine demands a local mirror for git integration :) But I see your point, I’ll look into making the call locally.

> Step 9 could be implemented on this branch right now, no? This would leave only steps 6-8 for the “immediately” part, which feels better.

Hm, not really, calling the api the first time will or will not work depending on system resources (it’s a hell of a lot of data to parse for one http request), I’m more comfortable with doing this manually that scripting something in puppet that calls the api, checks the response and, depending on the response, calls the api again (or not).
Eitherway, worst case, thanks to the hook, the api will be called anyway by any push to the repo, so it’ll get their eventually after two or three pushes to the repo.

#19 Updated by intrigeri 2018-10-17 08:39:58

• Assignee changed from intrigeri to groente
• QA Check changed from Info Needed to Dev Needed

>> * Please clean up the resources you’ve removed (File[/srv/gitosis/repositories], sudo::conf, etc.). I see no “removal of gitolite dependencies and sudo rights”, merely “stop setting them up but if they’re there already, do nothing”.

> Ah, the removal of the files i was planning to do manually in step 12, it seemed odd to me to add puppet code for the sole sake of removing a whole lot of stuff that was never in puppet in the first place and will not be there whenever we deploy this class on a clean system.

Agreed for the bits that were never managed by Puppet.

For the bits that were managed by Puppet and whose resources who simply removed from the code (e.g. File[/srv/gitosis/repositories]), usually we try to handle such migrations/cleanups via Puppet for two main reasons: supporting the case when someone has deployed a local replica of this piece of infra and allowing the migration/cleanup steps to be reviewed. In this case I can’t imagine someone having managed to replicate our Redmine setup (it’s simply not Puppetized enough) and I think it’s OK that you do the migration/cleanup manually with no review.

> As for the sudo rights, I thought config_file_replace => false in our base tails class implied that sudoers.d would be purged and filled with whatever is defined in puppet at every run, no?

Right! I totally forgot I had set it up this way but indeed, I’m happy I did :))

>> * Applying this on labs will put the system in a different state than applying this on a clean new system: user['www-data'] will be configured differently. If you want to set that user’s config to some different state (e.g. the Debian defaults), please handle the transition.

> How exactly do you mean? As far as I can tell there’s nothing to transition even if www-data is set back to Debian defaults?

For example, in the current state of the branch:

• applying this class on a clean new system will set www-data’s homedir to /var/www and its shell to /usr/sbin/nologin
• applying this class on labs will leave www-data’s homedir to /home/www-data and its shell to /bin/sh; also, that user will be in the gitolite group

So, either we need the current state (that was encoded in Puppet, which you removed) and then the corresponding code should stay; or we can live with Debian’s defaults, and then we need code to reset e.g. www-data’s homedir and shell to Debian’s defaults.

Am I clearer now?

>> * I don’t understand why you removed the ruby-* Redmine dependencies. Are they all only needed for the Gitolite plugin?

> That’s what I assumed based on the comment that was in puppet,

Ooops, I had missed that comment. The only exception might be ruby-haml-rails (see bb0712a104a2f306a80c53e43b809e7badc48783, not sure if it’s related to Gitolite or if it’s about another plugin and maybe I messed up the comment in 40e13f903c44e46cf87f1328d6633308506912b0).

> and the general assumption that the debian redmine package would already imply all the necessary dependencies.

FWIW, I’m not sure if redmine_git_hosting is the only non-packaged plugin we run.

> I’ll check on my local copy to see if things break when I remove these packages.

Great :)

>> * commit ddfe3f50faf0ddec16fba11397575334b0a4347d would work on a clean new system, but it won’t transition puppet-git to the new remote.

> Care to elaborate on that?

Sure. The proposed diff is:

git remote | grep -qs -x labs || \
-                  git remote add labs "gitolite@labs.riseup.net:$repository"
+                  git remote add labs "tails_git@labs.riseup.net:/srv/repositories/$repository"

tails.git on puppet-git.lizard already has a “labs” remote configured, so git remote | grep -qs -x labs will be true, so install_remotes.sh will leave it as-is and tails.git will keep trying to push to the old (Gitolite) mirror on labs. In this case we need transition code because there’s at least one existing local replica of this setup.

This should work:

# XXX: remove set-url migration handling once deployed everywhere

if git remote | grep -qs -x labs; then

git remote set-url labs “tails_git@labs.riseup.net:/srv/repositories/$repository”

else

git remote add labs “tails_git@labs.riseup.net:/srv/repositories/$repository”

fi
>> > - adapting the hook on puppet-git to indeed push the changes and call the redmine API

>>

>> Somehow I’m not a big fan of Gitolite pinging Redmine directly, which forces us to include Redmine-specific stuff in tails::gitolite. I would prefer Gitolite merely pushing --mirror and having /srv/repositories/tails.git/hooks/post-update on labs ping Redmine itself. Also, this would give us nicer separation of concerns & credentials. I can live with the current implementation but it feels a little bit wrong: I mean, if we’re happy with having Redmine updates be triggered remotely, why do we need a local mirror? If we’re going this far, we can as well have Redmine pull from a remote, already existing repo at step 8, no?
> We need a local mirror because redmine demands a local mirror for git integration :)
Of course, what was I thinking :S
> But I see your point, I’ll look into making the call locally.
Yeah (to clarify, that’s primarily for cleaner Puppet code design, and mostly because the ping needs a secret; if anyone could do the ping, then I wouldn’t care much, e.g. puppet-git pings ikiwiki and that’s fine IMO because it does not force us to add app-specific secrets/parameters to tails::gitolite).
>> Step 9 could be implemented on this branch right now, no? This would leave only steps 6-8 for the “immediately” part, which feels better.
> Hm, not really, calling the api the first time will or will not work depending on system resources (it’s a hell of a lot of data to parse for one http request), I’m more comfortable with doing this manually that scripting something in puppet that calls the api, checks the response and, depending on the response, calls the api again (or not).

> Eitherway, worst case, thanks to the hook, the api will be called anyway by any push to the repo, so it’ll get their eventually after two or three pushes to the repo.
Oops, I’ve been unclear. What I meant was not to have Puppet ping fetch_changesets itself but to deploy via Puppet the code that will do upon Git updates. But now I understand I got step 9 wrong: you meant pinging fetch_changesets by hand, not setting up the post-Git-update automation (which actually you did already, modulo it should move, as discussed above). So I’m not requesting any action from you here than what we already agreed on above, i.e. changing how step 10 is implemented on this branch :)

    
  

#21 Updated by groente 2018-10-17 10:24:55

• Assignee changed from groente to intrigeri
• QA Check changed from Dev Needed to Ready for QA

Can you check the branch again? Also, please let me know if you agree with uninstalling the bootstrap plugin. Thank you!

#22 Updated by intrigeri 2018-10-17 11:31:20

• Assignee changed from intrigeri to groente
• QA Check changed from Ready for QA to Dev Needed

> Can you check the branch again?

• What’s the rationale behind making $redmine_ws_api_key optional (i.e. giving it a default value)?
• typo in source => 'puppet:///modules/tails/gitolite/hooks/labs_mirror-post-update.hook'), (closing parenthesis)
• I think /srv/repositories/tails.git/hooks/post-update should use content + template, not “source”.

Other than this, looks good!

> Also, please let me know if you agree with uninstalling the bootstrap plugin.

As long as your assumption about its role holds true and this does not break the UI (e.g. it’s pretty responsive as-is), fine :)

#23 Updated by groente 2018-10-17 11:40:36

• Assignee changed from groente to intrigeri
• QA Check changed from Dev Needed to Ready for QA

> * What’s the rationale behind making $redmine_ws_api_key optional (i.e. giving it a default value)?

The idea is that I can deploy all this without needing a key yet. I’ll only have a key after I configure redmine to use git, but I don’t want to configure redmine to use git before I have the actual git repo set up. This way I can merge the branch, deploy, configure redmine, add the key to hiera, and deploy again to get the hook working properly.

> * typo in source => 'puppet:///modules/tails/gitolite/hooks/labs_mirror-post-update.hook'), (closing parenthesis)
> * I think /srv/repositories/tails.git/hooks/post-update should use content + template, not “source”.

both fixed.

> Other than this, looks good!

\o/

> > Also, please let me know if you agree with uninstalling the bootstrap plugin.
>
> As long as your assumption about its role holds true and this does not break the UI (e.g. it’s pretty responsive as-is), fine :)

I’ve not done very elaborate testing on my local copy, but I didn’t notice any difference in UI while creating tickets, adding references, etc.

#24 Updated by intrigeri 2018-10-31 08:44:02

• Assignee changed from intrigeri to groente
• QA Check deleted (Ready for QA)

Yeah, go ahead :)

#25 Updated by groente 2018-10-31 19:57:00

• % Done changed from 50 to 90

steps 1-10 are done, i’ll make an inventory of what to rm -rf and place the list here for QA

#27 Updated by intrigeri 2018-11-01 10:05:50

• Subject changed from Upgrade to Gitolite v3 on buse to Remove Gitolite from buse

#28 Updated by groente 2018-11-01 11:29:11

• Assignee changed from groente to intrigeri
• QA Check set to Ready for QA

Glad you like it :)

So, as for the cleanup, I’m proposing to dpkg —purge the following packages:

munin-common
munin-node
munin-plugins-core
munin-plugins-extra
gitolite

And then rm -rf whatever is left of:

/srv/gitosis
/srv/kgb-client.conf
/srv/redmine-3.1.1
/var/lib/git
/var/lib/gitolite
/usr/share/redmine/plugins/redmine_bootstrap_kit
/usr/share/redmine/plugins/redmine_git_hosting
/opt/acme

#29 Updated by intrigeri 2018-11-01 17:29:58

• Assignee changed from intrigeri to groente
• QA Check changed from Ready for QA to Dev Needed

> So, as for the cleanup, I’m proposing to dpkg —purge the following packages:

> munin-common
> munin-node
> munin-plugins-core
> munin-plugins-extra

IIRC it’s still monitored which could be useful. Sadly the corresponding Munin web interface seems to be broken currently so I cannot double check. Anyway, this seems rather unrelated to this ticket, no?

> gitolite

Yes.

> And then rm -rf whatever is left of:

ACK modulo maybe mv to somewhere under /root instead of rm, then rm a week or two later once we’re 200% sure we don’t need that stuff?

#30 Updated by groente 2018-11-01 17:40:36

• Priority changed from Elevated to Normal
• QA Check changed from Dev Needed to Ready for QA

okay, purged only the gitolite package and moved all the directories to /root/backups. let’s discuss munin in another ticket indeed.

i’ll wait a week to see if nothing broke before rm -rf’ing that and closing the ticket.

#32 Updated by groente 2018-11-13 09:22:59

• Assignee changed from groente to intrigeri

it’s been more than two weeks, i’m tempted to rm -rf /root/backups and close this ticket. any objections (just to be 200% sure)?

#33 Updated by intrigeri 2018-11-15 18:25:25

• Assignee changed from intrigeri to groente
• QA Check changed from Ready for QA to Dev Needed

> it’s been more than two weeks, i’m tempted to rm -rf /root/backups and close this ticket. any objections (just to be 200% sure)?

It’s a go!

#34 Updated by groente 2018-11-15 19:12:55

• Status changed from In Progress to Resolved

in the words of a particular birdy: nuked with extreme nuke force!