Bug #17417

Please adjust ISO history handling (git hook?)

Added by CyrilBrulebois 2020-01-09 19:36:41 . Updated 2020-04-06 21:28:47 .

Status:
Confirmed
Priority:
Normal
Assignee:
Sysadmins
Category:
Infrastructure
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

From our release process documentation:

ISO history
-----------

Push the released ISO and USB images and their artifacts (`.buildlog`,
`.build-manifest`, and `.packages` files) to our Tails ISO history git-annex
repo, so that:

 - The Jenkins `build_IUKs` can fetch them.
 - Our isotesters can fetch them from there for their testing

How to do so is described in the `ISO_history.mdwn` document in the RM team's
Git repo.

Then, wait (a few minutes) until the images appear
on <https://iso-history.tails.boum.org/>.

It seems it can take up to 10 minutes before the new version appears, once the git annex copy is finished (which can take a while already, see Bug #17414).

It would be great if this could be sped up, possibly by triggering the needed work in a git hook of some sort?

Thanks for your help.

Subtasks

Related issues

Related to Tails - Bug #17361: Streamline our release process Confirmed

History

#1 Updated by intrigeri 2020-01-11 10:00:22

  • related to Bug #17361: Streamline our release process added

#2 Updated by intrigeri 2020-01-12 19:57:55

  • Category set to Infrastructure
  • Status changed from New to Confirmed

The code that pulls from puppet-git.lizard is tails::git_annex::mirror, deployed on www.lizard. It is currently a cronjob, running every 15 minutes.

I fully agree that triggering the needed work automatically, after the RM has uploaded the images to gitolite3\puppet-git.lizard:isos.git@ with git annex, would be ideal. It would also save resources on puppet-git.lizard, which tends to be a bit overloaded these days by all the Git clients hitting large repos repeatedly.

I don’t remember how the gitolite / git-annex integration is done, but I hope it leaves us the possibility to add the hook we need here.

This design should be quite robust (e.g. concurrency is managed automatically), allows sysadmins to trivially start things by hand in the exact same context as when they happen automatically (as opposed to cron), and opens a very small attack surface:

  • The Git server is allowed to ssh into a non-privileged user on the web server and it’s only allowed to touch a flag file in there.
  • Pushing to the git annex repo triggers a hook that does the ssh www touch $FLAGFILE dance.
  • The web server has a systemd.path(5) unit that monitors said file and triggers the systemd.service(5) that does the pull. We can drop our current manual lock file handling as systemd does it by itself already.
  • The cronjob can go away.
  • If things get out-of-sync (e.g. a reboot or networking problem at the exact wrong time), the RM can trigger another sync by pushing a dummy change to the git annex repo.

If whoever has time first to implement this thinks it’s over-engineered or preferred another design, by all means, be my guest.

#3 Updated by CyrilBrulebois 2020-04-06 21:28:47

Note to myself but also others: I really should try and help here, as it’s a bit of shame to be waiting on a sync here while we have plenty of work to throw at Jenkins that depends on its completion (building all IUKs).