Feature #15510

Switch to the puppetlabs/apt module

Added by intrigeri 2018-04-09 15:02:48 . Updated 2019-08-11 11:19:39 .

Status:
Resolved
Priority:
Elevated
Assignee:
groente
Category:
Infrastructure
Target version:
Tails_3.16
Start date:
2018-04-09
Due date:
% Done:

100%

Feature Branch:
puppet-lizard-manifests:feature15510, puppet-tails:feature15510
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

The shared apt module is deprecated and the shared Puppet modules group decided to switch to https://github.com/puppetlabs/puppetlabs-apt.

See https://gitlab.com/shared-puppet-modules-group/apt/blob/master/UPGRADING.md for migration info. Lots of features we use are missing so this is going to take time. Let’s try to make it happen by the end of 2019Q1.

This migration can be split into steps:

  1. migrate away from functionality provided by the shared apt module but that’s not supported by puppetlabs/apt: either switch to already supported alternatives, or to “plugins”, or submit PRs to puppetlabs/apt
    1. listchanges done
    2. apt::apticron → APT::Periodic + monitoring check done
    3. apt::reboot_required_notify → install reboot-notifier ourselves done
    4. apt::dpkg_statoverride → import it in puppet-tails
  2. reach the point where the only functionality we use from the shared apt module is also supported either by puppetlabs/apt directly or by modules that depend on it
  3. switch to puppetlabs/apt; changes that must happen in lockstep when doing so:
    1. apt::apt_conf → apt::conf
    2. apt::custom_key_dir → apt class’ keys parameter but implementation looks scary (do the red flags apply to us?); worst case, follow https://wiki.debian.org/DebianRepository/UseThirdParty and drop the non-ascii-armorded key in /usr/share/keyrings/example-archive-keyring.gpg + point to it from sources.list
    3. -apt::cron → https://github.com/voxpupuli/puppet-unattended_upgrades-
    4. apt::package → package + apt::pin
    5. apt::preferences_snippet → apt::pin
    6. apt::proxy* → apt class’ proxy parameter
    7. apt::sources_list, apt::repos, apt::use_next_release, apt::use_volatile, tails::apt::repository::* → puppetlabs/apt provides several was to manage sources.list

To list the apt:: things we use:

git grep -h --only-matching --color=never \
  --recurse-submodules -E '\bapt::[^ ]+\b' -- \
  hieradata/ manifests/ modules/reprepro/ modules/tails* \
  2>/dev/null \
  | sort -u

Subtasks

Related issues

Blocks Tails - Feature #13284: Core work: Sysadmin (Adapt our infrastructure) Confirmed 2017-06-30

History

#1 Updated by intrigeri 2018-04-09 15:03:00

  • blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#2 Updated by intrigeri 2018-04-09 15:50:53

  • Description updated

#3 Updated by intrigeri 2018-06-05 11:06:06

  • Target version changed from Tails_3.9 to Tails_3.10.1

#4 Updated by intrigeri 2018-09-30 14:29:35

  • Target version changed from Tails_3.10.1 to Tails_3.11

#5 Updated by intrigeri 2018-10-12 12:02:53

#6 Updated by intrigeri 2018-10-12 14:15:25

  • Target version changed from Tails_3.11 to Tails_3.12

I’ve booked time to work on this around Dec 17-31.

#7 Updated by intrigeri 2019-01-02 05:00:35

  • Target version changed from Tails_3.12 to Tails_3.13

#8 Updated by intrigeri 2019-02-10 14:22:50

  • Description updated

#9 Updated by intrigeri 2019-03-10 15:08:28

  • Description updated
  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

Listed all the apt:: thing we use and what I think we should do with them. Migrated a few small things away from the old apt module.

#10 Updated by intrigeri 2019-03-10 15:21:45

  • Description updated

Migrated the last thing I could migrate without switching to puppetlabs/apt.

Next steps: prepare a topic branch (= environment) that switches to puppetlabs/apt + adjusts the bare minimum, put a couple non-critical nodes into that environment, fix stuff, and progressively port more code and nodes to that environment until we’re done and we can merge this branch into production and put all nodes back into the production environment.

#11 Updated by intrigeri 2019-03-11 19:04:55

  • Description updated

#12 Updated by intrigeri 2019-03-14 19:01:39

  • Target version changed from Tails_3.13 to Tails_3.14

I’ve been busy elsewhere and we’re now too close to a release for such a potentially disruptive change.

#13 Updated by intrigeri 2019-03-27 11:49:57

  • Feature Branch set to puppet-lizard-manifests:feature15510

#14 Updated by intrigeri 2019-05-02 17:24:10

  • Priority changed from Normal to Elevated

#15 Updated by intrigeri 2019-05-08 10:05:20

  • Target version changed from Tails_3.14 to Tails_3.15

#16 Updated by intrigeri 2019-06-11 09:13:48

  • Feature Branch changed from puppet-lizard-manifests:feature15510 to puppet-lizard-manifests:feature15510, puppet-tails:feature15510

#17 Updated by intrigeri 2019-06-11 09:46:12

  • Description updated

#18 Updated by intrigeri 2019-06-11 13:20:51

  • Description updated

#19 Updated by intrigeri 2019-06-11 13:24:20

  • Description updated

#20 Updated by intrigeri 2019-06-11 13:41:54

  • Description updated

#21 Updated by intrigeri 2019-06-11 14:00:14

  • Description updated

#22 Updated by intrigeri 2019-06-11 14:57:43

  • Description updated

#23 Updated by intrigeri 2019-06-11 15:00:24

  • % Done changed from 10 to 20

I think I’ve ported all the code we use to puppetlabs/apt and it works on local VMs that use tiny subsets of our Puppet codebase. Next steps: put each node, one after the other, into the feature15510 environment, from the least critical to the most, fix stuff as I go, until I’m done and we can merge this branch into production and put all nodes back into the production environment.

#24 Updated by intrigeri 2019-06-11 17:37:21

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to groente

Deployed! bertagaz, groente, anyone wants to review the code changes?

#25 Updated by intrigeri 2019-06-14 07:55:06

  • Status changed from Needs Validation to In Progress
  • Assignee changed from groente to intrigeri

/var/log/unattended-upgrades/unattended-upgrades.log on apt.lizard says:

2019-06-14 06:00:34,065 INFO Packages that will be upgraded: dbus dbus-x11 libdbus-1-3
2019-06-14 06:00:34,066 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log'
2019-06-14 06:00:45,561 INFO All upgrades installed

… and the corresponding monitoring checked recovered. So upgrades are applied, which is good. But the “apt” and “upgreadable” monitoring check respectively went to critical and warning state before that and that’s been noisy. I suspect our monitoring checks are scheduled in a way that matched what we did before (upgrading every 6 hours) but does not match unattended-upgrades’ scheduling. I’ll fix that.

#26 Updated by intrigeri 2019-06-14 08:23:51

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to groente

Tried to fix the monitoring scheduling issue. It may need some more fine tuning, I’ll keep an eye on it, but that shouldn’t block reviewing the rest of this work.

#27 Updated by intrigeri 2019-06-14 10:05:55

Next issue for me to triage/investigate: “Subject: [package on hold][reboot required] unattended-upgrades result for ‘puppet-git’: False”.

#28 Updated by intrigeri 2019-06-14 10:25:52

intrigeri wrote:
> Next issue for me to triage/investigate: “Subject: [package on hold][reboot required] unattended-upgrades result for ‘puppet-git’: False”.

Should be fixed by https://git-tails.immerda.ch/puppet-tails/commit/?id=a65ade061a5f739a4e5647b211c6f8d0b0fb56d8.

#29 Updated by intrigeri 2019-06-15 09:07:30

intrigeri wrote:
> Tried to fix the monitoring scheduling issue. It may need some more fine tuning,

It does: https://icingaweb2.tails.boum.org/monitoring/service/show?host=isobuilder1.lizard&service=apt%40isobuilder1.lizard. This time, I’ll (re×4)read Icinga check scheduling doc and will base whatever fix I try next on actual maths, instead of fiddling with it like I did 2 days ago.

#30 Updated by intrigeri 2019-06-19 06:47:31

intrigeri wrote:
> intrigeri wrote:
> > Tried to fix the monitoring scheduling issue. It may need some more fine tuning,
>
> It does: https://icingaweb2.tails.boum.org/monitoring/service/show?host=isobuilder1.lizard&service=apt%40isobuilder1.lizard. This time, I’ll (re×4)read Icinga check scheduling doc and will base whatever fix I try next on actual maths, instead of fiddling with it like I did 2 days ago.

In the end, I figured it was better to fix the root cause of the problem (upgrades applied less often than they used to be) rather than tweaking the monitoring, so I tried it (6a715c3363d35c7ef694fd8b4c12177a79a2b0f4 + ceac8b1890de68ec42bd8cf9ca753e233ee61dc9) and reverted my earlier fiddling with the monitoring config.

#31 Updated by CyrilBrulebois 2019-07-10 10:34:05

  • Target version changed from Tails_3.15 to Tails_3.16

#32 Updated by groente 2019-07-29 13:03:13

  • Status changed from Needs Validation to In Progress
  • Assignee changed from groente to intrigeri
  • % Done changed from 20 to 40

heya,
sorry for the long delay, but the code looks good!
one last issue is that the monitoring still triggers quite often before the upgrade is run. perhaps your calculations didn’t take the 60 minute randomisation on the upgrade timer into account?

#33 Updated by intrigeri 2019-08-06 12:44:10

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to groente

groente wrote:
> one last issue is that the monitoring still triggers quite often before the upgrade is run. perhaps your calculations didn’t take the 60 minute randomisation on the upgrade timer into account?

I think you’re right! https://git.tails.boum.org/puppet-tails/commit/?id=951713c6c11053fe5a6acfe94250f747ff1dbda9 should fix that. Please let me know if this sort of problems still happens :)

#34 Updated by groente 2019-08-11 11:18:52

  • Status changed from Needs Validation to Resolved

intrigeri wrote:
> groente wrote:
> > one last issue is that the monitoring still triggers quite often before the upgrade is run. perhaps your calculations didn’t take the 60 minute randomisation on the upgrade timer into account?
>
> I think you’re right! https://git.tails.boum.org/puppet-tails/commit/?id=951713c6c11053fe5a6acfe94250f747ff1dbda9 should fix that. Please let me know if this sort of problems still happens :)

Didn’t happen anymore during the last updates, I’m quite confident we’re good, so closing this ticket \o/

#35 Updated by groente 2019-08-11 11:19:39

  • % Done changed from 40 to 100