Bug #11242: Make puppet-concat generated Icinga2 config content sorted deterministically

Bug #11242

Make puppet-concat generated Icinga2 config content sorted deterministically

Added by bertagaz 2016-03-16 09:54:11 . Updated 2017-04-06 10:51:08 .

Status:

Resolved

Priority:

Elevated

Assignee:

Category:

Infrastructure

Target version:

Tails_2.12

Start date:

2016-03-16

Due date:

% Done:

100%

Feature Branch:

Type of work:

Sysadmin

Blueprint:

Starter:

Affected tool:

Deliverable for:

280

Description

While deploying our monitoring infra, we noticed (see ~~Feature #8648~~) that complex concatenation of fragments using the puppet-concat module is not always consistent. Their ordering is not really reliable, thus ending up in erratic changes in the resulting file.

Currently it seems that the two fragments that are badly sorted, are because their name is quite close:

200_zone_apt.lizard and 200_zone_apt-proxy.lizard as well as 200_endpoint_apt.lizard_apt.lizard and 200_endpoint_apt-proxy.lizard_apt-proxy.lizard

Sorting of fragments seems to have a long history of issues in this module (see https://github.com/puppetlabs/puppetlabs-concat/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed+sort) and the current module version we’re using is a bit old (commit somewhere between 0.1.0 and 0.2.0).

Might very well be that this bug has already been resolved, so an upgrade would help. Still, this module had a lot of changes, and some rewrites, so we should first see which version solves this issue (if any) and which version our modules that depend on it requires.

Subtasks

Related issues

Blocked by Tails - ~~Bug #11823~~: Upgrade puppet-concat to 2.2.0+

Resolved

2016-09-22

History

#1 Updated by bertagaz 2016-03-16 10:22:23

Status changed from Confirmed to In Progress

For some reasons I can’t yet explain, jenkins.lizard doesn’t seem to be subject to this bug, according to its etckeeper history.

#2 Updated by bertagaz 2016-03-16 12:15:56

% Done changed from 0 to 20

I’ve cherry-picked this commit from upstream repo, and it seems since then the zones.conf flip-flaping has stopped. The said commit has been included in puppet-concat v1.0.0, upgrading to this version should work. It doesn’t seem to have too much changes compared to the version we use. Upgrading to a much newer version is out of the scope of this ticket to me, and pertains to the sysadmin daily shits.

#3 Updated by intrigeri 2016-03-16 14:16:02

> I’ve cherry-picked […] and it seems since then the zones.conf flip-flaping has stopped.

Great!

> Upgrading to a much newer version is out of the scope of this ticket to me, and pertains to the sysadmin daily shits.

That’s totally right in theory. However, I’ve very rarely seen us (and especially you) update Puppet modules as part of daily sysadmin shifts, when not prompted by an actual fix we need to pull (I’m not doing a great job at it and I know why it’s this way, so let’s not debate that here). So I would appreciate if we took advantage of this case to update the concat module: it’s basically our only working way to upgrade Puppet modules currently. Now, I won’t insist, but if you really stand behind the “pertains to the sysadmin daily shits” justification, I expect you’ll be backing it with actions :)

#4 Updated by bertagaz 2016-03-16 14:32:56

Assignee changed from bertagaz to intrigeri
% Done changed from 20 to 80
QA Check set to Ready for QA

I’ve reset our puppet-concat repo to upstream tag 1.0.0, deployed that on a bunch of VMs, mostly the one which had the bug + puppet-git, and it does seem to resolve this issue, yay.

intrigeri wrote:
> > Upgrading to a much newer version is out of the scope of this ticket to me, and pertains to the sysadmin daily shits.
>
> That’s totally right in theory. However, I’ve very rarely seen us (and especially you) update Puppet modules as part of daily sysadmin shifts, when not prompted by an actual fix we need to pull (I’m not doing a great job at it and I know why it’s this way, so let’s not debate that here). So I would appreciate if we took advantage of this case to update the concat module: it’s basically our only working way to upgrade Puppet modules currently. Now, I won’t insist, but if you really stand behind the “pertains to the sysadmin daily shits” justification, I expect you’ll be backing it with actions :)

Right, this prehistoric concat module made me realize how bad we were at this. However, that’s quite a transition, we are quite outdated on this module that had many changes since then. I clearly won’t have time to spend before end of SponsorS_M4, and I could open a new ticket with another milestone, but I do think this should be part of the daily sysadmin shifts, where I’ll be more pro-active!

#5 Updated by intrigeri 2016-03-22 11:07:13

> I’ve reset our puppet-concat repo to upstream tag 1.0.0, deployed that on a bunch of VMs, mostly the one which had the bug + puppet-git, and it does seem to resolve this issue, yay.

Great! I’ll double-check later this week: I’ll be running puppet agent --test by hand a lot, and this constant Puppet churn was what highlighted the problem last time.

> However, that’s quite a transition, we are quite outdated on this module that had many changes since then. I clearly won’t have time to spend before end of SponsorS_M4, and I could open a new ticket with another milestone, but I do think this should be part of the daily sysadmin shifts, where I’ll be more pro-active!

I’m confused: earlier in the same comment you wrote that yo actually did that transition.

#6 Updated by intrigeri 2016-03-22 11:15:44

Subject changed from Investigate why puppet-concat module output is not consistent to Make puppet-concat -generated Icinga2 config content sorted deterministically

#7 Updated by bertagaz 2016-03-23 14:33:25

Subject changed from Make puppet-concat -generated Icinga2 config content sorted deterministically to Make puppet-concat generated Icinga2 config content sorted deterministically

intrigeri wrote:
> > I’ve reset our puppet-concat repo to upstream tag 1.0.0, deployed that on a bunch of VMs, mostly the one which had the bug + puppet-git, and it does seem to resolve this issue, yay.
>
> Great! I’ll double-check later this week: I’ll be running puppet agent --test by hand a lot, and this constant Puppet churn was what highlighted the problem last time.

Should work, didn’t see that problem since then.

> > However, that’s quite a transition, we are quite outdated on this module that had many changes since then. I clearly won’t have time to spend before end of SponsorS_M4, and I could open a new ticket with another milestone, but I do think this should be part of the daily sysadmin shifts, where I’ll be more pro-active!
>
> I’m confused: earlier in the same comment you wrote that yo actually did that transition.

Not completely, I just updated to the puppet-concat version that contained the commit we wanted. Going further in the module releases is a bit more complicated, as it had quite important changes iirc.

#8 Updated by intrigeri 2016-03-24 15:57:22

Assignee changed from intrigeri to bertagaz
QA Check changed from Ready for QA to Dev Needed

It seems that the problem hasn’t been fully solved, see on apt.lizard: cd /etc && sudo git log -p icinga2/zones.conf.

#9 Updated by intrigeri 2016-03-25 13:29:41

Same problem on apt-proxy.lizard.

#10 Updated by intrigeri 2016-03-25 17:20:01

commit 64983293c3ef5247e0611a90912a44005757853d in our puppet-concat might help, since the sort is locale-dependent (and the attempt at making it locale-independent is buggy). Note that current upstream master handles this totally differently, so their new stuff might fix the bug, or make it worse.

#11 Updated by intrigeri 2016-03-25 17:49:24

>> I’m confused: earlier in the same comment you wrote that yo actually did that transition.

> Not completely, I just updated to the puppet-concat version that contained the commit we wanted.

OK, got it.

#12 Updated by bertagaz 2016-04-26 05:00:07

Target version changed from Tails_2.3 to Tails_2.4

#13 Updated by bertagaz 2016-05-10 08:00:17

Target version changed from Tails_2.4 to Tails_2.5

#14 Updated by bertagaz 2016-07-19 04:17:26

Target version changed from Tails_2.5 to Tails_2.6

#15 Updated by anonym 2016-09-20 16:54:05

Target version changed from Tails_2.6 to Tails_2.7

#16 Updated by bertagaz 2016-09-22 05:47:03

Target version changed from Tails_2.7 to Tails_2.9.1

#17 Updated by intrigeri 2016-09-22 06:31:18

FYI concat 2.2.0 sorts stuff differently (it uses a Ruby implementation instead of the previous shell script). See ~~Bug #11823~~ for the progress I made towards this upgrade.