Feature #11833

Make our Puppet code compatible with the "future" parser

Added by intrigeri 2016-09-24 04:34:26 . Updated 2018-05-02 16:40:27 .

Status:
Resolved
Priority:
Normal
Assignee:
intrigeri
Category:
Infrastructure
Target version:
Tails_3.7
Start date:
2016-09-24
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

This will be the first necessary step towards migrating to Puppet 4.

Subtasks

Related issues

Blocked by Tails - Feature #11835: Upgrade Puppet master and clients to 3.8 Resolved 2016-09-24
Blocks Tails - Feature #13284: Core work: Sysadmin (Adapt our infrastructure) Confirmed 2017-06-30
Blocked by Tails - Feature #11836: Stop stringifying Puppet facts Resolved 2016-09-24

History

#1 Updated by intrigeri 2016-09-24 04:44:22

#2 Updated by intrigeri 2016-09-24 04:46:33

  • blocked by deleted (Feature #11834: Migrate our infrastructure to Puppet 4)

#3 Updated by intrigeri 2016-09-24 04:46:41

#4 Updated by intrigeri 2016-09-24 04:48:33

#5 Updated by intrigeri 2016-09-24 04:50:41

  • blocked by Feature #11835: Upgrade Puppet master and clients to 3.8 added

#6 Updated by intrigeri 2016-09-24 04:52:19

  • Assignee deleted (intrigeri)

#7 Updated by intrigeri 2017-04-09 11:02:09

  • Assignee set to intrigeri

#8 Updated by intrigeri 2017-06-05 13:39:33

  • Target version set to Tails_3.5

#9 Updated by intrigeri 2018-01-09 22:28:05

My plan (based on https://docs.puppet.com/puppet/4.5/upgrade_major_pre.html#enable-the-future-parser-and-fix-broken-code) is to set up a Puppet environment (cloned from our production one) with the future parser enabled and test each of our systems with it, one after the other, so that I have an initial list of problems I shall fix. I’ll minimize how long each system is managed by this test environment. Then I’ll fix these issues, push to that test environment, rince and repeat until I can enable the future parser on our current Puppet 3.x production environment, which should ensure we don’t write new code that’s not compatible with it.

#10 Updated by intrigeri 2018-01-09 23:00:37

  • blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#11 Updated by intrigeri 2018-01-23 16:47:28

  • Target version changed from Tails_3.5 to Tails_3.6

(See the parent ticket for the big picture & planning info.)

#12 Updated by intrigeri 2018-02-13 11:32:42

  • Target version changed from Tails_3.6 to Tails_3.7

#13 Updated by intrigeri 2018-04-04 13:11:49

#14 Updated by intrigeri 2018-04-04 13:24:11

intrigeri wrote:
> My plan (based on https://docs.puppet.com/puppet/4.5/upgrade_major_pre.html#enable-the-future-parser-and-fix-broken-code) is to set up a Puppet environment (cloned from our production one) with the future parser enabled and test each of our systems with it, one after the other, so that I have an initial list of problems I shall fix.

After improving our puppet-sync setup to support multiple environments, I realized there’s an easier way: run puppet agent --test --parser future on each system. Done everywhere and it works but the future parser is only partially used (types are not enforced) as long as we have stringify_facts = true, which is why https://docs.puppet.com/puppet/4.5/upgrade_major_pre.html recommends to set this to false (Feature #11836) before trying the future parser.

#15 Updated by intrigeri 2018-04-04 16:21:35

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 50

Deployed on all systems, seems to work.

#16 Updated by intrigeri 2018-04-05 08:05:10

  • Assignee changed from intrigeri to groente
  • QA Check set to Ready for QA

The “future” parser is now enabled for all our systems; we also now have stringify_facts = false everywhere, which is a prerequisite for the type checks performed by the “future” parser to be fully enabled.

I had to adjust quite a few bits in our manifests and in the tails module. Other modules were already fine thanks to the collective effort the shared Puppet modules group has put into making them compatible with Puppet 4 (including those I ported myself, see Feature #11834#note-17). I occasionally had to upgrade to the latest upstream version of a module, or to merge the “future” branch of a shared module (that’s the branch we use there in case the changes required for Puppet 4 compatibility don’t work without the future parser enabled), or to cherry-pick specific Puppet 4 compatibility fixes without upgrading to the latest upstream code (in the rare case when the upstream master branch only supports Puppet 4+).

All changes are tracked in our manifests repo => please review 415b6acf752831ef69e52dc42f9ea0508e8a2516..5ea8b7681089e40506d4e5e171d1ed2e2c514546 (and of course the 13 submodule updates referenced by these commits). https://docs.puppet.com/puppet/4.5/upgrade_major_pre.html#enable-the-future-parser-and-fix-broken-code is a good starting point to learn about the “future” parser; it should explain most, if not all, of the changes I had to apply.

The only remaining problem AFAICT is a dependency issue, see https://git-tails.immerda.ch/puppet-tails/commit/?id=d07bfba43f39deaffce08b49b1a68dca4c24a22a, https://git-tails.immerda.ch/puppet-tails/commit/?id=59fb58f7e1a2f613578f7e028b81e44d68d33931, and https://git-tails.immerda.ch/puppet-apt/commit/?id=edeab55358e73cb9cab42ec6cbe34b35dbbd7ef6. I’ve requested help about this on https://gitlab.com/shared-puppet-modules-group/apt/issues/26 and will implement whatever better solution I’m recommended there. I don’t think it’s a blocker nor worth a dedicated ticket: worst case we have to run Puppet one more time, during the initial setup of a system, until things converge; our stuff has never converged in one single run anyway. If you feel differently, I’m all ears :)

If anything is unclear / if you have any question, let me know.

#17 Updated by groente 2018-05-02 13:53:30

  • Assignee changed from groente to intrigeri
  • % Done changed from 50 to 80
  • QA Check changed from Ready for QA to Info Needed

the reprepro manifests have a lot of integers that are now declared as strings, particularly at commits

  • e3a073a3fedad32c0f2a539d13a259f5f16e54f5
  • 0839691743b2a1f193ef27b0a33607accc0d7fad
  • aa4a13b5b54b0c3b3109f6b43e4304e17453eaa5
  • 7563d33379e92dce6eb255e2502903e4fda84337
  • 4d65663061f818be20fa04a6c0b781f536e98318
  • 6d36bc8549885001e9dbb0c7b49cb5b6b0e0c43e

is there a particular reason these should be strings?

#18 Updated by groente 2018-05-02 13:54:35

  • blocked by deleted (Feature #11837: Upgrade Puppet master to Puppet 4)

#19 Updated by intrigeri 2018-05-02 15:04:41

  • Assignee changed from intrigeri to groente
  • QA Check changed from Info Needed to Ready for QA

> the reprepro manifests have a lot of integers that are now declared as strings, particularly at commits

All these were fixed later and are not strings anymore in current puppet-tails.git: they now have proper data types and often stronger validation constraints than what we used to have. For details, see git log -p --grep "Use more appropriate data types" :)

#20 Updated by groente 2018-05-02 16:40:28

  • Status changed from In Progress to Resolved
  • Assignee changed from groente to intrigeri
  • % Done changed from 80 to 100
  • QA Check changed from Ready for QA to Pass

okay, that was a bit too narrow a focus on that set of commits, thanks for clearing it up :)