Bug #11543: bitcoin.lizard cannot connect to our Puppet master

Bug #11543

bitcoin.lizard cannot connect to our Puppet master

Added by bertagaz 2016-06-23 05:50:40 . Updated 2016-09-24 04:53:49 .

Status:

Resolved

Priority:

High

Assignee:

Category:

Infrastructure

Target version:

Tails_2.7

Start date:

2016-06-23

Due date:

% Done:

100%

Feature Branch:

Type of work:

Sysadmin

Blueprint:

Starter:

Affected tool:

Deliverable for:

270

Description

We’ve received reports since a while that the puppet agent runs on bitcoin.lizard fail:

~$ sudo puppet agent --test
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Find /puppet/v3/node/bitcoin.lizard?environment=production&transaction_uuid=935f55d0-c55c-4d0e-9cf4-0b... \
resulted in 404 with the message: Not Found: Could not find environment 'puppet'
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///pluginfacts
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///plugins
Notice: /File[/var/lib/puppet/lib/facter]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/facter]: Skipping because of failed dependencies
Notice: /File[/var/lib/puppet/lib/facter/acpi_available.rb]: Dependency File[/var/lib/puppet/lib] has failures: true
[...]
Warning: /File[/var/lib/puppet/lib/puppet/parser/functions/any2array.rb]: Skipping because of failed dependencies
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/augeas.rb]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/puppet/parser/functions/augeas.rb]: Skipping because of failed dependencies
[...]
Notice: /File[/var/lib/puppet/lib/puppet/provider]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/puppet/provider]: Skipping because of failed dependencies
Notice: /File[/var/lib/puppet/lib/puppet/provider/file_line]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/puppet/provider/file_line]: Skipping because of failed dependencies
Notice: /File[/var/lib/puppet/lib/puppet/provider/file_line/ruby.rb]: Dependency File[/var/lib/puppet/lib] has failures: true
[...]
Notice: /File[/var/lib/puppet/lib/puppet/type]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/puppet/type]: Skipping because of failed dependencies
Notice: /File[/var/lib/puppet/lib/puppet/type/anchor.rb]: Dependency File[/var/lib/puppet/lib] has failures: true
[...]
Notice: /File[/var/lib/puppet/lib/puppet/util]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/puppet/util]: Skipping because of failed dependencies
Notice: /File[/var/lib/puppet/lib/puppet/util/external_iterator.rb]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/puppet/util/external_iterator.rb]: Skipping because of failed dependencies
[...]
Info: Loading facts
Error: Could not retrieve catalog from remote server: Find /puppet/v3/catalog/bitcoin.lizard?environment=production&facts_format=pson&facts=%257B%2522name%2... \
resulted in 404 with the message: Not Found: Could not find environment 'puppet'
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

It may be that the recent upgrade to puppet 4.x in Sid introduced this.

Subtasks

Related issues

Related to Tails - ~~Feature #11834~~: Migrate our infrastructure to Puppet 4

Resolved

2016-09-24

History

#1 Updated by intrigeri 2016-07-04 03:10:26

Subject changed from Sid puppet agent not compatible with Jessie puppet master to bitcoin.lizard cannot connect to our Puppet master
Priority changed from Normal to Elevated
Deliverable for set to 270

Note that Puppet 4 agents are not compatible with Puppet 3 masters, and we’re nowhere ready to upgrade our master to Puppet 4, so it seems to me that we need:

1. a short-term solution for our sid VM; options:

downgrade to Puppet 3 on bitcoin.lizard (that would be my preferred option I think)
maintain a private backport of bitcoind, and reinstall that VM on Jessie (given how we handle upgrading stuff like our Puppet modules and Jenkins plugins, I’m not confident in this solution, at least not in the current state of our sysadmin duty work sharing)
kill that service and let the accounting team handle our bitcoin themselves
others?

IMO this is the part that should be solved ASAP (I don’t know why this wasn’t done during your 4 weeks of sysadmin duty in June, but whatever: 2.5 target version is good enough; bumping priority though because this is now more than 1 month old, and setting the relevant “deliverable for”).

2. a migration plan to Puppet 4, that will take quite a bit longer (and some bits should be done before the Stretch freeze, i.e. about now)

our manifests and modules need to be ported (I would start with enabling the “future” parser)
we need to push puppetdb-termini to Debian (see the puppetdb RFP for details)
we need to handle the fact that we won’t upgrade everything to Stretch at the same time
https://fnord.no/2016/05/28/puppet4-debian-unstable/ has useful info for the migration

If you agree, then I’ll create tickets to track the migration plan, and in parallel we can discuss the short-term fix that this ticket is about (retitling to make it more specific). OK?

#2 Updated by bertagaz 2016-08-02 03:46:33

Target version changed from Tails_2.5 to Tails_2.6

#3 Updated by intrigeri 2016-08-05 07:54:38

Priority changed from Elevated to High

(This problem is now more than two months old, so bumping priority to make it show up higher than less critical stuff on your radar.)

#4 Updated by bertagaz 2016-09-19 04:06:12

Status changed from Confirmed to In Progress
% Done changed from 0 to 20

intrigeri wrote:
> 1. a short-term solution for our sid VM; options:
>
> * downgrade to Puppet 3 on bitcoin.lizard (that would be my preferred option I think)
> * maintain a private backport of bitcoind, and reinstall that VM on Jessie (given how we handle upgrading stuff like our Puppet modules and Jenkins plugins, I’m not confident in this solution, at least not in the current state of our sysadmin duty work sharing)
> * kill that service and let the accounting team handle our bitcoin themselves
> * others?
>
> IMO this is the part that should be solved ASAP (I don’t know why this wasn’t done during your 4 weeks of sysadmin duty in June, but whatever: 2.5 target version is good enough; bumping priority though because this is now more than 1 month old, and setting the relevant “deliverable for”).

So I’ve downgraded bitcoin’s Puppet to 3.x a while ago. There was still an email sent per Puppet run because of some warnings, but it seems to have disappeared lately.

> 2. a migration plan to Puppet 4, that will take quite a bit longer (and some bits should be done before the Stretch freeze, i.e. about now)
>
> * our manifests and modules need to be ported (I would start with enabling the “future” parser)
> * we need to push puppetdb-termini to Debian (see the puppetdb RFP for details)
> * we need to handle the fact that we won’t upgrade everything to Stretch at the same time
> * https://fnord.no/2016/05/28/puppet4-debian-unstable/ has useful info for the migration
>
> If you agree, then I’ll create tickets to track the migration plan, and in parallel we can discuss the short-term fix that this ticket is about (retitling to make it more specific). OK?

Agreed.

#5 Updated by anonym 2016-09-20 16:54:11

Target version changed from Tails_2.6 to Tails_2.7

#6 Updated by intrigeri 2016-09-24 04:31:11

> intrigeri wrote:
>> 1. a short-term solution for our sid VM; options:
>>
>> * downgrade to Puppet 3 on bitcoin.lizard (that would be my preferred option I think)
>> […]

> So I’ve downgraded bitcoin’s Puppet to 3.x a while ago.

Great!

> There was still an email sent per Puppet run because of some warnings, but it seems to have disappeared lately.

As said elsewhere a few days ago: that’s because I’ve fixed it manually (and told you over email).

>> 2. a migration plan to Puppet 4, that will take quite a bit longer (and some bits should be done before the Stretch freeze, i.e. about now)
>> […]
>> If you agree, then I’ll create tickets to track the migration plan, and in parallel
>> we can discuss the short-term fix that this ticket is about (retitling to make it
>> more specific). OK?

> Agreed.

Then I’ll go ahead with the ticket fun.

#7 Updated by intrigeri 2016-09-24 04:33:03

Assignee changed from bertagaz to intrigeri

#8 Updated by intrigeri 2016-09-24 04:45:03

related to ~~Feature #11834~~: Migrate our infrastructure to Puppet 4 added

#9 Updated by intrigeri 2016-09-24 04:53:33

intrigeri wrote:
> Then I’ll go ahead with the ticket fun.

Please have a look at ~~Feature #11834~~ and children, and check that it makes sense.

#10 Updated by intrigeri 2016-09-24 04:53:49

Status changed from In Progress to Resolved
Assignee deleted (~~intrigeri~~)
% Done changed from 20 to 100