Bug #15647: PuppetDB is broken since 2018-06-10

Bug #15647

PuppetDB is broken since 2018-06-10

Added by intrigeri 2018-06-11 05:37:58 . Updated 2018-10-31 07:59:20 .

Status:

Resolved

Priority:

Elevated

Assignee:

Category:

Infrastructure

Target version:

Start date:

2018-06-11

Due date:

% Done:

100%

Feature Branch:

Type of work:

Sysadmin

Blueprint:

Starter:

Affected tool:

Deliverable for:

Description

Tons of errors in puppet-git:/var/log/puppetdb which fills the root FS there which in turns breaks the Git service which in turn breaks ISO builds.

Subtasks

Related issues

Related to Tails - ~~Feature #11834~~: Migrate our infrastructure to Puppet 4	Resolved	2016-09-24
Related to Tails - ~~Bug #15648~~: puppet-git needs more diskspace	Resolved	2018-06-11
Related to Tails - ~~Bug #16083~~: APT snapshot used for libjetty9 is unreliable	Resolved	2018-10-31
Blocks Tails - Feature #13284: Core work: Sysadmin (Adapt our infrastructure)	Confirmed	2017-06-30

History

#1 Updated by intrigeri 2018-06-11 05:49:28

Actually I think that PuppetDB was broken because there was not enough disk space to create its shared memory file and then it logged noisily about it which made the problem worse. Deleted the logs that were about this recursive problem, restarted PuppetDB, things are a bit better now but requests to PuppetDB still fail. Investigating.

Regardless, the root FS on puppet-git.lizard is too tight. I’ll file a dedicated ticket about it for groente or will recycle this one once I’m done.

#2 Updated by intrigeri 2018-06-11 06:28:26

Status changed from Confirmed to In Progress
% Done changed from 0 to 10

intrigeri wrote:
> requests to PuppetDB still fail. Investigating.

OK, so apparently one of the recent upgrades of Java packages we pin from sid broke stuff. Not a big surprise, it had to happen at some point. I’ve downgraded them all manually (added an APT source from snapshot.d.o and apt install libmail-java=1.6.1-2 libxz-java=1.8-1 libxalan2-java=2.7.2-1 libpostgresql-jdbc-java=9.4.1212-1 libxerces2-java=2.11.0-8 libderby-java=10.14.1.0-1 libjetty9-extra-java=9.2.23-1 libjetty9-java=9.2.23-1 libecj-java=3.11.1-2 libyaml-snake-java=1.20-1 libjoda-time-java=2.9.9-1 libactivemq-java=5.15.3-2) and things are back in order. I’ll now try to upgrade them one after the other in order to identify which one(s) we should freeze to older versions. And if/when that does not work anymore I think we’ll have to run Buster on this system.

#3 Updated by intrigeri 2018-06-11 06:45:39

Found it! Things start breaking when we upgrade libjetty9-java and libjetty9-extra-java from 9.2.23-1 to 9.2.24-1. I’ll freeze them to the older version.

#4 Updated by intrigeri 2018-06-11 06:55:27

Assignee changed from intrigeri to groente
% Done changed from 10 to 50
QA Check set to Ready for QA

intrigeri wrote:
> Found it! Things start breaking when we upgrade libjetty9-java and libjetty9-extra-java from 9.2.23-1 to 9.2.24-1. I’ll freeze them to the older version.

Done: https://git-tails.immerda.ch/puppet-tails/commit/manifests/puppetdb.pp?id=96c3d4ba15a9710e6350839307e551ff1a709da3

Also, please give yourself a ticket about growing the root FS on puppet-git.lizard.