Bug #16025

jenkins-data-disk is running out of diskspace again

Added by bertagaz 2018-10-03 09:33:42 . Updated 2019-03-20 15:54:47 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
Start date:
2018-10-03
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

We’re received notifications that the jenkins-data LV is running out of space.

Subtasks

Related issues

Related to Tails - Feature #15780: Update server storage planning needs for 2019-2020 Rejected 2018-08-09
Related to Tails - Feature #9400: Evaluate future storage needs on lizard Resolved 2015-05-14
Related to Tails - Bug #14944: jenkins-data-disk is running out of diskspace Resolved 2017-11-09
Related to Tails - Bug #12119: Shrink lizard's jenkins-data LV Resolved 2017-01-07
Blocks Tails - Feature #13242: Core work: Sysadmin (Maintain our already existing services) Confirmed 2017-06-29

History

#1 Updated by bertagaz 2018-10-03 09:34:26

  • related to Feature #15780: Update server storage planning needs for 2019-2020 added

#2 Updated by bertagaz 2018-10-03 09:36:18

  • related to Feature #9400: Evaluate future storage needs on lizard added

#3 Updated by bertagaz 2018-10-03 09:36:56

  • related to Bug #14944: jenkins-data-disk is running out of diskspace added

#4 Updated by bertagaz 2018-10-03 09:48:34

  • related to #15779 added

#5 Updated by bertagaz 2018-10-03 09:56:23

So we had 28 build jobs running at the time this alert was raised, which explains why it happened. In Feature #9400, we calculated we needed 500G for 30 jobs, but then later we shrinked this LV to 300G (iirc because we didn’t use that much in reality). But it seems our math were quite good in the end.

I’ve updated the spreadsheet for Feature #15780 with the current datas. It seems we have some room left in term of storage (and we already have plans for the future) given we’ve been quite conservative with the storage needs for the APT snapshots. So I propose we grow this LV, from 50G as a first step. We’ll see if we reach this limit at some point.

#6 Updated by bertagaz 2018-10-03 09:56:43

  • related to Bug #12119: Shrink lizard's jenkins-data LV added

#7 Updated by bertagaz 2018-10-03 09:57:14

  • blocks Feature #13242: Core work: Sysadmin (Maintain our already existing services) added

#8 Updated by intrigeri 2018-10-10 21:19:06

> So I propose we grow this LV, from 50G as a first step.

Yes, we can totally afford it and hopefully it’ll be enough (and once we have the new drives set up, hopefully by the end of the month, we’ll have plenty of margin).
Please make sure to not allocate logical extents on the rotating drives because we’ll unplug them soon.

#9 Updated by intrigeri 2018-10-16 11:27:26

FYI this is a busy week for the Foundations Team with lots of branches to prepare, review and merge. Some sponsor deliverables teams are also struggling to get fixes in 3.10.. So, unless disk space issues come back, it would be nice to keep Jenkins nicely working until 3.10 is out and to postpone maintenance operations that require shutting it down. And in passing, one trick one can exceptionally use to bring disk usage down in case of emergency is to delete old ISOs for doc/* and web/* branches (there’s little chance one needs these ISO images), which I did when the 3.9.1 release process was blocked due to this problem.

#10 Updated by bertagaz 2018-10-17 10:52:59

intrigeri wrote:
> FYI this is a busy week for the Foundations Team with lots of branches to prepare, review and merge. Some sponsor deliverables teams are also struggling to get fixes in 3.10..

Ok, I’ll hold down for after the release then.

#11 Updated by intrigeri 2019-03-20 15:54:47

  • Status changed from Confirmed to Resolved
  • Assignee deleted (bertagaz)
  • % Done changed from 0 to 100

Looks like someone fixed the problem already.