Bug #11632

ISO builds from branch that need more RAM can break all our Jenkins isobuilders without us being notified

Added by intrigeri 2016-08-11 13:01:11 . Updated 2016-11-03 09:49:54 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Start date:
2016-08-11
Due date:
% Done:

0%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:
270

Description

I just had to restart the jenkins-slave service on 3 of our 4 isobuilders: it was down since yesterday because a feature/stretch build had triggered the OOM (by the way, this hints that we need an Icinga check that asks systemd if everything is running all-right; please add a low prio ticket if you agree). This has the potential to essentially kill our CI system within a few days, hence high priority.

Subtasks

Related issues

Related to Tails - Bug #11858: Monitor if isobuilders systems are running fine Resolved 2016-10-03

History

#1 Updated by bertagaz 2016-09-19 04:08:17

  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Info Needed

Is that still the case? I can’t find that kind of bug in the recent Jenkins history, but maybe I’m missing something.

#2 Updated by intrigeri 2016-09-19 06:26:44

  • Subject changed from ISO builds from feature/stretch trigger the OOM on Jenkins isobuilders to ISO builds from branch that need more RAM can break all our Jenkins isobuilders without us being notified
  • Assignee changed from intrigeri to bertagaz
  • Priority changed from High to Normal
  • Target version changed from Tails_2.6 to Tails_2.7
  • QA Check deleted (Info Needed)

bertagaz wrote:
> Is that still the case? I can’t find that kind of bug in the recent Jenkins history, but maybe I’m missing something.

Right, I “solved” this by changing things on feature/stretch so that less memory is required. Still, we have a problem. Let me rephrase it: “any branch that needs more RAM than usual to build can kill our CI system in a few days without us being clearly notified of it”. I’ve proposed a way to handle this problem, in the ticket description. Another way would be to reboot the isobuilders before each build, but I guess this is overkill given the monitoring check seems pretty cheap a (just good enough) workaround. What do you think?

#3 Updated by bertagaz 2016-10-03 08:43:04

  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Info Needed

intrigeri wrote:
> Right, I “solved” this by changing things on feature/stretch so that less memory is required. Still, we have a problem. Let me rephrase it: “any branch that needs more RAM than usual to build can kill our CI system in a few days without us being clearly notified of it”. I’ve proposed a way to handle this problem, in the ticket description. Another way would be to reboot the isobuilders before each build, but I guess this is overkill given the monitoring check seems pretty cheap a (just good enough) workaround. What do you think?

Alright, I agree a Icinga2 check would be much more relevant (cost/benefit ratio wise) than rebooting isobuilders. Created Bug #11858, do you want to take it or should I?

#4 Updated by intrigeri 2016-10-03 09:01:10

  • related to Bug #11858: Monitor if isobuilders systems are running fine added

#5 Updated by intrigeri 2016-10-03 09:02:08

  • Status changed from Confirmed to Resolved

> Created Bug #11858, do you want to take it or should I?

Please go ahead, it’ll fit nicely within your next few sysadmin shifts.

#6 Updated by intrigeri 2016-10-03 09:05:02

  • Assignee deleted (intrigeri)
  • Target version deleted (Tails_2.7)
  • QA Check deleted (Info Needed)

#7 Updated by intrigeri 2016-11-03 09:49:54

  • Deliverable for set to 270