Feature #9350

Jenkins workspaces for disabled jobs are not removed from ISO builders

Added by intrigeri 2015-05-06 10:05:39 . Updated 2015-08-26 06:10:21 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Tails_1.4.1
Start date:
2015-05-06
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:
266

Description

See /var/lib/jenkins/workspace on isobuilder{1,2}.lizard. This will grow out of control once we build all active branches. There’s surely a Jenkins config pref or plugin to handle that. And worst case, we can simply cron the removal of all workspaces that are not being used by a currently running build (the workspace directory is kept open for the entire build duration, as soon as build-tails is started, apparently — but perhaps there’ll be race conditions wrt. what happens before or after, so it would probably be more robust to implement this in another way).

Subtasks

Related issues

Related to Tails - Bug #8912: Nightly built artifacts for disabled jobs are not removed Resolved 2015-02-17
Blocks Tails - Feature #8658: Deploy the "build all active branches" system Resolved 2015-01-09

History

#1 Updated by intrigeri 2015-05-06 10:05:50

  • related to Bug #8912: Nightly built artifacts for disabled jobs are not removed added

#2 Updated by intrigeri 2015-05-06 10:30:06

  • Description updated

#3 Updated by intrigeri 2015-05-15 14:42:42

  • blocks Feature #8658: Deploy the "build all active branches" system added

#4 Updated by intrigeri 2015-05-29 11:38:03

  • blocks #8668 added

#5 Updated by bertagaz 2015-06-09 12:34:11

  • Assignee changed from bertagaz to intrigeri

When we last discussed about this topic, we thought about a script that would publish somewhere the current list of active jobs, list that would be retrieved by another script running on the builders to compare with the current list of job workspaces it has and delete those who don’t have a corresponding job.

While that’s still an option, there might be an easier way to remove this obsolete workspaces, more integrated into Jenkins: we have installed the workspace-cleanup plugin, but didn’t configure it. It enables us to setup a post-build step for each job that delete the workspace after the build. That way, workspaces would be deleted every time a job is finished, and there won’t be one left when the job is deleted. We already delete the workspaces at the beginning of the build to build from a clean tree, so that won’t change much I think.

If you agree, I’ll test this implementation first.

#6 Updated by bertagaz 2015-06-09 13:54:04

Correction: we do use the workspace-cleanup, but we did put it in the wrappers section of the job definition, where it should be in the publishers one. I’ll fix that ASAP, and it should fix this bug hopefully.

#7 Updated by intrigeri 2015-06-09 17:05:27

> If you agree, I’ll test this implementation first.

This sounds like an excellent idea to me.

#8 Updated by intrigeri 2015-06-10 15:13:36

  • Assignee changed from intrigeri to bertagaz

#9 Updated by bertagaz 2015-06-11 11:53:24

  • Status changed from Confirmed to In Progress
  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Ready for QA

I’ve fixed the workspace-cleanup plugin configuration in our jenkins-jobs repo. Pushed live and the workspaces are getting deleted after the builds ended as they should. Jobs that has run since then don’t have them left.

I’ve also fixed our generate_build_tails_iso_jobs script in our tails puppet module (commit 7265dba).

A way to test it is: delete all the workspaces on a builder and start a job on it. See if it remains after the job is completed.
Another one: find a job that has a workspace left, run it, check the workspace has been deleted.

This should fix this bug. If you agree, I’ll delete all the workspaces on the isobuilders.

#10 Updated by intrigeri 2015-06-11 20:41:43

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Ready for QA to Dev Needed

> I’ve fixed the workspace-cleanup plugin configuration in our jenkins-jobs repo. Pushed live and the workspaces are getting deleted after the builds ended as they should. Jobs that has run since then don’t have them left.

Excellent!

> This should fix this bug. If you agree, I’ll delete all the workspaces on the isobuilders.

I’ve confirmed that both a failed job and a successful one have their workspace deleted post-build => please go ahead and then mark this ticket as resolved :)

(Side note: apparently non-ISO-build jobs are not affected, but these have a way lower turnaround so IMO it’s fine this way.)

#11 Updated by bertagaz 2015-06-12 09:49:24

  • Status changed from In Progress to Resolved
  • Assignee deleted (bertagaz)
  • % Done changed from 0 to 100
  • QA Check deleted (Dev Needed)

intrigeri wrote:
> I’ve confirmed that both a failed job and a successful one have their workspace deleted post-build => please go ahead and then mark this ticket as resolved :)
>
> (Side note: apparently non-ISO-build jobs are not affected, but these have a way lower turnaround so IMO it’s fine this way.)

They are, but they have to be ran on the right builder that have a previous job workspace left to delete it for real. I’ve deleted all the remaining workspaces, and even the non iso building job don’t let their workspace after completion.

#12 Updated by intrigeri 2015-06-12 13:04:58

:)

#13 Updated by intrigeri 2015-08-26 06:10:21

  • Deliverable for set to 266