Bug #16158

Ensure that old Jenkins artifacts are cleaned up

Added by intrigeri 2018-11-28 07:27:13 . Updated 2018-12-17 17:06:22 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Tails_3.12
Start date:
2018-11-28
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:
316

Description

The naming scheme of artifacts has changed and the code to garbage collect them has been adjusted. Let’s check in a week or so that the new code still does its job on artifacts that predate these changes. Worst case, let’s clean them up manually.

Subtasks

History

#1 Updated by intrigeri 2018-12-04 10:58:25

  • Status changed from Confirmed to In Progress
  • Target version changed from Tails_3.11 to Tails_3.12
  • % Done changed from 0 to 50

I’ve looked at artifacts older than a week with find . -name *.iso -type f -mtime +7. Results seem to match our GC algo. I’ll take another look in a week or two.

In passing, I’ve seen artifacts dating back from May and August that should probably have been removed, so I think that our GC process has some bugs, regardless of the changes we did for this project.

#2 Updated by intrigeri 2018-12-17 17:06:23

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 50 to 100

intrigeri wrote:
> I’ve looked at artifacts older than a week with find . -name *.iso -type f -mtime +7. Results seem to match our GC algo. I’ll take another look in a week or two.

This time I’ve used find . \( -name *.iso -o -name *.img \) -a -type f -mtime +7 | sort which makes it easier to notice culprits. There were none in the area we touched as part of this project => closing!

> In passing, I’ve seen artifacts dating back from May and August that should probably have been removed, so I think that our GC process has some bugs, regardless of the changes we did for this project.

Confirmed… somewhat. I’ve found some images that should have been cleaned up already, regardless of the changes we made for this project. They’re all artifacts of reproducibly_build_Tails_ISO_ jobs, which are not managed by clean_old_jenkins_artifacts: they seem to be GC’ed by Jenkins own process (artifactDaysToKeep) which seems to work inconsistently, e.g. these ones are all older than 5 days so in my book they should have been deleted:

./reproducibly_build_Tails_ISO_devel/builds/2018-10-01_14-13-50/archive/build-artifacts/2/tails-amd64-devel-3.12-20181009T1420Z-51953efefe.iso
./reproducibly_build_Tails_ISO_feature-8514-7180-whisperback-remove-right-pane-and-refactor/builds/2018-08-15_17-54-55/archive/build-artifacts/2/tails-amd64-feature_8514-7180-whisperback-remove-right-pane-and-refactor-3.9-20180823T1759Z-adb771241b+devel@18b65bf2a6.iso
./reproducibly_build_Tails_ISO_feature-buster/builds/2018-11-22_11-17-23/archive/build-artifacts/2/tails-amd64-feature_buster-3.12-20181130T1120Z-09233368a3+devel@bfb341ab75.iso
./reproducibly_build_Tails_ISO_feature-buster/builds/2018-11-23_10-16-14/archive/build-artifacts/2/tails-amd64-feature_buster-3.12-20181201T1019Z-09233368a3+devel@bfb341ab75.iso
./reproducibly_build_Tails_ISO_feature-buster/builds/2018-11-24_10-19-58/archive/build-artifacts/2/tails-amd64-feature_buster-3.12-20181202T1023Z-09233368a3+devel@bfb341ab75.iso
./reproducibly_build_Tails_ISO_feature-buster/builds/2018-11-25_10-19-43/archive/build-artifacts/2/tails-amd64-feature_buster-3.12-20181203T1023Z-09233368a3+devel@bfb341ab75.iso
./reproducibly_build_Tails_ISO_feature-buster/builds/2018-11-26_10-20-03/archive/build-artifacts/2/tails-amd64-feature_buster-3.12-20181204T1023Z-09233368a3+devel@bfb341ab75.iso

But that’s off-topic here because at least the ones from August and October should have been cleaned up way before we changed anything for the USB image project. It might be caused by yet another race condition between deleting jobs and other operations (in this case, the GC process).