Feature #12576

Have Jenkins use basebox:clean_old instead of basebox:clean_all

Added by intrigeri 2017-05-22 07:48:41 . Updated 2017-10-08 03:58:40 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Tails_3.3
Start date:
2017-05-22
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:
301

Description

See parent ticket for details.

Subtasks

Related issues

Related to Tails - Feature #12002: Estimate hardware cost of reproducible builds in Jenkins Resolved 2016-11-28
Related to Tails - Feature #14811: Check if old vagrant baseboxes are cleaned up in Jenkins Resolved 2017-10-07
Blocked by Tails - Bug #12595: Not enough space in /var/lib/jenkins on isobuilders Resolved 2017-05-25
Blocked by Tails - Bug #12575: Fix basebox:clean_old Resolved 2017-05-22

History

#1 Updated by intrigeri 2017-05-22 07:48:50

  • blocked by Bug #12575: Fix basebox:clean_old added

#2 Updated by intrigeri 2017-05-24 07:01:01

  • blocks Feature #12002: Estimate hardware cost of reproducible builds in Jenkins added

#3 Updated by bertagaz 2017-05-26 13:14:14

  • blocked by Bug #12595: Not enough space in /var/lib/jenkins on isobuilders added

#4 Updated by bertagaz 2017-05-26 13:14:47

  • blocked by Bug #12599: /var/lib/libvirt/images gets filled on isobuilders added

#5 Updated by intrigeri 2017-05-27 08:57:45

  • Priority changed from High to Elevated

See notes 14 to 25 on Bug #12531, where there’s a discussion about how exactly we can make use of clean_old on Jenkins.

Also, downgrading severity a little bit: it’s important to address this build performance regression, but please first focus on what actually breaks builds (Bug #12531 and subtasks), and if this one has to be postponed to shortly after the 3.0 release, so be it (I’d rather not see this change be deployed to late in this cycle unless it’s been heavily tested locally first: early June won’t be a good time to break our CI, at all ;)

#6 Updated by intrigeri 2017-05-27 08:58:14

  • blocks deleted (Bug #12575: Fix basebox:clean_old)

#7 Updated by intrigeri 2017-05-27 08:59:36

#8 Updated by intrigeri 2017-05-27 09:00:05

  • blocked by deleted (Feature #12002: Estimate hardware cost of reproducible builds in Jenkins)

#9 Updated by intrigeri 2017-05-27 09:00:08

  • blocks deleted (Bug #12595: Not enough space in /var/lib/jenkins on isobuilders)

#10 Updated by intrigeri 2017-05-27 09:00:12

  • blocks deleted (Bug #12599: /var/lib/libvirt/images gets filled on isobuilders)

#11 Updated by intrigeri 2017-05-27 09:00:26

#12 Updated by intrigeri 2017-05-27 09:00:41

  • blocks Feature #12002: Estimate hardware cost of reproducible builds in Jenkins added

#13 Updated by intrigeri 2017-05-27 09:00:53

  • blocked by Bug #12595: Not enough space in /var/lib/jenkins on isobuilders added

#14 Updated by intrigeri 2017-05-27 09:01:08

  • blocked by Bug #12599: /var/lib/libvirt/images gets filled on isobuilders added

#15 Updated by intrigeri 2017-05-27 09:01:23

  • blocked by Bug #12575: Fix basebox:clean_old added

#16 Updated by bertagaz 2017-05-30 16:28:13

  • Status changed from Confirmed to In Progress

Applied in changeset commit:fc1bba108640e6057bcd0a34f052de1aebab5461.

#17 Updated by intrigeri 2017-05-30 17:09:10

Note: this should only be deployed once all active branches have the fix for clean_old. Perhaps the easiest way at this point is to wait for the 3.0 release, unless it’s still reasonably doable to apply all recent build system fixes on the stable branch (I think I’ve seen a bunch of changes pushed to testing/devel only).

#18 Updated by anonym 2017-06-01 00:34:15

intrigeri wrote:
> Note: this should only be deployed once all active branches have the fix for clean_old. Perhaps the easiest way at this point is to wait for the 3.0 release, unless it’s still reasonably doable to apply all recent build system fixes on the stable branch (I think I’ve seen a bunch of changes pushed to testing/devel only).

Perhaps we instead should just consider Tails 2.x, and temporarily disable CI for the stable branch (and make sure no other active branch has it as base branch), until 3.0 is released? That way we this blocker is gone.

Also, in case it helps (I don’t have the full picture of what the problem on Jenkins is), note that I could easily decrease the disk space needed for generating the basebox by reducing the basebox’ image size from 20GB to 15GB, or whatever that is enough to do a disk build (or even down to 5 GB or so if we decide to not support disk builds until we have a real fix…).

#19 Updated by intrigeri 2017-06-01 06:37:47

> Perhaps we instead should just consider Tails 2.x, and temporarily disable CI for the stable branch (and make sure no other active branch has it as base branch), until 3.0 is released? That way we this blocker is gone.

Right, this would work, but IMO improving performance of our CI environment 2 weeks earlier is not worth taking the risk of having no CI in case we want or have to release another 2.x. See Feature #12576#note-5 for my thoughts about the priority and timeline of this ticket.

> Also, in case it helps (I don’t have the full picture of what the problem on Jenkins is), note that I could easily decrease the disk space needed for generating the basebox […]

Thanks for the offer. I don’t think this would unblock anything fundamentally (it likely won’t be enough to allow us to store N baseboxes) but let’s keep it in mind in case we’re a little bit short on storage, and still lack data to purchase additional storage, when we address Bug #12595 which blocks this ticket anyway: I hope not, but saving 4 * a few GB might be just what we need to drop a blocker once we’re there.

#20 Updated by intrigeri 2017-06-01 06:41:21

  • blocked by deleted (Feature #12002: Estimate hardware cost of reproducible builds in Jenkins)

#21 Updated by intrigeri 2017-06-01 06:41:44

  • related to Feature #12002: Estimate hardware cost of reproducible builds in Jenkins added

#22 Updated by bertagaz 2017-06-01 10:39:53

intrigeri wrote:
> Note: this should only be deployed once all active branches have the fix for clean_old. Perhaps the easiest way at this point is to wait for the 3.0 release, unless it’s still reasonably doable to apply all recent build system fixes on the stable branch (I think I’ve seen a bunch of changes pushed to testing/devel only).

Indeed. I’ve investigated a bit what was not merged into stable, by looking at the diff between it and the testing branch in auto/, vagrant/ and Rakefile. There’s not so much: seems Bug #12556, Bug #12531, Bug #12525 and commit:505b31e28732f621795ac5aca4e0b9c65d5e4f02 are the only missing pieces (if I got it right, another look is welcome).

There are other differences (like the ntp/date setup in the build VM) which are related to reproducibility and can be skipped for the stable branch. The Vagrant build related differences I’ve listed above seem to be easy to merge or cherry-pick in stable given their simplicity.

Now I’m a bit undecided whether doing that now or right after 3.0. As you stated in #note-5, we’re not at the best moment to do that.

#23 Updated by intrigeri 2017-06-01 11:10:09

> Now I’m a bit undecided whether doing that now or right after 3.0. As you stated in #note-5, we’re not at the best moment to do that.

I’m in favour of having everything well tested and ready ASAP so it can be deployed first thing after the 3.0 release.

#24 Updated by bertagaz 2017-06-01 12:47:17

  • blocks deleted (Bug #12599: /var/lib/libvirt/images gets filled on isobuilders)

#25 Updated by intrigeri 2017-06-08 17:47:04

  • Target version changed from Tails_3.0 to Tails_3.1

I’ll build the 3.0 ISO in two days, so let’s not do potentially disruptive changes on our infra at this point.

#26 Updated by intrigeri 2017-06-25 15:22:25

  • Priority changed from Elevated to Normal

This will be nice but you have quite a few things with much higher prio to deal with.

#27 Updated by intrigeri 2017-07-06 14:52:28

  • Target version changed from Tails_3.1 to Tails_3.2

(Please focus on making builds robust again first, and postpone the performance improvements. Sorry we could not discuss this at the CI team meeting today.)

#28 Updated by intrigeri 2017-07-18 12:22:30

  • Deliverable for changed from 289 to 301

#29 Updated by intrigeri 2017-09-07 11:13:42

  • Target version changed from Tails_3.2 to Tails_3.3

I doesn’t feel realistic that all the dependency chain (Bug #12595 and Bug #13425) is resolved in time for 3.2.

#30 Updated by intrigeri 2017-10-06 05:05:36

FWIW on my local Jenkins I have done this yesterday night:

--- a/macros/builders.yaml
+++ b/macros/builders.yaml
@@ -39,8 +39,7 @@
 - builder:
     name: clean_old_baseboxes
     builders:
-      - shell: "rake basebox:clean_all"
-      - shell: "rm -rf /var/lib/jenkins/.vagrant.d"
+      - shell: "rake basebox:clean_old"

 - builder:
     name: move_artifacts_to_dir_1

If I notice anything weird or broken, I’ll report back here.

#31 Updated by bertagaz 2017-10-07 15:25:21

  • related to Feature #14811: Check if old vagrant baseboxes are cleaned up in Jenkins added

#32 Updated by bertagaz 2017-10-07 15:28:43

  • Status changed from In Progress to Resolved
  • Assignee deleted (bertagaz)
  • % Done changed from 0 to 100

intrigeri wrote:
> If I notice anything weird or broken, I’ll report back here.

Yes, I’ve just enabled the basebox:clean_old rake task in Jenkins. I’ve done basic testing, and the basebox is no longer deleted, build is not broken. I’ve created Feature #14811 to check if baseboxes older than 4 monthes are deleted or not. Closing this ticket.

#33 Updated by intrigeri 2017-10-08 03:58:41

Woohoo!