Bug #10123

Autobuild ISOs cleanup loses all history for topic branches post-release

Added by intrigeri 2015-08-30 10:34:44 . Updated 2015-12-15 23:04:23 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Start date:
2015-08-30
Due date:
% Done:

100%

Feature Branch:
pythonlib:bugfix/10123-dont-lose-all-history-for-topic-branches-post-release
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

The way we currently clean up autobuilt ISOs deletes all topic branches’ artifacts and Jenkins job history after each release, presumably because they’re not considered to be “active” anymore, and thus their Jenkins job is deleted, and in turn the corresponding artifacts are deleted (Bug #8912). That’s problematic because it makes it harder to track down regressions in a given topic branch, that would have been introduced by updates elsewhere in our source code. If we see the same behavior the Jenkins history of test suite jobs in the future, we may also lose valuable data there.


Subtasks


Related issues

Related to Tails - Bug #8912: Nightly built artifacts for disabled jobs are not removed Resolved 2015-02-17

History

#1 Updated by intrigeri 2015-08-30 10:34:55

  • related to Bug #8912: Nightly built artifacts for disabled jobs are not removed added

#2 Updated by intrigeri 2015-08-30 10:40:52

  • Assignee set to bertagaz
  • QA Check set to Info Needed

I’m not sure it’s a real blocker, I can live with “let’s see what happens for a while”, but I already find it a bit painful to have to push something to my WIP branches (e.g. after an emergency release) so that Jenkins adds them back to its autobuild setup, and so that in turn autobuild ISOs are made available again. Granted, if I’m not pushing anything to them they’re not under active development, but:

  • I might prioritize them higher if I notice that they suddendly start FTBFS’ing;
  • we sometimes pass calls for testing and point people to autobuilt ISOs; when this happens, presumably the corresponding work is basically done, and the branch developer has little reasons to push stuff to the topic branch; in the current situation, there’s a window during which said autobuilt ISOs are not available anymore.

Perhaps a simple tweak to our definition of active branches would be enough to make most of these concerns moot, e.g. “that are not merged into any base branch, and have had changes in Git in the last N weeks”. At first glance it sounds quite trivial to implement.

bertagaz, what do you think? (assigning to you merely to get your opinion, just clear assignee once you’ve replied)

#3 Updated by intrigeri 2015-10-16 02:19:08

  • Target version set to Tails_1.8

(Setting a milestone to put this more clearly on your radar.)

#4 Updated by bertagaz 2015-11-04 09:09:53

intrigeri wrote:
> Perhaps a simple tweak to our definition of active branches would be enough to make most of these concerns moot, e.g. “that are not merged into any base branch, and have had changes in Git in the last N weeks”. At first glance it sounds quite trivial to implement.
>
> bertagaz, what do you think? (assigning to you merely to get your opinion, just clear assignee once you’ve replied)

Ok, that sounds pretty relevant. The 1.7 release finished to convinced me. I agree we should change that to your proposal (might be that it was the initial one even in fact). I’ll keep the assignment for this ticket, as you say it’s pretty straightforward so I’ll take care of that.

#5 Updated by bertagaz 2015-11-04 09:10:22

  • QA Check changed from Info Needed to Dev Needed

#6 Updated by intrigeri 2015-12-05 15:25:48

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 30
  • QA Check changed from Dev Needed to Ready for QA
  • Feature Branch set to pythonlib:bugfix/10123-dont-lose-all-history-for-topic-branches-post-release

I gave it a try, please review and deploy if it seems good enough.

#7 Updated by bertagaz 2015-12-07 03:46:24

  • Assignee changed from bertagaz to intrigeri
  • QA Check changed from Ready for QA to Info Needed

intrigeri wrote:
> I gave it a try, please review and deploy if it seems good enough.

Nice! From my testing it works well.

Now there’s one question that is refraining me from merging it: I’m not sure to like that the number of days is a constant. If like you explain in your commit this variable can help in lowering the load on our Jenkins instance (which is a nice feature), then I think I’d prefer it to be configurable (class argument) so that our script can set this delay. That way, if we decide to change it, we would just have to modify it in our puppet module, rather than in this pythonlib. I think in term of maintenance it would be easier. Want me to propose something like that, or you feel like playing with that?

#8 Updated by intrigeri 2015-12-07 05:35:01

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Info Needed to Dev Needed

> I think I’d prefer it to be configurable (class argument) so that our script can set this delay.

Assuming you mean a class attribute that can be set via the constructor: lovely, yes, excellent idea, please go ahead!

#9 Updated by bertagaz 2015-12-08 04:22:47

  • Assignee changed from bertagaz to intrigeri
  • % Done changed from 30 to 80
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:
> > I think I’d prefer it to be configurable (class argument) so that our script can set this delay.
>
> Assuming you mean a class attribute that can be set via the constructor: lovely, yes, excellent idea, please go ahead!

Pushed two commits (sorry, forgot the Refs on this ones) in the pythonlib repo branch.

Also pushed a new branch in the puppet-tails repo with the same name that adapts our scripts and manifests.

Hope you’ll like it!

#10 Updated by intrigeri 2015-12-08 18:19:33

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Ready for QA to Dev Needed

I see you’ve made active days default to 49 for tails::jenkins::master, we’ll see if we have the resources to support that. All this looks good at first glance, feel free to deploy and to keep an eye on resources consumption :)

#11 Updated by bertagaz 2015-12-10 05:19:35

  • Assignee changed from bertagaz to intrigeri
  • % Done changed from 80 to 90
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:
> I see you’ve made active days default to 49 for tails::jenkins::master, we’ll see if we have the resources to support that. All this looks good at first glance, feel free to deploy and to keep an eye on resources consumption :)

Fixed two issues before deploying, one in the Tails pythonlib and the other in the generate_tails_iso_jobs script. It’s now live and seems to work so far. I’ve set the active_days limit to 36, which is the number that outputs the same job list than with the previous last release tag limit.

#12 Updated by intrigeri 2015-12-13 05:26:18

  • Target version changed from Tails_1.8 to Tails_2.0

Looks good, I’ll take a look after 1.8 is out to confirm that the actual problem this is about did not happen again.

#13 Updated by bertagaz 2015-12-15 23:04:23

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 90 to 100
  • QA Check deleted (Ready for QA)

Taking the liberty and happiness to close this ticket: Jenkins didn’t reset the configuration for the branches it is taking care of, yay.