Bug #10068

Upgrade to Jenkins 2.x, using upstream packages

Added by bertagaz 2015-08-20 03:56:51 . Updated 2019-09-23 08:24:52 .

Status:
Resolved
Priority:
High
Assignee:
Sysadmins
Category:
Continuous Integration
Target version:
Tails_4.0
Start date:
2018-01-08
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
0
Affected tool:
Deliverable for:

Description

The current state of the Jenkins Debian package is quite scary: it’s lagging a lot behind Jenkins’ LTS version, and it has quite a bunch of known security bugs (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=781223)

The Debian package state will probably not be fixed, given the Jenkins LTS release fast pace. In this situation, we’re quite stuck using outdated Jenkins plugins too, given they often depends on precise Jenkins versions. This doesn’t help our Jenkins plugins upgrade sysadmin task.

jenkins.debian.net is using upstream’s LTS package. We probably should discuss the situation with them, as they’ll probably stumble upon the same problem as they want DSA to take care of the sysadmin maintenance of their instance.

Some discussions are planned at the 2015 Debconf about jenkins.d.n. Could be good to follow what happen on this front.

We could use the upstream APT repo and their Debian package, but it would need some review from our side, to see how the packaging is done and what it really install. Sadly, the Debian package sources don’t seem to be available.

Subtasks

Feature #15155: Upgrade the jenkins Puppet module Resolved Sysadmins

0

Related issues

Related to Tails - Feature #10117: Design how to run our test suite in Jenkins Resolved 2015-08-28
Related to Tails - Feature #11739: Upgrade our isotesters to Stretch Resolved 2016-08-28
Related to Tails - Bug #17088: Test suite became unreliable on Jenkins: OOM kills QEMU, OpenJDK memory allocation failure aborts the test suite run Resolved
Blocks Tails - Feature #10328: Clean up features with Scenario Outlines Confirmed 2015-10-03
Blocks Tails - Feature #13284: Core work: Sysadmin (Adapt our infrastructure) Confirmed 2017-06-30
Blocks Tails - Bug #11295: Test jobs sometimes get scheduled on a busy isotester while there are available ones Resolved 2016-03-31
Blocks Tails - Bug #10601: isotesterN:s are sometimes put offline and never back online Needs Validation 2015-11-23
Blocks Tails - Feature #16954: Update Jenkins plugins: 2019Q3 → 2019Q4 edition Rejected
Blocks Tails - Feature #16955: Update Jenkins plugins: 2020Q1 → 2020Q2 edition Needs Validation
Blocks Tails - Feature #6270: Publish our Jenkins read-only on the web Confirmed 2013-09-10
Blocks Tails - Bug #17080: Upgrade Cucumber on Jenkins isotesters Resolved

History

#1 Updated by bertagaz 2015-08-29 06:39:53

  • Type of work changed from Sysadmin to Research

#2 Updated by bertagaz 2015-08-31 10:38:56

  • Target version changed from Tails_1.6 to Tails_1.7

Delaying, that’s a long research/discussion I won’t have time to work on until the Feature #5288 is deployed.

#3 Updated by intrigeri 2015-09-01 02:05:47

Note that recent versions of plugins (e.g. ParameterizedTrigger) require a version of Jenkins that’s not in Debian.

#4 Updated by intrigeri 2015-09-01 02:06:28

bertagaz wrote:
> jenkins.debian.net is using upstream’s LTS package. We probably should discuss the situation with them, as they’ll probably stumble upon the same problem as they want DSA to take care of the sysadmin maintenance of their instance.

Actually not: DSA is fine with having the upstream package used when transitioning to jenkins.debian.org.

#5 Updated by intrigeri 2015-09-01 02:06:48

  • related to Feature #10117: Design how to run our test suite in Jenkins added

#6 Updated by bertagaz 2015-09-01 03:17:43

intrigeri wrote:
> bertagaz wrote:
> > jenkins.debian.net is using upstream’s LTS package. We probably should discuss the situation with them, as they’ll probably stumble upon the same problem as they want DSA to take care of the sysadmin maintenance of their instance.
>
> Actually not: DSA is fine with having the upstream package used when transitioning to jenkins.debian.org.

Ah, interesting.

I realized lately that https://jenkins.openstack.org was using the same old version we do btw. And they do expose their instances in the WildWildWeb, using Jenkins’ security matrix setup.

Infos about their deployment can be found here

#7 Updated by intrigeri 2015-09-01 03:41:41

> I realized lately that https://jenkins.openstack.org was using the same old version we do btw.

Wow! That’s surprising, since https://git.openstack.org/cgit/openstack-infra/puppet-jenkins/tree/manifests/master.pp explicitly enables the upstream’s APT repo, which currently proposes 1.609.2. It might be because the repo’s key URLs (both the one in that Puppet manifest, and the one advertised on http://pkg.jenkins-ci.org/debian-stable/) give me a 403. Anyway.

#8 Updated by bertagaz 2015-09-01 04:20:14

Yes, I’ve been surprised too after reading their manifest. Maybe they use some kind of static or self-generated webpage with an outdated infos regarding the version.

#9 Updated by bertagaz 2015-10-12 02:28:32

  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Info Needed

I’ve discussed a bit with weasel about this. They are using the upstream LTS Debian package in the Torproject infra. He didn’t audited that package much (e.g was suprised of its 50M size), but says he is happy with it and works well. He confirmed that DSA is willing to use that package too when jenkins.d.n will be taken care of by them.
So if both of this projects decided to do so, maybe it’s worth considering doing so too (even if I’m a bit afraid of such an upgrade ;)). If we do, I don’t think I’ll do this upgrade soon anyway, let just finish the auto test deployment first.

#10 Updated by intrigeri 2015-10-12 11:53:16

> So if both of this projects decided to do so, maybe it’s worth considering doing so too

I find it baffling that there’s no Debian solution to this problem in sight, especially with all the big players involved who rely on that package for mission critical gatekeeping tasks. I am worried that we start relying on essentially non-free software ourselves, which will be problematic if/once we want to trust a Jenkins instance of ours more than we currently do. But all in all, it seems potentially worse to be using for long periods of time a version of Jenkins with known security issues, especially once we want to make it more public.

So if you think we should go ahead and do the switch, feel free to.

#11 Updated by intrigeri 2015-10-12 11:53:27

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Info Needed to Dev Needed

#12 Updated by bertagaz 2015-11-01 08:25:30

  • Target version changed from Tails_1.7 to Tails_1.8

Postponing, as that’s clearly something that won’t happen before 1.7.

#13 Updated by intrigeri 2015-11-05 06:35:01

  • related to Feature #6270: Publish our Jenkins read-only on the web added

#14 Updated by bertagaz 2015-12-15 03:34:21

Postponing

#15 Updated by bertagaz 2015-12-15 03:35:12

  • Target version changed from Tails_1.8 to Tails_2.0

#16 Updated by bertagaz 2016-01-06 13:15:15

  • Target version changed from Tails_2.0 to Tails_2.2

Postponing, won’t work on that for the rest of 2.0 cycle.

#17 Updated by bertagaz 2016-02-14 13:23:03

  • Target version changed from Tails_2.2 to Tails_2.3

Postponing, this ticket won’t be worked on during this release.

#18 Updated by intrigeri 2016-02-27 18:56:59

Jenkins was removed from Debian: https://bugs.debian.org/811522. So this now blocks Feature #11113 (I can’t set up new isotesters given I can’t install jenkins-slave on them). I may then give it a try in the next 1-3 days. Given that this has been regularly postponed since 4+ months, I guess you won’t mind if I do it.

#19 Updated by intrigeri 2016-02-27 18:57:27

  • Subject changed from Use a more recent Jenkins version to Upgrade to upstream packages for Jenkins

#20 Updated by intrigeri 2016-02-27 18:57:43

#21 Updated by intrigeri 2016-02-27 19:48:13

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10
  • Feature Branch set to puppet-tails:feature/10068-upstream-jenkins-deb
  • master: see some preliminary work (untested, probably broken in many ways) in the topic branch; I have no plans to work more on this, it’s back on your plate
  • slaves: I’m going to upload the jenkins-slave package to our own APT repo, it’s just some glue that gets the JAR from the master and turns it into a service; I would like to complete this part as it is what blocks me for Feature #11113.

#22 Updated by intrigeri 2016-02-27 20:09:09

The jenkins-slave side of things is done, I’ll let you handle the master part, or shout for help.

#23 Updated by intrigeri 2016-02-27 20:09:21

#24 Updated by bertagaz 2016-04-23 06:27:33

  • Target version changed from Tails_2.3 to Tails_2.4

#25 Updated by bertagaz 2016-06-07 12:31:34

  • Target version changed from Tails_2.4 to Tails_2.5

#26 Updated by bertagaz 2016-06-10 16:49:21

  • Target version changed from Tails_2.5 to Tails_2.6

Other things are claiming me for the next release.

#27 Updated by anonym 2016-09-20 16:53:56

  • Target version changed from Tails_2.6 to Tails_2.7

#28 Updated by bertagaz 2016-09-22 05:36:15

  • Target version changed from Tails_2.7 to Tails_2.9.1

#29 Updated by intrigeri 2016-11-18 09:12:47

  • blocks Feature #10328: Clean up features with Scenario Outlines added

#30 Updated by anonym 2016-12-14 20:11:20

  • Target version changed from Tails_2.9.1 to Tails 2.10

#31 Updated by anonym 2017-01-24 20:48:48

  • Target version changed from Tails 2.10 to Tails_2.11

#32 Updated by bertagaz 2017-02-28 16:31:22

Note for myself (and the reviewer): that will be the right time to document which plugin we should not update. Most likely it will only be the priority sorter plugin.

#33 Updated by bertagaz 2017-03-08 10:38:04

  • Target version changed from Tails_2.11 to Tails_2.12

#34 Updated by bertagaz 2017-04-06 14:27:53

  • Target version changed from Tails_2.12 to Tails_3.0

#35 Updated by intrigeri 2017-04-16 17:27:16

#36 Updated by bertagaz 2017-05-15 13:33:18

  • Target version changed from Tails_3.0 to Tails_3.1

#37 Updated by bertagaz 2017-05-16 14:29:58

Note to myself: when upgrading to the upstream package, it’s likely that we’ll update the cucumber test report plugin too, and then we’ll be able to remove the custom cucumber package we’ve installed on our isotesters. See Feature #11739 for details.

#38 Updated by bertagaz 2017-05-21 16:37:36

  • Target version changed from Tails_3.1 to Tails_3.2

#39 Updated by intrigeri 2017-06-30 11:25:56

  • Subject changed from Upgrade to upstream packages for Jenkins to Upgrade to Jenkins 2.x, using upstream packages

#40 Updated by intrigeri 2017-06-30 11:26:03

  • blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#41 Updated by bertagaz 2017-08-11 16:16:28

  • Target version changed from Tails_3.2 to Tails_3.3

#42 Updated by intrigeri 2017-10-02 15:11:39

Note that my plans for Bug #11680 might require Jenkins plugins that want a newer Jenkins (and I might not be in the mood to cope with bugs in older version of these plugins if they’ve been fixed in newer versions already). I want to work on this mid-December. Can you please give me an ETA for this ticket? It’s been postponed to “next release” regularly since almost two years, so you’ll understand I take the current Target version with a grain of salt :)

Other options (if you can’t give an ETA or if it’s in too long):

  • I postpone Bug #11680 and focus on the Puppet 4 migration first, to give you some more time here.
  • groente or I takes it over.

#43 Updated by bertagaz 2017-10-03 10:56:50

  • Target version changed from Tails_3.3 to Tails_3.5

#44 Updated by intrigeri 2017-10-22 06:29:30

Ping wrt. the question I’ve asked 3 weeks ago? I want to make sure you’re at least aware of the fallback options that I may have to go with if you can’t give me a suitable ETA.

#45 Updated by bertagaz 2017-10-22 11:47:48

  • blocks Bug #14875: Build reproducibility Jenkins tests: confusing UX and implementation added

#46 Updated by bertagaz 2017-10-23 10:44:58

intrigeri wrote:
> Ping wrt. the question I’ve asked 3 weeks ago? I want to make sure you’re at least aware of the fallback options that I may have to go with if you can’t give me a suitable ETA.

I had a look at my future schedules and it should be doable if that’s the next big task I’m tackling, meaning I’ll have to probably delay a few others.

#47 Updated by intrigeri 2017-10-29 07:48:26

> I had a look at my future schedules and it should be doable […]

I’ve asked you for an ETA and you tell me “it should be doable”. Does this implicitly mean it’ll be done (tested, debugged, deployed, fixed) by mid-December?

#48 Updated by bertagaz 2017-10-30 10:03:17

intrigeri wrote:
> I’ve asked you for an ETA and you tell me “it should be doable”. Does this implicitly mean it’ll be done (tested, debugged, deployed, fixed) by mid-December?

Yes, that’s what I meant.

#49 Updated by intrigeri 2017-10-30 16:58:56

> intrigeri wrote:
>> I’ve asked you for an ETA and you tell me “it should be doable”. Does this implicitly mean it’ll be done (tested, debugged, deployed, fixed) by mid-December?

> Yes, that’s what I meant.

Thanks for clarifying :)

#50 Updated by intrigeri 2017-12-18 16:40:22

  • Priority changed from Normal to High

(As per sysadmin team sprint.)

#51 Updated by intrigeri 2018-01-08 17:44:59

FWIW I’ve noticed today, while working on Feature #15154, that the Puppet module we use to manage Jenkins “does not presently support Jenkins 2.x due to incompatible changes with 1.x. Support is planned for a future release” as of 1.7.0 (last upstream release, August 2016).

#52 Updated by anonym 2018-01-23 19:52:33

  • Target version changed from Tails_3.5 to Tails_3.6

#53 Updated by intrigeri 2018-01-26 20:13:58

#54 Updated by bertagaz 2018-03-14 11:32:08

  • Target version changed from Tails_3.6 to Tails_3.7

#55 Updated by intrigeri 2018-04-08 13:05:06

  • blocks Feature #15502: Update Jenkins modules: 2018Q2 → 2018Q3 edition added

#56 Updated by intrigeri 2018-04-08 13:33:36

  • blocks Feature #15501: Server hardware (2017-2019 edition): evaluate some of the options added

#57 Updated by intrigeri 2018-04-08 13:40:18

I’d like to plan my sysadmin work for this year and this ticket blocks Feature #15501, which I’d like to tackle in 2018Q4 to the latest. So, let’s do the ETA dance again. Are you in a position to:

  1. open your agenda
  2. schedule/block enough time for doing this work, with some safety margin to take into account unscheduled AFK emergencies/unavailability and unexpected technical issues
  3. tell me when you are confident this will be done

?

If not, well, let’s come back to this topic in ~August or so and then we’ll see how we can organize this in a way that works for everyone.

Thanks in advance.

#58 Updated by intrigeri 2018-04-08 18:09:39

  • blocked by deleted (Bug #14875: Build reproducibility Jenkins tests: confusing UX and implementation)

#59 Updated by bertagaz 2018-05-10 11:09:13

  • Target version changed from Tails_3.7 to Tails_3.8

#60 Updated by intrigeri 2018-06-26 16:27:51

  • Target version changed from Tails_3.8 to Tails_3.9

#61 Updated by intrigeri 2018-08-19 16:13:00

#62 Updated by intrigeri 2018-09-05 16:26:51

  • Target version changed from Tails_3.9 to Tails_3.10.1

#63 Updated by intrigeri 2018-09-12 17:07:06

Hi bertagaz, welcome back! It would be very useful if you could reply to Bug #10068#note-57 aka. “let’s do the ETA dance again” one of these days.

From my side, two data points:

  • I won’t work on Feature #15501 this year but I’d like to schedule it for the first half of 2019 and that work is still blocked by the upgrade to a recent Jenkins.
  • As I’ve just reported on Feature #10328 the old version of Jenkins we have forces us to write Gherkin scenarios in suboptimal ways. I’m excited at the idea of being able to clean this up :)

#64 Updated by bertagaz 2018-09-13 15:26:19

intrigeri wrote:
> Hi bertagaz, welcome back! It would be very useful if you could reply to Bug #10068#note-57 aka. “let’s do the ETA dance again” one of these days.
>
> From my side, two data points:
>
> * I won’t work on Feature #15501 this year but I’d like to schedule it for the first half of 2019 and that work is still blocked by the upgrade to a recent Jenkins.
> * As I’ve just reported on Feature #10328 the old version of Jenkins we have forces us to write Gherkin scenarios in suboptimal ways. I’m excited at the idea of being able to clean this up :)

Ack, I’ll think about that and will come back with a plan.

#65 Updated by intrigeri 2018-09-25 09:17:45

  • related to deleted (Feature #15798: Jenkins access for new FT members)

#66 Updated by intrigeri 2018-10-24 17:03:36

  • Target version changed from Tails_3.10.1 to Tails_3.11

#67 Updated by intrigeri 2018-12-01 16:41:12

  • blocked by deleted (Feature #15501: Server hardware (2017-2019 edition): evaluate some of the options)

#68 Updated by intrigeri 2018-12-01 16:49:30

> Blocks deleted (Feature Feature #15501: Server hardware (2017-2019 edition): evaluate some of the options)

Explanation: I’ve downgraded a bit the cloud option of Feature #15501, see commit:da82e1f3b14d30dfd24b156f05db4475bc73b799 for details. If the newly hired sysadmin is excited by cloud things, then we shall change the plan back to the original one, re-add the blocking relationship, and organize things in a way that the corresponding work does happen.

#69 Updated by intrigeri 2018-12-02 13:08:36

  • blocked by deleted (Feature #15502: Update Jenkins modules: 2018Q2 → 2018Q3 edition)

#70 Updated by bertagaz 2018-12-03 20:27:34

Makes sense. Meanwhile I’ve been reluctant to give a deadline for this regarding how ASP work was going on. I was right, my first feeling was to tell “at the end of 2018Q4”, but now it seems end of 2019Q1 is much more realistic.

#71 Updated by CyrilBrulebois 2018-12-16 13:52:50

  • Target version changed from Tails_3.11 to Tails_3.12

#72 Updated by anonym 2019-01-30 11:59:14

  • Target version changed from Tails_3.12 to Tails_3.13

#73 Updated by Anonymous 2019-03-14 13:22:09

  • blocks Bug #11295: Test jobs sometimes get scheduled on a busy isotester while there are available ones added

#74 Updated by Anonymous 2019-03-14 13:22:43

  • blocks Bug #10601: isotesterN:s are sometimes put offline and never back online added

#75 Updated by CyrilBrulebois 2019-03-20 14:35:09

  • Target version changed from Tails_3.13 to Tails_3.14

#76 Updated by CyrilBrulebois 2019-05-23 21:23:20

  • Target version changed from Tails_3.14 to Tails_3.15

#77 Updated by anonym 2019-06-17 09:13:17

When this is done, please revert commit:eccc25460099522ad26d33c93f3d70601ae63ee9 (currently only in feature/buster). Background: Bug #16747.

#78 Updated by CyrilBrulebois 2019-07-10 10:33:56

  • Target version changed from Tails_3.15 to Tails_3.16

#79 Updated by zen 2019-08-09 15:25:51

  • Assignee changed from bertagaz to Sysadmins
  • Target version changed from Tails_3.16 to Tails_3.17

On today’s Tails Sysadmins meeting we collectively decided to re-scheduled this work to happen on September 19-22 from 12:00 (noon) to 16:00 UTC and to be done by a number of us together. Zen and intrigeri volunteered. @bertagaz is invited to join us. :)

#80 Updated by intrigeri 2019-08-09 16:42:13

  • blocks Feature #16954: Update Jenkins plugins: 2019Q3 → 2019Q4 edition added

#81 Updated by intrigeri 2019-08-09 16:43:09

  • blocks Feature #16955: Update Jenkins plugins: 2020Q1 → 2020Q2 edition added

#82 Updated by intrigeri 2019-09-12 14:25:09

  • Target version changed from Tails_3.17 to Tails_4.0

#83 Updated by intrigeri 2019-09-19 06:56:18

  • related to deleted (Feature #6270: Publish our Jenkins read-only on the web)

#84 Updated by intrigeri 2019-09-19 06:56:25

  • blocks Feature #6270: Publish our Jenkins read-only on the web added

#85 Updated by intrigeri 2019-09-20 10:32:58

  • blocks Bug #14875: Build reproducibility Jenkins tests: confusing UX and implementation added

#86 Updated by intrigeri 2019-09-21 08:49:46

  • blocks Bug #17080: Upgrade Cucumber on Jenkins isotesters added

#87 Updated by intrigeri 2019-09-21 17:22:17

The upgrade went quite well in our dev environment and it was thus deployed to production 1 or 2 hours ago.

#88 Updated by intrigeri 2019-09-21 17:24:31

  • Feature Branch deleted (puppet-tails:feature/10068-upstream-jenkins-deb)
  • Type of work changed from Research to Sysadmin

bertagaz wrote:
> Note for myself (and the reviewer): that will be the right time to document which plugin we should not update. Most likely it will only be the priority sorter plugin.

In the end, we had to upgrade the Priority Sorter plugin: older versions are not compatible with current Jenkins. Our config was adapted to work with the current version.

#89 Updated by intrigeri 2019-09-21 17:27:16

anonym wrote:
> When this is done, please revert commit:eccc25460099522ad26d33c93f3d70601ae63ee9 (currently only in feature/buster). Background: Bug #16747.

This is now tracked on Bug #17080: upgrading to Jenkins 2.x is necessary but not sufficient, in itself, to revert that commit.

#90 Updated by intrigeri 2019-09-22 05:53:56

  • blocked by deleted (Bug #14875: Build reproducibility Jenkins tests: confusing UX and implementation)

#91 Updated by zen 2019-09-22 15:52:16

  • Status changed from In Progress to Resolved

Jenkins is now running version 2.176.3. :-)

#92 Updated by intrigeri 2019-09-23 08:24:52

FTR we had to patch Schleuder in place (yeah, yeah) on mail.lizard so it lets Jenkins failure messages through to the RM mailing list. The patch we applied was submitted upstream: https://0xacab.org/schleuder/schleuder/merge_requests/300.

#93 Updated by intrigeri 2019-09-23 18:47:24

  • related to Bug #17088: Test suite became unreliable on Jenkins: OOM kills QEMU, OpenJDK memory allocation failure aborts the test suite run added