Feature #10119

Adapt the job generation script to also output corresponding test jobs

Added by bertagaz 2015-08-28 09:39:11 . Updated 2015-10-16 03:22:02 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Continuous Integration
Target version:
Tails_1.7
Start date:
2015-08-28
Due date:
% Done:

100%

Feature Branch:
{pythonlib,puppet-tails,jenkins-jobs}:feature/10119-adapt-job-generation-script
Type of work:
Code
Blueprint:

Starter:
0
Affected tool:
Deliverable for:
267

Description

Once we have the library code ready, we’ll need to adapt our generate_build_Tails_ISO_jobs script to also generate the corresponding test jobs.

Subtasks

Related issues

Blocked by Tails - Feature #10118: Write library code that maps Jenkins jobs from building to testing Resolved 2015-08-28
Blocks Tails - Feature #10287: Set up limited email notification on automatic test failure for the initial deployment Resolved 2015-09-27

History

#1 Updated by bertagaz 2015-08-28 09:39:26

#2 Updated by bertagaz 2015-08-28 09:39:52

  • blocked by Feature #10118: Write library code that maps Jenkins jobs from building to testing added

#3 Updated by bertagaz 2015-08-28 09:46:04

  • Subject changed from Adapt the job generation script to also output test jobs to Adapt the job generation script to also output corresponding test jobs

#4 Updated by intrigeri 2015-08-28 10:39:23

It’s not obvious to me that using the same script is a good idea, but I don’t have the entire problem space in mind. If the answer is “that’s needed to avoid code duplication”, then it may mean that we need to move more stuff to the library ;)

#5 Updated by bertagaz 2015-08-28 11:17:03

intrigeri wrote:
> It’s not obvious to me that using the same script is a good idea, but I don’t have the entire problem space in mind. If the answer is “that’s needed to avoid code duplication”, then it may mean that we need to move more stuff to the library ;)

It’s not really because of code duplication, but rather because I think it will probably be a bit more difficult to run two scripts sequentially with cron, rather than one. I think we would end up doing some hackish stuff to ensure the first has finished its work before the other can enter in the game. Or write a yet another wrapper when we could avoid the use of it. To me it makes sense that the script generating the build job for a branch also generates the test job of this branch at the same time.

#6 Updated by intrigeri 2015-08-28 13:58:25

> It’s not really because of code duplication, but rather because I think it will probably be a bit more difficult to run two scripts sequentially with cron, rather than one.

I’m curious why you think they can’t run concurrently. They’re working on the same Git repo, but touching different files, so the only race condition is if they try to push at the same time, no? Sounds like this can be re-attempted (pull+push) a couple times and it should work fine. OK, perhaps a bit too hackish, unless they push to different remotes and some other cronjob takes care of merging both into the production repo, which is perhaps too complicated for what it’s worth.

> To me it makes sense that the script generating the build job for a branch also generates the test job of this branch at the same time.

JFTR: I’m not convinced by this argument, that seems to couple building and testing for a given branch a bit too much for my taste. Our initial design may couple them that strongly, but let’s avoid locking us with code into such a tight coupling.

I’m OK with a single script if it simplifies the implementation (which seems to be the case), but please beware of such coupling problems :)

#7 Updated by intrigeri 2015-09-16 14:14:03

  • blocks #8668 added

#8 Updated by bertagaz 2015-09-22 15:17:43

  • Target version changed from Tails_1.6 to Tails_1.7

#9 Updated by sajolida 2015-10-01 07:26:37

  • Priority changed from Normal to Elevated

Note that this is due on October 15 which is actually before Tails 1.7. Raising priority accordingly.

#10 Updated by bertagaz 2015-10-12 02:22:41

  • Assignee changed from bertagaz to intrigeri
  • % Done changed from 0 to 50
  • QA Check changed from Dev Needed to Ready for QA

I’ve implemented such a job generation script in the feature/10119-adapt-job-generation-script branch of the pythonlib, puppet-tails and jenkins-jobs repo.

Basically, with all this code changes, we’d now use much simpler job definitions and code that rely on JJB job-template and project features.

For each active branch, it creates a related project file that contains the three needed jobs definition (build_Tails_ISO, wrap_test_Tails_ISO and test_Tails_ISO). This prevent us to have to write and maintain one big generation script for all this jobs or several of them.

Here’s a way to test it:

  • In a temporary directory TEMP_DIR, create a bare clone of the Tails repo.
  • Create two sub-directory: one for the current code state (e.g master), one for this ticket code state (e.g 10119)
  • In each of this sub-directory, create three sub-directory:
  • One that host a jenkins-jobs.git clone (e.g 10019/jjb and master/jjb)
  • One that host the pythonlib code (e.g 10019/pythonlib and master/pythonlib)
  • One that will hold the resulting XML files (e.g master/result and 10119/result)
  • Copy in each sub-directory the code of each related repo branch, e.g:
cd jenkins-jobs.git
cp -a . TEMP_DIR/master/jjb/
git checkout feature/10119-adapt-job-generation-script
cp -a . TEMP_DIR/10119/jjb/
  • Copy the generation script version from puppet-tails’ repo related branch where it belongs:
cp files/jenkins/master/generate_build_tails_iso_jobs master/pythonlib
git checkout feature/10119-adapt-job-generation-script
cp files/jenkins/master/generate_tails_iso_jobs master/pythonlib
  • Now, for each subdir at the top-level of the temporary directory, generate the JJB yaml:
cd master/jjb
rm -f build_Tails_ISOS/*.yaml
cd ../pythonlib
./generate_build_Tails_iso_jobs ../../Tails.git/ ../jjb/
cd ../../10119/jjb
rm -f Tails_ISOS/*.yaml
cd ../pythonlib
./generate_tails_iso_jobs ../Tails.git/ ../jjb/
  • Compare the resulting generated job definitions. You should get the same number of created files, but named differently and with different content. Git can help you for that.
  • Now generate the Jenkins jobs xml files with JJB:
cd master/jjb
python2-jenkins-job-builder test -o ../result/ .
cd ../../10119/
python2-jenkins-job-builder test -o ../result/ .

Compare the resulting generated job definition in both result directory:

  • Each active branch should now get three dedicated jobs.
  • The build jobs for feature branches should be almost identical from the one in master, except they contain what is necessary to chain the test job (a copyartifact property, a postbuildscript that collect the build env variables, and a trigger to the corresponding wrap_test_Tails_ISO job).
  • The build jobs for base branches only contain a change in that it now has some additionnal EnvInject related stuffs. To check that you can diff -Naur master/result/build_Tails_ISO_devel 10119/result/build_Tails_ISO_devel.
  • The test job definitions created with the 10119 branch should differ from the one in master only in that some variables aren’t noted in the same way (${UPSTREAMJOB_BUILD_NUMBER} is now noted $UPSTREAMJOB_BUILD_NUMBER). I’ve tested that, it doesn’t prevent the variable to work. You can use an already existing test job for that: diff -Naur master/result/test_Tails_ISO_devel 10119/result/test_Tails_ISO_devel

If these changes make sense and you’re happy with the result, I think we could go on with deploying this thing.

#11 Updated by bertagaz 2015-10-12 06:37:32

Just tested on my local Jenkins instance to apply both resulting configurations, and it applied cleanly. Once this feature branch is used, it updates the build jobs as necessary to chain the newly created corresponding test jobs. They seem to be configured correctly. No jobs are loosed but the one that were used to test the deployment and that we don’t need anymore.

#12 Updated by intrigeri 2015-10-13 23:41:47

  • Feature Branch set to {pythonlib,puppet-tails,jenkins-jobs}:feature/10119-adapt-job-generation-script

#13 Updated by intrigeri 2015-10-13 23:53:59

  • Assignee changed from intrigeri to bertagaz
  • % Done changed from 50 to 60
  • QA Check changed from Ready for QA to Dev Needed

> I’ve implemented such a job generation script in the feature/10119-adapt-job-generation-script branch of the pythonlib, puppet-tails and jenkins-jobs repo.

I like the design very much! Thanks for the lengthy explanation (that could become a good piece of design doc, with minimal effort, once copied to the blueprint, no?).

> If these changes make sense and you’re happy with the result, I think we could go on with deploying this thing.

Please go ahead :)

#14 Updated by intrigeri 2015-10-14 00:01:11

  • blocks Feature #10287: Set up limited email notification on automatic test failure for the initial deployment added

#15 Updated by intrigeri 2015-10-14 00:01:56

  • Status changed from Confirmed to In Progress

#16 Updated by bertagaz 2015-10-14 02:37:06

  • Assignee changed from bertagaz to intrigeri
  • QA Check changed from Dev Needed to Info Needed

intrigeri wrote:
> I like the design very much! Thanks for the lengthy explanation (that could become a good piece of design doc, with minimal effort, once copied to the blueprint, no?).

Like it very much too. Glad to have found so easily this way to reduce our number of lines of code. Expected this ticket to be much more painful to implement. :)

Just to be sure: which lengthy explanation are you talking about? Is that the whole Feature #10119#note-10, with all the steps to test this feature branch? Or just the first lines?

#17 Updated by intrigeri 2015-10-14 05:39:17

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Info Needed to Dev Needed

> Just to be sure: which lengthy explanation are you talking about? Is that the whole Feature #10119#note-10, with all the steps to test this feature branch? Or just the first lines?

Almost all of that note, without the exact steps to test the whole thing. The goal is for people like me, or your next team mate, or you in 1 year, to be able to get the big picture in 3 minutes by reading this, instead of having to reverse engineer it from the content + history of 2 or 3 Git repos.

#18 Updated by bertagaz 2015-10-15 01:08:27

  • Assignee changed from bertagaz to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:
> Almost all of that note, without the exact steps to test the whole thing. The goal is for people like me, or your next team mate, or you in 1 year, to be able to get the big picture in 3 minutes by reading this, instead of having to reverse engineer it from the content + history of 2 or 3 Git repos.

Make sense. Documented that in the blueprint

#19 Updated by intrigeri 2015-10-15 02:42:51

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Ready for QA to Dev Needed

> Make sense. Documented that in the blueprint

Looks great, thanks!

Please proceed with deployment! \o/

#20 Updated by bertagaz 2015-10-15 16:26:55

  • Assignee changed from bertagaz to intrigeri
  • % Done changed from 60 to 100
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:
> Please proceed with deployment! \o/

Done. This new script is now merged and deployed on our infra. It means that now any branch that gets a new commit will be build AND tested! I’ll notify later the dev mailing list.

#21 Updated by intrigeri 2015-10-16 03:09:54

  • Status changed from In Progress to Resolved
  • QA Check changed from Ready for QA to Pass

> intrigeri wrote:
>> Please proceed with deployment! \o/

> Done. This new script is now merged and deployed on our infra. It means that now any branch that gets a new commit will be build AND tested! I’ll notify later the dev mailing list.

Wooohoooo!

I there have been some failures already (which is totally expected).
anonym and kytv have committed to track those that are about fragile
tests, and file tickets accordingly. Great! But we also need to track
the issues that have less clear causes, e.g. that may have roots in the
infra side of things.

=> anonym and kytv, in the next two weeks, can you please also track
issues that you don’t think are on your plate, file tickets about them,
and make sure bertagaz is aware of them?

In particular, those that may be blockers for turning on the real
“notify developers” thing (Feature #10382) should be marked as such.

Reminder: all new such created tickets MUST block #8668.

Thanks! You rock.

[Please drop Redmine from Cc on next reply.]

#22 Updated by intrigeri 2015-10-16 03:22:02

  • Assignee deleted (intrigeri)