Feature #12391

Run full test suite when building from a release tag

Added by anonym 2017-03-20 12:02:05 . Updated 2018-08-21 11:43:21 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Tails_3.9
Start date:
2017-03-20
Due date:
% Done:

100%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

When jenkins notices a tag has been pushed or modified (from git push -f --tags) it should check if the changelog at the tagged commit has its first entry != UNRELEASED. Then we know that that is a commit/tag the we are releasing, so it should add a build from that exact tag. And in the test suite it should run all tests (i.e. drop do not do --tag ~fragile@).

This will be a step towards involving Jenkins in our release process for real, and the full test suite run will gather some interesting data about how well the @fragile tagged tests are running on Jenkins. We could save these (ideally Jenkins should be told to keep builds of release tags longer than other builds) and look at them once or twice a year to see if it seems some tests are not fragile on Jenkins any more.

Subtasks

History

#1 Updated by anonym 2017-03-20 12:02:34

  • Description updated

#2 Updated by anonym 2017-03-20 12:03:24

  • Assignee set to bertagaz

What do you think? :)

#3 Updated by Anonymous 2017-06-27 15:59:11

ping @bertagaz :)

#4 Updated by intrigeri 2017-06-30 19:24:53

  • Status changed from New to Confirmed
  • QA Check set to Info Needed

#5 Updated by intrigeri 2017-07-17 16:40:01

  • Target version set to Tails_3.2

#6 Updated by intrigeri 2017-07-17 16:40:40

(Just so the initial question is on bertagaz’ radar. The goal is not to implement this for 3.2.)

#7 Updated by bertagaz 2017-09-07 13:03:40

  • Target version changed from Tails_3.2 to Tails_3.3

#8 Updated by bertagaz 2017-10-23 09:53:19

  • Target version changed from Tails_3.3 to Tails_3.5

Realistically reschedule for 3.4.

#9 Updated by Anonymous 2018-01-16 08:37:49

  • Target version changed from Tails_3.5 to Tails_3.6

Postpone this again.

#10 Updated by bertagaz 2018-03-14 11:32:13

  • Target version changed from Tails_3.6 to Tails_3.7

#11 Updated by bertagaz 2018-05-10 11:09:18

  • Target version changed from Tails_3.7 to Tails_3.8

#12 Updated by intrigeri 2018-06-10 08:00:24

  • Assignee changed from bertagaz to intrigeri
  • QA Check deleted (Info Needed)

anonym wrote:
> When jenkins notices a tag has been pushed or modified (from git push -f --tags) it should check if the changelog at the tagged commit has its first entry != UNRELEASED. Then we know that that is a commit/tag the we are releasing, so it should add a build from that exact tag.

We already build from the tag in that case (IIRC that was done for reproducible builds) so the ISO build part is covered.

> And in the test suite it should run all tests (i.e. drop do not do --tag ~fragile@).

I’m on it (because I have no other simple way to run the full test suite as part of the 3.7.1 release process due to Bug #15460).

#13 Updated by intrigeri 2018-06-10 08:33:58

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

Tried something (https://git-tails.immerda.ch/puppet-tails/commit/files/jenkins/slaves/isotesters/wrap_test_suite?id=8ad403f1fceadef4f6f2267685af6dab410a2016), let’s see how it goes.

#14 Updated by intrigeri 2018-06-10 08:41:31

  • Assignee changed from intrigeri to bertagaz
  • % Done changed from 10 to 50
  • QA Check set to Ready for QA

Seems to work (https://jenkins.tails.boum.org/job/manual_test_Tails_ISO_stable/18/):

08:38:13 + grep -q -E \+force-all-tests$
08:38:13 + git describe --tags --exact-match 0314a6c499dce0f36a317fb11e455d47e177f30a
08:38:13 3.7.1
08:38:13 + TAGS_ARGS=
08:38:13 + as_root_do ./run_test_suite --old-iso tmp/tails-amd64-3.7.1.iso --iso tmp/tails-amd64-3.7.1.iso --artifacts-base-uri https://jenkins.tails.boum.org/job/manual_test_Tails_ISO_stable/18/artifact/build-artifacts/ --capture --
08:38:13 + sudo -n WORKSPACE=/var/lib/jenkins/workspace/manual_test_Tails_ISO_stable TMP_DIR=/tmp/TailsToaster UPSTREAMJOB_ISO=tmp/tails-amd64-3.7.1.iso PREVIOUS_ISO=tmp/tails-amd64-3.7.1.iso ./run_test_suite --old-iso tmp/tails-amd64-3.7.1.iso --iso tmp/tails-amd64-3.7.1.iso --artifacts-base-uri https://jenkins.tails.boum.org/job/manual_test_Tails_ISO_stable/18/artifact/build-artifacts/ --capture --

#15 Updated by intrigeri 2018-06-10 08:50:40

anonym wrote:
> We could save these (ideally Jenkins should be told to keep builds of release tags longer than other builds) and look at them once or twice a year to see if it seems some tests are not fragile on Jenkins any more.

One cheap-ish way would be to have a script running regularly, that looks for jobs that were run from a tag (clone tails.git and run git describe --tags --exact-match on GIT_COMMIT for each jenkins.lizard:/var/lib/jenkins/jobs/test_Tails_ISO_stable/builds/*-*-*_*-*-*/injectedEnvVars.txt), and that copies their cucumber.json to some place that’s not affecting by Jenkins’ GC. Let’s not block on this here. I assume this topic will resurface once development on our test suite resumes.

#16 Updated by intrigeri 2018-06-26 16:27:55

  • Target version changed from Tails_3.8 to Tails_3.9

#17 Updated by Anonymous 2018-08-17 09:35:03

This is ready for QA since quite some time. May you please have a look?

#18 Updated by intrigeri 2018-08-21 11:43:21

  • Status changed from In Progress to Resolved
  • Assignee deleted (bertagaz)
  • % Done changed from 50 to 100
  • QA Check deleted (Ready for QA)

It has worked fine again during the 3.9~rc1 release process, calling this done. Whoever wants to review the implementation can do it at their own pace, whenever they want.