Bug #17080

Upgrade Cucumber on Jenkins isotesters

Added by intrigeri 2019-09-21 08:49:26 . Updated 2019-09-23 08:37:06 .

Status:
Resolved
Priority:
Normal
Assignee:
intrigeri
Category:
Continuous Integration
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
puppet-tails:cucumber2, jenkins-jobs:cucumber2
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Once we’ve done Bug #10068 we might be able to upgrade to Cucumber 2.x from Stretch, while we’re currently running 1.3 from Jessie.

Goals:

See Feature #10328 for things that might need to happen at the same time.

Finally, as part of this ticket, we should revert commit:eccc25460099522ad26d33c93f3d70601ae63ee9.


Subtasks


Related issues

Related to Tails - Feature #11739: Upgrade our isotesters to Stretch Resolved 2016-08-28
Blocked by Tails - Bug #10068: Upgrade to Jenkins 2.x, using upstream packages Resolved 2018-01-08
Blocks Tails - Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report Confirmed 2018-11-23
Blocks Tails - Feature #10328: Clean up features with Scenario Outlines Confirmed 2015-10-03
Blocks Tails - Feature #17083: Remove compatibility code for Cucumber << 2.4.0 from our test suite Resolved

History

#1 Updated by intrigeri 2019-09-21 08:49:36

  • related to Feature #10328: Clean up features with Scenario Outlines added

#2 Updated by intrigeri 2019-09-21 08:49:40

  • related to Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report added

#3 Updated by intrigeri 2019-09-21 08:49:47

  • blocked by Bug #10068: Upgrade to Jenkins 2.x, using upstream packages added

#4 Updated by intrigeri 2019-09-21 17:36:44

  • Description updated

#5 Updated by intrigeri 2019-09-22 04:19:43

#6 Updated by intrigeri 2019-09-22 04:38:09

  • related to deleted (Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report)

#7 Updated by intrigeri 2019-09-22 04:38:14

  • blocks Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report added

#8 Updated by intrigeri 2019-09-22 04:50:47

  • related to deleted (Feature #10328: Clean up features with Scenario Outlines)

#9 Updated by intrigeri 2019-09-22 04:50:53

  • blocks Feature #10328: Clean up features with Scenario Outlines added

#10 Updated by intrigeri 2019-09-22 05:14:02

  • Description updated
  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri
  • Feature Branch set to puppet-tails:cucumber2

The problem that made us stick to Cucumber 1.x last time (https://issues.jenkins-ci.org/browse/JENKINS-29328) still occurs. Three years later, the corresponding PR (https://github.com/jenkinsci/cucumber-testresult-plugin/pull/9) still hasn’t been finalized nor merged. Note that this PR workarounds the crash bug but does not give us correct reporting of failure in after/before that are part of Background (Bug #16150), which is one of the main incentives of upgrading Cucumber on our CI.

We could post-process the JSON file to drop the new before/after arrays before we pass it to the Cucumber Test Result Jenkins plugin. This would allow us to upgrade Cucumber, which is a first step. But we’ll still not be in a position to fix Bug #16150 and I suspect this will create even more confusion for developers.

Given the Cucumber Test Result Jenkins plugin is up for adoption and mostly unmaintained, I think our options are:

  • Give up on the goals of this ticket and revisit in a year or so, when we’ll rethink the future of our CI and how Jenkins fits into it.
  • Switch to another Cucumber plugin for Jenkins, namely Cucumber Reports (GitHub, which is actively maintained and is supported by JJB. I’ll give it a quick try.
  • Have Cucumber output JUnit XML and use whatever Jenkins has to display the results from such a data source, without particular knowledge of Cucumber concepts. Possible drawbacks:
    • I don’t know if the test results will be presented in a usable way. I have my doubts.
    • I don’t know if the JUnit XML output provides all the info we need to fix Bug #16150 and unblock Feature #10328.

#11 Updated by intrigeri 2019-09-22 07:00:30

  • Feature Branch changed from puppet-tails:cucumber2 to puppet-tails:cucumber2, jenkins-jobs:cucumber2

#12 Updated by intrigeri 2019-09-22 15:47:11

  • Status changed from In Progress to Needs Validation
  • Target version set to Tails_4.0

This was deployed: upgraded to Cucumber 2.4.0 from Stretch, switched to Cucumber Reports plugin.

Next steps:

  • check that it works as intended on lizard (worked fine on sib so far)
  • tell users of Jenkins about the web UI change
  • tell users of Jenkins that they can upgrade Cucumber to Stretch’s in their test systems
  • prepare a branch that reverts commit:eccc25460099522ad26d33c93f3d70601ae63ee9

#13 Updated by intrigeri 2019-09-23 08:36:17

  • blocks Feature #17083: Remove compatibility code for Cucumber << 2.4.0 from our test suite added

#14 Updated by intrigeri 2019-09-23 08:37:06

  • Status changed from Needs Validation to Resolved

intrigeri wrote:
> Next steps:
>
> * check that it works as intended on lizard (worked fine on sib so far)
> * tell users of Jenkins about the web UI change
> * tell users of Jenkins that they can upgrade Cucumber to Stretch’s in their test systems

Done.

> * prepare a branch that reverts commit:eccc25460099522ad26d33c93f3d70601ae63ee9

See you on Feature #17083 :)