Tails

#1 Updated by intrigeri 2019-09-21 08:49:36

• related to Feature #10328: Clean up features with Scenario Outlines added

#2 Updated by intrigeri 2019-09-21 08:49:40

• related to Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report added

#3 Updated by intrigeri 2019-09-21 08:49:47

• blocked by Bug #10068: Upgrade to Jenkins 2.x, using upstream packages added

#4 Updated by intrigeri 2019-09-21 17:36:44

• Description updated

#5 Updated by intrigeri 2019-09-22 04:19:43

• related to Feature #11739: Upgrade our isotesters to Stretch added

#6 Updated by intrigeri 2019-09-22 04:38:09

• related to deleted (Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report)

#7 Updated by intrigeri 2019-09-22 04:38:14

• blocks Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report added

#8 Updated by intrigeri 2019-09-22 04:50:47

• related to deleted (Feature #10328: Clean up features with Scenario Outlines)

#9 Updated by intrigeri 2019-09-22 04:50:53

• blocks Feature #10328: Clean up features with Scenario Outlines added

#10 Updated by intrigeri 2019-09-22 05:14:02

• Description updated
• Status changed from Confirmed to In Progress
• Assignee set to intrigeri
• Feature Branch set to puppet-tails:cucumber2

The problem that made us stick to Cucumber 1.x last time (https://issues.jenkins-ci.org/browse/JENKINS-29328) still occurs. Three years later, the corresponding PR (https://github.com/jenkinsci/cucumber-testresult-plugin/pull/9) still hasn’t been finalized nor merged. Note that this PR workarounds the crash bug but does not give us correct reporting of failure in after/before that are part of Background (Bug #16150), which is one of the main incentives of upgrading Cucumber on our CI.

We could post-process the JSON file to drop the new before/after arrays before we pass it to the Cucumber Test Result Jenkins plugin. This would allow us to upgrade Cucumber, which is a first step. But we’ll still not be in a position to fix Bug #16150 and I suspect this will create even more confusion for developers.

Given the Cucumber Test Result Jenkins plugin is up for adoption and mostly unmaintained, I think our options are:

• Give up on the goals of this ticket and revisit in a year or so, when we’ll rethink the future of our CI and how Jenkins fits into it.
• Switch to another Cucumber plugin for Jenkins, namely Cucumber Reports (GitHub, which is actively maintained and is supported by JJB. I’ll give it a quick try.
• Have Cucumber output JUnit XML and use whatever Jenkins has to display the results from such a data source, without particular knowledge of Cucumber concepts. Possible drawbacks:
  • I don’t know if the test results will be presented in a usable way. I have my doubts.
  • I don’t know if the JUnit XML output provides all the info we need to fix Bug #16150 and unblock Feature #10328.

#11 Updated by intrigeri 2019-09-22 07:00:30

• Feature Branch changed from puppet-tails:cucumber2 to puppet-tails:cucumber2, jenkins-jobs:cucumber2

#12 Updated by intrigeri 2019-09-22 15:47:11

• Status changed from In Progress to Needs Validation
• Target version set to Tails_4.0

This was deployed: upgraded to Cucumber 2.4.0 from Stretch, switched to Cucumber Reports plugin.

Next steps:

• check that it works as intended on lizard (worked fine on sib so far)
• tell users of Jenkins about the web UI change
• tell users of Jenkins that they can upgrade Cucumber to Stretch’s in their test systems
• prepare a branch that reverts commit:eccc25460099522ad26d33c93f3d70601ae63ee9

#13 Updated by intrigeri 2019-09-23 08:36:17

• blocks Feature #17083: Remove compatibility code for Cucumber << 2.4.0 from our test suite added

#14 Updated by intrigeri 2019-09-23 08:37:06

• Status changed from Needs Validation to Resolved

intrigeri wrote:
> Next steps:
>
> * check that it works as intended on lizard (worked fine on sib so far)
> * tell users of Jenkins about the web UI change
> * tell users of Jenkins that they can upgrade Cucumber to Stretch’s in their test systems

Done.

> * prepare a branch that reverts commit:eccc25460099522ad26d33c93f3d70601ae63ee9

See you on Feature #17083 :)