Bug #16768

Failing scenarios in After scenario hooks can lead to inconsistency

Added by anonym 2019-05-30 21:55:46 . Updated 2019-06-01 08:42:47 .

Status:
Duplicate
Priority:
Normal
Assignee:
Category:
Test suite
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
Type of work:
Research
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

I just noticed a discrepancy between this run’s debug.log’s summary:

Failing Scenarios:
cucumber features/additional_software_packages.feature:69 # Scenario: Recovering in offline mode after Additional Software previously failed to upgrade and then succeed to upgrade when online
cucumber features/electrum.feature:15 # Scenario: Using a persistent Electrum configuration
cucumber features/tor_stream_isolation.feature:9 # Scenario: tails-security-check is using the Tails-specific SocksPort
cucumber features/totem.feature:50 # Scenario: Watching a WebM video over HTTPS

219 scenarios (4 failed, 215 passed)
1658 steps (3 failed, 1655 passed)
346m16.087s

and Jenkins’ cucumber plugin’s summary:

All Failed Scenarios
====================
Test Name    Duration    Age
---------    --------    ---
Electrum Bitcoin client ǂ Using a persistent Electrum configuration    7 min 53 sec    1
Tor stream isolation is effective ǂ tails-security-check is using the Tails-specific SocksPort    20 sec    1
Using Totem ǂ Watching a WebM video over HTTPS    26 min    1

Notice how the Recovering in offline mode after Additional Software previously failed to upgrade and then succeed to upgrade when online scenario is present in the former but absent from the latter.

In cucumber.json I can see that this scenario’s result has status “passed”. My theory is that the result is recorded before we run our After Scenario hook, so if we fail the scenario there, as happened in this scenario’s case thanks to the @check_tor_leaks tag and the firewall leak detector hitting Bug #16148.

I have attached the run’s logs for future investigation.


Files

cucumber.json (933441 B) anonym, 2019-05-30 21:39:48
debug.log (2783689 B) anonym, 2019-05-30 21:40:05

Subtasks


Related issues

Is duplicate of Tails - Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report Confirmed 2018-11-23

History

#1 Updated by anonym 2019-05-30 21:56:07

#2 Updated by anonym 2019-05-30 21:56:55

  • Subject changed from Jenkins cucumber plugin can ignore failures to Failing scenarios in After scenario hooks can lead to inconsistency

#3 Updated by intrigeri 2019-06-01 08:42:32

  • is duplicate of Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report added

#4 Updated by intrigeri 2019-06-01 08:42:35

#5 Updated by intrigeri 2019-06-01 08:42:47

  • Status changed from Confirmed to Duplicate