Feature #17621

Keep latest test suite screenshot

Added by segfault 2020-04-15 12:31:45 . Updated 2020-05-15 09:24:58 .

Status:
Resolved
Priority:
Normal
Assignee:
intrigeri
Category:
Target version:
Tails_4.7
Start date:
Due date:
% Done:

100%

Feature Branch:
feature/17621-keep-latest-screenshot
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

For easier debugging.

Subtasks

History

#1 Updated by segfault 2020-04-15 12:32:55

  • Status changed from Confirmed to In Progress

Applied in changeset commit:tails|e1ee41bd6808573e7d36e4855491ee8ddac82bb4.

#2 Updated by segfault 2020-04-15 12:33:24

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (segfault)
  • Feature Branch set to feature/17621-keep-latest-screenshot

#3 Updated by intrigeri 2020-04-16 05:57:04

How does this relate to / combine with the screenshots our test suite already takes, and keeps, on failure, such as https://jenkins.tails.boum.org/view/RM/job/test_Tails_ISO_stable/2299/artifact/build-artifacts/01%3A31%3A20_Installing_Tails_with_GNOME_Disks_from_a_USB_image.png?
I guess my question really is: in the cases where we would need screenshot.png for debugging, why don’t we get the info we need on the failure snapshot that we take+save already?

#4 Updated by segfault 2020-04-16 09:39:31

intrigeri wrote:
> How does this relate to / combine with the screenshots our test suite already takes, and keeps, on failure, such as https://jenkins.tails.boum.org/view/RM/job/test_Tails_ISO_stable/2299/artifact/build-artifacts/01%3A31%3A20_Installing_Tails_with_GNOME_Disks_from_a_USB_image.png?
> I guess my question really is: in the cases where we would need screenshot.png for debugging, why don’t we get the info we need on the failure snapshot that we take+save already?

I don’t know where this screenshot is getting created in the code and I don’t know where/if I can find that screenshot when a test run fails on my local system. Also, I assume that this screenshot gets created after a test run fails, so it’s not the same screenshot that was being used to search for the image.

#5 Updated by intrigeri 2020-04-16 10:27:13

> I don’t know […] where/if I can find that screenshot when a test run fails on my local system.

It will be with all other test suite artifacts: /tmp/TailsToaster/run-*/*.png

> Also, I assume that this screenshot gets created after a test run fails,

Correct.

> so it’s not the same screenshot that was being used to search for the image.

Indeed, in some cases they could be different. For example, when the code that took that “latest test suite screenshot”, which your branch keeps around, did not result in a scenario failure.
If that’s the sort of failure modes this branch is meant to debug, and the existing /tmp/TailsToaster/run-*/*.png don’t fit the bill, then: +1 from me to merge the branch.

#6 Updated by intrigeri 2020-04-18 06:55:39

  • Assignee set to segfault

intrigeri wrote:
> If that’s the sort of failure modes this branch is meant to debug, and the existing /tmp/TailsToaster/run-*/*.png don’t fit the bill, then: +1 from me to merge the branch.

Please merge if that’s indeed the case :)

#7 Updated by intrigeri 2020-05-12 15:31:57

  • Assignee changed from segfault to intrigeri

Re-reading the discussion, “it’s not the same screenshot that was being used to search for the image” makes it clear that this branch addresses a need that our current screenshots don’t, even after having pointed segfault to the location where these existing screenshots live.

So I’ll save on paperwork and will review this :)

Sorry for the hassle and extra delay :/

#8 Updated by intrigeri 2020-05-15 09:15:47

  • Status changed from Needs Validation to Resolved
  • % Done changed from 0 to 100

Applied in changeset commit:tails|2a164d26b38cb612f8e9b78520614ecdcfa62ec7.

#9 Updated by intrigeri 2020-05-15 09:24:58

  • Target version set to Tails_4.7

This branch felt harmless and it solves a real problem for segfault, so I’ve merged it.

Food for thought follows.

Note, however, that just like another, occasionally useful, temporary file we save as a test suite artifact (last_opencv_match.png), in the current implementation, screenshot.png is:

  • unique when running the test suite locally ⇒ if I re-run the test suite, I’ll lose the previous version of that file
  • per-test-suite run on Jenkins

This implies it’s only useful when one runs 1 scenario at a time, and makes little sense in the context of Jenkins most of the time. If we wanted to make these 2 image artifacts even more useful, on scenario failure we could move them to the per-run artifacts directory and rename them so 1 separate pair of images is saved for every failed scenario, just like every other failure artifact we’re saving. On top of this, it would make it easier to find these files, because they would be stored in the artifacts directory that we print at the end of a failed scenario, together with every other failure artifact.