Bug #17472

Jenkins does not run the full test suite for tags

Added by anonym 2020-02-11 10:27:27 . Updated 2020-03-28 09:51:30 .

Status:
Needs Validation
Priority:
Normal
Assignee:
anonym
Category:
Continuous Integration
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
Type of work:
Research
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

When Jenkins tested the Tails 4.3 tag, it didn’t run @fragile scenarios, which is a bug. See e.g. https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/2205/

In the console output I see this trace:

14:17:21 + echo origin/stable
14:17:21 + grep -q -E \+force-all-tests$
14:17:21 + [ stable = testing ]
14:17:21 + [ stable = feature/tor-nightly-master ]
14:17:21 + [ stable = devel ]
14:17:21 + git describe --tags --exact-match 4904a1fde8887488167a8a6c34b34c45b51aa112
14:17:21 + [ 4904a1fde8887488167a8a6c34b34c45b51aa112 != 4904a1fde8887488167a8a6c34b34c45b51aa112 ]
14:17:21 + TAGS_ARGS=--tag ~@fragile


for this part of wrap_test_suite:

if echo "${UPSTREAMJOB_GIT_BRANCH}" | grep -q -E '\+force-all-tests$' \
   || [ "${GIT_BRANCH#origin/}" = testing ] \
   || [ "${GIT_BRANCH#origin/}" = feature/tor-nightly-master ] \
   || [ "${GIT_BRANCH#origin/}" = devel ] ; then
    TAGS_ARGS=""
elif git describe --tags --exact-match "${UPSTREAMJOB_GIT_COMMIT}" >/dev/null 2>&1; then
    TAGS_ARGS=""
elif [ "${UPSTREAMJOB_GIT_COMMIT}" != "${UPSTREAMJOB_GIT_BASE_BRANCH_HEAD}" ] && \
       only_changes_doc "$UPSTREAMJOB_GIT_BASE_BRANCH_HEAD"; then
    TAGS_ARGS="--tag ~@fragile --tag @doc"
else
    TAGS_ARGS="--tag ~@fragile"
fi

So $UPSTREAMJOB_GIT_COMMIT is commit:4904a1f, which is not the 4.3 tag (it is commit:2797053), but just one of the commits I made yesterday, so the correct if-case for releases was not hit. Note that commit:4904a1f is not the commit that was initially tagged 4.3 (FWIW, that was commit:4172c53), so there was no mix up with the tag update. I have no clue how commit:4904a1f entered the picture since it doesn’t seem very special. Curious!

Subtasks

History

#1 Updated by intrigeri 2020-02-16 10:16:04

  • Status changed from Confirmed to New
  • Assignee changed from intrigeri to anonym

> When Jenkins tested the Tails 4.3 tag, it didn’t run @fragile scenarios, which is a bug. See e.g. https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/2205/

What makes you think that this specific run was supposed to test the 4.3 tag?

That test suite run has been triggered by https://jenkins.tails.boum.org/job/build_Tails_ISO_stable/3140/, which was about commit:4904a1f, which I believed was triggered by the Git push at the end of the “Update other base branches” section of the release process (the timing matches: commit done on Feb 10 10:02:43 2020, build started at Feb 10, 2020 10:17:36 AM). So to me, it seems that not running fragile scenarios in this case was the expected behavior.

The first build from the tag was https://jenkins.tails.boum.org/job/build_Tails_ISO_stable/3142/, started a few hours later.
It triggered https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/2207/, which did run the full test suite, which is what we want.

#2 Updated by anonym 2020-02-17 09:33:24

  • Assignee changed from anonym to intrigeri

intrigeri wrote:
> > When Jenkins tested the Tails 4.3 tag, it didn’t run @fragile scenarios, which is a bug. See e.g. https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/2205/
>
> What makes you think that this specific run was supposed to test the 4.3 tag?

That test_Tails_ISO job says “Revision: 2797053a53225134a74e0bcc80682e05bfcb7edd” which is the commit for the real 4.3 tag. I have always assumed that build_Tails_ISO will trigger a test_Tails_ISO from the exact same commit; I never thought revisions could be mixed like that, and it in fact seems like a source of bad test results. What is going on here? What am I missing?

> That test suite run has been triggered by https://jenkins.tails.boum.org/job/build_Tails_ISO_stable/3140/, which was about commit:4904a1f, which I believed was triggered by the Git push at the end of the “Update other base branches” section of the release process (the timing matches: commit done on Feb 10 10:02:43 2020, build started at Feb 10, 2020 10:17:36 AM). So to me, it seems that not running fragile scenarios in this case was the expected behavior.
>
> The first build from the tag was https://jenkins.tails.boum.org/job/build_Tails_ISO_stable/3142/, started a few hours later.
> It triggered https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/2207/, which did run the full test suite, which is what we want.

Ah, I see! Cased closed, but I’ll wait for an answer to the above before closing the ticket.

#3 Updated by intrigeri 2020-02-17 09:49:01

  • Assignee changed from intrigeri to anonym

Hi,

anonym wrote:
> intrigeri wrote:
>> > When Jenkins tested the Tails 4.3 tag, it didn’t run @fragile scenarios, which is a bug. See e.g. https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/2205/
>>
>> What makes you think that this specific run was supposed to test the 4.3 tag?
>
> That test_Tails_ISO job says “Revision: 2797053a53225134a74e0bcc80682e05bfcb7edd” which is the commit for the real 4.3 tag. I have always assumed that build_Tails_ISO will trigger a test_Tails_ISO from the exact same commit; I never thought revisions could be mixed like that, and it in fact seems like a source of bad test results. What is going on here? What am I missing?

OK, I think I now understand the source of the misunderstanding!

I think the confusion comes from the mismatch between:

  • Your correct understanding of the intended high-level behavior: build_Tails_ISO does trigger a test_Tails_ISO from the exact same commit. In the test job run log you’ll see git reset --hard 4904a1fde8887488167a8a6c34b34c45b51aa112.
  • Low-level implementation “details”, i.e.:
    • When it starts a run of test_Tails_ISO, Jenkins itself does not know which commit we want to test. Hence the confusing “Revision: 2797053a53225134a74e0bcc80682e05bfcb7edd” information it displays.
    • The code that implements that job is currently responsible for resetting the working copy to the desired state.

Unfortunately, I don’t think there’s any way to hide the confusing “Revision:” info via configuration, short of patching the corresponding Jenkins plugin; and the awful HTML+CSS implementation (OMG!) makes it hard to hide it via CSS :/

Would it help if I documented this in our doc aimed at developers?
Or are we bound to forget occasionally about it and not refer to the doc anyway?

#4 Updated by intrigeri 2020-03-28 09:49:53

  • Status changed from New to In Progress

Applied in changeset commit:tails|7d39c7fb6c1ec4db5ed8f4fb468034b9cc72a445.

#5 Updated by intrigeri 2020-03-28 09:51:30

  • Status changed from In Progress to Needs Validation

Hi,

> Would it help if I documented this in our doc aimed at developers?

I went ahead and did it. Please review commit:7d39c7fb6c1ec4db5ed8f4fb468034b9cc72a445.