Bug #17056

Make test suite robust with Tor Browser 9.0

Added by intrigeri 2019-09-14 12:54:39 . Updated 2019-10-05 07:54:59 .

Status:
Resolved
Priority:
High
Assignee:
Category:
Test suite
Target version:
Tails_4.0
Start date:
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Browser
Deliverable for:

Description

I see lots of failures which seem to be clear test suite robustness regressions triggered by the upgrade to Tor Browser 9.
This makes it hard to tell what else we need to do on Feature #16356, and whether TB 9 fixes Bug #17007.

Subtasks

Related issues

Related to Tails - Bug #17121: Tor Browser 9 sometimes won't load new URLs Resolved
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by intrigeri 2019-09-14 12:54:54

#2 Updated by intrigeri 2019-09-14 16:51:10

  • Status changed from Confirmed to In Progress

Applied in changeset commit:tails|75524b705749bcc605fc9431c3b7be403d4e856d.

#3 Updated by intrigeri 2019-09-14 16:52:14

commit:75524b705749bcc605fc9431c3b7be403d4e856d and commit:5066426db1f6c30bffc5555f60ea25d013023814 may help.

#4 Updated by intrigeri 2019-09-14 17:54:53

  • blocks Bug #17007: JavaScript sometimes blocked on Tor Browser first start ⇒ "Watching a WebM video over HTTPS" and "Playing an Ogg audio track" scenarios are fragile: blocked by NoScript click-to-play added

#5 Updated by intrigeri 2019-09-15 08:02:25

I had some hope that commit:465900996ee0a74a57eb891ee553a3c4b79721ef and commit:5066426db1f6c30bffc5555f60ea25d013023814 would help, but unfortunately they don’t: the last test runs on Jenkins show that opening an URL in a new tab (“I open the address”) is still extremely fragile.

#6 Updated by intrigeri 2019-09-16 15:37:44

intrigeri wrote:
> the last test runs on Jenkins show that opening an URL in a new tab (“I open the address”) is still extremely fragile.

… while it always works fine on my local Jenkins. This suggests a race condition that we may win or lose, depending on the particularities of the isotester (most likely: resources, performance, workload).

#7 Updated by intrigeri 2019-09-23 18:15:26

  • Assignee deleted (intrigeri)

I won’t be able to work on this before Saturday.

#8 Updated by intrigeri 2019-09-30 17:37:08

  • Assignee set to intrigeri

#9 Updated by intrigeri 2019-10-03 09:25:55

  • blocked by deleted (Bug #17007: JavaScript sometimes blocked on Tor Browser first start ⇒ "Watching a WebM video over HTTPS" and "Playing an Ogg audio track" scenarios are fragile: blocked by NoScript click-to-play)

#10 Updated by intrigeri 2019-10-03 10:07:05

>> the last test runs on Jenkins show that opening an URL in a new tab (“I open the address”) is still extremely fragile.

> … while it always works fine on my local Jenkins. This suggests a race condition that we may win or lose, depending on the particularities of the isotester (most likely: resources, performance, workload).

My latest batch of test suite runs seem to confirm my initial guess:

  • I never see this problem on my fastest local VMs (workerN.ant01).
  • I have seen this problem a couple times on my slower local VM (isotester1.sib).
  • This problem happens a bunch of times during every test suite run on lizard.

When this problem happens, there’s a weird graphical glitch earlier, when we load the /home/testing page: there’s some white space between the URL bar and the Tails logo and at the bottom of the page. Maybe that’s just the new letterboxing feature and it’s fully unrelated.

Next thing I want to try: instead of pasting the URL and then pressing Enter, I’ll try opening the address bar’s right-click contextual menu and then clicking “Paste & Go”.

#11 Updated by intrigeri 2019-10-03 10:24:11

  • Feature Branch set to test/17056-tor-browser-9.0+force-all-tests

I’ll use this topic branch for experiments, and will feel free to rewrite its history. Once happy I’ll merge into the branch for Feature #16356.

#12 Updated by intrigeri 2019-10-03 17:37:18

  • Feature Branch deleted (test/17056-tor-browser-9.0+force-all-tests)

Merged something that seems robust enough locally into the branch for Feature #16356. It only fixes the problem for the Tor Browser. OTOH I don’t know yet if the Unsafe Browser is affected — we’ll see once it works well enough so our test suite can exercise more of it.

#13 Updated by intrigeri 2019-10-04 05:43:05

intrigeri wrote:
> Merged something that seems robust enough locally into the branch for Feature #16356. It only fixes the problem for the Tor Browser.

Confirmed, this makes the Tor Browser scenarios pass on Jenkins! \o/

> OTOH I don’t know yet if the Unsafe Browser is affected

Hmm, looks like I broke some Unsafe Browser tests while fixing the Tor Browser ones. Will investigate & fix.

#14 Updated by intrigeri 2019-10-04 05:49:13

> Hmm, looks like I broke some Unsafe Browser tests while fixing the Tor Browser ones. Will investigate & fix.

commit:6f71531208bac5ab92a9dc83bb3d9c4af4b383d2 might fix that.

#15 Updated by intrigeri 2019-10-05 07:51:37

  • related to Bug #17121: Tor Browser 9 sometimes won't load new URLs added

#16 Updated by intrigeri 2019-10-05 07:54:59

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)

It’s now robust, thanks to the workaround for Bug #17121 that I’ve added (removal of this hack is tracked on that other ticket).

Of course, the Unsafe Browser tests fail, but that’s because it was not fully ported to Tor Browser 9 yet, which is tracked on Feature #17055.