Feature #15701

Adapt and run our test suite for Tor Browser 8

Added by intrigeri 2018-07-03 12:53:24 . Updated 2018-07-10 15:55:12 .

Status:
Resolved
Priority:
High
Assignee:
Category:
Target version:
Start date:
2018-07-03
Due date:
% Done:

100%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Browser
Deliverable for:

Description

(Both the manual and automatic test suites.)


Subtasks


Related issues

Blocks Tails - Feature #13241: Core work: Test suite maintenance Rejected 2017-06-29

History

#1 Updated by intrigeri 2018-07-03 12:53:39

#2 Updated by intrigeri 2018-07-03 20:41:20

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 20

#3 Updated by intrigeri 2018-07-03 22:36:02

“Starting the Unsafe Browser works as it should” is blocked by the fact search engines are enabled (Bug #15708) (Update: scratch that, that’s Feature #15701#note-5 which also affects Tor Browser.)

#4 Updated by intrigeri 2018-07-04 10:21:08

The “Watching a WebM video” test fails (“No video with supported format and MIME type found.”) and I see this in the Journal:

Jul 04 09:25:39 amnesia tor-browser.desktop[9040]: [Child 9313, MediaPlayback #3] WARNING: Decoder=7e6a6a774200 state=DECODING_METADATA Decode metadata failed, shutting down decoder: file /var/tmp/build/firefox-a0efd2fcd6e9/dom/media/MediaDecoderStateMachine.cpp, line 379
Jul 04 09:25:39 amnesia tor-browser.desktop[9040]: [Child 9313, MediaPlayback #3] WARNING: Decoder=7e6a6a774200 Decode error: NS_ERROR_DOM_MEDIA_METADATA_ERR (0x806e0006): file /var/tmp/build/firefox-a0efd2fcd6e9/dom/media/MediaDecoderStateMachine.cpp, line 3445

But the video plays just fine in my test VM booted from the same ISO.

#5 Updated by intrigeri 2018-07-04 10:29:56

The browsers address bar seems to have an initialization problem: initially it says “Search or enter address” and after some actions it says “Search with DuckDuckGo or enter address”, both in Tor Browser and Unsafe Browser. This breaks parts of our test suite. Next step: check if it’s a real bug that should be fixed or merely a minor annoyance that our test suite needs to cope with.

#6 Updated by intrigeri 2018-07-05 14:01:29

intrigeri wrote:
> The “Watching a WebM video” test fails (“No video with supported format and MIME type found.”) and I see this in the Journal:

“Fixed” with commit:209de345a373163f2dd62404cd655f4920678021.

#7 Updated by intrigeri 2018-07-05 14:09:08

intrigeri wrote:
> The browsers address bar seems to have an initialization problem: initially it says “Search or enter address” and after some actions it says “Search with DuckDuckGo or enter address”, both in Tor Browser and Unsafe Browser. This breaks parts of our test suite. Next step: check if it’s a real bug that should be fixed or merely a minor annoyance that our test suite needs to cope with.

I can reproduce this with a freshly unpacked Tor Browser 8.0a9 but not with a new Firefox 61 profile. It’s not important enough to warrant investigating this further so I’ll work it around in the test suite.

#8 Updated by intrigeri 2018-07-05 20:45:18

  • % Done changed from 20 to 50

Almost there!

#9 Updated by intrigeri 2018-07-07 09:07:38

I’ve seen all scenarios pass locally! But running the full test suite on lizard shows a lot of robustness issues; nothing new but I’ve tried to fix a few ones so hopefully the next runs will fare better.

Next step: go through the manual test suite.

#10 Updated by intrigeri 2018-07-10 15:55:12

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 50 to 100

intrigeri wrote:
> I’ve seen all scenarios pass locally! But running the full test suite on lizard shows a lot of robustness issues; nothing new but I’ve tried to fix a few ones so hopefully the next runs will fare better.

Indeed it works better. I’m now satisfied by the current state of things, even on lizard when it’s overloaded (it’s not exactly great but at least it’s not worse than usual).

> Next step: go through the manual test suite.

Manual test suite passes.