Bug #10442

Totem "Watching a WebM video over HTTPS" test never passes on Jenkins

Added by kytv 2015-10-28 07:33:52 . Updated 2019-09-05 00:03:21 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Test suite
Target version:
Start date:
2015-10-28
Due date:
% Done:

100%

Feature Branch:
test/10442-totem-watching-webm-over-https
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

An error dialog is displayed:

  • title: “An error occurred”
  • text: “The movie could not be read.”

Nothing obvious in the Journal (e.g. no AppArmor error) except that for each attempt, I see this: org.gnome.Shell.desktop[7280]: Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x3200013 ().

Running torsocks totem https://tails.boum.org/lib/test_suite/test.webm works fine in a local VM (2 vCPUs, 2GB RAM) running Tails 4.0~beta1 on my sid system:

  • virtio GPU with 3d acceleration enabled
  • virtio GPU without 3d accel
  • same GPU as the one we use on Jenkins: type='qxl' ram='65536' vram='131072' vgamem='16384'

I can’t reproduce either when I run the test suite manually in a Stretch VM on my system; so nested virtualization is not the only explanation.

But I can reproduce this problem on my local Jenkins (worker1.ant01, nested virtualization), which shows that the problem is not specific to lizard.

wget https://tails.boum.org/lib/test_suite/test.webm works just fine when run by hand on worker1.ant01 and isotesterN.lizard. The former uses our VPN to connect to tails.b.o while the latter uses direct inter-lizard-VM IP connectivity to do so. Both resolve tails.b.o to a RFC1918 address.


Subtasks


Related issues

Related to Tails - Feature #11403: Migrate to Tor Browser 6.0.x based on Firefox 45.2 Resolved 2016-05-09
Related to Tails - Feature #14588: Self-host our website Resolved 2018-10-03
Blocked by Tails - Feature #9521: Use the chutney Tor network simulator in our test suite Resolved 2016-04-15
Blocked by Tails - Bug #10497: wait_until_tor_is_working helper is fragile Resolved 2015-11-06
Blocked by Tails - Bug #10381: The "I open the address" steps are fragile Resolved 2015-10-15
Blocked by Tails - Feature #6729: Bump the number of CPU cores the testing VM has Resolved 2014-02-19
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by kytv 2015-10-28 07:34:17

#2 Updated by kytv 2015-10-28 07:35:56

  • Feature Branch set to kytv/test/10442-watching-webm-over-https-is-fragile

#3 Updated by intrigeri 2015-10-31 05:51:32

  • blocks #8668 added

#4 Updated by intrigeri 2015-11-06 06:07:44

  • Assignee set to kytv
  • Deliverable for set to 270

#5 Updated by anonym 2015-11-06 06:12:04

  • Assignee changed from kytv to anonym
  • Target version set to Tails_1.8

There might be something smart to do that will fix all of {Bug #10442, Bug #10381, Bug #10376} at the same time, and increase browser page loading throughout the test suite.

#6 Updated by intrigeri 2015-12-05 13:22:26

  • Target version changed from Tails_1.8 to Tails_2.0

(We’re going to mark as fragile all tests that depend on Tor to have bootstrapped for the moment => not so urgent.)

#7 Updated by intrigeri 2015-12-07 16:47:19

See commit:f0e00b9c31a77d021509821a433ab80c30fe24a2.

#8 Updated by anonym 2016-01-06 14:06:09

  • Target version changed from Tails_2.0 to Tails_2.2

#9 Updated by intrigeri 2016-02-13 04:17:49

I’ve seen this one fail immediately after loading our webpage worked. IIRC everything online breaks anyway in our test suite if our website is not up, so I suggest we add an ikiwiki underlay to our website, that ships the smallest WebM video that can fit out test suite’s needs, and then we use it from there.

#10 Updated by anonym 2016-02-20 14:18:34

  • Priority changed from Normal to Elevated
  • Target version changed from Tails_2.2 to Tails_2.4

#11 Updated by anonym 2016-02-20 14:56:38

  • Priority changed from Elevated to Normal
  • Target version deleted (Tails_2.4)
  • Deliverable for deleted (270)

I’m gonna gamble and focus on Chutney (Feature #9521) hoping it will fix this issue.

#12 Updated by anonym 2016-02-20 14:57:17

  • blocked by Feature #9521: Use the chutney Tor network simulator in our test suite added

#13 Updated by intrigeri 2016-05-14 13:18:53

  • Feature Branch changed from kytv/test/10442-watching-webm-over-https-is-fragile to test/10442-watching-webm-over-https-is-fragile

#14 Updated by intrigeri 2016-05-18 13:16:14

  • Assignee changed from anonym to intrigeri
  • Target version set to Tails_2.4

Will give it a try.

#15 Updated by intrigeri 2016-05-18 15:13:21

  • blocked by deleted (#8668)

#16 Updated by intrigeri 2016-05-18 15:13:42

  • blocked by Bug #10497: wait_until_tor_is_working helper is fragile added

#17 Updated by intrigeri 2016-05-18 15:19:23

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to anonym
  • % Done changed from 0 to 50
  • QA Check set to Ready for QA

(Or should it be for bertagaz?)

Seems to be fixed by chutney.

#18 Updated by intrigeri 2016-05-20 19:22:41

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

Same as a few similar tickets: too many false positives so far.

#19 Updated by intrigeri 2016-05-25 10:24:15

  • % Done changed from 50 to 60

This now seems to be pretty robust (after flagging more tests as fragile), so here we’re only blocking on Bug #10497 before this can be reviewed’n’merged.

#20 Updated by intrigeri 2016-05-27 13:59:18

  • related to Feature #6729: Bump the number of CPU cores the testing VM has added

#21 Updated by intrigeri 2016-05-27 14:00:02

So, with Tor Browser 6.x it seems that this ticket is blocked by the cheap version of Feature #6729 (i.e. just bumping the number of vcpus to 2).

#22 Updated by intrigeri 2016-05-27 14:01:11

Actually, might be blocked: I haven’t see “file is corrupted” issue on Jenkins yet.

#23 Updated by intrigeri 2016-05-28 09:43:56

  • blocked by Bug #10381: The "I open the address" steps are fragile added

#24 Updated by intrigeri 2016-05-28 09:45:28

intrigeri wrote:
> Actually, might be blocked: I haven’t see “file is corrupted” issue on Jenkins yet.

Seen it: https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10381-fix-i-open-the-address-test-is-fragile/27/artifact/build-artifacts/01%3A32%3A29_Watching_a_WebM_video.png

Of course I didn’t happen on this branch: it can only happen when we run these scenarios, that were flagged fragile due to Bug #10381. So I’ve merged the branch for Bug #10381 into the one for Bug #10442, and added a blocking relationship.

#25 Updated by intrigeri 2016-05-28 09:45:38

  • related to deleted (Feature #6729: Bump the number of CPU cores the testing VM has)

#26 Updated by intrigeri 2016-05-28 09:45:51

  • blocked by Feature #6729: Bump the number of CPU cores the testing VM has added

#27 Updated by intrigeri 2016-05-31 13:09:13

  • related to Feature #11403: Migrate to Tor Browser 6.0.x based on Firefox 45.2 added

#28 Updated by intrigeri 2016-05-31 13:09:58

  • Assignee changed from intrigeri to anonym

Moving to anonym’s plate, since Feature #11403 broke this for real this time.

#29 Updated by anonym 2016-06-08 01:34:58

  • Target version changed from Tails_2.4 to Tails_2.5

#30 Updated by intrigeri 2016-07-21 06:37:42

  • Target version changed from Tails_2.5 to Tails_2.6

#31 Updated by intrigeri 2016-07-30 15:30:32

  • Target version changed from Tails_2.6 to Tails_2.7

(And feel free to drop the target version for this one, and instead tackle Bug #10381, since there’s no way to have this one done before Bug #10381.)

#32 Updated by bertagaz 2016-11-17 17:38:35

  • Target version changed from Tails_2.7 to Tails_2.9.1

#33 Updated by anonym 2016-12-05 20:43:37

  • Assignee deleted (anonym)
  • Target version deleted (Tails_2.9.1)

#34 Updated by Anonymous 2017-06-28 14:01:44

  • Assignee set to anonym

#35 Updated by Anonymous 2017-06-30 10:19:43

  • Assignee deleted (anonym)

#36 Updated by intrigeri 2019-03-08 15:38:58

  • Status changed from In Progress to Confirmed

#37 Updated by intrigeri 2019-03-08 15:39:13

#38 Updated by intrigeri 2019-03-20 14:48:15

#39 Updated by intrigeri 2019-03-20 14:48:29

  • blocked by deleted (Feature #13241: Core work: Test suite maintenance)

#40 Updated by intrigeri 2019-03-25 07:51:53

  • Subject changed from Watching a WebM video over HTTPS is fragile to Totem "Watching a WebM video over HTTPS" test is fragile

#41 Updated by intrigeri 2019-03-25 08:00:29

  • Status changed from Confirmed to In Progress

Applied in changeset commit:tails|8a978562ef2b574a191788f05ab1d5ca1123cd7f.

#42 Updated by intrigeri 2019-03-25 08:02:05

  • Assignee set to CyrilBrulebois
  • Target version set to Tails_3.14
  • % Done changed from 60 to 0
  • QA Check changed from Dev Needed to Ready for QA
  • Feature Branch changed from test/10442-watching-webm-over-https-is-fragile to test/10442-webm-video-update-fragile-status

@CyrilBrulebois, here as well, the branch I’m pushing merely updates the test suite fragile status and comments, in order to avoid further confusion. Please review, merge if happy, and reset Status to Confirmed. Then we shall investigate why the Totem test fails, but that’ll be for another day (and actually, I’m curious to see what happens on Buster).

#43 Updated by anonym 2019-04-02 08:54:04

  • Assignee changed from CyrilBrulebois to anonym

#44 Updated by anonym 2019-04-02 10:42:43

  • Status changed from In Progress to Confirmed
  • Assignee deleted (anonym)
  • QA Check deleted (Ready for QA)

#45 Updated by intrigeri 2019-04-05 13:19:25

  • Target version deleted (Tails_3.14)

#46 Updated by hefee 2019-06-19 07:54:01

I checked many different build on Jenkins and could not found any successful build for cucumber features/totem.feature:50 # Scenario: Watching a WebM video over HTTPS. In detail I checked the range at the end are the buildlogs I looked at:

test_Tails_ISO_bugfix-16471-drop-time-synchronization-hacks-force-all-tests range(1, 12) 
test_Tails_ISO_feature-16356-tor-browser-9.0-force-all-tests range(1, 15) 
test_Tails_ISO_feature-16792-update-chutney-force-all-tests range(1, 5) 
test_Tails_ISO_feature-buster-force-all-tests range(40, 55) 
test_Tails_ISO_hefee-bugfix-16471-drop-time-synchronization-hacks-force-all-tests range(19, 34) 
test_Tails_ISO_test-16820-uefi-force-all-tests range(1, 1) 
test_Tails_ISO_test-anonym-force-all-tests range(1, 4)

#47 Updated by intrigeri 2019-06-22 13:33:48

  • Feature Branch deleted (test/10442-webm-video-update-fragile-status)

#48 Updated by intrigeri 2019-06-22 13:35:35

  • Subject changed from Totem "Watching a WebM video over HTTPS" test is fragile to Totem "Watching a WebM video over HTTPS" test (almost?) never passes on Jenkins

#49 Updated by intrigeri 2019-08-10 14:55:38

  • Description updated

#50 Updated by intrigeri 2019-08-10 16:02:08

  • Description updated
  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri
  • Target version set to Tails_3.16

Reporting about a bunch of situations in which I can, and cannot, reproduce this.

The problem happens on all systems that resolve tails.b.o to a RFC1918 address, and I can’t reproduce it anywhere else, so I think it is caused by:

       ClientDNSRejectInternalAddresses 0|1
           If true, Tor does not believe any anonymously retrieved DNS answer that tells
           it that an address resolves to an internal address (like 127.0.0.1 or
           192.168.0.1). This option prevents certain browser-based attacks; it is not
           allowed to be set on the default network. (Default: 1)

       ClientRejectInternalAddresses 0|1
           If true, Tor does not try to fulfill requests to connect to an internal
           address (like 127.0.0.1 or 192.168.0.1) unless an exit node is specifically
           requested (for example, via a .exit hostname, or a controller request). If
           true, multicast DNS hostnames for machines on the local network (of the form
           *.local) are also rejected. (Default: 1)

I think we need to disable these options at least for the affected scenario.

#51 Updated by intrigeri 2019-08-10 16:09:19

We use Chutney and TestingTorNetwork, which disables the 2 aforementioned options, but in features/step_definitions/chutney.rb we re-enable ClientRejectInternalAddresses (that we need enabled for at least one scenario).

#52 Updated by intrigeri 2019-08-10 16:28:39

  • Priority changed from Normal to Elevated
  • Feature Branch set to test/10442-totem-watching-webm-over-https

(We now run fragile tests on Jenkins for our main branches, so having a scenario that always fails there makes the output of our CI more painful to use.)

#53 Updated by intrigeri 2019-08-10 19:00:04

#54 Updated by intrigeri 2019-08-10 20:34:54

This fixes this bug on my local Jenkins (the video is played in Totem), except the check for Tor leaks of course makes a fuss out of it:

Unexpected connections were made:
  #<OpenStruct mac_saddr="50:54:00:ae:aa:8a", mac_daddr="52:54:00:d7:c9:e3", protocol="tcp", sport=48124, dport=443, saddr="10.2.1.247", daddr="192.168.122.6"> (FirewallAssertionFailedError)
  /var/lib/jenkins/workspace/test_Tails_ISO_test-10442-totem-watching-webm-over-https/features/support/helpers/firewall_helper.rb:97:in `After'

#55 Updated by intrigeri 2019-08-11 09:17:50

In passing, I’m surprised that the Tor Browser’s “Watching a WebM video” test does not fail in the same way; ditto for checking for upgrades: they all connect to tails.boum.org:443 as well. I suspect that the way they resolve this hostname (using Tor’s SOCKS support and presumably proper “the SOCKS proxy resolves DNS”) yields different results than torsocks totem, which will do DNS resolution separately i.e. go through tor’s DNS port. Still, I’m surprised: regardless of how exactly DNS resolution is requested, at the end of the day it should be performed by some exit node in our Chutney network, so I would assume it should yield the same results. Anyways.

#56 Updated by intrigeri 2019-08-11 09:24:16

  • Subject changed from Totem "Watching a WebM video over HTTPS" test (almost?) never passes on Jenkins to Totem "Watching a WebM video over HTTPS" test never passes on Jenkins

#57 Updated by intrigeri 2019-08-11 09:51:09

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to anonym

@anonym, I came up with something that works on my local Jenkins (which as explained above & in the description, is affected by the bug for the same reasons as lizard’s isotesters). I’m not super happy with it but I could not find a better fix. This being said, given this bug makes all base branch tests fail on Jenkins, I’m in favor of a quick resolution, even if slightly suboptimal.

I’ll verify today that this works on lizard too but if you happen to pop up here on a Sunday (which I don’t expect), feel free to beat me to it.

#58 Updated by intrigeri 2019-08-12 09:47:34

Worked 2/2 times on lizard :)

#59 Updated by anonym 2019-08-14 10:30:30

  • Status changed from Needs Validation to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 0 to 100

intrigeri wrote:
> In passing, I’m surprised that the Tor Browser’s “Watching a WebM video” test does not fail in the same way; ditto for checking for upgrades: they all connect to tails.boum.org:443 as well. I suspect that the way they resolve this hostname (using Tor’s SOCKS support and presumably proper “the SOCKS proxy resolves DNS”) yields different results than torsocks totem, which will do DNS resolution separately i.e. go through tor’s DNS port. Still, I’m surprised: regardless of how exactly DNS resolution is requested, at the end of the day it should be performed by some exit node in our Chutney network, so I would assume it should yield the same results.

Frankly, this sounds slightly concerning, almost like there’s a DNS leak or similar. Or do you see anything redeeming?


Other than that, code looks good. My tests do not look good, however, as I very often see @check_tor_leaks:

      Unexpected connections were made:
        #<OpenStruct mac_saddr="00:00:00:00:00:00", mac_daddr="00:00:00:00:00:00", protocol="tcp", sport=5015, dport=38432, saddr="10.2.1.1", daddr="10.2.1.202"> (FirewallAssertionFailedError)


but it turns out that is not caused by your branch. I’ve at least seen the scenario pass once, so I’m happy. Merged!

#60 Updated by intrigeri 2019-08-14 11:41:38

> Other than that, code looks good. My tests do not look good, however, as I very often see @check_tor_leaks:
>

>       Unexpected connections were made:
>         #<OpenStruct mac_saddr="00:00:00:00:00:00", mac_daddr="00:00:00:00:00:00", protocol="tcp", sport=5015, dport=38432, saddr="10.2.1.1", daddr="10.2.1.202"> (FirewallAssertionFailedError)
> 


> but it turns out that is not caused by your branch.

@anonym, I’m curious now :) What is this caused by?

#61 Updated by intrigeri 2019-08-15 07:48:41

Hi @anonym !

> Frankly, this sounds slightly concerning, almost like there’s a DNS leak or similar. Or do you see anything redeeming?

I am also slightly concerned by the fact I don’t understand what’s going on.

But given this test passed just fine (without the firewall leak checker raising hell) on systems without a “special” name resolution setup, I’m not too worried: this shows that Tails behaves as expected. Note that “as expected” here can plausibly include “Totem’s DNS resolution mechanism escapes torsocks and ends up querying the Tor DNS port itself”, which would not be a big surprise (torsocks tries it best but it gives no guarantee and there are well-known ways to escape it, be it maliciously or simply by using techniques that torsocks can’t catch).

#62 Updated by CyrilBrulebois 2019-09-05 00:03:22

  • Status changed from Fix committed to Resolved