Totem "Watching a WebM video over HTTPS" test never passes on Jenkins
An error dialog is displayed:
- title: “An error occurred”
- text: “The movie could not be read.”
Nothing obvious in the Journal (e.g. no AppArmor error) except that for each attempt, I see this:
org.gnome.Shell.desktop: Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x3200013 ().
torsocks totem https://tails.boum.org/lib/test_suite/test.webm works fine in a local VM (2 vCPUs, 2GB RAM) running Tails 4.0~beta1 on my sid system:
- virtio GPU with 3d acceleration enabled
- virtio GPU without 3d accel
- same GPU as the one we use on Jenkins:
type='qxl' ram='65536' vram='131072' vgamem='16384'
I can’t reproduce either when I run the test suite manually in a Stretch VM on my system; so nested virtualization is not the only explanation.
But I can reproduce this problem on my local Jenkins (worker1.ant01, nested virtualization), which shows that the problem is not specific to lizard.
wget https://tails.boum.org/lib/test_suite/test.webm works just fine when run by hand on worker1.ant01 and isotesterN.lizard. The former uses our VPN to connect to tails.b.o while the latter uses direct inter-lizard-VM IP connectivity to do so. Both resolve tails.b.o to a RFC1918 address.
Related to Tails -
Related to Tails -
Blocked by Tails -
Blocked by Tails -
Blocked by Tails -
Blocked by Tails -
|Blocks Tails - Feature #16209: Core work: Foundations Team||Confirmed|
#9 Updated by intrigeri 2016-02-13 04:17:49
I’ve seen this one fail immediately after loading our webpage worked. IIRC everything online breaks anyway in our test suite if our website is not up, so I suggest we add an ikiwiki underlay to our website, that ships the smallest WebM video that can fit out test suite’s needs, and then we use it from there.
#24 Updated by intrigeri 2016-05-28 09:45:28
> Actually, might be blocked: I haven’t see “file is corrupted” issue on Jenkins yet.
Of course I didn’t happen on this branch: it can only happen when we run these scenarios, that were flagged fragile due to
Bug #10381. So I’ve merged the branch for Bug #10381 into the one for Bug #10442, and added a blocking relationship.
#42 Updated by intrigeri 2019-03-25 08:02:05
- Assignee set to CyrilBrulebois
- Target version set to Tails_3.14
- % Done changed from 60 to 0
- QA Check changed from Dev Needed to Ready for QA
- Feature Branch changed from test/10442-watching-webm-over-https-is-fragile to test/10442-webm-video-update-fragile-status
@CyrilBrulebois, here as well, the branch I’m pushing merely updates the test suite fragile status and comments, in order to avoid further confusion. Please review, merge if happy, and reset Status to Confirmed. Then we shall investigate why the Totem test fails, but that’ll be for another day (and actually, I’m curious to see what happens on Buster).
#46 Updated by hefee 2019-06-19 07:54:01
I checked many different build on Jenkins and could not found any successful build for cucumber
features/totem.feature:50 # Scenario: Watching a WebM video over HTTPS. In detail I checked the range at the end are the buildlogs I looked at:
test_Tails_ISO_bugfix-16471-drop-time-synchronization-hacks-force-all-tests range(1, 12) test_Tails_ISO_feature-16356-tor-browser-9.0-force-all-tests range(1, 15) test_Tails_ISO_feature-16792-update-chutney-force-all-tests range(1, 5) test_Tails_ISO_feature-buster-force-all-tests range(40, 55) test_Tails_ISO_hefee-bugfix-16471-drop-time-synchronization-hacks-force-all-tests range(19, 34) test_Tails_ISO_test-16820-uefi-force-all-tests range(1, 1) test_Tails_ISO_test-anonym-force-all-tests range(1, 4)
#50 Updated by intrigeri 2019-08-10 16:02:08
- Description updated
- Status changed from Confirmed to In Progress
- Assignee set to intrigeri
- Target version set to Tails_3.16
Reporting about a bunch of situations in which I can, and cannot, reproduce this.
The problem happens on all systems that resolve tails.b.o to a RFC1918 address, and I can’t reproduce it anywhere else, so I think it is caused by:
ClientDNSRejectInternalAddresses 0|1 If true, Tor does not believe any anonymously retrieved DNS answer that tells it that an address resolves to an internal address (like 127.0.0.1 or 192.168.0.1). This option prevents certain browser-based attacks; it is not allowed to be set on the default network. (Default: 1) ClientRejectInternalAddresses 0|1 If true, Tor does not try to fulfill requests to connect to an internal address (like 127.0.0.1 or 192.168.0.1) unless an exit node is specifically requested (for example, via a .exit hostname, or a controller request). If true, multicast DNS hostnames for machines on the local network (of the form *.local) are also rejected. (Default: 1)
I think we need to disable these options at least for the affected scenario.
#52 Updated by intrigeri 2019-08-10 16:28:39
- Priority changed from Normal to Elevated
- Feature Branch set to test/10442-totem-watching-webm-over-https
(We now run fragile tests on Jenkins for our main branches, so having a scenario that always fails there makes the output of our CI more painful to use.)
#54 Updated by intrigeri 2019-08-10 20:34:54
This fixes this bug on my local Jenkins (the video is played in Totem), except the check for Tor leaks of course makes a fuss out of it:
Unexpected connections were made: #<OpenStruct mac_saddr="50:54:00:ae:aa:8a", mac_daddr="52:54:00:d7:c9:e3", protocol="tcp", sport=48124, dport=443, saddr="10.2.1.247", daddr="192.168.122.6"> (FirewallAssertionFailedError) /var/lib/jenkins/workspace/test_Tails_ISO_test-10442-totem-watching-webm-over-https/features/support/helpers/firewall_helper.rb:97:in `After'
#55 Updated by intrigeri 2019-08-11 09:17:50
In passing, I’m surprised that the Tor Browser’s “Watching a WebM video” test does not fail in the same way; ditto for checking for upgrades: they all connect to tails.boum.org:443 as well. I suspect that the way they resolve this hostname (using Tor’s SOCKS support and presumably proper “the SOCKS proxy resolves DNS”) yields different results than
torsocks totem, which will do DNS resolution separately i.e. go through tor’s DNS port. Still, I’m surprised: regardless of how exactly DNS resolution is requested, at the end of the day it should be performed by some exit node in our Chutney network, so I would assume it should yield the same results. Anyways.
#57 Updated by intrigeri 2019-08-11 09:51:09
- Status changed from In Progress to Needs Validation
- Assignee changed from intrigeri to anonym
@anonym, I came up with something that works on my local Jenkins (which as explained above & in the description, is affected by the bug for the same reasons as lizard’s isotesters). I’m not super happy with it but I could not find a better fix. This being said, given this bug makes all base branch tests fail on Jenkins, I’m in favor of a quick resolution, even if slightly suboptimal.
I’ll verify today that this works on lizard too but if you happen to pop up here on a Sunday (which I don’t expect), feel free to beat me to it.
#59 Updated by anonym 2019-08-14 10:30:30
- Status changed from Needs Validation to Fix committed
- Assignee deleted (
- % Done changed from 0 to 100
> In passing, I’m surprised that the Tor Browser’s “Watching a WebM video” test does not fail in the same way; ditto for checking for upgrades: they all connect to tails.boum.org:443 as well. I suspect that the way they resolve this hostname (using Tor’s SOCKS support and presumably proper “the SOCKS proxy resolves DNS”) yields different results than
torsocks totem, which will do DNS resolution separately i.e. go through tor’s DNS port. Still, I’m surprised: regardless of how exactly DNS resolution is requested, at the end of the day it should be performed by some exit node in our Chutney network, so I would assume it should yield the same results.
Frankly, this sounds slightly concerning, almost like there’s a DNS leak or similar. Or do you see anything redeeming?
Other than that, code looks good. My tests do not look good, however, as I very often see @
Unexpected connections were made: #<OpenStruct mac_saddr="00:00:00:00:00:00", mac_daddr="00:00:00:00:00:00", protocol="tcp", sport=5015, dport=38432, saddr="10.2.1.1", daddr="10.2.1.202"> (FirewallAssertionFailedError)
but it turns out that is not caused by your branch. I’ve at least seen the scenario pass once, so I’m happy. Merged!
#60 Updated by intrigeri 2019-08-14 11:41:38
> Other than that, code looks good. My tests do not look good, however, as I very often see @
> Unexpected connections were made: > #<OpenStruct mac_saddr="00:00:00:00:00:00", mac_daddr="00:00:00:00:00:00", protocol="tcp", sport=5015, dport=38432, saddr="10.2.1.1", daddr="10.2.1.202"> (FirewallAssertionFailedError) >
> but it turns out that is not caused by your branch.
@anonym, I’m curious now :) What is this caused by?
#61 Updated by intrigeri 2019-08-15 07:48:41
Hi @anonym !
> Frankly, this sounds slightly concerning, almost like there’s a DNS leak or similar. Or do you see anything redeeming?
I am also slightly concerned by the fact I don’t understand what’s going on.
But given this test passed just fine (without the firewall leak checker raising hell) on systems without a “special” name resolution setup, I’m not too worried: this shows that Tails behaves as expected. Note that “as expected” here can plausibly include “Totem’s DNS resolution mechanism escapes torsocks and ends up querying the Tor DNS port itself”, which would not be a big surprise (torsocks tries it best but it gives no guarantee and there are well-known ways to escape it, be it maliciously or simply by using techniques that torsocks can’t catch).