Tails

#3 Updated by intrigeri 2015-01-03 10:40:58

• Assignee set to boklm

#5 Updated by intrigeri 2015-04-11 09:39:16

• Assignee changed from boklm to intrigeri
• Target version set to Tails_1.4.1

I’ll meet boklm next month to look at this problem and work on it if doable.

#6 Updated by intrigeri 2015-05-21 16:53:05

• Feature Branch set to test/7066-run-tbb-tests

#7 Updated by intrigeri 2015-05-30 09:22:18

• Status changed from Confirmed to In Progress
• Assignee changed from intrigeri to anonym
• % Done changed from 0 to 50
• QA Check set to Ready for QA

So, here’s a quick summary of the current state of things in the test/7066-run-tbb-tests branch:

• that’s a quite ugly proof-of-concept only, in no way ready to be merged;
• at test suite runtime, we clone (our own clone of) the Tor Browser’s test suite Git repos and run its install-deps and setup-virtualenv scripts; this is not very nice, and it’s also quite slow, but I’m not sure how we can do better without bloating the Tails ISO;
• the Tails firewall is disabled, as a shortcut to allow Tor Browser to talk to whatever local web server is set up by Mozmill and Selenium; one should check if they can use a fixed port, and if so, instead we should add a test suite step that inserts the iptables/netfilter rules needed to allow the test suite to run, instead of entirely disabling the firewall;
• Selenium and Mozmill tests are run in a way that bypasses the tbb-testsuite wrapper, IIRC because back then there was no way to enable only the tests we want; once this is fixed (which might be the case already, see the new --testsuite= option) we should run something like ./tbb-testsuite --tor-control-port=9052 --tor-socks-port=9150 --no-starttor --enable-tests=...;
• we run all Selenium tests;
• we run some Mozmill tests, excluding those that depend on tbb-testsuite to have prepared things ahead.

Now zooming out. The Tor Browser’s and Tails’ test suites have a very different general approach:

• the Tails test suite (almost) only interacts with the browser in the very same way as a user would; and it checks success/failure either from the outside (blackbox style), or by looking at things a user would see;
• the Tor Browser’s test suite interacts with the browser using JS that Mozmill/Selenium have injected into the browser; and it checks success/failure the same way, asking the browser about its internal state.

As a result, on the one hand many kinds of tests seem to be way easier to develop within the Tor Browser’s test suite framework. On the other hand, of course the browser under testing is modified (which we generally avoid) and whatever it says about success/failure is blindly trusted (while we generally try to check things from the outside). IMO we should integrate the TB’s test suite into our own (as the PoC branch does), and whenever we want to add browser tests, we should decide, on a case-by-case basis, which approach is more appropriate (in terms of “cost/benefit”, as they would say).

To end with, I see a few major reasons why we should run the TB test suite inside our own:

1. it cheaply increases test coverage of the Tor Browser as shipped in Tails;
2. it allows boklm to easily and continuously check that TB changes don’t break Tails use case;
3. if we combined this with automatically building ISO images that include dev snapshots of the Tor Browser, then we could identify problems earlier in the dev process than when we import a new TB;
4. it widens the collaboration area between Tails and TB teams, by providing more reasons to communicate and keep each other informed of what we’re up to.

Now, if we agree that it’s the way to go, then I’ll close this ticket and create one (or more) new tickets about making this real and bringing it to a mergeable state.

Thoughts? (this research is “Ready for QA”, not the branch itself :)

#8 Updated by anonym 2015-06-03 14:18:44

• Assignee changed from anonym to intrigeri
• QA Check changed from Ready for QA to Dev Needed

intrigeri wrote:
> * the Tails firewall is disabled, as a shortcut to allow Tor Browser to talk to whatever local web server is set up by Mozmill and Selenium; one should check if they can use a fixed port, and if so, instead we should add a test suite step that inserts the iptables/netfilter rules needed to allow the test suite to run, instead of entirely disabling the firewall;

Doesn’t the TB test suite at all times capture the network and look for leaks? Any way we should be good if the scenario in our test suite that run’s the TB test suite is tagged @check_tor_leaks unless we have leaks that the firewall blocks, which then will be discovered. Right?

> Now zooming out. The Tor Browser’s and Tails’ test suites have a very different general approach:
>
> * the Tails test suite (almost) only interacts with the browser in the very same way as a user would; and it checks success/failure either from the outside (blackbox style), or by looking at things a user would see;
> * the Tor Browser’s test suite interacts with the browser using JS that Mozmill/Selenium have injected into the browser; and it checks success/failure the same way, asking the browser about its internal state.
>
> As a result, on the one hand many kinds of tests seem to be way easier to develop within the Tor Browser’s test suite framework. On the other hand, of course the browser under testing is modified (which we generally avoid) and whatever it says about success/failure is blindly trusted (while we generally try to check things from the outside). IMO we should integrate the TB’s test suite into our own (as the PoC branch does), and whenever we want to add browser tests, we should decide, on a case-by-case basis, which approach is more appropriate (in terms of “cost/benefit”, as they would say).

I agree.

> To end with, I see a few major reasons why we should run the TB test suite inside our own:
>
> # it cheaply increases test coverage of the Tor Browser as shipped in Tails;
> # it allows boklm to easily and continuously check that TB changes don’t break Tails use case;
> # if we combined this with automatically building ISO images that include dev snapshots of the Tor Browser, then we could identify problems earlier in the dev process than when we import a new TB;
> # it widens the collaboration area between Tails and TB teams, by providing more reasons to communicate and keep each other informed of what we’re up to.

Awesome!

> Now, if we agree that it’s the way to go, then I’ll close this ticket and create one (or more) new tickets about making this real and bringing it to a mergeable state.
>
> Thoughts? (this research is “Ready for QA”, not the branch itself :)

Let’s do it!

#10 Updated by intrigeri 2015-06-11 19:48:10

• blocks Feature #9561: Run the Tor Browser's own test suite inside our own added

#11 Updated by intrigeri 2015-06-11 19:48:48

• Status changed from In Progress to Resolved
• Assignee deleted (intrigeri)
• % Done changed from 50 to 100
• QA Check changed from Dev Needed to Pass
• Affected tool set to Browser