Feature #10394

Identify which of the remaining manual tests have the best cost/benefit to automate

Added by sajolida 2015-10-20 09:29:28 . Updated 2016-11-13 12:01:02 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Test suite
Target version:
Start date:
2015-10-20
Due date:
% Done:

50%

Feature Branch:
Type of work:
Research
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

| Ticket | Title                                                                                | Benefit for testers | Benefit for devs | Cost | Cost / benefit |
|--------+--------------------------------------------------------------------------------------+---------------------+------------------+------+----------------|
| <a class='issue tracker-2 status-9 priority-4 priority-default child' href='/code/issues/10264' title='Automate Tor Browser WebRTC tests '>Feature #10264</a> | Automate Tor Browser WebRTC tests                                                    |                   2 |                1 |    1 |     0.33333333 |
| <del><a class='issue tracker-2 status-5 priority-4 priority-default closed child' href='/code/issues/10253' title='Test that Google is the default search plugin in Unsafe Browser'>Feature #10253</a></del> | Test that Google is the default search plugin in Unsafe Browser                      |                   2 |                1 |    1 |     0.33333333 |
| <a class='issue tracker-2 status-7 priority-4 priority-default child' href='/code/issues/10278' title='Automatically test that services that we disable are not started'>Feature #10278</a> | Automatically test that services that we disable are not started                     |                   3 |                2 |    2 |            0.4 |
| <a class='issue tracker-2 status-9 priority-4 priority-default child' href='/code/issues/10259' title='Automate test for checking whether Tor is up-to-date'>Feature #10259</a> | Automate test for checking whether Tor is up-to-date                                 |                   1 |                1 |    1 |            0.5 |
| <a class='issue tracker-2 status-9 priority-4 priority-default child' href='/code/issues/10274' title='Automatically test that we cannot login as root'>Feature #10274</a> | Automatically test that we cannot login as root                                      |                   1 |                1 |    1 |            0.5 |
| <a class='issue tracker-2 status-7 priority-4 priority-default child' href='/code/issues/8788' title='Automatically test the &#34;Tails documentation&#34; link on the Desktop'>Feature #8788</a>  | Automatically test the "Tails documentation" link on the Desktop                     |                   3 |                1 |    2 |            0.5 |
| i18n   | Internationalization <a class='issue tracker-2 status-9 priority-4 priority-default child' href='/code/issues/10261' title='Test that the correct keyboard layout is set '>Feature #10261</a> + <a class='issue tracker-2 status-9 priority-4 priority-default child' href='/code/issues/10263' title='Test that the on-screen keyboard works and its layout is correctly set'>Feature #10263</a> + <a class='issue tracker-2 status-7 priority-4 priority-default child' href='/code/issues/10265' title='Test that DDG is the default search engine in Tor Browser'>Feature #10265</a> + <del><a class='issue tracker-2 status-5 priority-4 priority-default closed child' href='/code/issues/10267' title='Test that the search plugins (disconnect.me, WP, and StartPage) are localized'>Feature #10267</a></del>                               |                  10 |                8 |   10 |           0.56 |
| <a class='issue tracker-2 status-9 priority-4 priority-default child' href='/code/issues/10276' title='Automatically test that the Greeter is displayed correctly in low screen resolutions'>Feature #10276</a> | Automatically test that the Greeter is displayed correctly in low screen resolutions |                   3 |                2 |    3 |            0.6 |
| <a class='issue tracker-2 status-9 priority-4 priority-default child' href='/code/issues/10260' title='Automate the Tor Browser folks&#39; manual test suite'>Feature #10260</a> | Automate the Tor Browser folks' manual test suite                                    |                   4 |                2 |    4 |     0.66666667 |
| <del><a class='issue tracker-2 status-5 priority-4 priority-default closed child' href='/code/issues/10268' title='Automate test of /etc/default/htpdate.user-agent'>Feature #10268</a></del> | Automate test of getTorBrowserUserAgent                                              |                   1 |                0 |    1 |              1 |
| <a class='issue tracker-1 status-9 priority-3 priority-default child' href='/code/issues/9097' title='Automatic test: measure boot time'>Bug #9097</a>  | Automatic test: measure boot time                                                    |                   4 |                1 |    5 |              1 |
| <a class='issue tracker-2 status-9 priority-4 priority-default child' href='/code/issues/10262' title='Automate Tor Browser fingerprint tests'>Feature #10262</a> | Automate Tor Browser fingerprint tests                                               |                   3 |                2 |    5 |              1 |
| <a class='issue tracker-2 status-9 priority-4 priority-default child' href='/code/issues/10254' title='Test that the internal links in the documentation included with Tails work fine'>Feature #10254</a> | Test that the documentation included with Tails is correct                           |                   1 |                0 |    1 |              1 |
| <a class='issue tracker-2 status-9 priority-4 priority-default child' href='/code/issues/10252' title='Test that FTP connections to LAN IPs works in Unsafe Browser'>Feature #10252</a> | Test that FTP connections to LAN IPs works in Unsafe Browser                         |                   1 |                1 |    2 |              1 |
| <a class='issue tracker-2 status-9 priority-4 priority-default child' href='/code/issues/10269' title='Test that spellcheck works in Tor Browser for each of our supported languages'>Feature #10269</a> | Test that spellcheck works in Tor Browser for each of our supported languages        |                   2 |                1 |    3 |              1 |
| <a class='issue tracker-2 status-7 priority-4 priority-default child' href='/code/issues/10277' title='Automatically test that Tails has booted correctly and without errors'>Feature #10277</a> | Automatically test that Tails has booted correctly and without errors                |                   4 |                3 |    9 |      1.2857143 |
| <a class='issue tracker-2 status-9 priority-4 priority-default child' href='/code/issues/10271' title='Test that spellcheck&#39;s suggestions in Tor Browser are for the correct language'>Feature #10271</a> | Test that spellcheck's suggestions in Tor Browser are for the correct language       |                   2 |                1 |    5 |      1.6666667 |

Benefit = benefit for testers + benefit for developers

Subtasks

History

#1 Updated by sajolida 2015-10-20 09:30:52

I’ll start with a list of the most time-consuming tests, then ask for confirmation to emmapeel and mercedes, than hand this to anonym and kytv for the “cost” part.

#2 Updated by sajolida 2015-11-02 15:12:58

  • Target version changed from Tails_1.7 to Tails_1.8

#3 Updated by sajolida 2015-11-04 11:52:17

  • Description updated
  • Status changed from Confirmed to In Progress
  • Assignee changed from sajolida to mercedes508
  • QA Check set to Info Needed

Done as a first draft. mercedes508, do you mind having a second look and adjusting my numbers? The idea is to rate the benefit of removing each of the remaining manual tests on a scale from 1 to 5. I took into account the time it takes to execute the test (do you have to reboot, multiple times, etc.), the complexity of understanding what’s expected from the test and misinterpret it, and whether the test requires special setup, conditions, hardware, etc.

Do you mind having a look?

Then please reassign it to kytv or anonym so they can rate the cost of automating them. Then they can reassign it to me for the final cost/benefit computation!

#4 Updated by sajolida 2015-11-04 11:53:28

  • Description updated

#5 Updated by sajolida 2015-11-04 11:53:53

  • Description updated

#6 Updated by mercedes508 2015-11-04 16:09:34

  • Assignee changed from mercedes508 to anonym

I would change some to the following:

Automate Tor Browser WebRTC tests 2
Automatically test behavior on boot medium removal 2
Test that the OpenPGP Applet’s text is readable when Windows Camouflage is enabled 1
Test that Browsers in Tails use the IE theme when Windows Camouflage is enabled 1
Test that Vidalia does not start when Windows Camouflage is enabled 1
Test that Google is the default search plugin in Unsafe Browser 1

#7 Updated by sajolida 2015-11-06 02:13:40

  • Description updated

I updated the description according to your feedback, then. Thanks!

#8 Updated by intrigeri 2015-12-05 13:59:40

  • Target version changed from Tails_1.8 to Tails_2.2

Postponing to after January, since times will be crazy until then.

#9 Updated by anonym 2016-03-03 16:44:58

  • Target version changed from Tails_2.2 to Tails_2.3

#10 Updated by anonym 2016-04-26 09:36:38

  • Target version changed from Tails_2.3 to Tails_2.4

#11 Updated by anonym 2016-06-02 12:18:42

So I’ve been postponing this one for 6 months. One reason is that we’re already overloaded with making existing tests work reliably, and this being a step towards adding more tests haven’t felt very motivating.

Also, it can be unexpectedly hard to estimate the cost of implementing a test — the tools we work with have strange limitations, and the modern desktop is an unholy mess when you get down to the details. Stuff doesn’t work reliably/deterministically, but we, as intelligent users, are so used to react to these tiny glitches and are creative enough to improvise solutions almost subconsiously. But the situation is very different when trying to interact with these things in an automated/scripted way, where nothing can be left out to improvisation.

So, I’m not sure what to do here. Personally I’d like to wait with this estimation (and the thought of new tests) until we have the current set of implemented tests under control. What do you think about this, sajolida and intrigeri.

#12 Updated by intrigeri 2016-06-02 12:59:54

> Personally I’d like to wait with this estimation (and the thought of new tests) until we have the current set of implemented tests under control.

Full ACK, this totally makes sense to me. It would feel quite futile and theoretical to do such future planning at this point.

#13 Updated by anonym 2016-06-02 13:40:52

  • Assignee changed from anonym to sajolida

intrigeri wrote:
> > Personally I’d like to wait with this estimation (and the thought of new tests) until we have the current set of implemented tests under control.
>
> Full ACK, this totally makes sense to me. It would feel quite futile and theoretical to do such future planning at this point.

Ok. Waiting for sajolida’s opinion, then.

I guess we could express this proposal in Redmine as “Blocked by: Bug #10288” (Bug #10288 == “Fix newly identified issues to make our test suite more robust and faster”) but I have my doubts we’ll be in a place where this makes sense when SponsorS is done.

#14 Updated by sajolida 2016-06-02 14:40:43

Thanks a lot for communicating about this!

All the work on the test suite is visible to me only through the sponsor reports and stuff I hear from intrigeri, so it’s super important for me when you share the big picture like this. What you’re saying totally makes sense and what I understood from your recent backend work for the test suite sounds much more important to finish than writing more fragile tests. All this is equally important to translate to the funders: we’ve done that in the final report for SponsorR and should continue doing that for SponsorS.

I’ll setup the blocking relationship.

#15 Updated by sajolida 2016-06-02 14:40:58

  • blocked by Bug #10288: Fix newly identified issues to make our test suite more robust and faster added

#16 Updated by sajolida 2016-06-04 10:15:56

  • Assignee deleted (sajolida)
  • Target version deleted (Tails_2.4)
  • QA Check deleted (Info Needed)

#17 Updated by sajolida 2016-06-04 10:16:41

  • Assignee set to anonym

Actually, I think this should still be assigned to anonym even if blocked for now and with no ETA. Feel free to change if you think that’s wrong.

#18 Updated by anonym 2016-06-04 10:52:15

sajolida wrote:
> Thanks a lot for communicating about this!
>
> All the work on the test suite is visible to me only through the sponsor reports and stuff I hear from intrigeri, so it’s super important for me when you share the big picture like this.

I see. Don’t be afraid to ask me of this when you need such input — it’s not clear to me when/how I should communicate this to you.

> What you’re saying totally makes sense and what I understood from your recent backend work for the test suite sounds much more important to finish than writing more fragile tests. All this is equally important to translate to the funders: we’ve done that in the final report for SponsorR and should continue doing that for SponsorS.

Ack.

> I’ll setup the blocking relationship.

Thanks!

> Actually, I think this should still be assigned to anonym even if blocked for now and with no ETA. Feel free to change if you think that’s wrong.

Sure, I assigned it to you only to get your opinion.

#19 Updated by intrigeri 2016-11-10 14:58:24

  • Description updated

#20 Updated by intrigeri 2016-11-10 14:58:56

  • Description updated

#21 Updated by intrigeri 2016-11-10 15:00:35

  • Assignee changed from anonym to sajolida
  • % Done changed from 0 to 50

We did what was requested from us. We grouped some stuff together and modified some of the cost (for testers) estimates that felt wrong. We’re going to use this data to prepare our next SponsorS proposal. Anything else you need from us?

#22 Updated by intrigeri 2016-11-10 15:01:10

Also, note that test cases that have “Benefit for devs” == 0 should go away in our opinion.

#23 Updated by sajolida 2016-11-13 12:00:46

  • blocks deleted (Bug #10288: Fix newly identified issues to make our test suite more robust and faster)

#24 Updated by sajolida 2016-11-13 12:01:02

  • Status changed from In Progress to Resolved
  • Assignee deleted (sajolida)

Works for me.

I’m all for tests going away :)