Feature #9519

Make the test suite more deterministic through network simulation

Added by anonym 2015-06-02 14:15:49 . Updated 2016-08-20 10:56:49 .

Status:
In Progress
Priority:
Normal
Assignee:
anonym
Category:
Test suite
Target version:
Start date:
2015-06-02
Due date:
% Done:

100%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

As a Tails contributor
When Jenkins tells me that I broke something
Then I want to be able to trust it
And I won't ignore such notifications

This would eliminate all or parts of the issues we have with false test failures due to transient network errors.

Subtasks

Feature #9520: Investigate the Shadow network simulator for use in our test suite Rejected

0
Feature #9521: Use the chutney Tor network simulator in our test suite Resolved

100

Related issues

Related to Tails - Bug #9478: How to deal with transient network errors in the test suite? Resolved 2015-05-27
Related to Tails - Bug #12211: Adapt GnuPG automated tests after switching to an Onion keyserver Resolved 2017-02-03
Related to Tails - Bug #14770: "Fetching OpenPGP keys" scenarios are fragile: communication failure with keyserver Resolved 2017-10-04

History

#1 Updated by anonym 2015-06-02 14:17:24

  • related to Bug #9478: How to deal with transient network errors in the test suite? added

#2 Updated by anonym 2015-06-02 14:35:15

Mostly copy-pasted from Bug #9478:

Some problems with this general approach is:

When only simulating the Tor network:

  • We’d have to reconfigure the Tor client to use this fake network. All such reconfigurations are of course bad, since it makes the system under testing deviate from the real situation. However, since we already are using a different Tor network, we’ve deviated quite a lot already. This would be somewhat alleviated with @release tests using the real Internet + Tor network, that we only run for releases or something like that.

When simulating the Tor network and complete internet, we in addition have:

  • We will have to reconfigure many parts of Tails and run servers for at least:
    • the Tails security/upgrade check
    • incremental upgrades
    • Tor Browser home page (and other pages, since we sometimes need more than this one)
    • APT repos
    • whisperback
    • OpenPGP keyserver
    • (perhaps?) check.torproject.org
    • Gobby server
    • (soon) mail servers
  • We probably cannot do anything for I2P.

#3 Updated by anonym 2015-06-03 16:47:17

For a part of the performance side of this, I just ran the full test suit (in branch test/wip-improved-snapshots, which should just be a bit better than the situation in stable) and discovered that out of the full 211 minutes it took to run the test suite, 50 minutes (so ~25%) was spent on waiting for Tor to bootstrap (either initially, or after restoring from a snapshot). This was even with some hacks to restart Tor if the bootstrap progress seemed to stall (see Feature #9516#note-1). With a simulated network, where bootstrapping is ~instant, most of this would be eliminated.

#5 Updated by anonym 2015-06-05 13:24:05

intrigeri wrote:
> Do we really want this to block Feature #8539 (as a subtask, it currently does)? If you feel that it will indeed prevent us from usefully run the test suite in a CI setup, fine with me. Just keep in mind that Feature #8539 is a deliverable with a fixed deadline :)

That was definitely a mistake. Removed!

#6 Updated by intrigeri 2015-09-27 08:56:58

  • Description updated

#7 Updated by BitingBird 2016-06-26 10:43:57

  • Status changed from Confirmed to In Progress

#8 Updated by intrigeri 2017-12-03 21:06:08

  • related to Bug #12211: Adapt GnuPG automated tests after switching to an Onion keyserver added

#9 Updated by intrigeri 2017-12-03 21:06:20

  • related to Bug #14770: "Fetching OpenPGP keys" scenarios are fragile: communication failure with keyserver added