Tails

#1 Updated by anonym 2015-09-27 06:37:21

• Description updated

#2 Updated by intrigeri 2015-09-28 02:02:07

• blocks #8668 added

#3 Updated by intrigeri 2015-09-28 02:02:44

• Deliverable for set to 270

#4 Updated by anonym 2015-09-28 04:52:13

• related to Feature #10287: Set up limited email notification on automatic test failure for the initial deployment added

#5 Updated by anonym 2015-10-04 12:53:19

• File <del>missing: analysis-summary.txt</del> added
• File <del>missing: tailstester1-json.tar.bz2</del> added
• File json-analysis added
• Assignee changed from anonym to kytv
• QA Check set to Dev Needed

Edit: Removing comment. So much was wrong with these early isotester1 runs that it’s just confusing the discussion on this ticket.

#6 Updated by anonym 2015-10-12 03:48:23

• File deleted (analysis-summary.txt)

#7 Updated by anonym 2015-10-12 03:48:30

• File deleted (tailstester1-json.tar.bz2)

#8 Updated by anonym 2015-10-12 04:51:14

• File full-summary.txt added

Here’s a summary for the last 12 runs on isotester1 (simply json-analysis *.json):

Step failure breakdown (total: 55):
* 24    Step: the Unsafe Browser works in all supported languages
  - 24    Scenario: The Unsafe Browser can be used in all languages supported in Tails
* 10    Step: Tor is ready
  - 5     Scenario: Clock way in the future
  - 2     Scenario: Using obfs2 pluggable transports
  - 1     Scenario: Clock with host's time
  - 1     Scenario: The tor process should be confined with Seccomp
  - 1     Scenario: I2P is enabled when the "i2p" boot parameter is added
* 6     Step: Pidgin successfully connects to the "irc.oftc.net" account
  - 4     Scenario: Connecting to the #tails IRC channel with the pre-configured account
  - 2     Scenario: Using a persistent Pidgin configuration
* 4     Step: the OpenPGP keys shipped with Tails will be valid for the next 3 months
  - 4     Scenario: The shipped Tails OpenPGP keys are up-to-date
* 2     Step: the Tor Browser has started and loaded the Tails roadmap
  - 2     Scenario: Connecting to the #tails IRC channel with the pre-configured account
* 1     Step: I see "SSHAuthVerification.png" after at most 60 seconds
  - 1     Scenario: SSH is using the default SocksPort
* 1     Step: I configure some Bridge pluggable transports in Tor Launcher
  - 1     Scenario: Clock way in the future in bridge mode
* 1     Step: I create a new bitcoin wallet
  - 1     Scenario: Using a persistent Electrum configuration
* 1     Step: I open "/home/amnesia/Persistent/default-testpage.pdf" with Evince
  - 1     Scenario: I can view and print a PDF file stored in persistent /home/amnesia/Persistent but not /home/amnesia/.gnupg
* 1     Step: I click the blocked video icon
  - 1     Scenario: Watching a WebM video
* 1     Step: I fetch the "10CC5BC7" OpenPGP key using the GnuPG CLI without any signatures
  - 1     Scenario: Syncing OpenPGP keys using Seahorse started from the Tails OpenPGP Applet should work and be done over Tor.
* 1     Step: I connect Gobby to "gobby.debian.org"
  - 1     Scenario: Gobby is using the default SocksPort
* 1     Step: the Tor Browser has started and loaded the startup page
  - 1     Scenario: Importing an OpenPGP key from a website
* 1     Step: I see "WindowsStartMenu.png" after at most 10 seconds
  - 1     Scenario: The panel menu should look like Microsoft Windows's start menu

Some more detailed analysis:

* 24    Step: the Unsafe Browser works in all supported languages
  - 24    Scenario: The Unsafe Browser can be used in all languages supported in Tails

This is because isotester1 doesn’t set an UTF-8 locale. See Bug #10359.

Also, we apparently have duplicated the 'The Unsafe Browser can be used in all languages supported in Tails' scenario in both localization.feature and unsafe_browser.feature.

* 10    Step: Tor is ready
  - 5     Scenario: Clock way in the future
  - 2     Scenario: Using obfs2 pluggable transports
  - 1     Scenario: Clock with host's time
  - 1     Scenario: The tor process should be confined with Seccomp
  - 1     Scenario: I2P is enabled when the "i2p" boot parameter is added

7 of these happen in wait_until_tor_is_working indicating that Feature #9516 didn’t solve everything. In fact, all those errors occur in the ‘Tor is ready’ step, implying that none seem to occur when we restore from a snapshot. My impression is that before Feature #9516 the Tor bootstrapping issues happened just as often when restoring from snapshot. Add to that that the case where we run wait_until_tor_is_working after restoring a snapshot happens a lot more frequently than in the ‘Tor is ready’ step. This seems to indicate that Feature #9516 has issues with the initial bootstrap, e.g. when combined with tordate.

The other failures happened in the (sub) step ‘the time has synced’, and specifically it was htpdate. We should add some improved error logging (e.g. dump contents of /var/log/htpdate.log) and possibly logic for retrying htpdate on failure. Although that should be thought about carefully since these errors affect users as much and hence are real issues.

Any way, we cannot mark all scenarios using this step (incl. those depending on snapshots using it) @fragile since it would disable all tests using the network, essentially.

* 6     Step: Pidgin successfully connects to the "irc.oftc.net" account
  - 4     Scenario: Connecting to the #tails IRC channel with the pre-configured account
  - 2     Scenario: Using a persistent Pidgin configuration

All are of the type: “The operation failed (despite forcing 5 new Tor circuits) with: RuntimeError: Connecting to account irc.oftc.net failed. (TorFailure)”.

OFTC blocking is still a problem. We could try bumping the retries from 5 to 10. Thoughts?

* 4     Step: the OpenPGP keys shipped with Tails will be valid for the next 3 months
  - 4     Scenario: The shipped Tails OpenPGP keys are up-to-date

Expected. We need to be more proactive about updating the key… :)

* 2     Step: the Tor Browser has started and loaded the Tails roadmap
  - 2     Scenario: Connecting to the #tails IRC channel with the pre-configured account

The ‘I see the Tails roadmap URL’ doesn’t use the retrying-magic we have in the 'I open the address ...' step. We should refactor out that code from the latter so it can be used in the former step.

* 1     Step: I see "SSHAuthVerification.png" after at most 60 seconds
  - 1     Scenario: SSH is using the default SocksPort

I’m unsure what’s wrong here. For some reason isotester1 doesn’t have any artifacts except the json log for run 41, so I cannot investigate further.

* 1     Step: I configure some Bridge pluggable transports in Tor Launcher
  - 1     Scenario: Clock way in the future in bridge mode

Known to be very fragile. We should probably mark all the “way in the past/future” scenarios as @fragile right away.

* 1     Step: I create a new bitcoin wallet
  - 1     Scenario: Using a persistent Electrum configuration

This was on run 45, and the error screenshot shows an image of a prompt saying “You are offline”. Interesting.

* 1     Step: I open "/home/amnesia/Persistent/default-testpage.pdf" with Evince
  - 1     Scenario: I can view and print a PDF file stored in persistent /home/amnesia/Persistent but not /home/amnesia/.gnupg

Here we failed to find GnomeTerminalWindow.png. Can’t investigate further due to missing artifacts in run 38.

* 1     Step: I click the blocked video icon
  - 1     Scenario: Watching a WebM video

Run 35. Looking at the trace is interesting:

    And I open the address "https://webm.html5.org/test.webm" in the Tor Browser      # features/step_definitions/common_steps.rb:550
    And I click the blocked video icon                                                # features/step_definitions/common_steps.rb:838
      FindFailed: can not find TorBrowserBlockedVideo.png on the screen.

In the error screenshot I can see that the Tor Browser is showing the “The connection has timed out” page. Something must be wrong with our recent improvements in the ‘I open the address ...’ step since it accepted this page. Possibly the existing condition isn’t enough, but we should also check that we do not see this particular error page, or perhaps any error page. Thoughts?

* 1     Step: I fetch the "10CC5BC7" OpenPGP key using the GnuPG CLI without any signatures
  - 1     Scenario: Syncing OpenPGP keys using Seahorse started from the Tails OpenPGP Applet should work and be done over Tor.

Another “The operation failed (despite forcing 5 new Tor circuits)”. Either we have to bump the retries, or start investigating the OpenPGP server issue again.

* 1     Step: I connect Gobby to "gobby.debian.org"
  - 1     Scenario: Gobby is using the default SocksPort

Run 43. The error screenshot shows that Gobby is still trying to resolve gobby.debian.net. I guess we need Tor retry magic here.

* 1     Step: the Tor Browser has started and loaded the startup page
  - 1     Scenario: Importing an OpenPGP key from a website

Happened in run 39, which lacks artifacts so I cannot investigate. Our recent retry magic should have fixed it, but maybe it’s another instance of the “The connection has timed out” page messing things up, like above?

* 1     Step: I see "WindowsStartMenu.png" after at most 10 seconds
  - 1     Scenario: The panel menu should look like Microsoft Windows's start menu

Happened in run 39, which lacks artifacts so I cannot investigate.

#9 Updated by anonym 2015-10-12 04:55:08

• Status changed from Confirmed to In Progress
• Target version changed from Tails_1.8 to Tails_1.7
• QA Check changed from Dev Needed to Info Needed

Actually, we need to deal with the @fragile tagging and branch creation ASAP since the jenkins deplyment is imminent.

kytv, can you please go through my analysis above? Could you create tickets where you agree that there is a problem? I think we should skip those where we didn’t get artifacts (and hence have no clue what the issue is) so they will be re-run and fail again soon with artifacts. I can then do the @fragile tagging + branch creation since that’s easier with commit rights.

#14 Updated by kytv 2015-10-15 06:29:27

• blocked by Bug #10378: The "Tails OpenPGP keys" scenario is fragile added

#15 Updated by kytv 2015-10-15 06:53:52

• blocked by Bug #10380: gobby tests are fragile added

#16 Updated by kytv 2015-10-15 07:05:01

• blocked by Bug #10381: The "I open the address" steps are fragile added

#18 Updated by intrigeri 2015-10-16 02:51:42

• Subject changed from Fix newly identified robustness issues in the automated test suite to Fix newly identified issues to make our test suite more robust and faster, phase 1
• Deliverable for changed from 270 to 267

anonym, kytv: we actually have a deliverable on this for milestone III, and another for milestone VI. The second one already had a ticket (Bug #10096) so I’m tweaking this one to track the milestone III goals. In practice, of course you’ll be continuously fixing stuff as part of intermediary milestones as well. That’s a big pile of work (whose size is not known yet) to be done over the courses of 9.5 months, so it’ll need to be split into smaller batches, so that it’s clear what are the ones you want to fix during the current cycle, etc. I suggest you two draft that in two weeks, when we know more about how large the problem is, but still in time for the next CI team meeting.

#19 Updated by intrigeri 2015-10-16 02:52:02

• related to Bug #10096: Fix newly identified issues to make our test suite more robust and faster, phase 2 added

#20 Updated by intrigeri 2015-10-16 03:14:06

• blocks deleted (Bug #10380: gobby tests are fragile)

#21 Updated by intrigeri 2015-10-16 03:14:56

• blocks deleted (Bug #10378: The "Tails OpenPGP keys" scenario is fragile)

#22 Updated by intrigeri 2015-10-16 03:15:41

• blocks deleted (Bug #10381: The "I open the address" steps are fragile)

#23 Updated by anonym 2015-10-16 07:45:57

• Description updated
• Status changed from In Progress to Confirmed

#24 Updated by anonym 2015-10-16 10:19:16

• Description updated

#27 Updated by intrigeri 2015-11-06 06:03:40

• Subject changed from Fix newly identified issues to make our test suite more robust and faster, phase 1 to Fix newly identified issues to make our test suite more robust and faster
• Target version changed from Tails_1.7 to Tails_2.5
• QA Check deleted (Info Needed)
• Deliverable for changed from 267 to 270

#28 Updated by intrigeri 2015-12-07 07:08:11

• File <del>missing: summary-20151201-to-20151207.txt</del> added

Here’s the output of json-analysis for the test suite runs since the beginning of December.

#29 Updated by intrigeri 2015-12-07 07:09:20

• Description updated

(Documenting how to create such a summary without too much by-hand painful operations).

#30 Updated by intrigeri 2015-12-07 07:43:43

• File deleted (summary-20151201-to-20151207.txt)

#31 Updated by intrigeri 2015-12-07 07:44:23

• File summary-20151201-to-20151207.txt added
• Description updated

(Updated summary + doc to include details requested by bertagaz.)

#32 Updated by intrigeri 2016-02-20 14:49:10

• Subject changed from Fix newly identified issues to make our test suite more robust and faster to Fix newly identified issues to make our test suite more robust and faster
• Assignee changed from kytv to anonym

#33 Updated by sajolida 2016-06-02 14:40:58

• blocks Feature #10394: Identify which of the remaining manual tests have the best cost/benefit to automate added

#34 Updated by BitingBird 2016-06-26 10:38:29

• Status changed from Confirmed to In Progress

#35 Updated by anonym 2016-07-18 06:42:53

• Target version changed from Tails_2.5 to Tails_2.7

#36 Updated by intrigeri 2016-07-18 06:50:36

• Deliverable for changed from 270 to SponsorS_Internal

#39 Updated by intrigeri 2016-08-18 07:33:58

• Assignee deleted (anonym)
• Target version deleted (Tails_2.7)
• Deliverable for deleted (SponsorS_Internal)

#40 Updated by intrigeri 2016-08-18 07:42:48

• related to Feature #11355: Re-enable Jenkins notifications on ISO build/test failure added

#41 Updated by sajolida 2016-11-13 12:00:46

• blocked by deleted (Feature #10394: Identify which of the remaining manual tests have the best cost/benefit to automate)

#42 Updated by Anonymous 2017-06-29 08:55:45

• Assignee set to anonym

Assigning the parent ticket to anonym for tracking.

#43 Updated by Anonymous 2018-01-15 11:36:02

• blocked by deleted (#8668)

#44 Updated by intrigeri 2019-10-19 16:19:21

• Assignee deleted (anonym)

(I’m not sure what this ticket is useful for at this point. Either way, presuming that anonym will fix All The Things™ matches neither recent reality, nor current plans.)

#45 Updated by intrigeri 2019-12-21 15:16:17

• related to Bug #16959: Gather usability data about our current CI added