Feature #7563

Update the automated test suite for Jessie ISO images

Added by BitingBird 2014-07-11 13:09:54 . Updated 2016-02-05 13:36:34 .

Status:
Resolved
Priority:
High
Assignee:
Category:
Test suite
Target version:
Tails_2.0
Start date:
2014-11-26
Due date:
2016-01-15
% Done:

100%

Feature Branch:
feature/7563-test-suite-on-jessie
Type of work:
Code
Blueprint:

Starter:
0
Affected tool:
Deliverable for:
268

Description

Subtasks

Bug #8317: fillram isn't effective for filling the memory in Jessie Resolved

100
Bug #8782: Update the "all notifications have disappeared" test suite step for Jessie Resolved

100
Bug #9046: Update the encryption test for Jessie Resolved

100
Bug #9695: "I start the Tor Browser in offline mode" step starts... Pidgin on Jessie Resolved

100
Bug #9692: Test suite's --retry-find option breaks wait_for_gnome_window Rejected

100
Bug #9704: iptables_parse is buggy for IPv6 Resolved

100
Bug #9705: Update the memory erasure automated tests for Jessie Resolved

100
Feature #9706: Jessie: applications menu handling in the test suite is fragile Resolved

100
Bug #9791: Update torified_gnupg feature for Jessie Resolved

100
Bug #10325: Update the "Connect to server" tests for Jessie Rejected

100
Bug #10336: Update "blocked wireless device" trace for Jessie Rejected

0
Feature #10402: Make sure the new snapshot system works as intended in Jessie Resolved

100
Bug #10403: Screen blanking breaks some tests in Jessie Resolved

100
Bug #10467: "I open the address" step is fragile on Jessie Resolved

100
Bug #10782: Regression in "I synchronize keys in Seahorse" recovery_proc Resolved

100
Bug #10907: usb_install.feature fails when run as part of the entire test suite Resolved

100
Bug #10926: The I2P AppArmor confinement test case succeeds even when I2P is not confined Resolved

100

Related issues

Blocked by Tails - Feature #7424: Have Tails based on Jessie building and starting Resolved 2014-07-14
Blocked by Tails - Feature #7882: Fonts look ugly on Jessie Resolved 2014-09-08 2014-10-20
Blocked by Tails - Bug #8256: Running pkexec in Jessie doesn't open the GNOME prompt Resolved 2014-11-13
Blocked by Tails - Bug #8274: liveusb-creator's gdisk scripts returns error in Jessie Resolved 2014-12-18 2015-03-05
Blocked by Tails - Bug #8275: tails-persistence-setup broken in Jessie Resolved 2014-11-18
Blocked by Tails - Bug #8372: Shutdown/reboot broken in Jessie Resolved 2014-12-03
Blocked by Tails - Bug #9053: syslinux/ldlinux.c32 differs from installation source to destination on Jessie Resolved 2015-03-14
Blocked by Tails - Bug #10537: Fix restart-tor and tor-has-bootstrapped semantics vs. test suite on Jessie Resolved 2015-11-12
Blocked by Tails - Bug #10571: The 'application/pgp-keys' MIME type has bad application associations Resolved 2015-11-17
Blocked by Tails - Bug #10539: "Clone and upgrade" on Jessie pretends my Tails was not installed with our Installer Resolved 2015-11-12
Blocked by Tails - Bug #10591: CUPS fails to start with read-only persistent configuration on Jessie Resolved 2015-11-20
Blocked by Tails - Bug #10593: Totem + torsocks can't read videos on Jessie Resolved 2015-11-20
Blocked by Tails - Bug #10660: Tails Installer 4.x crashes when a pristine USB drive is plugged Resolved 2015-11-25
Blocked by Tails - Bug #10696: Tor bridge mode fails on Jessie when time is way off Resolved 2015-11-30
Blocked by Tails - Bug #10732: Fix tor-has-bootstrapped semantics on network reconnect Resolved 2015-12-09

History

#1 Updated by intrigeri 2014-07-11 14:47:24

  • Subject changed from Update the automated test suite to Jessie to Update the automated test suite for Jessie
  • Parent task set to Bug #7423

#2 Updated by intrigeri 2014-07-11 14:52:36

The rationale for doing this before Jessie is frozen is to detect issues ASAP and hopefully fix them in Jessie.

#3 Updated by intrigeri 2014-07-12 10:23:01

  • blocked by Feature #7424: Have Tails based on Jessie building and starting added

#4 Updated by intrigeri 2014-07-12 10:32:09

  • Tracker changed from Bug to Feature

#5 Updated by anonym 2014-09-08 11:29:13

#6 Updated by intrigeri 2014-10-11 03:46:40

  • Due date set to 2014-10-20
  • Priority changed from Normal to High

Raising priority and setting a due date, as the freeze is now very close, and we’ll need a few days before the upload deadline to file bugs and help get them fixed. anonym, does that sound realistic, or should we rethink who/when does this?

#7 Updated by intrigeri 2014-11-10 16:19:16

  • Due date changed from 2014-10-20 to 2014-12-01

intrigeri wrote:
> Raising priority and setting a due date, as the freeze is now very close, and we’ll need a few days before the upload deadline to file bugs and help get them fixed. anonym, does that sound realistic, or should we rethink who/when does this?

This wasn’t realistic. Postponing deadline. anonym, do you think this will be enough, or should we rething who does this? (Dropping the ball and asking for help is fine. Silently dropping the ball is less fine :)

#8 Updated by anonym 2014-11-13 16:43:49

  • blocked by Bug #8256: Running pkexec in Jessie doesn't open the GNOME prompt added

#9 Updated by anonym 2014-11-13 22:00:28

  • % Done changed from 0 to 20
  • Feature Branch set to test/jessie

intrigeri wrote:
> This wasn’t realistic. Postponing deadline. anonym, do you think this will be enough, or should we rething who does this? (Dropping the ball and asking for help is fine. Silently dropping the ball is less fine :)

Sorry. I’ve had it on my TODO for quite some time. I’ve at least started something now, although it’s quite a bit more work left.

#10 Updated by anonym 2014-11-14 17:39:23

  • Feature Branch changed from test/jessie to feature/jessie

Working directly in feature/jessie from now on, since the test suite work reveals a lot of regressions that I’m fixing along the way.

At the moment the following features passes:

  • features/torified_browsing.feature
  • features/root_access_control.feature
  • features/dhcp.feature

#11 Updated by intrigeri 2014-11-14 17:52:46

> Working directly in feature/jessie from now on, since the test suite work reveals a lot of regressions that I’m fixing along the way.

I think you forgot to push that branch.

#12 Updated by anonym 2014-11-19 17:49:38

  • blocked by Bug #8274: liveusb-creator's gdisk scripts returns error in Jessie added

#13 Updated by anonym 2014-11-19 17:49:55

  • blocked by Bug #8275: tails-persistence-setup broken in Jessie added

#14 Updated by anonym 2014-11-21 17:20:51

At the moment (commit 906940a in feature/jessie) the progress is as follows:

Passes completely:

  • features/apt.feature
  • features/build.feature
  • features/dhcp.feature
  • features/firewall_leaks.feature
  • features/i2p.feature
  • features/root_access_control.feature
  • features/time_syncing.feature
  • features/torified_browsing.feature
  • features/torified_gnupg.feature
  • features/unsafe_browser.feature
  • features/untrusted_partitions.feature

Failures because all GNOME 2.x notification area (i.e. “systray”) applets are mostly broken in GNOME Shell:

  • features/checks.feature:
    - Scenario: The emergency shutdown applet can shutdown Tails
    - Scenario: The emergency shutdown applet can reboot Tails
  • features/encryption.feature

Failures because USB install (Bug #8274) and persistence (Bug #8275) are broken:

  • features/evince.feature
  • features/pidgin.feature
  • features/totem.feature
  • features/usb_install.feature

Other failures:

  • features/erase_memory.feature: See Bug #7183.
  • features/windows_camouflage.feature: our GNOME flashback theme hasn’t been adapted to GNOME Shell.

Ignore:

  • features/truecrypt.feature: we are going to remove truecrypt.

#15 Updated by intrigeri 2014-11-21 18:07:32

> At the moment (commit 906940a in feature/jessie) the progress is as follows:

Congrats! We now know what we have to fix, thanks :)

#16 Updated by intrigeri 2014-11-22 11:01:06

  • Category changed from Test suite to 174

Regarding “features/erase_memory.feature: See Bug #7183”, Bug #7183 is actually now resolved, so the remaining problem is that the screen is too garbled in the test suite environment, and then we can’t find the “Happy dumping!” message, right? Not sure if it’s easier to fix in the runtime code or in the test suite itself.

#17 Updated by intrigeri 2014-11-23 17:07:34

anonym wrote:
> Failures because all GNOME 2.x notification area (i.e. “systray”) applets are mostly broken in GNOME Shell:
> * features/checks.feature:
> - Scenario: The emergency shutdown applet can shutdown Tails
> - Scenario: The emergency shutdown applet can reboot Tails
> * features/encryption.feature

FYI I’ve added the topIcons extension, so this problem shouldn’t be blocking your work anymore :)

#18 Updated by anonym 2014-11-24 16:55:17

  • Category changed from 174 to Test suite

intrigeri wrote:
> Regarding “features/erase_memory.feature: See Bug #7183”, Bug #7183 is actually now resolved, so the remaining problem is that the screen is too garbled in the test suite environment, and then we can’t find the “Happy dumping!” message, right? Not sure if it’s easier to fix in the runtime code or in the test suite itself.

Yeah I see that both Bug #7183 and Bug #8098 are resolved, but nothing has changed for me w.r.t. the garbled screen, so I opened Bug #8300 which is awaiting your clarification. And yes, the issue is that we cannot see the “Happy dumping!” message. I’m not sure how we reliably could wait for the memory wiping to be done otherwise.

#19 Updated by anonym 2014-11-25 12:24:41

intrigeri wrote:
> anonym wrote:
> > Failures because all GNOME 2.x notification area (i.e. “systray”) applets are mostly broken in GNOME Shell:
> > * features/checks.feature:
> > - Scenario: The emergency shutdown applet can shutdown Tails
> > - Scenario: The emergency shutdown applet can reboot Tails
> > * features/encryption.feature
>
> FYI I’ve added the topIcons extension, so this problem shouldn’t be blocking your work anymore :)

Thanks! In commit 4ea49b2 I’ve fixed encryption.feature.

However, there’s still no sign of the emergency shutdown helper applet. Even if we could get it to work with topIcons I do not think it’s what we want. I’m opening a new ticket for how to deal with it in Jessie: Bug #8302

#20 Updated by intrigeri 2014-11-26 07:36:20

> However, there’s still no sign of the emergency shutdown helper applet.

Indeed, since Tails 1.1 it’s not hijacking the notification bar anymore.

#21 Updated by anonym 2014-11-26 21:26:21

With the fixes from Bug #8300, Bug #8302 and Bug #8306, I pushed test suite updates so checks.feature passes.

I also did some work for erase_memory.feature, and the two first (“modern computer”) scenarios of passes but the rest (“old computer”) fails due to low pattern coverage. It becomes something like 55% now (and we end up requiring ~60%) which is getting seriously low, and this is despite there still being ~1300 MiB of free (non-buffer, non-cache) memory. This is getting way too low, so I have opened Bug #8317.

#22 Updated by intrigeri 2014-11-27 09:24:48

  • blocked by Bug #8317: fillram isn't effective for filling the memory in Jessie added

#23 Updated by intrigeri 2014-11-27 23:23:22

  • Status changed from Confirmed to In Progress

#24 Updated by intrigeri 2014-12-18 12:26:35

  • Due date changed from 2014-12-01 to 2015-02-15
  • % Done changed from 20 to 40

#25 Updated by intrigeri 2014-12-21 10:16:04

  • blocked by Bug #8372: Shutdown/reboot broken in Jessie added

#26 Updated by intrigeri 2015-01-06 12:57:22

  • blocks #8538 added

#27 Updated by BitingBird 2015-01-07 18:19:10

  • Subject changed from Update the automated test suite for Jessie to Update the automated test suite for Jessie ISO images

more precisions (“Jessie ISO images”) because I keep confusing with Feature #8165)

#28 Updated by anonym 2015-01-09 14:44:13

  • Target version changed from Tails_2.0 to Tails_1.8

#29 Updated by intrigeri 2015-02-26 10:09:54

Note that I’ve just merged devel into feature/jessie (post-1.3 release) and there were lots of changes in the test suite, and quite a few merge conflicts (e.g. the Synaptic tests have been rewritten both on feature/jessie and in the devel branch, fortunately more or less converging at least conceptually to similar implementations). So I expect that a lot of new and updated tests will now fail on Jessie ISOs.

#30 Updated by intrigeri 2015-03-05 09:44:38

Note that Bug #8274 has been resolved, which should allow us to adapt more parts of the test suite to Jessie.

#31 Updated by intrigeri 2015-03-14 15:18:19

I think the blockers have been resolved, and work on this can be resumed. And anyway, even without fixing those blockers, I suspect that the test suite of Tails/Jessie ISO images was broken enough by changes we’ve introduced in the devel branch in the last few months.

#32 Updated by intrigeri 2015-03-14 19:48:45

  • blocked by Bug #9053: syslinux/ldlinux.c32 differs from installation source to destination on Jessie added

#33 Updated by intrigeri 2015-03-15 19:23:02

I’ve updated a bunch of things for the usb_install feature, since I wanted to validate the work Alan and I have been doing in the last few days. In particular, it’s been ported to UDisks 2, and a few additional low-level tests were added. Now, I gave up after unsuccessfully trying to get rid of FindFailed for: PersistenceWizardWindow.png, that I can’t make sense of. So I’m eagerly waiting for this feature to be updated, so that Alan and I know how much work we have left on Feature #8290 and friends, if any :)

#34 Updated by intrigeri 2015-04-01 13:15:45

#35 Updated by intrigeri 2015-07-06 09:20:27

intrigeri wrote:
> I’ve updated a bunch of things for the usb_install feature, since I wanted to validate the work Alan and I have been doing in the last few days. In particular, it’s been ported to UDisks 2, and a few additional low-level tests were added. Now, I gave up after unsuccessfully trying to get rid of FindFailed for: PersistenceWizardWindow.png, that I can’t make sense of. So I’m eagerly waiting for this feature to be updated, so that Alan and I know how much work we have left on Feature #8290 and friends, if any :)

Fixed in commit:f12419c. This is related to Bug #9692.

#36 Updated by intrigeri 2015-08-02 09:02:41

  • blocks #8668 added

#38 Updated by intrigeri 2015-10-01 07:31:03

We’ll have Jessie sprints in November (9-13 and 16-20). It would be awesome if the test suite was a bit more up-to-date by then, and/or if some of you fixed stuff during these sprints.

#39 Updated by intrigeri 2015-10-01 07:32:51

Given our timeline, it would be good if someone did a full test suite run soon against a Tails/Jessie ISO and added subtasks for broken/missing bits, so that we have an idea of how much work is left to update (and it might be that only doing that finds some bugs in feature/jessie, which I’d love to hear about ASAP :)

#40 Updated by anonym 2015-10-20 09:16:00

intrigeri wrote:
> We’ll have Jessie sprints in November (9-13 and 16-20). It would be awesome if the test suite was a bit more up-to-date by then, and/or if some of you fixed stuff during these sprints.

Indeed. I plan to have this finished by then. Yay!

> Given our timeline, it would be good if someone did a full test suite run soon against a Tails/Jessie ISO and added subtasks for broken/missing bits, so that we have an idea of how much work is left to update (and it might be that only doing that finds some bugs in feature/jessie, which I’d love to hear about ASAP :)

I’ve done a full run. Massive breakage! :) I’ve killed ~half of it so far, and I’d rather spend time fixing the issues than redmine wrangling. Sorry!

I’ve also merged devel in commit commit:b7b2a9a, which had some tricky merge conflicts. It’d be good if someone had a second look.

The rest, commit:b7b2a9a..commit:c63d4b5, is pretty straightforward, except commit:e901c7f (for Bug #9705), although some thought and testing likely will be needed for commit:51a9e41 (see its commit message). I’ll open a ticket for this one, at least.

#41 Updated by intrigeri 2015-10-21 01:00:07

> I’ve also merged devel in commit commit:b7b2a9a, which had some tricky merge conflicts. It’d be good if someone had a second look.

Thanks. Please file a ticket for me to look at it.

#42 Updated by anonym 2015-10-21 03:13:36

anonym wrote:
> […] some thought and testing likely will be needed for commit:51a9e41 (see its commit message). I’ll open a ticket for this one, at least.

Filed as Bug #10395.

intrigeri wrote:
> > I’ve also merged devel in commit commit:b7b2a9a, which had some tricky merge conflicts. It’d be good if someone had a second look.
>
> Thanks. Please file a ticket for me to look at it.

Filed as Feature #10402.

#43 Updated by intrigeri 2015-11-05 02:33:56

  • blocks deleted (Bug #8317: fillram isn't effective for filling the memory in Jessie)

#44 Updated by intrigeri 2015-11-12 01:41:04

  • blocked by Bug #10537: Fix restart-tor and tor-has-bootstrapped semantics vs. test suite on Jessie added

#45 Updated by anonym 2015-11-17 15:26:04

  • blocked by Bug #10571: The 'application/pgp-keys' MIME type has bad application associations added

#46 Updated by intrigeri 2015-11-18 03:41:20

  • blocked by Bug #10539: "Clone and upgrade" on Jessie pretends my Tails was not installed with our Installer added

#47 Updated by intrigeri 2015-11-20 01:07:00

  • blocked by Bug #10591: CUPS fails to start with read-only persistent configuration on Jessie added

#48 Updated by intrigeri 2015-11-20 06:11:06

  • blocked by Bug #10593: Totem + torsocks can't read videos on Jessie added

#49 Updated by intrigeri 2015-11-25 05:07:50

  • blocked by Bug #10660: Tails Installer 4.x crashes when a pristine USB drive is plugged added

#50 Updated by intrigeri 2015-11-30 13:50:47

  • blocked by Bug #10696: Tor bridge mode fails on Jessie when time is way off added

#51 Updated by intrigeri 2015-12-07 13:26:06

#52 Updated by intrigeri 2015-12-07 14:10:34

  • blocked by Bug #9707: Jessie: System sometimes does not poweroff after memory erasure added

#53 Updated by intrigeri 2015-12-09 05:47:38

Anyone looking at this ticket and wondering why it’s still open, while all subtasks are resolved: we’re tracking progress (and things left to do) in this area on a pad elsewhere.

#54 Updated by intrigeri 2015-12-09 06:10:31

  • blocked by Bug #10732: Fix tor-has-bootstrapped semantics on network reconnect added

#55 Updated by intrigeri 2015-12-14 13:10:53

  • Target version changed from Tails_1.8 to Tails_2.0

#56 Updated by anonym 2015-12-14 13:53:55

  • blocked by Bug #10754: Install Electrum 0.2.5.x in Tails/Jessie added

#57 Updated by anonym 2015-12-15 11:26:52

  • blocks deleted (Bug #10754: Install Electrum 0.2.5.x in Tails/Jessie)

#58 Updated by anonym 2015-12-16 11:39:51

  • Feature Branch changed from feature/jessie to feature/7563-test-suite-on-jessie

feature/jessie was merged, creating dedicated branch.

#59 Updated by intrigeri 2015-12-22 13:32:55

  • blocked by deleted (Feature #10583: Release a first Jessie-based beta)

#60 Updated by anonym 2016-01-27 14:36:48

  • Target version changed from Tails_2.0 to Tails_2.2

#61 Updated by intrigeri 2016-02-05 13:35:40

  • blocks deleted (Bug #9707: Jessie: System sometimes does not poweroff after memory erasure)

#62 Updated by intrigeri 2016-02-05 13:36:34

  • Status changed from In Progress to Resolved
  • Assignee deleted (anonym)
  • Target version changed from Tails_2.2 to Tails_2.0

I think we’re done here. Now that 2.0 is out, let’s deal with remaining problems (that are only robustness ones) on Bug #10288.