Bug #16004

Make our automated test suite take into account USB image updates wrt. supported installation & update methods

Added by Anonymous 2018-09-28 10:53:48 . Updated 2019-08-29 11:50:11 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Test suite
Target version:
Tails_4.0
Start date:
2018-09-28
Due date:
% Done:

100%

Feature Branch:
test/16004-adapt-usb-scenarios-to-usb-images+force-all-tests
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:
316

Description

- .img + .iso as inputs,

- run different tests on them,
- add scenarios for new cases

Subtasks

Related issues

Blocked by Tails - Bug #16419: devel branch FTBFS because time-based snapshots of the "debian" archive are not updated since Jan 28 Resolved 2019-02-05
Blocks Tails - Bug #16976: Run Dogtail using Python 3 Resolved

History

#1 Updated by Anonymous 2018-09-28 10:55:03

  • blocked by Bug #16005: Code review & rubber-duck added

#2 Updated by intrigeri 2018-10-12 11:58:24

  • blocks deleted (Bug #16005: Code review & rubber-duck)

#3 Updated by CyrilBrulebois 2018-11-27 16:14:19

Initial brainstorming:

  • keep the run_test_suite interface as it is, meaning --iso and --old-iso on the .iso files, and no new options for the .img files.
  • check if there’s an .img file alongside the .iso one, and use it for tests that are specific to the USB image.

#4 Updated by intrigeri 2018-11-28 16:26:07

  • Status changed from Confirmed to In Progress

Applied in changeset commit:tails|c81ef61ea9a852f5d6559cea83cf87443ae4dbbc.

#5 Updated by intrigeri 2018-12-07 13:29:32

#6 Updated by intrigeri 2018-12-16 11:43:06

  • Target version changed from Tails_3.11 to Tails_3.12

#7 Updated by Anonymous 2018-12-27 09:49:55

I’m unsure about the progress of this ticket, if you can, it would be helpful to understand what remains to be done. Thanks!

#8 Updated by intrigeri 2019-01-02 06:39:44

  • Feature Branch set to test/16003-growing-system-partition

#9 Updated by Anonymous 2019-01-03 16:22:42

Hi kibi! i’m still unsure about this one, can you let me know how far you’ve come and when this would be ready? We need to release a beta that is now late. I’m fine doing these tests later, but I need a clear ETA.

#10 Updated by intrigeri 2019-01-04 14:01:35

#11 Updated by Anonymous 2019-01-08 10:59:13

Hi Cyril, could you try to review this for January 21st latest delay please?
We need this to be done on February 1st and I would like to leave some time for intrigeri in case something needs fixing after the review. Thanks!

#12 Updated by intrigeri 2019-01-08 11:07:17

> Hi Cyril, could you try to review this for January 21st latest delay please?

To clarify, what’s needed here is implementation, not reviewing.
We made an implementation plan during our sprint early December.

#13 Updated by Anonymous 2019-01-08 11:41:53

intrigeri wrote:
> > Hi Cyril, could you try to review this for January 21st latest delay please?
>
> To clarify, what’s needed here is implementation, not reviewing.
> We made an implementation plan during our sprint early December.

Thanks for the clarification. This does not change what I said about the deadline though :)
Do you want to share the implementation plan with me (if it contains any ETAs that is, so that I know what your plans are)?

#14 Updated by intrigeri 2019-01-08 14:56:20

> Do you want to share the implementation plan with me (if it contains any ETAs that is, so that I know what your plans are)?

It’s only a technical plan: clarifying specification of what should be done + ideas/guidance for kibi about how to do it. Nothing like ETAs and such in there. We’ve pushed it as comments in the affected test suite files on the topic branch.

#15 Updated by CyrilBrulebois 2019-01-08 17:24:39

FWIW getting back to the test suite (be it for Buster or for USB image) was done during the FT sprint, and Bug #16003 should be in a rather good shape now. I’ll continue with Bug #16004 (this bug) during this week, and send an update on Sunday.

#16 Updated by intrigeri 2019-01-09 18:31:56

  • Subject changed from Adjust automated test suite to Make our automated test suite take into account USB image updates wrt. supported installation & update methods

(Slightly more specific ticket title to better express the plan we did in commit:05f8adba2cee269b03843af1fc8ac89b5d665b3f and commit:c81ef61ea9a852f5d6559cea83cf87443ae4dbbc, which is what this ticket is about.)

#17 Updated by intrigeri 2019-01-09 18:32:13

  • Feature Branch deleted (test/16003-growing-system-partition)

#18 Updated by CyrilBrulebois 2019-01-13 20:09:25

  • Feature Branch set to test/16004-adapt-usb-scenarios-to-usb-images

I’ve pushed the test/16004-adapt-usb-scenarios-to-usb-images branch earlier, which addresses a bunch of the XXX’s added in the “design doc” commits.

One thing I’ve left out so far is:

# XXX: rename to tails_installer.feature and move things that don't use Tails
# Installer elsewhere?

which looked a bit cosmetic/clean-up, compared to the commits I pushed, actually changing some codepaths/checkpoint paths.

Regarding the documented use cases, that should be checked by the test suite, this list was gathered:

* install from Linux: “dd” with GNOME Disks, done; same on Windows/macOS, modulo s/GNOME Disks/Etcher/

* expert Linux install: dd the .img, done

* upgrade from “inside” Tails, i.e. without having another up-to-date Tails: from my Tails d/l USB image and install it with GNOME Disks to an intermediary Tails, then boot on that intermediary Tails, and from there upgrade my Tails with Tails Installer

* Upgrade or install from another up-to-date Tails: restart on the up-to-date Tails, clone with Tails Installer, no ISO/IMG involved

I haven’t checked yet what pieces would be missing in the current set of tests.

#19 Updated by intrigeri 2019-01-13 21:39:31

Here’s a first review, at commit:7f4dac0d8b85e24a334f5b3e97e9ed95d6e45b5f.

My only comments are about commit:90041e5ed41377484c688be718fdd4821aba824f:

  • Maybe add a comment to explain why we bother booting the USB stick? (i.e. so that the repartitioning code runs; no other reason I guess?)
  • I’m not sure but I seem to remember that unplugging the USB drive was done on purpose, to get the VM definition back to a clean state avoid tainting the following scenarios. Given what the current next scenario is, it does not matter, but as things are moved / removed, it might matter some day and I don’t want to debug weird failures that happen because a scenario is using a different version of Tails from the one I’m expecting. It’s possible I’m totally wrong (e.g. if unplugging is a no-op in practice).

That’s all for now, great job! Thanks in particular for the great commit messages: this saves me lots of time.

A first job run started on Jenkins, if the disk usage has grown too much, we’ll know soon enough. In any case, I’ll run the full test suite overnight, measuring the maximum disk space usage: if it has grown, this might indicate regressions (e.g. disks created not as temporarily as they could/should) or be entirely justified. And this full test suite run will also tell us if/how these changes affect other features.

> I haven’t checked yet what pieces would be missing in the current set of tests.

… and perhaps, what pieces we can drop :)

#20 Updated by intrigeri 2019-01-14 07:50:46

> A first job run started on Jenkins, if the disk usage has grown too much, we’ll know soon enough.

The 2 first (partial) test suite runs on Jenkins went fine: no unexpected breakage.

> In any case, I’ll run the full test suite overnight, measuring the maximum disk space usage: […]

I forgot to run the measurements but before I started this, my /tmp/TailsToaster had 22G available, i.e. same as our Jenkins isotesters (modulo units discrepancies and rounding), and this full test suite run went fine (no unexpected breakage) so I think we’re good on this side.

#21 Updated by intrigeri 2019-01-14 18:46:09

  • % Done changed from 0 to 50

Merged into feature/15292-usb-image because this is already better than what we had there.

#22 Updated by intrigeri 2019-01-27 10:50:50

  • Target version changed from Tails_3.12 to Tails_3.13

#23 Updated by CyrilBrulebois 2019-01-31 11:25:44

From Bug #16004#note-19:
> Maybe add a comment to explain why we bother booting the USB stick? (i.e. so that the repartitioning code runs; no other reason I guess?)

Hrm. I guess there aren’t that many reasons, except it was done in the usb-install-tails-greeter snapshot I’ve adapted, which makes it possible to check a few things like the udev-watchdog monitoring.

> I’m not sure but I seem to remember that unplugging the USB drive was done on purpose, to get the VM definition back to a clean state avoid tainting the following scenarios. Given what the current next scenario is, it does not matter, but as things are moved / removed, it might matter some day and I don’t want to debug weird failures that happen because a scenario is using a different version of Tails from the one I’m expecting. It’s possible I’m totally wrong (e.g. if unplugging is a no-op in practice).

I can definitely implement unplugging instead of shutting down, to get unplug + shutdown in a single operation, just to be on the safe side. Checking runtime right now.

#24 Updated by CyrilBrulebois 2019-01-31 13:12:32

  • Assignee changed from CyrilBrulebois to intrigeri
  • QA Check set to Ready for QA

I’ve pushed two extra commits with a switch back to unplugging, and with added comments at the top of the scenario.

Back to “Ready for QA” as I think I’ve addressed everything; thanks for the review!

#25 Updated by intrigeri 2019-02-05 10:58:29

  • Assignee changed from intrigeri to CyrilBrulebois
  • QA Check changed from Ready for QA to Dev Needed

This branch FTBFS on Jenkins. I guess you need to merge current devel into it :)

#26 Updated by intrigeri 2019-02-07 08:06:05

Hi, I’ve noticed you were not a watcher for this ticket so you might have missed my previous comment => added you as a watcher + here’s a gentle poke to make sure you know this is back on your plate :)

#27 Updated by intrigeri 2019-02-08 07:26:55

  • blocked by Bug #16419: devel branch FTBFS because time-based snapshots of the "debian" archive are not updated since Jan 28 added

#28 Updated by intrigeri 2019-02-11 08:50:45

Merged origin/stable into the topic branch, made sure config/base_branch says “stable” (there’s a chance we have no major release for a while and I want this to be merged soon).

#29 Updated by intrigeri 2019-02-11 15:18:27

  • Assignee changed from CyrilBrulebois to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

@CyrilBrulebois incidentally, my last Git mangling fixed the FTBFS => happy to check the test results myself on Jenkins, as an exception :)

#30 Updated by intrigeri 2019-02-12 07:37:10

Jenkins is happy, will code-review.

#31 Updated by intrigeri 2019-02-12 11:19:20

  • Assignee changed from intrigeri to CyrilBrulebois
  • % Done changed from 50 to 60
  • QA Check changed from Ready for QA to Dev Needed

Code review passes at commit:a40123ad5bc451d59b2e52a035bef56c8f9218cf.

Next step: deal with the leftovers of Bug #16004#note-18.

#32 Updated by Anonymous 2019-03-04 14:52:57

hi @CyrilBrulebois : we still miss this part on this ticket: deal with the leftovers of Bug #16004#note-18. Can you please schedule this?

#33 Updated by Anonymous 2019-03-06 11:54:14

#34 Updated by intrigeri 2019-03-15 11:04:55

Feel free to skip “rename to tails_installer.feature and move things that don’t use Tails Installer elsewhere?”, it’s not super useful and indeed, mostly cosmetic.

#35 Updated by intrigeri 2019-03-15 11:07:31

And regarding the last bit, i.e. “I haven’t checked yet what pieces would be missing in the current set of tests”: this shouldn’t take very long and I’m happy to do it myself, as part of my review / rubber-duck work, if you don’t manage to get it done in time for 3.13.

#36 Updated by intrigeri 2019-03-18 09:30:37

  • Assignee changed from CyrilBrulebois to intrigeri
  • Target version changed from Tails_3.13 to Tails_3.14
  • QA Check changed from Dev Needed to Info Needed

CyrilBrulebois wrote:
> Regarding the documented use cases, that should be checked by the test suite, this list was gathered:
>
> * install from Linux: “dd” with GNOME Disks, done; same on Windows/macOS, modulo s/GNOME Disks/Etcher/

AFAICT we have no test about installing Tails with GNOME Disks. This is about GNOME Disks outside of Tails anyway and I don’t think we’re ready to add test cases that boot other operating systems than Tails to our test suite.

> * expert Linux install: dd the .img, done

Indeed, basically all our test cases that boot from USB exercise this.

> * upgrade from “inside” Tails, i.e. without having another up-to-date Tails: from my Tails d/l USB image and install it with GNOME Disks to an intermediary Tails, then boot on that intermediary Tails, and from there upgrade my Tails with Tails Installer

We do exercise the “boot on that intermediary Tails, and from there upgrade my Tails with Tails Installer” part in “Upgrading an old Tails USB installation from another Tails USB drive” but we have no test about “install it with GNOME Disks to an intermediary Tails”: the drive that plays the role of the intermediary Tails was prepared in a test suite ad-hoc way, outside of Tails. If we have time left on the budget, let’s add a test for this (I’ll send you my own clocking data and you can do the maths). Otherwise, forget it.

> * Upgrade or install from another up-to-date Tails: restart on the up-to-date Tails, clone with Tails Installer, no ISO/IMG involved

Cloning with Tails installer: “Installing Tails to a pristine USB drive” exercises the installation; “Re-installing Tails over an existing USB installation with a persistent partition” exercises re-installing; “Upgrading an old Tails USB installation from a Tails DVD” exercises upgrading. So we’re good here.

> I haven’t checked yet what pieces would be missing in the current set of tests.

Done!

#37 Updated by intrigeri 2019-03-18 09:37:40

  • Assignee changed from intrigeri to CyrilBrulebois
  • Priority changed from Normal to Elevated
  • Parent task changed from Feature #15292 to Bug #16002

#39 Updated by CyrilBrulebois 2019-03-18 13:51:39

  • Private changed from No to Yes

@intrigeri Please get back to me on this topic when it’s no longer release days.

#40 Updated by intrigeri 2019-03-18 14:21:24

  • Assignee changed from CyrilBrulebois to intrigeri

OK.

#41 Updated by Anonymous 2019-03-19 10:52:04

  • Private changed from Yes to No

#42 Updated by intrigeri 2019-03-23 17:19:31

  • Assignee changed from intrigeri to CyrilBrulebois

@CyrilBrulebois, 3.13 is in the past so I’m sending this back to your plate wrt. my question on Bug #16004#note-38. As discussed elsewhere, no big hurry. But I’d like an answer early April so we can decide what’s the next step and eventually wrap this project up :)

#43 Updated by Anonymous 2019-04-03 17:34:43

simply unparenting this ticket.

#44 Updated by intrigeri 2019-04-03 18:59:32

u wrote:
> simply unparenting this ticket.

@u, FYI this breaks the “estimated time” data of the parent task, which is why I reverted the exact same change 2 weeks ago.

#45 Updated by Anonymous 2019-04-04 10:12:27

intrigeri wrote:
> u wrote:
> > simply unparenting this ticket.
>
> @u, FYI this breaks the “estimated time” data of the parent task,

I do apparently not understand why/how this breaks the estimated time" data of the parent task. Each task has their own estimated time. According to #note-38 you seem to consider the estimated time of both tasks together, which is fine by me, but could simply be modified in the “estimated time” field of this ticket task to reflect exactly that thinking. So currently to me it sounds more like a conceptual/UX issue with Redmine. But maybe I need more explanations here.

> which is why I reverted the exact same change 2 weeks ago.

I see that you reverted a different change (different parent task) in #note-37, without explaining that you reverted it for the reason you are stating right now.

#47 Updated by CyrilBrulebois 2019-04-05 12:37:24

  • Private changed from No to Yes

[Note deleted, sorry for the noise.]

#48 Updated by CyrilBrulebois 2019-04-05 16:01:58

  • Private changed from Yes to No

#50 Updated by intrigeri 2019-04-08 15:07:55

> I do apparently not understand why/how this breaks the estimated time" data of the parent task. Each task has their own estimated time. According to #note-39 you seem to consider the estimated time of both tasks together, which is fine by me, but could simply be modified in the “estimated time” field of this ticket task to reflect exactly that thinking. So currently to me it sounds more like a conceptual/UX issue with Redmine. But maybe I need more explanations here.

@u, you’re of course entirely correct that there are several equally valid ways in which we could handle our tickets here :) Now, switching right now to a new organization has a cost for me (I would need to “simply” adjust ticket metadata and then update my habits)… for no perceived value for me. I understand that you did these changes for a reason, presumably to help organize your management work. So I guess we’re hitting a typical case of friction here, as in “a change that helps the manager has drawbacks for a worker who does not understand the value of the change and gets defensive”. Were we closer to the beginning of this project, it would be interesting to spend more time understanding each other’s needs and reach a consensus. Given we’re almost done, I’d rather not spend more time on this so I’ll reapply your “simply unparenting this ticket” change. We can discuss later and elsewhere how best to handle these things in a way that every affected person understands.

#51 Updated by intrigeri 2019-04-08 15:13:04

  • Assignee changed from CyrilBrulebois to intrigeri
  • QA Check changed from Info Needed to Dev Needed

#53 Updated by intrigeri 2019-04-08 15:13:53

(As per Bug #16004#note-50, to avoid process getting in the way of doing the work.)

#54 Updated by intrigeri 2019-04-08 15:35:19

  • Assignee changed from intrigeri to CyrilBrulebois

@CyrilBrulebois, I wrote Gherkin for the missing scenario: commit:58d83ab8b8a2ce154b144a46771f0292a2b3fedc.

Note that we already have code to do “I plug and mount a USB drive containing…”; presumably it’s not too hard to extend it to support what we need here. We also have some code to interact with GNOME Disks (features/veracrypt.feature, features/step_definitions/veracrypt.rb).

Happy hacking!

#55 Updated by Anonymous 2019-04-24 15:41:10

@CyrilBrulebois is there an ETA for this on your side?

#56 Updated by CyrilBrulebois 2019-05-23 21:23:28

  • Target version changed from Tails_3.14 to Tails_3.15

#57 Updated by Anonymous 2019-06-21 13:53:05

@CyrilBrulebois: I would like to close this ticket before July 10th. Is it possible for you to make that happen?

#58 Updated by intrigeri 2019-07-04 16:36:21

Replying to:

> Was just wondering about the “I plug and mount a USB drive containing a Tails USB image” part of Bug #16004. Should that be a pristine USB drive with just a copy of the IMG file, with no specific size (8+GB), and without having been booted at all, so that we copy/duplicate the exact content of that USB drive?

The way I understand the comments above that I’ve only skimmed above quickly, this is about:

> upgrade from “inside” Tails, i.e. without having another up-to-date Tails: from my Tails d/l USB image and install it with GNOME Disks to an intermediary Tails, then boot on that intermediary Tails, and from there upgrade my Tails with Tails Installer

… and what’s missing is “install it with GNOME Disks to an intermediary Tails”. IIRC in the context of our upgrading end-user doc, which we’re trying to follow as closely as possible here, what we need here is a USB drive that has a filesystem that contains a copy of the USB image file, i.e. what a user would get if they downloaded our USB image to some FAT32 USB stick from Tails.

But I would suggest you double-check what I’m saying vs. Bug #16004#note-36 and our end-user upgrade doc.

#59 Updated by CyrilBrulebois 2019-07-10 10:34:09

  • Target version changed from Tails_3.15 to Tails_3.16

#61 Updated by intrigeri 2019-08-08 15:00:31

  • Assignee changed from CyrilBrulebois to intrigeri

@u, kibi and I have discussed this and we agreed to move this to my plate.

#62 Updated by intrigeri 2019-08-18 19:02:20

Drafted something. The two affected scenarios Work On My Machine™. Let’s see what Jenkins thinks (it tends to disagree with me).

#63 Updated by intrigeri 2019-08-19 11:37:10

  • blocked by Bug #12092: The Greeter keeps eating lots of memory after logging in added

#64 Updated by intrigeri 2019-08-19 11:38:28

This new scenario fails on Jenkins for the same reason as other USB install/upgrade ones (see Bug #16281) which should be solved by Bug #12092.

#65 Updated by intrigeri 2019-08-19 18:48:52

  • blocks Bug #16976: Run Dogtail using Python 3 added

#66 Updated by intrigeri 2019-08-19 18:51:55

  • Feature Branch changed from test/16004-adapt-usb-scenarios-to-usb-images to test/16004-adapt-usb-scenarios-to-usb-images+force-all-tests

#67 Updated by intrigeri 2019-08-19 19:06:52

  • Target version changed from Tails_3.16 to Tails_4.0

Oh well, I mistakenly rebased this branch on devel and now it has my work for Bug #16976, that is Buster-only. Whatever.

#68 Updated by intrigeri 2019-08-20 06:51:54

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to anonym

Full test suite passes on my local Jenkins except 2 GnuPG key fetches failures. Please review and merge into devel :)

#69 Updated by intrigeri 2019-08-27 18:46:07

  • Assignee deleted (anonym)

(anonym encouraged me to look for other reviewers.)

#70 Updated by segfault 2019-08-29 11:50:11

  • Status changed from Needs Validation to Resolved
  • % Done changed from 60 to 100

Applied in changeset commit:tails|3d8a965c97fc4489dc103527ecd228645d7f674a.

#71 Updated by intrigeri 2019-08-29 12:12:34

  • blocks deleted (Bug #12092: The Greeter keeps eating lots of memory after logging in)