Feature #5472: Automatically test behavior on boot medium removal

Feature #5472

Automatically test behavior on boot medium removal

Added by Tails 2013-07-18 07:41:34 . Updated 2016-06-26 10:39:55 .

Status:

Resolved

Priority:

Normal

Assignee:

Category:

Test suite

Target version:

Tails_2.0

Start date:

Due date:

2016-01-15

% Done:

100%

Feature Branch:

test/5472-shutdown-on-boot-medium-removal

Type of work:

Code

Blueprint:

Starter:

Affected tool:

Deliverable for:

268

Description

While it’s possible to tell libvirt to forcefully remove both DVD and USB media, it doesn’t result in the signal (or whatever) that our udev watchdog looks for. The media just gets broken with lots of IO errors happening. This prevents implementation of the tests:

erase memory on shutdown
- After booting from DVD, remove Tails boot medium and check that the memory erasure process is started (Loading new kernel, at least). The USB tests are bellow, in the USB installer section.
USB Installer/Upgrader
- The installed or upgraded Tails medium shall have the "emergency shutdown on boot medium removal" feature working: remove Tails boot medium and check that the memory erasure process is started (Loading new kernel, at least).

We need to research if we can fix this, or if it’s an upstream bug.

Subtasks

Related issues

Related to Tails - ~~Feature #6106~~: test suite: erase memory feature	Resolved
Blocks Tails - ~~Feature #7468~~: Replace our memory erasure implementation with the wiperam package	Rejected	2014-06-29

History

#1 Updated by intrigeri 2013-10-04 08:11:13

Category set to Test suite
Starter set to No

#2 Updated by intrigeri 2014-03-07 13:16:24

Good news! virsh change-media $DOMAIN hdc --eject successfully triggers shutdown for me (host system = Debian sid, libvirt 1.2.1-1; guest system = Tails 0.23~rc1).

#3 Updated by intrigeri 2014-04-17 10:17:29

Subject changed from test suite: simulate media removal to Automatically test behavior on boot medium removal
Type of work changed from Research to Code

#4 Updated by intrigeri 2014-05-25 11:42:43

I think this would be a good candidate for the 2.0 milestone, on the grounds that it is one of our last manual tests that require booting Tails on bare metal due to lack of automatization (and when one only has a single dev laptop, they have to shutdown their “normal” system to try that).

#5 Updated by sajolida 2014-11-03 20:53:58

Target version set to Sustainability_M1

#6 Updated by anonym 2015-01-10 06:06:58

Target version changed from Sustainability_M1 to Tails_2.4

#7 Updated by anonym 2015-01-10 17:18:10

blocks #8668 added

#8 Updated by anonym 2015-01-10 18:26:33

Assignee set to anonym
Target version changed from Tails_2.4 to Tails_1.8

#9 Updated by intrigeri 2015-07-08 10:47:42

blocks ~~Feature #7468~~: Replace our memory erasure implementation with the wiperam package added

#10 Updated by intrigeri 2015-08-02 11:01:28

Due date set to 2016-01-15

#11 Updated by anonym 2015-09-26 07:29:56

Parent task set to Bug #10250

#12 Updated by intrigeri 2015-12-05 13:56:51

Priority changed from Normal to Elevated
Target version changed from Tails_1.8 to Tails_2.0

Writing new tests shall wait post-1.8, but bumping priority so it’s on top of the 2.0 plate.

#13 Updated by anonym 2016-01-06 19:51:16

Status changed from Confirmed to In Progress

Applied in changeset commit:f691afa6014147f1918150ec7e08c955b7cad57a.

#14 Updated by anonym 2016-01-06 19:55:50

Assignee changed from anonym to kytv
% Done changed from 0 to 50
QA Check set to Ready for QA

So I also wanted to test that the memory was properly wiped on medium removal but the graphics become corrupted very often (p = 0.5 or so) making it impossible to find the “Happy dumping!” message on the screen. Hm. I’m not sure that can easily be solved, so I hope we can accept this test without that part. What do you think?

Here’s the commit I tried:

commit 8f569987a6ef462b96b9dd26b28fcc1394454a3e
Author: anonym <anonym@riseup.net>
Date:   Wed Jan 6 19:14:00 2016 +0100

    Automatically test that the memory is wiped on boot medium removal.

    Will-fix: <del><a class='issue tracker-2 status-3 priority-4 priority-default closed child' href='/code/issues/5472' title='Automatically test behavior on boot medium removal'>Feature #5472</a></del>

diff --git a/features/erase_memory.feature b/features/erase_memory.feature
index d3425fd..1c7afaa 100644
--- a/features/erase_memory.feature
+++ b/features/erase_memory.feature
@@ -34,7 +34,8 @@ Feature: System memory erasure on shutdown
     And process "udev-watchdog" is running
     And udev-watchdog is monitoring the correct device
     When I fill the guest's memory with a known pattern
-    And I shutdown and wait for Tails to finish wiping the memory
+    And I eject the boot medium
+    And I wait for Tails to finish wiping the memory
     Then I find very few patterns in the guest's memory

   Scenario: Anti-test: no memory erasure on an old computer
@@ -67,5 +68,6 @@ Feature: System memory erasure on shutdown
     And process "udev-watchdog" is running
     And udev-watchdog is monitoring the correct device
     When I fill the guest's memory with a known pattern
-    And I shutdown and wait for Tails to finish wiping the memory
+    And I eject the boot medium
+    And I wait for Tails to finish wiping the memory
     Then I find very few patterns in the guest's memory
diff --git a/features/step_definitions/erase_memory.rb b/features/step_definitions/erase_memory.rb
index 24b21e0..5d324e0 100644
--- a/features/step_definitions/erase_memory.rb
+++ b/features/step_definitions/erase_memory.rb
@@ -206,8 +206,7 @@ When /^I stop the boot at the bootloader menu$/ do
   @screen.waitVanish(bootsplash_tab_msg, 1)
 end

-When /^I shutdown and wait for Tails to finish wiping the memory$/ do
-  $vm.spawn("halt")
+When /^I wait for Tails to finish wiping the memory$/ do
   nr_gibs_of_ram = convert_from_bytes($vm.get_ram_size_in_bytes, 'GiB').ceil
   try_for(nr_gibs_of_ram*5*60, { :msg => "memory wipe didn't finish, probably the VM crashed" }) do
     # We spam keypresses to prevent console blanking from hiding the

#15 Updated by anonym 2016-01-16 15:45:01

Feature Branch set to test/5472-shutdown-on-boot-medium-removal

#16 Updated by intrigeri 2016-01-16 15:51:01

kytv: in case you can’t handle this review within a week from now, please let me know, so I take it over. Note that I won’t be able to do it after Jan 21, so I need to know pretty quickly.

#17 Updated by anonym 2016-01-18 03:11:51

Assignee changed from kytv to intrigeri

intrigeri wrote:
> kytv: in case you can’t handle this review within a week from now, please let me know, so I take it over. Note that I won’t be able to do it after Jan 21, so I need to know pretty quickly.

Since we haven’t heard anything from kytv yet I’m assigning it to you, as decided earlier.

#18 Updated by intrigeri 2016-01-18 11:23:25

% Done changed from 50 to 60

Code review passes, will now test.

#19 Updated by intrigeri 2016-01-18 11:50:24

Assignee changed from intrigeri to anonym
% Done changed from 60 to 70
QA Check changed from Ready for QA to Dev Needed

Pushed a commit on top, please take a look.
Passes just fine locally.
It always fails on Jenkins so far, so I can’t merge as-is. If it’s not realistic to make it work on Jenkins this month, please 1. mark the test as fragile in the topic branch; 2. merge yourself into testing+devel (so at least we benefit from the automated test at releas time; 3. file a ticket about the fact it’s fragile, under Bug #10288.

#20 Updated by anonym 2016-01-22 00:51:10

intrigeri wrote:
> * Pushed a commit on top, please take a look.

Ah, forgot that important and very satisfying part. :) Thanks!

> * Passes just fine locally.

On Debian Unstable with libvirt 1.3.0? That’s what I use.

> * It always fails on Jenkins so far, so I can’t merge as-is.
The failure is:

    When I eject the boot medium                                      # features/step_definitions/common_steps.rb:963
      Call to virDomainUpdateDeviceFlags failed: internal error: unable to execute QEMU command 'eject': Device 'drive-ide0-1-0' is locked (Libvirt::Error)
      ./features/support/helpers/vm_helper.rb:164:in `update_device'
      ./features/support/helpers/vm_helper.rb:164:in `block in set_cdrom_image'
      ./features/support/helpers/vm_helper.rb:153:in `set_cdrom_image'
      ./features/support/helpers/vm_helper.rb:173:in `remove_cdrom'
      ./features/step_definitions/common_steps.rb:968:in `/^I eject the boot medium$/'

So it seems whatever libvirt/QEMU setup we have on the isotesters does not support this. Indeed, on Debian Jessie there’s libvirt 1.2.9, and in 1.2.18 there was a fix to try harder to eject media, which seems relevant, so with Debian Unstable’s version 1.3.0 we’re definitely fine.

However, I distinctly remember being able to remove the DVD boot medium via virt-manager when I still ran Debian Jessie. Hm. It could be that I had installed testing’s/unstable’s libvirt at the time. I should try downgrading.

I tried adding VIR_DOMAIN_DEVICE_MODIFY_FORCE, which is documented to “[f]orcibly modify device (ex. force eject a cdrom)”, which sounds nice, but on my system it seems akin to ripping out the data cable to the drive, resulting in I/O errors without the OS being notified of an ejection event so the emergency shutdown doesn’t even trigger.

> If it’s not realistic to make it work on Jenkins this month, please 1. mark the test as fragile in the topic branch; 2. merge yourself into testing+devel (so at least we benefit from the automated test at releas time; 3. file a ticket about the fact it’s fragile, under Bug #10288.

I’ll return on this after testing Jessie’s libvirt.

#21 Updated by anonym 2016-01-22 02:04:27

anonym wrote:
> > If it’s not realistic to make it work on Jenkins this month, please 1. mark the test as fragile in the topic branch; 2. merge yourself into testing+devel (so at least we benefit from the automated test at releas time; 3. file a ticket about the fact it’s fragile, under Bug #10288.
>
> I’ll return on this after testing Jessie’s libvirt.

I did, and it still works. But then I also downgraded qemu to Jessie’s version, and then I could reproduce the isotester’s error.

#22 Updated by anonym 2016-01-22 11:19:45

% Done changed from 70 to 80

It turns out that with an older qemu the strange error that libvirt throws is different. I’ve pushed commit:bf5f92b and will check how it runs on our Jenkins instances. If it passes there I’ll just merge this as-is, other wise I add @fragile and merge etc.

#23 Updated by anonym 2016-01-22 14:46:37

Status changed from In Progress to Fix committed
% Done changed from 80 to 100

Applied in changeset commit:4c92f445fc2327971cfcbbe1303376c5a6091a9e.

#24 Updated by anonym 2016-01-22 14:48:19

Assignee deleted (~~anonym~~)
QA Check changed from Dev Needed to Pass

anonym wrote:
> It turns out that with an older qemu the strange error that libvirt throws is different. I’ve pushed commit:bf5f92b and will check how it runs on our Jenkins instances. If it passes there I’ll just merge this as-is, other wise I add @fragile and merge etc.

I’ve now seen the scenario pass on Jenkins too!

#25 Updated by anonym 2016-01-27 13:31:24

Status changed from Fix committed to Resolved

#26 Updated by BitingBird 2016-06-26 10:39:56

Priority changed from Elevated to Normal