Bug #12731

Instructions to run the automated test suite fail

Added by dachary 2017-06-18 19:03:24 . Updated 2017-06-23 11:39:18 .

Status:
Resolved
Priority:
Normal
Assignee:
intrigeri
Category:
Test suite
Target version:
Start date:
2017-06-18
Due date:
% Done:

0%

Feature Branch:
Type of work:
Contributors documentation
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

https://tails.boum.org/contribute/release_process/test/setup/ is incorrect and should be fixed as follows, assuming Debian GNU/Linux 9:

sudo apt-get update
sudo apt install \
    cucumber \
    devscripts \
    dnsmasq-base \
    gawk \
    git \
    i18nspector \
    libav-tools \
    libcap2-bin \
    libsikuli-script-java \
    libvirt-clients \
    libvirt-daemon-system \
    libvirt-dev \
    libvirt0 \
    obfs4proxy \
    openssh-server \
    ovmf \
    pry \
    python-jabberbot \
    python-potr \
    qemu-kvm \
    qemu-system-common \
    qemu-system-x86 \
    qemu-utils \
    redir \
    ruby-guestfs \
    ruby-json \
    ruby-libvirt \
    ruby-net-irc \
    ruby-packetfu \
    ruby-rb-inotify \
    ruby-rjb \
    ruby-rspec \
    ruby-test-unit \
    seabios \
    tcpdump \
    tor \
    unclutter \
    virt-viewer \
    x11vnc \
    tigervnc-viewer \
    xvfb 
$ sudo systemctl restart libvirtd

The proposed usage also needs fixing at https://tails.boum.org/contribute/release_process/test/usage/

$ sudo apt-get install git
$ git clone --recursive -b stable https://git-tails.immerda.ch/tails
$ sudo ./run_test_suite --view --iso ../*.iso features/apt.feature

but I was not able to figure out how to fix the last failure:

debian@test-loic-9-tests:~/tails$ sudo ./run_test_suite --view --iso ../*.iso features/apt.feature
Virtual X framebuffer started on display :0
VNC server running on: localhost:5900
[info] runcmd: lsb_release -i -r -s 
@product
Feature: Installing packages through APT
  As a Tails user
  when I set an administration password in Tails Greeter
  I should be able to install packages using APT and Synaptic
  and all Internet traffic should flow only through Tor.

  Scenario: APT sources are configured correctly                                                                                         # features/apt.feature:8
    Given I have started Tails from DVD without network and logged in                                                                    # features/step_definitions/snapshots.rb:172
      FindFailed: can not find GnomeSystrayFlorence.png in S(0)[0,0 1024x768]
        Line 2171, in file Region.java
       (RuntimeError)
      ./features/step_definitions/common_steps.rb:273:in `/^I log in to a new session(?: in )?(|German)$/'
      ./features/step_definitions/snapshots.rb:154:in `block in reach_checkpoint'
      ./features/step_definitions/snapshots.rb:152:in `each'
      ./features/step_definitions/snapshots.rb:152:in `reach_checkpoint'
      ./features/step_definitions/snapshots.rb:173:in `/^I\ have\ started\ Tails\ from\ DVD\ without\ network\ and\ logged\ in$/'
      features/apt.feature:9:in `Given I have started Tails from DVD without network and logged in'
    Then the only hosts in APT sources are "vwakviie2ienjx6t.onion,sgvtcaew4bxjd7ln.onion,jenw7xbd6tf7vfhp.onion,sdscoq7snqtznauu.onion" # features/step_definitions/apt.rb:3
    And no proposed-updates APT suite is enabled                                                                                         # features/step_definitions/apt.rb:17
      Scenario failed at time 00:02:08

      Screenshot: /tmp/TailsToaster/run-2017-06-18_17:35:45_+0000-git_stable_7e12308-7rWWw4/00:02:08_APT_sources_are_configured_correctly.png

  @check_tor_leaks
  Scenario: Install packages using apt                                                                             # features/apt.feature:14
    Given I have started Tails from DVD and logged in with an administration password and the network is connected # features/step_definitions/snapshots.rb:172
      FindFailed: can not find GnomeSystrayFlorence.png in S(0)[0,0 1024x768]
        Line 2171, in file Region.java
       (RuntimeError)
      ./features/step_definitions/common_steps.rb:273:in `/^I log in to a new session(?: in )?(|German)$/'
      ./features/step_definitions/snapshots.rb:154:in `block in reach_checkpoint'
      ./features/step_definitions/snapshots.rb:152:in `each'
      ./features/step_definitions/snapshots.rb:152:in `reach_checkpoint'
      ./features/step_definitions/snapshots.rb:139:in `reach_checkpoint'
      ./features/step_definitions/snapshots.rb:173:in `/^I\ have\ started\ Tails\ from\ DVD\ and\ logged\ in\ with\ an\ administration\ password\ and\ the\ network\ is\ connected$/'
      features/apt.feature:15:in `Given I have started Tails from DVD and logged in with an administration password and the network is connected'
    When I configure APT to use non-onion sources                                                                  # features/step_definitions/apt.rb:24
    And I update APT using apt                                                                                     # features/step_definitions/apt.rb:41
    And I install "cowsay" using apt                                                                               # features/step_definitions/apt.rb:54
    Then the package "cowsay" is installed                                                                         # features/step_definitions/common_steps.rb:543
      Scenario failed at time 00:02:59

      Screenshot: /tmp/TailsToaster/run-2017-06-18_17:35:45_+0000-git_stable_7e12308-7rWWw4/00:02:59_Install_packages_using_apt.png

  <del><a class='issue tracker-1 status-5 priority-4 priority-default closed child' href='/code/issues/12586' title='Synaptic test is fragile on Stretch'>Bug #12586</a></del>
  @check_tor_leaks @fragile
  Scenario: Install packages using Synaptic                                                                        # features/apt.feature:23
    Given I have started Tails from DVD and logged in with an administration password and the network is connected # features/step_definitions/snapshots.rb:172
      FindFailed: can not find GnomeSystrayFlorence.png in S(0)[0,0 1024x768]
        Line 2171, in file Region.java
       (RuntimeError)
      ./features/step_definitions/common_steps.rb:273:in `/^I log in to a new session(?: in )?(|German)$/'
      ./features/step_definitions/snapshots.rb:154:in `block in reach_checkpoint'
      ./features/step_definitions/snapshots.rb:152:in `each'
      ./features/step_definitions/snapshots.rb:152:in `reach_checkpoint'
      ./features/step_definitions/snapshots.rb:139:in `reach_checkpoint'
      ./features/step_definitions/snapshots.rb:173:in `/^I\ have\ started\ Tails\ from\ DVD\ and\ logged\ in\ with\ an\ administration\ password\ and\ the\ network\ is\ connected$/'
      features/apt.feature:24:in `Given I have started Tails from DVD and logged in with an administration password and the network is connected'
    When I configure APT to use non-onion sources                                                                  # features/step_definitions/apt.rb:24
    And I start Synaptic                                                                                           # features/step_definitions/apt.rb:68
    And I update APT using Synaptic                                                                                # features/step_definitions/apt.rb:79
    When I install "cowsay" using Synaptic                                                                         # features/step_definitions/apt.rb:100
    Then the package "cowsay" is installed                                                                         # features/step_definitions/common_steps.rb:543
      Scenario failed at time 00:03:49

      Screenshot: /tmp/TailsToaster/run-2017-06-18_17:35:45_+0000-git_stable_7e12308-7rWWw4/00:03:49_Install_packages_using_Synaptic.png

Artifacts directory: /tmp/TailsToaster/run-2017-06-18_17:35:45_+0000-git_stable_7e12308-7rWWw4

Debug log:           /tmp/TailsToaster/run-2017-06-18_17:35:45_+0000-git_stable_7e12308-7rWWw4/debug.log

Failing Scenarios:
cucumber features/apt.feature:8 # Scenario: APT sources are configured correctly
cucumber features/apt.feature:14 # Scenario: Install packages using apt
cucumber features/apt.feature:23 # Scenario: Install packages using Synaptic

3 scenarios (3 failed)
14 steps (3 failed, 11 skipped)
3m47.891s

Subtasks


History

#1 Updated by intrigeri 2017-06-23 11:22:54

  • Subject changed from instructions to run the tests fail to Instructions to run the automated test suite fail
  • Category set to Test suite
  • Assignee set to intrigeri
  • Type of work changed from Pass Test to Contributors documentation

#2 Updated by intrigeri 2017-06-23 11:39:18

  • Status changed from New to Resolved

> https://tails.boum.org/contribute/release_process/test/setup/ is incorrect and should be fixed as follows, assuming Debian GNU/Linux 9:

Fixed, thanks. I was not expecting this would be needed, but why not :)

> The proposed usage also needs fixing at https://tails.boum.org/contribute/release_process/test/usage/

>

> $ sudo apt-get install git
> $ git clone --recursive -b stable https://git-tails.immerda.ch/tails
> $ sudo ./run_test_suite --view --iso ../*.iso features/apt.feature
> 

I’ve added a link to our Git doc in commit:f769033, as I’d rather not duplicate such information all over the place.

Regarding sudo: see the “Running the test suite as a non-root user” section on https://tails.boum.org/contribute/release_process/test/setup/. I’d rather not encourage everyone to run this as root when it’s not really needed.

> but I was not able to figure out how to fix the last failure:

I guess that’s Bug #11872, but I would need to see the screenshot or video to be sure.