Feature #8988

What to do with run_test_suite's dependency checking

Added by anonym 2015-03-01 20:21:04 . Updated 2015-05-12 18:43:22 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Test suite
Target version:
Start date:
2015-03-01
Due date:
% Done:

100%

Feature Branch:
test/8988-up-to-date-dep-checks
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Currently it’s out-of-date — one must follow the setup instructions to get a working environment that’s able to run the automated test suite.

Personally I think we should just remove it so it will be up to the user of the test suite to follow the setup instructions. The only benefit I can see in keeping it (+ enforcing that we keep it up-to-date) is that it would be handy to notify the user of new dependencies in a new branch, for instance. But still, its dependency checking is crude and cannot specify that “we need at least this version of $pkg” and similar.


Subtasks


Related issues

Related to Tails - Bug #9288: Dependency checking in run_test_suite is still broken Resolved 2015-04-28

History

#1 Updated by kytv 2015-03-02 03:26:19

I automatically go to the set-up docs when/if things aren’t working, so I’ve no strong preference here.

#2 Updated by intrigeri 2015-03-03 11:53:43

  • Assignee set to anonym

On the one hand, we have a tails::tester Puppet module already, that I keep up-to-date since that’s what sets up isotesterN.lizard. Maybe maintaing these bits in two different places is enough, and those who want to have their testing setup always up-to-date can use the Puppet module.

On the other hand, I’ve seen e.g. Alan struggle with keeping his testing environment up-to-date, and with keeping up with the unwritten requirements our automated test suite has (I’ve filed at least one ticket about it recently). Removing these checks would lead to more and more obscure errors and discouragement for such people who don’t run the test suite often => even if they are crude and not entirely up-to-date, they probably help.

So I’m unsure. Maybe ask Alan and others on -dev@ how they would feel about it?

#3 Updated by anonym 2015-03-27 15:12:19

  • Status changed from Confirmed to In Progress

Applied in changeset commit:904cde1266f84b06df12c39c1f4839490d893239.

#4 Updated by anonym 2015-03-27 15:13:36

  • Assignee changed from anonym to kytv
  • % Done changed from 0 to 50
  • QA Check set to Ready for QA
  • Feature Branch set to test/8988-up-to-date-dep-checks

I ended up updating the deps. I’ll be harder checking that updates to the setup instructions are reflected in the script when reviewing stuff, suggest that you are too, kytv and intrigeri. :)

#5 Updated by kytv 2015-04-06 15:21:15

  • Assignee changed from kytv to anonym
  • Target version changed from Tails_1.3.2 to Tails_1.4
  • QA Check changed from Ready for QA to Dev Needed

The dependencies look good but in it fails for me with

# ./run_test_suite --view --iso tails-i386-1.3.2.iso features/unsafe_browser.feature 
run_test_suite: error: 'libvirt-daemon-system' is missing, please install it and run again. Aborting...

(I just upgraded my test VM to Jessie)

#6 Updated by intrigeri 2015-04-06 15:25:50

> The dependencies look good but in it fails for me with

>

> # ./run_test_suite --view --iso tails-i386-1.3.2.iso features/unsafe_browser.feature 
> run_test_suite: error: 'libvirt-daemon-system' is missing, please install it and run again. Aborting...
> 

That’s Bug #9156.

#7 Updated by kytv 2015-04-06 15:33:27

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:
> > The dependencies look good but in it fails for me with
>
> > […]
>
> That’s Bug #9156.

Ah..never mind then. I worked around it locally by commenting out the pipefail line but

 package_installed() {
+   set +o pipefail
    dpkg -s "${1}" 2>/dev/null | grep -q "^Status:.*installed"
+   set -o pipefail

looks like it’ll be a proper fix.

Re-assigning to intrigeri for his perusal.

#8 Updated by intrigeri 2015-04-06 15:46:03

  • Assignee changed from intrigeri to anonym
  • % Done changed from 50 to 60
  • QA Check changed from Ready for QA to Dev Needed

I see this branch introduces the full list of dependencies, that has changed since we’ve merged Feature #8165, so it can’t really be merged as-is => please merge current stable into this branch, resolve any conflict, perhaps merge my branch for Bug #9156 into it as well, and then we should be good to go.

#9 Updated by anonym 2015-04-29 01:48:18

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:
> I see this branch introduces the full list of dependencies, that has changed since we’ve merged Feature #8165, so it can’t really be merged as-is => please merge current stable into this branch, resolve any conflict, perhaps merge my branch for Bug #9156 into it as well, and then we should be good to go.

I’ve merged stable, and beware: the conflict resolution was a quite hairy.

Also noticed that something was off with package_installed() again… => Bug #9288.

#10 Updated by intrigeri 2015-04-29 02:55:54

  • Assignee changed from intrigeri to kytv
  • % Done changed from 60 to 70
  • Type of work changed from Discuss to Code

anonym wrote:
> I’ve merged stable,

Thanks!

> and beware: the conflict resolution was a quite hairy.

Then reassigning to kytv for another round of review and test before I look at it.

#11 Updated by kytv 2015-05-01 11:55:57

  • Assignee changed from kytv to intrigeri

Review and tests pass, but the branch in Bug #9288 is needed for the dependency checking to work (as noted above)

#12 Updated by intrigeri 2015-05-01 12:13:04

Going to merge that, with commit:b389b93 and commit:1f466c5 on top.

#13 Updated by intrigeri 2015-05-01 12:13:58

  • Status changed from In Progress to Fix committed
  • % Done changed from 70 to 100

Applied in changeset commit:c6ab155fece99c1d9a1044fb631a2dcdef14f2b0.

#14 Updated by BitingBird 2015-05-01 12:15:12

  • related to Bug #9288: Dependency checking in run_test_suite is still broken added

#15 Updated by BitingBird 2015-05-01 12:15:33

  • Assignee deleted (intrigeri)
  • QA Check changed from Ready for QA to Pass

#16 Updated by BitingBird 2015-05-12 18:43:22

  • Status changed from Fix committed to Resolved