Bug #10523

whois test is fragile

Added by kytv 2015-11-09 15:56:54 . Updated 2015-12-16 11:33:47 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Test suite
Target version:
Tails_1.8
Start date:
2015-11-09
Due date:
% Done:

100%

Feature Branch:
kytv/test/10523-whois-test-is-fragile
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:
270

Description

Seen in a few recent jobs from jenkins.

calling as amnesia: whois 'torproject.org'
call returned: [0, "WHOIS LIMIT EXCEEDED - SEE WWW.PIR.ORG/WHOIS FOR DETAILS\n", ""]
    When I query the whois directory service for "torproject.org"                  # features/step_definitions/torified_misc.rb:1
    Then the whois command is successful                                           # features/step_definitions/torified_misc.rb:25
    Then the whois standard output contains "The Tor Project"                      # features/step_definitions/torified_misc.rb:34
      The whois standard output does not contain The Tor Project:
      WHOIS LIMIT EXCEEDED - SEE WWW.PIR.ORG/WHOIS FOR DETAILS

      .
      <nil> is not true. (Test::Unit::AssertionFailedError)
      ./features/step_definitions/torified_misc.rb:35:in `/^the (wget|whois) standard output contains "([^"]+)"$/'
      features/torified_misc.feature:24:in `Then the whois standard output contains "The Tor Project"'

This was addressed previously but when the retry_tor code was refactored this regression sneaked in. The problem stems from the fact that whois still “succeeds” because it exits 0 and it exits 0 because whois itself did what it was supposed to; it returned what the server sent.

Subtasks

History

#1 Updated by kytv 2015-11-09 15:57:16

#2 Updated by kytv 2015-11-09 16:04:27

  • Description updated

#3 Updated by kytv 2015-11-09 16:10:11

  • Assignee changed from kytv to anonym
  • % Done changed from 10 to 60
  • QA Check set to Ready for QA
  • Feature Branch set to kytv/test/10523-whois-test-is-fragile

Now:

calling as amnesia: whois 'torproject.org'
call returned: [0, "WHOIS LIMIT EXCEEDED - SEE WWW.PIR.ORG/WHOIS FOR DETAILS\n", ""]
Tor operation failed (Try 1 of 50) with:
RuntimeError: Looking up whois info for torproject.org failed with:
WHOIS LIMIT EXCEEDED - SEE WWW.PIR.ORG/WHOIS FOR DETAILS
Forcing new Tor circuit...
calling as root: . /usr/local/lib/tails-shell-library/tor.sh && tor_control_send "signal NEWNYM"
call returned: [0, "250 OK\n250 OK\n250 closing connection\n", ""]
calling as amnesia: whois 'torproject.org'
call returned: [0, "Domain Name:TORPROJECT.ORG\nDomain ID: D130971538-LROR\nCreation Date: 2006-10-17T22:02:50Z\nUpdated Date: 2015-08-20T01:06:22Z\nRegistry Expiry Date:
    When I query the whois directory service for "torproject.org"                  # features/step_definitions/torified_misc.rb:1
    Then the whois command is successful                                           # features/step_definitions/torified_misc.rb:25
    Then the whois standard output contains "The Tor Project"                      # features/step_definitions/torified_misc.rb:34

That’s more like it. :)

#4 Updated by bertagaz 2015-11-12 04:31:55

Hmm, I wonder if we shouldn’t just consider that if we get the “LIMIT EXCEEDED” message, then the test passes too. After all it successfully communicated with the whois server, and it would probably make the test suite more robust.

#5 Updated by intrigeri 2015-11-16 04:27:43

> Hmm, I wonder if we shouldn’t just consider that if we get the “LIMIT EXCEEDED” message, then the test passes too. After all it successfully communicated with the whois server, and it would probably make the test suite more robust.

Let’s please not introduce such mismatches between the Gherkin phrasing, and what the step definitions actually do. If we ever have to downgrade what we’re actually testing, then we’ll have to update the wording in Gherkin accordingly.

#6 Updated by kytv 2015-11-16 06:41:41

bertagaz wrote:
> Hmm, I wonder if we shouldn’t just consider that if we get the “LIMIT EXCEEDED” message, then the test passes too. After all it successfully communicated with the whois server, and it would probably make the test suite more robust.

My 2¢: I can understand this argument if we only want to test whether we’re stopping whois from working because of what we put into Tails (apparmor profiles, iptables rules, etc.). If we want to test whether whois is actually useful for users, we couldn’t do this.

#7 Updated by bertagaz 2015-12-05 13:20:22

  • Assignee changed from anonym to bertagaz

Taking over, want to merge this.

#8 Updated by bertagaz 2015-12-10 06:58:45

  • Status changed from In Progress to Fix committed
  • % Done changed from 60 to 100

Applied in changeset commit:ed01924c90617a929e213969fbf91c5d95c71529.

#9 Updated by bertagaz 2015-12-10 07:02:44

  • QA Check changed from Ready for QA to Pass

Ran it 200 times, only 6 of them failed, but because of other issues (Bug #10495). Merged then, congrats and sorry for the delay!

#10 Updated by intrigeri 2015-12-15 04:20:06

  • Assignee deleted (bertagaz)

#11 Updated by anonym 2015-12-16 11:33:47

  • Status changed from Fix committed to Resolved