Feature #5836

Test suite: verify time changes

Added by Tails 2013-07-18 07:46:59 . Updated 2015-02-24 22:51:50 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Test suite
Target version:
Tails_1.3
Start date:
Due date:
% Done:

100%

Feature Branch:
test/5836-verify-time-changes
Type of work:
Code
Blueprint:

Starter:
0
Affected tool:
Deliverable for:

Description

After setting the date to something wrong, one should verify that the date change actually took place, else it makes little sense verifying later that our stuff indeed has "fixed" the date: if we don’t know for sure that it was broken, we’re not really verifying anything. This should be implemented at least in features/step_definitions/time_syncing.rb.

Subtasks

History

#1 Updated by intrigeri 2013-10-04 06:05:15

  • Category set to Test suite
  • Starter set to No

#2 Updated by BitingBird 2014-06-09 11:11:25

  • Subject changed from test suite: verify time changes to Test suite: verify time changes

#3 Updated by intrigeri 2014-08-26 23:19:01

  • Assignee set to anonym
  • Target version set to Tails_1.2.2
  • Parent task set to Feature #6298

#4 Updated by intrigeri 2014-10-20 14:08:11

  • blocks #8117 added

#5 Updated by anonym 2014-12-12 16:41:53

  • Target version changed from Tails_1.2.2 to Tails_1.2.3

#6 Updated by anonym 2015-01-13 14:14:34

  • Target version changed from Tails_1.2.3 to Tails_1.3

#7 Updated by anonym 2015-01-23 18:39:13

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 20
  • QA Check set to Dev Needed
  • Feature Branch set to test/5836-verify-time-changes

Just pushing some initial work. I guess we don’t only want to verify by checking the exit status of date (done), but also run date again (after waiting a few seconds?) and see that we get an expected value.

#8 Updated by intrigeri 2015-01-23 18:54:23

> Just pushing some initial work. I guess we don’t only want to verify by checking
> the exit status of date (done), but also run date again (after waiting a few
> seconds?) and see that we get an expected value.

It would indeed feel safer :)

#9 Updated by Tails 2015-01-26 19:14:25

Applied in changeset commit:0f5dcdc403ed4514f18dafdd858ad513635c13ab.

#10 Updated by anonym 2015-01-26 19:17:33

  • Assignee changed from anonym to kytv
  • % Done changed from 20 to 50
  • QA Check changed from Dev Needed to Ready for QA

#11 Updated by kytv 2015-02-08 12:28:01

  • Assignee changed from kytv to anonym
  • % Done changed from 50 to 40
  • QA Check changed from Ready for QA to Dev Needed

Unfortunately this is consistently failing for me.

  Scenario: Clock is one day in the past                                 # features/time_syncing.feature:17
    When I bump the system time with "-1 day"                            # features/step_definitions/time_syncing.rb:14
      The guest's time was suppsed to be bumped to '2015-02-07 00:00:00 +0000' but is '2015-02-07 12:17:47 +0000' (MiniTest::Assertion)
      ./features/step_definitions/time_syncing.rb:24:in `/^I bump the system time with "([^"]+)"$/'
      features/time_syncing.feature:18:in `When I bump the system time with "-1 day"'
    And the network is plugged                                           # features/step_definitions/common_steps.rb:104
    And Tor is ready                                                     # features/step_definitions/common_steps.rb:291
    Then Tails clock is less than 5 minutes incorrect                    # features/step_definitions/time_syncing.rb:29
Scenario failed at time 00:00:41

…

  Scenario: Clock is one day in the future                               # features/time_syncing.feature:29
    When I bump the system time with "+1 day"                            # features/step_definitions/time_syncing.rb:14
      The guest's time was suppsed to be bumped to '2015-02-09 00:00:00 +0000' but is '2015-02-09 12:17:47 +0000' (MiniTest::Assertion)
      ./features/step_definitions/time_syncing.rb:24:in `/^I bump the system time with "([^"]+)"$/'
      features/time_syncing.feature:30:in `When I bump the system time with "+1 day"'
    And the network is plugged                                           # features/step_definitions/common_steps.rb:104
    And Tor is ready                                                     # features/step_definitions/common_steps.rb:291
    Then Tails clock is less than 5 minutes incorrect                    # features/step_definitions/time_syncing.rb:29
Scenario failed at time 00:03:17

…

Failing Scenarios:
cucumber features/time_syncing.feature:17 # Scenario: Clock is one day in the past
cucumber features/time_syncing.feature:29 # Scenario: Clock is one day in the future

#12 Updated by kytv 2015-02-18 14:27:39

With debug info:

calling as root: date
call returned: [0, "Wed Feb 18 14:23:27 UTC 2015\n", ""]
calling as root: date -s 'now -1 day'
call returned: [0, "Tue Feb 17 14:23:27 UTC 2015\n", ""]
calling as root: date
call returned: [0, "Tue Feb 17 14:23:28 UTC 2015\n", ""]
    When I bump the system time with "-1 day"                            # features/step_definitions/time_syncing.rb:14
      The guest's time was supposed to be bumped to '2015-02-17 14:23:27 +0000' but is '2015-02-17 14:23:28 +0000' (MiniTest::Assertion)
      ./features/step_definitions/time_syncing.rb:24:in `/^I bump the system time with "([^"]+)"$/'
      features/time_syncing.feature:18:in `When I bump the system time with "-1 day"'
    And the network is plugged                                           # features/step_definitions/common_steps.rb:110
    And Tor is ready                                                     # features/step_definitions/common_steps.rb:297
    Then Tails clock is less than 5 minutes incorrect                    # features/step_definitions/time_syncing.rb:29
Scenario failed at time 00:03:28
Took screenshot "/tmp/TailsToaster/time_syncing-2015-02-18T14:24:19+00:00.png"
calling as root: date
call returned: [0, "Wed Feb 18 14:23:27 UTC 2015\n", ""]
calling as root: date -s 'now +1 day'
call returned: [0, "Thu Feb 19 14:23:28 UTC 2015\n", ""]
calling as root: date
call returned: [0, "Thu Feb 19 14:23:28 UTC 2015\n", ""]
    When I bump the system time with "+1 day"                            # features/step_definitions/time_syncing.rb:14
      The guest's time was supposed to be bumped to '2015-02-19 14:23:27 +0000' but is '2015-02-19 14:23:28 +0000' (MiniTest::Assertion)
      ./features/step_definitions/time_syncing.rb:24:in `/^I bump the system time with "([^"]+)"$/'
      features/time_syncing.feature:30:in `When I bump the system time with "+1 day"'
    And the network is plugged                                           # features/step_definitions/common_steps.rb:110
    And Tor is ready                                                     # features/step_definitions/common_steps.rb:297
    Then Tails clock is less than 5 minutes incorrect                    # features/step_definitions/time_syncing.rb:29
Scenario failed at time 00:05:57
Took screenshot "/tmp/TailsToaster/time_syncing-2015-02-18T14:26:48+00:00.png"

#13 Updated by anonym 2015-02-18 15:25:07

  • Assignee changed from anonym to kytv
  • QA Check changed from Dev Needed to Ready for QA

kytv wrote in Feature #5836#note-11:
> The guest’s time was suppsed to be bumped to ‘2015-02-07 00:00:00 +0000’ but is ‘2015-02-07 12:17:47 +0000’ (MiniTest::Assertion)

I find it suspicious that the expected time is exactly 00:00:00. Especially since they’re not in you next run. I have no good explanation for this.

kytv wrote in Feature #5836#note-12:
> The guest’s time was supposed to be bumped to ‘2015-02-17 14:23:27 +0000’ but is ‘2015-02-17 14:23:28 +0000’ (MiniTest::Assertion)

I guess my 1 second upper bound wasn’t enough. Let’s bump it to 5 seconds.

#14 Updated by kytv 2015-02-18 16:32:45

  • Assignee changed from kytv to anonym
  • QA Check changed from Ready for QA to Dev Needed

anonym wrote:
> kytv wrote in Feature #5836#note-11:
> > The guest’s time was suppsed to be bumped to ‘2015-02-07 00:00:00 +0000’ but is ‘2015-02-07 12:17:47 +0000’ (MiniTest::Assertion)
>
> I find it suspicious that the expected time is exactly 00:00:00. Especially since they’re not in you next run. I have no good explanation for this.

Same. That was incredibly peculiar.

>
> kytv wrote in Feature #5836#note-12:
> > The guest’s time was supposed to be bumped to ‘2015-02-17 14:23:27 +0000’ but is ‘2015-02-17 14:23:28 +0000’ (MiniTest::Assertion)
>
> I guess my 1 second upper bound wasn’t enough. Let’s bump it to 5 seconds.

I was able to complete the test successfully (woohoo) but it’s occasionally failing for me during subsequent runs:

calling as root: date
call returned: [0, "Wed Feb 18 16:19:16 UTC 2015\n", ""]
calling as root: date -s 'now -1 day'
call returned: [0, "Tue Feb 17 16:19:17 UTC 2015\n", ""]
calling as root: date
call returned: [0, "Tue Feb 17 16:19:17 UTC 2015\n", ""]
    When I bump the system time with "-1 day"                            # features/step_definitions/time_syncing.rb:21
      The guest's time was supposed to be bumped to '2015-02-17 16:19:16 +0000' but is '2015-02-17 16:19:17 +0000' (MiniTest::Assertion)
      ./features/step_definitions/time_syncing.rb:30:in `/^I bump the system time with "([^"]+)"$/'
      features/time_syncing.feature:18:in `When I bump the system time with "-1 day"'
    And the network is plugged                                           # features/step_definitions/common_steps.rb:110
    And Tor is ready                                                     # features/step_definitions/common_steps.rb:297
    Then Tails clock is less than 5 minutes incorrect                    # features/step_definitions/time_syncing.rb:35
Scenario failed at time 00:04:17

#15 Updated by kytv 2015-02-19 15:16:36

  • % Done changed from 40 to 50
  • QA Check changed from Dev Needed to Pass

With the latest changes (rev ddadef9) this branch is now working well for me.

#16 Updated by intrigeri 2015-02-19 15:44:14

  • Assignee changed from anonym to intrigeri

#17 Updated by Tails 2015-02-19 16:05:26

  • Status changed from In Progress to Fix committed
  • % Done changed from 50 to 100

Applied in changeset commit:ebc19f96f7d64a15467b0852529df94d2b109f6d.

#18 Updated by intrigeri 2015-02-19 16:06:14

  • Assignee deleted (intrigeri)

#19 Updated by BitingBird 2015-02-24 22:51:50

  • Status changed from Fix committed to Resolved