Feature #6304

Automate the most important bits of the Icedove tests

Added by bertagaz 2013-09-26 05:43:32 . Updated 2016-11-15 18:23:31 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Test suite
Target version:
Tails_2.7
Start date:
2013-09-26
Due date:
% Done:

100%

Feature Branch:
test/6304-icedove-tests
Type of work:
Code
Blueprint:

Starter:
0
Affected tool:
Email Client
Deliverable for:
268

Description

This should be enough: set up an account with the configuration wizard, check email over IMAP and POP, send email over SMTP, all this using one server (no need to do clearnet + Onion service).

Subtasks

Related issues

Related to Tails - Feature #10332: Write initial Icedove tests Resolved 2015-10-03
Related to Tails - Bug #11890: Checking credentials in Thunderbird autoconfig wizard sometimes fails in the test suite In Progress 2016-10-31
Related to Tails - Bug #10912: Tails Installer fails to install on USB stick that has a isohybrid dd'ed to it Resolved 2016-01-12
Related to Tails - Bug #11659: Add more manual tests for Icedove Rejected 2016-08-18
Blocked by Tails - Feature #6301: Securely store secrets needed by the automated test suite Resolved 2013-09-26
Blocked by Tails - Feature #8920: Have and use a repo shared between core developers for storing test suite secrets Resolved 2015-04-10
Blocked by Tails - Feature #9493: Write Icedove manual tests for common usecases and security requirements Resolved 2015-05-29

History

#1 Updated by intrigeri 2013-09-26 06:16:11

  • Target version set to Sustainability_M1

#2 Updated by BitingBird 2014-06-01 13:16:12

Feature #5663 does not have a milestone, and blocks this one - so either Feature #5663 is aimed for 2.0, or this ticket can loose its milestone…

#3 Updated by intrigeri 2014-06-03 07:05:30

  • Target version deleted (Sustainability_M1)

#4 Updated by intrigeri 2014-07-19 20:34:49

  • Target version set to Sustainability_M1

Setting back the milestone, now that Feature #5663 was set to 2.0.

#5 Updated by BitingBird 2015-01-07 18:13:07

  • Affected tool set to Email Client

#6 Updated by anonym 2015-01-09 14:46:17

  • Target version changed from Sustainability_M1 to Tails_1.8

#7 Updated by anonym 2015-01-09 15:53:32

  • blocks deleted (Feature #6300: Framework to set up throw-away services for the test suite)

#8 Updated by anonym 2015-01-09 15:54:10

  • blocked by Feature #6301: Securely store secrets needed by the automated test suite added

#9 Updated by anonym 2015-01-09 16:27:53

See Feature #6301#note-8 for a pointer on how this should be implemented.

#10 Updated by anonym 2015-01-10 18:08:55

  • Assignee set to kytv

#11 Updated by intrigeri 2015-02-19 15:31:48

  • blocked by Feature #8920: Have and use a repo shared between core developers for storing test suite secrets added

#12 Updated by intrigeri 2015-05-29 12:47:33

#13 Updated by intrigeri 2015-05-29 12:47:54

#14 Updated by intrigeri 2015-05-29 12:48:07

  • blocks #8668 added

#15 Updated by intrigeri 2015-05-29 12:49:09

  • Subject changed from Write Icedove tests to Automate the most important bits of the Icedove tests

#16 Updated by intrigeri 2015-05-29 12:49:32

  • blocked by Feature #9493: Write Icedove manual tests for common usecases and security requirements added

#17 Updated by intrigeri 2015-10-23 06:58:39

#18 Updated by sajolida 2015-11-27 04:45:14

  • Target version changed from 246 to Tails_2.0

#19 Updated by Anonymous 2015-12-22 07:11:13

Hi kytv,

this would be nice to have for 2.0. However, if you feel like postponing it to concentrate on a working PoC during January, please go ahead.

#20 Updated by kytv 2015-12-25 08:39:47

I hope to have progress here within the next few days.

#21 Updated by Anonymous 2016-02-05 16:47:44

  • Priority changed from Normal to Elevated
  • Target version changed from Tails_2.0 to Tails_2.2

Postponing and raising priority. Everything needed to work on this ticket has been done imo.

#22 Updated by Anonymous 2016-02-24 11:23:20

  • Target version changed from Tails_2.2 to Tails_2.3

#23 Updated by anonym 2016-03-10 12:57:45

  • Assignee changed from kytv to anonym

#24 Updated by anonym 2016-03-10 15:18:05

  • Target version changed from Tails_2.3 to Tails_2.4

#25 Updated by intrigeri 2016-05-10 04:40:45

  • blocked by deleted (#8668)

#26 Updated by intrigeri 2016-05-10 04:42:54

  • Deliverable for changed from 268 to SponsorS_Internal

#28 Updated by anonym 2016-05-26 15:41:26

  • Status changed from Confirmed to In Progress
  • Feature Branch set to test/6304-icedove-tests

#29 Updated by anonym 2016-06-08 01:34:51

  • Target version changed from Tails_2.4 to Tails_2.5

#30 Updated by intrigeri 2016-07-18 06:27:57

  • Target version changed from Tails_2.5 to Tails_2.7

#31 Updated by intrigeri 2016-08-18 07:42:02

  • Description updated

#33 Updated by anonym 2016-09-16 04:34:29

I have pushed something now where “most important” is:

  • Scenario: Only the expected addons are installed
  • Scenario: Enigmail is configured to use the correct keyserver
  • Scenario: Torbirdy is configured to use Tor
  • Scenario: Icedove’s autoconfiguration wizard defaults to IMAP and secure protocols
  • Scenario: Icedove can send emails, and receive emails over IMAP
  • Scenario: Icedove can send emails, and receive emails over POP3

Let’s see what jenkins thinks.

Note: I also wrote all the steps for testing sending/receiving mail over hidden services, but given that Chutney doesn’t give us access to real world hidden services, we would have to set up our own hidden services.

#34 Updated by anonym 2016-09-16 04:34:42

  • % Done changed from 0 to 40

#35 Updated by intrigeri 2016-09-16 06:12:25

  • Deliverable for changed from SponsorS_Internal to 268

#37 Updated by anonym 2016-09-22 06:14:25

  • Assignee changed from anonym to bertagaz
  • % Done changed from 40 to 50
  • QA Check set to Ready for QA

It runs reliably on Jenkins from what I can tell.

#38 Updated by anonym 2016-10-04 13:21:58

#39 Updated by bertagaz 2016-10-13 12:13:58

  • Assignee changed from bertagaz to anonym
  • QA Check changed from Ready for QA to Dev Needed

Sorry it took me a while to review this branch. I ran it extensively at home (over a hundred of times), and had to get it more runs in Jenkins to confirm my findings.

On the code review side I like it a lot. This feature now sounds like a perfect dogtail-ification example for other ones.

On the code testing side, I found a few glitches:

  • At home, Icedove seems to segfault quite often. It mostly happen in the step where we send an email, at different moments of this sending, but not only. Didn’t test with 45.4.0 yet though. I didn’t see that in Jenkins runs, so I guess we can put this aside.
  • There seem to sometimes have some problems related to at-spi: this run and this run are good example. The output of each dogtail command is then bloated with warnings that makes step parsing this output fail. Also testified this during runs at home.
  • In jenkins (never seen home), an attachment pop-up is sometimes raised when trying to send an email. I don’t know where it comes from. This run and this run exhibit this problem.
  • Sometimes, probably because of network troubles, the email account login and password verification step in Icedove fails. We probably need to detect this and add a retry here. You can find such failure in this run and this run

Other than that, this branch clearly ameliorates the whole feature. Dogtail seems much more reliable.

#40 Updated by anonym 2016-10-19 22:31:47

  • Assignee changed from anonym to bertagaz
  • % Done changed from 50 to 60
  • QA Check changed from Dev Needed to Ready for QA

bertagaz wrote:
> Sorry it took me a while to review this branch. I ran it extensively at home (over a hundred of times), and had to get it more runs in Jenkins to confirm my findings.
>
> On the code review side I like it a lot. This feature now sounds like a perfect dogtail-ification example for other ones.
>
> On the code testing side, I found a few glitches:
>
> * At home, Icedove seems to segfault quite often. It mostly happen in the step where we send an email, at different moments of this sending, but not only. Didn’t test with 45.4.0 yet though. I didn’t see that in Jenkins runs, so I guess we can put this aside.

Yes, it is known that Icedove recently got very segfault-prone. I don’t see how we can write tests where we restart icedove if this happens without making all scenarios into single massive steps. So I don’t think there’s much we want to do about this, even if it turns out to be a problem. :/

> * There seem to sometimes have some problems related to at-spi: this run and this run are good example. The output of each dogtail command is then bloated with warnings that makes step parsing this output fail. Also testified this during runs at home.

See: https://bugzilla.redhat.com/show_bug.cgi?id=972257

I feel the right course of action is to file a ticket about this Dogtail and/or at-spi bug, and hope that it will be fixed in the versions shipped in Tails/Stretch. If the Icedove tests fail too often due to this on Jessie I guess we mark them @fragile (and still benefit from them for release testing) while we hopefully can have them untagged in feature/stretch. Ok?

If so, feel free to review’n’merge, but reassign it to me until I’ve filed this ticket. :)

> * In jenkins (never seen home), an attachment pop-up is sometimes raised when trying to send an email. I don’t know where it comes from. This run and this run exhibit this problem.

If you look at the screenshots, you can see the message “Found an attachment keyword: CV”. :) Let’s disable this feature in the test suite. Fixed in commit commit:de62e7a.

> * Sometimes, probably because of network troubles, the email account login and password verification step in Icedove fails. We probably need to detect this and add a retry here. You can find such failure in this run and this run

Right. According to Mozilla bug #555448 it is a catch-all error message, but we can safely assume that if we see it, it is because of Tor issue. The only problem is that Dogtail cannot handle the “status_area” widget the error is stored inside. Meh. I added some retry_tor logic that is good enough. We should have retry_tor logic at every place where we depend on the network, but I will only add it at this step, because:

  1. if we succeed here, we’ll have a working circuit for the next actions too thanks to the ten minute circuit life time. Well, more or less.
  2. it’s hard to add it in most other places. (Actually, one other place is easy, so I added it in commit commit:96b5118.)

Done in commit commit:756e711.

> Other than that, this branch clearly ameliorates the whole feature.

Yay, that was my (secret) intention! :)

> Dogtail seems much more reliable.

… but it sure has its own share of shortcomings too. :/

#41 Updated by intrigeri 2016-10-26 14:19:17

  • Assignee changed from bertagaz to intrigeri

The deadline we set for bertagaz’ review has passed, so I’m taking this over.

#43 Updated by intrigeri 2016-10-26 15:06:31

  • Assignee changed from intrigeri to anonym
  • % Done changed from 60 to 70
  • QA Check changed from Ready for QA to Dev Needed

Code review passes, so the test failures I’ve reported above are the only blockers I can think of. Good job!

#44 Updated by anonym 2016-10-26 15:11:21

intrigeri wrote:
> Out of the last 10 runs of icedove.feature from this branch on Jenkins:
>
> * 8 passed
> * 2 had 4 scenarios (out of 6) failing
> https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_test-6304-icedove-tests/95/cucumberTestReport/icedove-email-client/only-the-expected-addons-are-installed/ while the picture looks OK: https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_test-6304-icedove-tests/95/artifact/build-artifacts/01%3A34%3A20_Only_the_expected_addons_are_installed.png
> https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_test-6304-icedove-tests/95/cucumberTestReport/icedove-email-client/enigmail-is-configured-to-use-the-correct-keyserver/ looks like a dogtail issue
> https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_test-6304-icedove-tests/95/cucumberTestReport/icedove-email-client/icedove-can-send-emails,-and-receive-emails-over-imap/ and https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_test-6304-icedove-tests/95/cucumberTestReport/icedove-email-client/icedove-can-send-emails,-and-receive-emails-over-pop3/ have “I can find the email I sent to myself in my inbox” failing but the screenshot shows presumably that very email
>
> anonym, what do you want to do about those?

All of them has this Dogtail warning of an APT-SPI error:

WARNING **: AT-SPI: Error in GetItems, sender=org.freedesktop.DBus, error=The name :1.0 was not provided by any .service files


so it is another instance of the issues mentioned in Feature #6304#note-40 (look for the link to the upstream redhat ticket). What do you think we should do about this situation?

#45 Updated by anonym 2016-10-26 15:12:02

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Dev Needed to Info Needed

#46 Updated by intrigeri 2016-10-26 15:30:22

  • Assignee changed from intrigeri to anonym

> All of them has this Dogtail warning of an APT-SPI error:
> […]
> so it is another instance of the issues mentioned in Feature #6304#note-40 (look for the link to the upstream redhat ticket). What do you think we should do about this situation?

> See: https://bugzilla.redhat.com/show_bug.cgi?id=972257

So in theory it’s fixed in dogtail 0.9.1 (testing/sid) but not in 0.9.0 (Jessie).

> I feel the right course of action is to file a ticket about this Dogtail and/or at-spi bug, and hope that it will be fixed in the versions shipped in Tails/Stretch. If the Icedove tests fail too often due to this on Jessie I guess we mark them @fragile (and still benefit from them for release testing) while we hopefully can have them untagged in feature/stretch. Ok?

Indeed, a 20% failure rate feels too high not to mark these tests as fragile. I could agree to merge this branch with the affected tests marked as fragile, but first I’d like to have a better understanding of what a real solution could be (without any occurrence of “hope” in the description of the solution ;)

But, wait: can’t we install python-dogtail from testing, that supposedly fixes the problem, in our ISO? I’ve just upgraded the package manually in a running Tails and at least APT-wise it went just fine.

#47 Updated by anonym 2016-10-26 16:31:22

  • QA Check changed from Info Needed to Dev Needed

intrigeri wrote:
> > All of them has this Dogtail warning of an APT-SPI error:
> > […]
> > so it is another instance of the issues mentioned in Feature #6304#note-40 (look for the link to the upstream redhat ticket). What do you think we should do about this situation?
>
> > See: https://bugzilla.redhat.com/show_bug.cgi?id=972257
>
> So in theory it’s fixed in dogtail 0.9.1 (testing/sid) but not in 0.9.0 (Jessie).

Yes. Or maybe the atspi2 version in Stretch works better.

> > I feel the right course of action is to file a ticket about this Dogtail and/or at-spi bug, and hope that it will be fixed in the versions shipped in Tails/Stretch. If the Icedove tests fail too often due to this on Jessie I guess we mark them @fragile (and still benefit from them for release testing) while we hopefully can have them untagged in feature/stretch. Ok?
>
> Indeed, a 20% failure rate feels too high not to mark these tests as fragile. I could agree to merge this branch with the affected tests marked as fragile, but first I’d like to have a better understanding of what a real solution could be (without any occurrence of “hope” in the description of the solution ;)

Well, I’m a bit lost, since I have no clue how atspi2 works, and that’s where the bug is. So, I suppose fixing the bug in atspi2 is the real solution.

> But, wait: can’t we install python-dogtail from testing, that supposedly fixes the problem, in our ISO? I’ve just upgraded the package manually in a running Tails and at least APT-wise it went just fine.

I pushed a commit doing that, so let’s see what happens for 10 more runs.

#48 Updated by intrigeri 2016-10-26 16:36:34

>> So in theory it’s fixed in dogtail 0.9.1 (testing/sid) but not in 0.9.0 (Jessie).

> Yes. Or maybe the atspi2 version in Stretch works better.

Just to clarify, I meant that https://bugzilla.redhat.com/show_bug.cgi?id=972257 is marked as resolved in 0.9.1 upstream.

> Well, I’m a bit lost, since I have no clue how atspi2 works, and that’s where the bug is. So, I suppose fixing the bug in atspi2 is the real solution.

… and as written above, it seems that this has already been done 2 years ago :)

>> But, wait: can’t we install python-dogtail from testing, that supposedly fixes the problem, in our ISO? I’ve just upgraded the package manually in a running Tails and at least APT-wise it went just fine.

> I pushed a commit doing that, so let’s see what happens for 10 more runs.

Cool!

#49 Updated by bertagaz 2016-10-27 11:21:23

Yay, I missed what happened in this ticket, I was busy outside, but meanwhile I’ve still run this branch locally and in Jenkins, and analyzed the errors I’ve seen.

First, this branch greatly enhanced the runs. The failure ratio has lowered a lot more.

As you noted one big remaining issue is the dogtail/at-spi failures. If the upgrades you’re trying to test is not solving the problem, we may still workaround the issue: what I’ve seen is that if this error happens, it always does at the beginning of the feature, and then is spread to all the scenarios thanks to our snapshot system.

So maybe we can add a step right at the beginning of the session opening, after the “available upgrades have been checked” that check if dogtail works fine, and restart the snapshot process if it isn’t. But well, let’s hope that the simpler upgrading at-spi and dogtail works. If we need to use this trick, I think we should open a dedicated ticket and merge this branch.

The only other issue I’ve seen when Icedove is checking the credentials in the wizard. Even retry_tor() it doesn’t prevent some failures from time to time. One thing we may do is just ignore this failures: in this step, what we need to know is that there’s no network traffic outside of Tor, not that the login we know works does. So we may just try it in the related scenario, ensure there were traffic outside of Tails and only through Tor no matter what the result of this credentials check is, and then in the other scenario simply drop a configuration file in icedove directory. But maybe that’s not required, or pertains to another ticket.

On the code side, I don’t like so much this:

     When I start Icedove
     And Icedove has started

I think we could just write “When I start Icedove”.

That’s the only remaining troubles I’ve seen (apart from one segfault out of 40 local runs). We’re getting close !

Now, I’ll switch to Bug #11874 :)

#50 Updated by anonym 2016-10-27 13:38:56

anonym wrote:
> intrigeri wrote:
> > But, wait: can’t we install python-dogtail from testing, that supposedly fixes the problem, in our ISO? I’ve just upgraded the package manually in a running Tails and at least APT-wise it went just fine.
>
> I pushed a commit doing that, so let’s see what happens for 10 more runs.

After this I’ve seen two runs where the error is logged, but Dogtail doesn’t fail:

I guess that is a promising sign!

#51 Updated by anonym 2016-10-27 18:30:32

anonym wrote:
> After this I’ve seen two runs where the error is logged, but Dogtail doesn’t fail:

Here’s a third:

Now we’re up in nine runs, and 3/9 is pretty consistent with the previous 20% error rate reported by intrigeri (Feature #6304#note-46) given the sample size, but it gives confidence that this new Dogtail version catches these errors and prevents the failure. What a relief! Over the past months I’ve started building up a feeling that Dogtail has subtle issues like this one that makes it less robust than what we require, and perhaps worse (accuracy-wise) than sikuli. Now my confidence is returning! :)

bertagaz, I’ll look at your comment Feature #6304#note-49 soon, but not today.

#52 Updated by anonym 2016-10-28 10:56:15

bertagaz wrote:
> As you noted one big remaining issue is the dogtail/at-spi failures. If the upgrades you’re trying to test is not solving the problem, we may still workaround the issue: what I’ve seen is that if this error happens, it always does at the beginning of the feature, and then is spread to all the scenarios thanks to our snapshot system.
>
> So maybe we can add a step right at the beginning of the session opening, after the “available upgrades have been checked” that check if dogtail works fine, and restart the snapshot process if it isn’t. But well, let’s hope that the simpler upgrading at-spi and dogtail works. If we need to use this trick, I think we should open a dedicated ticket and merge this branch.

I guess we don’t need this now.

> The only other issue I’ve seen when Icedove is checking the credentials in the wizard. Even retry_tor() it doesn’t prevent some failures from time to time.

For the record I’ve never seen this locally or on Jenkins. I admittedly have not looked at all Jenkins runs — have you seen it there?

> One thing we may do is just ignore this failures: in this step, what we need to know is that there’s no network traffic outside of Tor, not that the login we know works does. So we may just try it in the related scenario, ensure there were traffic outside of Tails and only through Tor no matter what the result of this credentials check is, and then in the other scenario simply drop a configuration file in icedove directory. But maybe that’s not required, or pertains to another ticket.

The credentials checking is part of the account creation process, which must succeed for the rest of the scenarios steps, so this must be robust any way. I would not be surprised if there’s a bug in the JavaScript mess that is the autoconfiguration wizard that prevents it from recovering (to a state where it can retest the credentials), hence making our try_tor magic not work. I’m gonna look into reproducing this. Do you happen to have a debug log of when this happens?

Still, I propose that if the Jenkins results continue to look this good we give merging it (without @fragile tags) a try, and possibly revisit this if it turns out to be a problem.

> On the code side, I don’t like so much this:
>
> […]
>
> I think we could just write “When I start Icedove”.

Fixed!

> That’s the only remaining troubles I’ve seen (apart from one segfault out of 40 local runs).

So far I’ve seen no segfault on Jenkins, by the way.

Thanks for the review and test data!

Everything’s been looking good on Jenkins, except an error in this log:

Note that it has the AT-SPI error but handles it nicely. The error I am talking about is:

calling as amnesia: echo '#!/usr/bin/python
from dogtail import tree
from dogtail.config import config
config.searchShowingOnly = True
application = tree.root.application('"'"'Icedove'"'"')
from dogtail import predicate
node = None
for n in application.child('"'"'Add-ons Manager - Icedove Mail/News'"'"', roleName='"'"'frame'"'"').child('"'"'amnesia branding'"'"', roleName='"'"'label'"'"').parent.parent.findChildren(predicate.GenericPredicate()):
    if str(n.path) == '"'"'/org/a11y/atspi/accessible/228'"'"':
        node = n
        break
assert(node)
print(node.name)' >> '/tmp/tmp.nvQs511obP'
call returned: [1744, 0, "", ""]
calling as amnesia: /usr/bin/python '/tmp/tmp.nvQs511obP'
call returned: [1745, 0, "Creating logfile at /tmp/dogtail-amnesia/logs/tmp.nvQs511obP_20161028-004613_debug ...\nDogtail: warning: application may be hanging\nTorBirdy This extension configures Thunderbird to make connec\u2026 More Release Notes: Loading\u2026 Preferences Disable\n", "\n** (tmp.nvQs511obP:5663): WARNING **: AT-SPI: Error in GetItems, sender=org.freedesktop.DBus, error=The name :1.0 was not provided by any .service files\n"]
calling as root: rm -f '/tmp/tmp.nvQs511obP'
call returned: [1746, 0, "", ""]
    Then I see that only the amnesia branding, Enigmail and TorBirdy addons are enabled in Icedove # features/step_definitions/icedove.rb:59
      <nil> expected to not be nil. (Test::Unit::AssertionFailedError)

The interesting part is the remote shell stdout result that starts with “Creating logfile” and then has “TorBirdy This extension […]”. Normally we’d only get this latter part, and that’s what we expect, and why we get an assertion failure here. We need to make sure that Dogtail never writes to stdout unless we tell it to, since we depend on the exact stdout ouput many times. I’ll look into this.

#53 Updated by anonym 2016-10-28 11:08:09

anonym wrote:
> We need to make sure that Dogtail never writes to stdout unless we tell it to, since we depend on the exact stdout ouput many times. I’ll look into this.

Fixed in commit:1eb8566.

Also, commit:55cdd5f might be interesting.

I’m waiting for results on Jenkins before I put this one back for review.

#54 Updated by intrigeri 2016-10-28 11:22:32

> Still, I propose that if the Jenkins results continue to look this good we give merging it (without @fragile tags) a try, and possibly revisit this if it turns out to be a problem.

ACK

#55 Updated by bertagaz 2016-10-28 15:50:17

anonym wrote:
> > So maybe we can add a step right at the beginning of the session opening, after the “available upgrades have been checked” that check if dogtail works fine, and restart the snapshot process if it isn’t. But well, let’s hope that the simpler upgrading at-spi and dogtail works. If we need to use this trick, I think we should open a dedicated ticket and merge this branch.
>
> I guess we don’t need this now.

Seems so.

> > The only other issue I’ve seen when Icedove is checking the credentials in the wizard. Even retry_tor() it doesn’t prevent some failures from time to time.
>
> For the record I’ve never seen this locally or on Jenkins. I admittedly have not looked at all Jenkins runs — have you seen it there?

I usually have more network transient errors than in Jenkins, so that’s probably why. I don’t remember having seen that in Jenkins, specially since your last commits with the retry_tor wrapping (and I’ve look at most of the last runs since then). I’ll have a quick look just in case, but I think it works better on lizard than $HOME. My connectivity is probably a bot more shitty.

> The credentials checking is part of the account creation process, which must succeed for the rest of the scenarios steps, so this must be robust any way. I would not be surprised if there’s a bug in the JavaScript mess that is the autoconfiguration wizard that prevents it from recovering (to a state where it can retest the credentials), hence making our try_tor magic not work. I’m gonna look into reproducing this. Do you happen to have a debug log of when this happens?

Yeah I think I can find one or two. I’ll dig and post later.

> Still, I propose that if the Jenkins results continue to look this good we give merging it (without @fragile tags) a try, and possibly revisit this if it turns out to be a problem.

I agree, we’ll see in the November fragile tag round if that’s a problem.

> > That’s the only remaining troubles I’ve seen (apart from one segfault out of 40 local runs).
>
> So far I’ve seen no segfault on Jenkins, by the way.

Me neither. That’s good news. :)

> Thanks for the review and test data!

Thanks for the code! :)

#56 Updated by anonym 2016-10-29 02:50:39

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

anonym wrote:
> anonym wrote:
> > We need to make sure that Dogtail never writes to stdout unless we tell it to, since we depend on the exact stdout ouput many times. I’ll look into this.
>
> Fixed in commit:1eb8566.
>
> Also, commit:55cdd5f might be interesting.
>
> I’m waiting for results on Jenkins before I put this one back for review.

4/4 Jenkins runs with the new commits has finished without any Icedove or Dogtail issues. Review’n’merge, please!

bertagaz wrote:
> anonym wrote:
> > > The only other issue I’ve seen when Icedove is checking the credentials in the wizard. Even retry_tor() it doesn’t prevent some failures from time to time.
> >
> > For the record I’ve never seen this locally or on Jenkins. I admittedly have not looked at all Jenkins runs — have you seen it there?
>
> I usually have more network transient errors than in Jenkins, so that’s probably why. I don’t remember having seen that in Jenkins, specially since your last commits with the retry_tor wrapping (and I’ve look at most of the last runs since then). I’ll have a quick look just in case, but I think it works better on lizard than $HOME. My connectivity is probably a bot more shitty.

Ok. Let’s open a ticket about it, but only if you manage to get some useful log to attach to it — otherwise I have nothing to try to understand the problem.

> > The credentials checking is part of the account creation process, which must succeed for the rest of the scenarios steps, so this must be robust any way. I would not be surprised if there’s a bug in the JavaScript mess that is the autoconfiguration wizard that prevents it from recovering (to a state where it can retest the credentials), hence making our try_tor magic not work. I’m gonna look into reproducing this. Do you happen to have a debug log of when this happens?
>
> Yeah I think I can find one or two. I’ll dig and post later.

… to the above mentioned new ticket, please!

#57 Updated by intrigeri 2016-10-29 12:23:25

  • Status changed from In Progress to Fix committed
  • % Done changed from 70 to 100

Applied in changeset commit:695dad03f49452d93e4ecea0c6ae4aadbec93b73.

#58 Updated by intrigeri 2016-10-29 12:24:18

  • Assignee deleted (intrigeri)
  • QA Check changed from Ready for QA to Pass

#59 Updated by intrigeri 2016-10-31 08:42:04

  • related to Bug #11890: Checking credentials in Thunderbird autoconfig wizard sometimes fails in the test suite added

#60 Updated by intrigeri 2016-10-31 08:42:38

anonym wrote:
> bertagaz wrote:
> > anonym wrote:
> > > > The only other issue I’ve seen when Icedove is checking the credentials in the wizard. Even retry_tor() it doesn’t prevent some failures from time to time.
> > >
> > > For the record I’ve never seen this locally or on Jenkins. I admittedly have not looked at all Jenkins runs — have you seen it there?
> >
> > I usually have more network transient errors than in Jenkins, so that’s probably why. I don’t remember having seen that in Jenkins, specially since your last commits with the retry_tor wrapping (and I’ve look at most of the last runs since then). I’ll have a quick look just in case, but I think it works better on lizard than $HOME. My connectivity is probably a bot more shitty.
>
> Ok. Let’s open a ticket about it, but only if you manage to get some useful log to attach to it — otherwise I have nothing to try to understand the problem.
>
> > > The credentials checking is part of the account creation process, which must succeed for the rest of the scenarios steps, so this must be robust any way. I would not be surprised if there’s a bug in the JavaScript mess that is the autoconfiguration wizard that prevents it from recovering (to a state where it can retest the credentials), hence making our try_tor magic not work. I’m gonna look into reproducing this. Do you happen to have a debug log of when this happens?
> >
> > Yeah I think I can find one or two. I’ll dig and post later.
>
> … to the above mentioned new ticket, please!

Bug #11890

#61 Updated by bertagaz 2016-11-15 18:23:31

  • Status changed from Fix committed to Resolved

#62 Updated by intrigeri 2016-12-06 16:22:15

  • related to Bug #10912: Tails Installer fails to install on USB stick that has a isohybrid dd'ed to it added

#63 Updated by intrigeri 2016-12-06 16:23:16

  • related to Bug #11659: Add more manual tests for Icedove added