#3 Updated by intrigeri 2019-02-09 10:43:04

Looks like this will require refreshing config/chroot_local-patches/apparmor-adjust-thunderbird-profile.diff: https://jenkins.tails.boum.org/job/build_Tails_ISO_feature-buster/581/console

#5 Updated by CyrilBrulebois 2019-03-08 17:57:13

I’m working on this and initial investigations suggest it will likely take more than the usual 2 hours. Expecting up to 4 hours (maybe more with some testing), due to some needed rework for a bunch of the secure-account-creation patches.

@intrigeri: ACK regarding the extended time budget?

#6 Updated by intrigeri 2019-03-09 06:58:25

> ACK regarding the extended time budget?

CyrilBrulebois, yes. Note that anonym has recently sent an updated set of patches upstream (probably based on something more recent than 60.x though) so it might be that some of this patch reworking has been done already. It would be great if you managed to coordinate with him but don’t block on this for too long :)

#7 Updated by intrigeri 2019-03-09 07:10:09

Also, note that 60.5.3 is ready upstream. I don’t know if it has changes we want nor when Carsten will package it.

#8 Updated by CyrilBrulebois 2019-03-10 16:21:20

OK, thanks, will prod by mail too.

60.5.3’s release notes seem to be mostly about: “Problem when using ”Send to > Mail recipient" on Windows introduced in version 60.5.2.”

60.5.2’s release notes seem to be about Windows issues mostly, except for S/MIME support. Do we support that? → “Problem with S/MIME certificate verification when receiving email from Outlook (issue introduced in version 60.5.1)”

#9 Updated by intrigeri 2019-03-11 07:13:30

> except for S/MIME support. Do we support that?

We explicitly support OpenPGP in Thunderbird but we don’t support S/MIME ⇒ ignore the S/MIME issues and proceed :)

#17 Updated by intrigeri 2019-03-18 06:08:18

> The resulting image almost passes the features/thunderbird.feature test suite, except for this part:

@CyrilBrulebois, that’s Bug #16555. Glad you found a fix for it!

#18 Updated by intrigeri 2019-03-18 06:34:10

@CyrilBrulebois, code review passes except one thing:

It looks like the mailnews.auto_config.account_constraints.ssl_only pref was renamed to mailnews.auto_config.ssl_only_mail_servers in the updated patchset. In config/chroot_local-includes/etc/thunderbird/pref/thunderbird.js we’re setting the former but not the latter, so I think that on this branch, we haven’t enabled 100% of the things we expect to get out of anonym’s Feature #6156 patchset. I’ve renamed this pref on the topic branch; now building, then will run thunderbird.feature and report back here.

#19 Updated by intrigeri 2019-03-18 06:34:20

> I’ve managed to fix the test suite thanks to accerciser, and learned a bit more about how to print debug information in the process (I struggled a bit with Dogtail::Node objects initially…).

Congrats \o/

#24 Updated by intrigeri 2019-03-18 07:44:43

> Will now test manually with & without persistence, using a non-Riseup email account

@CyrilBrulebois, unfortunately I see a regression in the account wizard. I first see “Configuration found at email provider” with correct settings, so far so good; on Tails 3.12.1 the wizard would (correctly) stop here. But on this branch, then the wizard keeps trying other things (at least “Looking up configuration: Trying common server names”) and ends up telling me “Thunderbird failed to find the settings for your email account” and pre-filling the config settings with imap.$DOMAIN and smtp.$DOMAIN, that won’t work in this case.

With pristine 1:60.5.1-1~deb9u1 I see the same buggy behaviour except one important thing: while I do see “Thunderbird failed to find the settings for your email account”, the pre-filled config settings are the correct ones fetched from the email provider, so clicking “Done” is enough to get a working account.

So it seems that Thunderbird 60.5.1 brings part, but not all, of the regression (the wizard keeps looking in other sources after it has already found settings in an authoritative place, and then it gives confusing error feedback to the user while everything is configured just fine). And the other part of the regression seems to be brought either by the latest patchset prepared by anonym (if he tested only with Riseup account, he may have missed this problem: imap.riseup.net and smtp.riseup.net will work), or by the backporting to 60.5.1.

In any case, I think it’s too late to try and fix the patchset in time for 3.13 so IMO our best option at this point is to:

1. Drop the patchset, ship 1:60.5.1-1~deb9u1 from Debian in Tails 3.13. I’ll prepare a branch right away.
2. File a ticket about this for 3.14 (and if we’re fast enough there + unlucky enough to have to release 3.13.1 after Pwn2Own, this could actually be fixed within a week or two instead of in 2 months).
3. Document the known issue in the release notes.

#27 Updated by anonym 2019-03-18 10:32:12

Looking at the branch I see no changes to any prefs. This is concerning because the new patch series changes many of the pref names (upstream request) so you probably tested something strange. It’s unclear how this will have effected your test results, but my guess is that it didn’t affect and your tests are still good.

intrigeri wrote:
> > Will now test manually with & without persistence, using a non-Riseup email account
>
> CyrilBrulebois, unfortunately I see a regression in the account wizard. I first see “Configuration found at email provider” with correct settings, so far so good; on Tails 3.12.1 the wizard would (correctly) stop here. But on this branch, then the wizard keeps trying other things (at least “Looking up configuration: Trying common server names”) and ends up telling me “Thunderbird failed to find the settings for your email account” and pre-filling the config settings with imap.$DOMAIN and smtp.$DOMAIN, that won’t work in this case.

I noticed this myself when I was rebasing and partly rewriting the patch set for Thunderbird 66 but only in a odd edge case: I tested against ProtonMail whose config points to servers at 127.0.0.1 (I suppose you are supposed to run some local proxy to ProtonMail). Thunderbird would consider it a success in the GUI as soon as a valid config is fetched, but would fail after a few seconds when it tries the config (since we don’t run any mail servers on 127.0.0.1 and even block any such connection attempt), and then fallback to the remaining auto-config methods that still are running in the background. The fix is to no consider a config successful (not even in the GUI!) until it has been tested.

Care to share which service you tried with so I can investigate if it’s similar?

> With pristine 1:60.5.1-1~deb9u1 I see the same buggy behaviour except one important thing: while I do see “Thunderbird failed to find the settings for your email account”, the pre-filled config settings are the correct ones fetched from the email provider, so clicking “Done” is enough to get a working account.

Interesting! How many times did you reproduce this?

Also note this patch from our series:

    Invalidate config when restarting autoconfiguration.

    Otherwise residual data from previous attempts can leak into following
    attempts, e.g.:

    1. Try "a@a.a"
    2. All methods fail, so we end up in "manual edit" mode
    3. Note that Server hostname == "a.a", which is expected
    4. Then start over by changing the email to "b@b.b"
    5. All methods fail, so we end up in "manual edit" mode
    6. Note that Server hostname == "a.a", but should be "b.b"

So I hope you didn’t trigger that bug in a very unlucky scenario with the non-patched Thunderbird.

I also wonder if the prefs descrepency could cause problems with your test, but I sort of doubt it and think a bug in my patches is more likely.

> So it seems that Thunderbird 60.5.1 brings part, but not all, of the regression (the wizard keeps looking in other sources after it has already found settings in an authoritative place, and then it gives confusing error feedback to the user while everything is configured just fine). And the other part of the regression seems to be brought either by the latest patchset prepared by anonym (if he tested only with Riseup account, he may have missed this problem: imap.riseup.net and smtp.riseup.net will work), or by the backporting to 60.5.1.

This is my “test suite” to cover the auto-config methods we care about:

• tails@riseup.net → ‘Configuration found on Thunderbird installation’
• tails@boum.org → ‘Configuration found at email provider’
• tails@gmail.com → ‘Configuration found in Mozilla ISP database’
• tails@herbesfolles.org → ‘Configuration found by trying common server names’ (they serve a .well-known config over insecure http only, so in Tails we won’t fetch it since we require https, and get protocol guessing instead)

Since Thunderbird 60.5.1/66 makes all auto-config methods run in parallel you might have to rerun tests to get the expected result.

> In any case, I think it’s too late to try and fix the patchset in time for 3.13 so IMO our best option at this point is to:
>
> # Drop the patchset, ship 1:60.5.1-1~deb9u1 from Debian in Tails 3.13. I’ll prepare a branch right away.

Ack! However…

☣️☠⚠️ Please also disable the auto-configuration wizard! ⚠️☠☣️

By default auto-configuration is catastrophically unsafe (after all, that is why the patch series exist in the first place), and protocol guessing is broken since it bypasses the proxy configuration => blocked. Sadly, since our IMAP/POP/SMTP using Thunderbird scenarios arrive there by using the configuration wizard, these tests will have to be removed until we can safely enable the wizard again.

Implementation hint: in config/chroot_local-includes/etc/xul-ext/torbirdy.js set extensions.torbirdy.emailwizard to false.

#29 Updated by anonym 2019-03-18 13:57:00

intrigeri wrote:
> anonym wrote:
> > intrigeri wrote:
> > > With pristine 1:60.5.1-1~deb9u1 I see the same buggy behaviour except one important thing: while I do see “Thunderbird failed to find the settings for your email account”, the pre-filled config settings are the correct ones fetched from the email provider, so clicking “Done” is enough to get a working account.
>
> > Interesting! How many times did you reproduce this?
>
> At least 5 times over 5 tries (3 tries with a non-Riseup account, 2 tries with a Riseup account).

I had a quick look. The config is fetched from the ISP (autistici) just fine, but an exception is thrown (TypeError: radiogroup is null, yet to be investigated) invalidating it => fallback to protocol guessing. This happens both with our patches and without, so it’s an upstream bug. The difference is that without our patches protocol guessing will fail immediately and some how the server settings from the invalidated config is kept, but with our patches the guessing will initiate and fail after exhaustive testing, and when it fails like that it settles on exactly ${PROTOCOL}.${EMAILDOMAIN} (e.g. imap.boum.org, yeah it extracted the domain straight from the provided email address).

So our patches might not have a bug after all! :)

> So I’m taking back my proposal to ship 60.5.1-1~deb9u1 ⇒ kibi, please review and merge the updated bugfix-16422-thunderbird-60.5.1 branch.

Yeah, that makes most sense.