Tails

#7 Updated by BitingBird 2015-01-02 19:29:41

I think the type of work is communicate, since at least in Debian I see no big report, but I’m not sure so I let it as is for now.

#14 Updated by Anonymous 2015-12-22 07:00:25

Setting realistic target version to 2.4, maximum latest delay would be 2.5.

#15 Updated by intrigeri 2015-12-22 09:33:04

Today we discussed that this should be started during the 2.2 cycle, span over the 2.3 and 2.4 ones. I would suggest setting a milestone that matches when you should start working on it, instead of one that matches when it should be finished, to prevent any risk that it’s forgotten until 2.3 is out. Your call, of course :)

#18 Updated by Anonymous 2016-01-09 18:19:53

The current plan is to send our patches with a text to Torbirdy people for a first review with ETA Jan 24th 2016.

Then ideally we would be able to send them to the Thunderbird people and also update the corresponding TPO ticket. I’d also open a Debian bug with the same patches and link it to the upstream bug, so eventually the patches could be included in Debian directly - until upstream has them released.

#19 Updated by Anonymous 2016-02-27 12:39:55

I’ve posted the patches upstream today.
See https://bugzilla.mozilla.org/show_bug.cgi?id=669238 and https://bugzilla.mozilla.org/show_bug.cgi?id=971347

#20 Updated by intrigeri 2016-02-27 12:51:43

> I’ve posted the patches upstream today.

Great! A link would be useful next time we need to report about the progress of this task :)

#25 Updated by intrigeri 2016-06-07 02:18:30

> There has been quite some progress. Our patches have all been reviewed now and only code style improvements were asked for as a result of this review. This makes me think that we’re nearly there! We’ll work on a modified patchset taking upstream’s remarks into account later this week.

Excellent news, glad to read that!

#26 Updated by Anonymous 2016-06-10 06:19:54

We’ve resubmitted a new patchset today which was thouroughly tested by anonym. Waiting for a reply from upstream now. Let’s hope for the best.

#31 Updated by Anonymous 2016-09-15 07:47:42

I’ve submitted the latest patchset today. Waiting for upstream again now.

#35 Updated by intrigeri 2016-10-04 16:52:54

> I think that this ticket’s deliverable has been accomplished “try hard to upstream the icedove autoconfig wizard”. Thus, we’ll continue to keep track of it, but not as part of the deliverable.

Woohoo! \o/

#36 Updated by bertagaz 2016-10-04 18:23:23

intrigeri wrote:
> Woohoo! \o/

+1 :)

#43 Updated by intrigeri 2018-01-07 15:50:37

I see lots of discussion happening on the corresponding upstream bug in the past three months. Someone posted an updated patch and somewhat aggressively engaged with upstream. Upstream reacted (very understandably) somewhat defensively. If we still plan to get this merged some day, I think we need to intervene and help de-escalate the situation. anonym & u, did you follow this discussion, do you plan to get involved? If not, please let me/us know that you need help there: I don’t think we can afford doing nothing about it on the long term, the risk of having to ship our patched Thunderbird package forever is just too great. Thanks in advance!

#44 Updated by Anonymous 2018-01-15 11:23:06

See https://labs.riseup.net/code/issues/6156#note-35 for why I am unblocking #8668.

#46 Updated by Anonymous 2018-01-15 11:25:45

I will take care of the upstream discussion.

However, our patches have not been merged because we were unable to provide a version that worked in Thunderbird’s test suite (which we did not manage to run). This is Bug #12151.

#49 Updated by Anonymous 2018-02-26 15:11:14

I replied on the bug - there was a lot of activity recently and it seems like people are still willing to have these patches reviewed and incorporated into Thunderbird. See https://bugzilla.mozilla.org/show_bug.cgi?id=971347

#50 Updated by intrigeri 2018-02-27 11:11:19

> it seems like people are still willing to have these patches reviewed and incorporated into Thunderbird.

I wonder who these people are and whether I missed something because my understanding of the situation is very different, and still the same as Feature #6156#note-43 (nothing happened since then anyway until your own comment): among the two Thunderbird maintainers who commented, one nack-ed and the other one has strong doubts. I see someone else (alta88) arguing and insisting in favour of our patches but I’m not sure that will help much, quite the opposite IMO.

#51 Updated by Anonymous 2018-02-27 15:15:23

intrigeri wrote:
> > it seems like people are still willing to have these patches reviewed and incorporated into Thunderbird.
>
> I wonder who these people are and whether I missed something because my understanding of the situation is very different,

Looks like it.

> and still the same as Feature #6156#note-43 (nothing happened since then anyway until your own comment): among the two Thunderbird maintainers who commented, one nack-ed

I’m not sure what “nack’ing” is, may you please explain that?

> and the other one has strong doubts.

This one says “Of course SSL/TLS connections would be preferred” and “I see the patch add proxy support in some of the paths, maybe at least that part could be accepted?”. So the way you phrase it (“strong doubts”) seems slightly pessimistic to me.

> I see someone else (alta88) arguing and insisting in favour of our patches but I’m not sure that will help much, quite the opposite IMO.

Absolutely, but I sincerely hope Thunderbird people can make the distinction between this person, us, and the problem we’re trying to solve.

#52 Updated by intrigeri 2018-02-27 16:03:18

> I’m not sure what “nack’ing” is, may you please explain that?

Sure, sorry! ACK = agreeing/approving != NACK = disagreeing/rejecting

>> and the other one has strong doubts.

> This one says “Of course SSL/TLS connections would be preferred” and “I see the patch add proxy support in some of the paths, maybe at least that part could be accepted?”. So the way you phrase it (“strong doubts”) seems slightly pessimistic to me.

I think we’ll have to agree to disagree on this one :)

>> I see someone else (alta88) arguing and insisting in favour of our patches but I’m not sure that will help much, quite the opposite IMO.

> Absolutely, but I sincerely hope Thunderbird people can make the distinction between this person, us, and the problem we’re trying to solve.

+1

#54 Updated by anonym 2018-03-07 15:02:04

BTW, I’ve noticed that the Mozilla configuration database (https://live.mozillamessaging.com/autoconfig/v1.1/) often presents a CAPTCHA when the request originates from Tor => that method fails => Bug #15387

#58 Updated by Anonymous 2018-04-04 11:33:34

I added those to the upstream bug today.

#62 Updated by intrigeri 2018-08-15 06:46:37

I think Bug #15788 should be done first thing before any more work is put into this project :)

#65 Updated by Anonymous 2018-08-17 06:25:31

> AFAICT it was never published, let alone submitted upstream, so far; 4 people have access to it with the attached comments that explain things: anonym, Ulrike, kibi and myself (the 2 latter in <335bfd60-7be2-b0e3-9fbd-8a58db17e10d451f.org>@). Apparently anonym wanted the patches and the associated comments to go on the upstream bug report, so perhaps I could attach all this here to start with, so at least the material to submit is not kept secret, but there may be a reason why this did not happen yet. Ulrike?

It has not happened because it’s not based on the current development version of TB, so these patches are very likely to be rejected - something I’d like to avoid, after all the complicated discussions we’ve gone through with upstream until now. And I was still expecting anonym to rebase the patches on a newer version of TB…

But yes, indeed, you can attach the patches here, and we can improve upon them from this point on. Please not the comments on the upstream bug: some patches have already been reviewed and are considered mergeably by them. If possible I’d like to add this in a comment somewhere, so that we don’t redo work that has already been done.

#67 Updated by Anonymous 2018-08-17 06:27:53

I would love us to rework on these patches and resubmit them upstream for good. Let me know if you need my help for this or if FT will handle it from this point on.

#69 Updated by intrigeri 2018-08-17 12:28:48

> Let me know if you need my help

Anyone moving this forward is warmly welcome :)

> for this or if FT will handle it from this point on.

We’ll see at the summit. In the current state of things I doubt the FT has the budget to do that.

#70 Updated by intrigeri 2018-09-14 11:26:06

u wrote:
> But yes, indeed, you can attach the patches here, and we can improve upon them from this point on.

Note to myself: once this is done, set target version to 2019 and reassign to lamby, as per summit 2018.

#71 Updated by lamby 2018-09-14 13:50:30

Again, Did we discuss this? I don’t recall doing so, thus just wondering if it’s a mistake to assign it over to me specifically? (If it’s “to be discussed” feel free to assign back - I note the “2019” target.)

#72 Updated by intrigeri 2018-09-14 14:04:23

> Again, Did we discuss this?

Yes (but very briefly!) during the FT meeting when we were brainstorming the tasks we want to add to our roadmap for the next years. Here’s what I remember: IIRC kibi expressed interest because he’s already worked on our Thunderbird patched package, then I asked “how is your JavaScript”, his answer suggested it would be better to assign the task to someone else, and after a show of hands you mentioned your JavaScript was OK. I probably jumped too fast to conclusions: the fact your JS is OK does not necessarily imply that you want to work on this specific task. Sorry! Either way, I think there’s quite some info I need to share with you before you can tell whether you want to take it or not, so let’s take this assignment with a grain of salt. If you prefer, feel free to de-assign yourself until we’ve had this conversation :)

#78 Updated by Anonymous 2019-02-28 17:36:05

anonym has rebased our patches - once again - on the current code base of Thunderbird. I’ve posted them all to the bug at https://bugzilla.mozilla.org/show_bug.cgi?id=971347 (also see https://bugzilla.mozilla.org/show_bug.cgi?id=971347#c197 for explanations).

Let’s hope that’s the last time we need to do that.

#85 Updated by Anonymous 2019-03-27 13:28:26

Thundebird people are organizing a bugday.
They have a document where they collect bugs to focus on: https://docs.google.com/document/d/1jQfOIuu_9KYPw_hfrPjwGrhQWKv-bzBDr214_tstMbc/edit#
I’ve added “our” bug there.

#90 Updated by anonym 2019-05-09 17:56:59

Big news! Most patches have been upstreamed!

Summary of upstreamed patches

You can look here for reference

• Minor stuff:
  • Improve logging of guess instances.
  • Add comment for pref.
  • Invalidate config when restarting autoconfiguration.
• Tor compatibility
  • Add pref for setting the autoconfiguration guess timeout.
  • Add SOCKS proxy support for account guessing.
• Security
  • Also fetch ISP configuration using SSL.
  • New patch! Add pref for guessing only SSL configs. (This is ~half of “Add pref for whether to accept plaintext protocols during autoconfiguration” that we will discuss below)

Patches that are still not upstreamed

Make use of non-SSL Exchange AutoDiscover methods optional.

Thunderbird very recently (60.6.0) added a new autoconfig method, Microsoft’s AutoDiscover, which use two secure mehtods and one insecure HTTP fallback. On short notice I tried to secure it (i.e. only do the secure methods), but upstream didn’t like it (“it breaks the protocol”).

I propose we just disable this method entirely (mailnews.auto_config.fetchFromExchange.enabled → false).

Add pref for whether we accept OAuth2 during autoconfiguration.

This is pretty bad, UX-wise, which can be demonstrated in current Tails:

1. Start Thunderbird
2. Quit autoconfig wizard
3. Flip mailnews.auto_config.account_constraints.allow_oauth2 to true
4. Start autoconfig wizard for $account@gmail.com (must be a valid one that you have the password for)
5. Pick one of the two results (imap or pop, it doesn’t matter) you get; due to its priority Mozilla’s database will win, and its first pick is OAuth2 so that’s what you get (it’s not displayed unless you click the “Manual config” button)
6. A browser window will open, asking you to login to the Google account, so enter $account@gmail.com and its password
7. Observe the error message stating that you need to enable JavaScript
8. Closing that browser window brings you back to the result screen from step 5, but now it says “the username or password is wrong”.

You are stuck in 8: autoconfig will bring you exactly here again no matter what you try. Your only option is to click “Manual config” and change authentication from “OAuth2” to “Normal password” (both for pop/imap and smtp). Good luck! :)

This breaks several providers, namely these Mozilla database entries that all employ OAuth2 as its first authentication option: gmail.com, google.com, googlemail.com, bk.ru, corp.mail.ru, inbox.ru, jazztel.es, list.ru, mail.ru. In addition to this any provider not in Mozilla’s database that serve a .well-known config with OAuth2 first is also affected.

Add pref for whether to accept plaintext protocols during autoconfiguration.

This pref did two things:

1. When fetching configs, discard all configs using plaintext protocols
2. When guessing a config, only guess ssl protocols

If you look among the upstreamed patches you can see that the “new” patch adds a pref doing 2, so lets focus on 1.

1 is not so bad given that the upstreamed patch that makes us able to only fetch configs over ssl; an active attacked cannot inject a non-ssl config, so the provider must offer it. Let’s say a provider offers only plaintext (I have never seen this in the wild) then when a user accepts its plaintext config the wizard will turn to a bit scary red warning explaining the problem, and requiring a confirmation that the user actually wants to proceed.

Personally I think this is good enough.

Prefer fetched configurations using SSL over plaintext.

In Tails we only fetch over SSL, so this means nothing to us. But it’s a bit unfortunate for all other Thunderbird users. :/

Next steps:

1. Agree to just disable Microsoft AutoDiscover.
2. Discuss whether we are fine with only half of the “accept plaintext protocols” patch (i.e. the scary red warning is good enough).
3. Probably continue as usual (manual patching + attempt to upstream) with the OAuth2 patch.
4. …
5. Celebrate?

#93 Updated by Anonymous 2019-06-07 13:22:23

During our last meeting we decided that we still want to upstream the OAuth patch and apply anonym’s proposal above, ie.

disable

mailnews.auto_config.fetchFromExchange.enabled → false

and disregard
Prefer fetched configurations using SSL over plaintext.
for Tails.

#97 Updated by anonym 2019-06-19 10:59:21

(12:54:24) intrigeri: anonym: btw, wrt. the Thunderbird patch that they asked to file separately and we were not sure what was going on, I took a look a couple weeks ago, and I think their reply is easy to understand in the context of the commit message of the patch they’re refering to, which was a bit unclear: it felt you were refering to other code that was not submitted yet (which is not the case AFAICT), so they asked to submit that other code.
(12:55:11) intrigeri: anonym: with this context in mind, I think their reply (based on a misunderstanding) makes sense. clarifying the commit message should fix the misunderstanding, hopefully :)

#98 Updated by anonym 2019-06-25 10:36:53

anonym wrote:
> (12:54:24) intrigeri: anonym: btw, wrt. the Thunderbird patch that they asked to file separately and we were not sure what was going on, I took a look a couple weeks ago, and I think their reply is easy to understand in the context of the commit message of the patch they’re refering to, which was a bit unclear: it felt you were refering to other code that was not submitted yet (which is not the case AFAICT), so they asked to submit that other code.
> (12:55:11) intrigeri: anonym: with this context in mind, I think their reply (based on a misunderstanding) makes sense. clarifying the commit message should fix the misunderstanding, hopefully :)

I think I get it. This is the problematic commit message:

Add pref for whether we accept OAuth2 during autoconfiguration.

For many providers JavaScript is required for OAuth2 to work; with it
disabled autoconfiguration then result in a terrible UX (e.g. the web
login fails, has to manually alter the authentication method). Let's
provide a pref that discards OAuth2 configurations so e.g. extensions
that disables JavaScript (like TorBirdy) can provide a workaround.

If I understand correctly the problematic part is “can provide a workaround”. Ben understood it as something involving extra code for this “workaround”, but what I really meant was simply: “can provide autoconfiguration that doesn’t fail get stuck and fail as soon as OAuth is presented as a possible authentication method”.

#99 Updated by Anonymous 2019-06-26 09:19:10

I opened a new bug for the OAuth issue here: https://bugzilla.mozilla.org/show_bug.cgi?id=1561542

#102 Updated by hefee 2019-12-13 15:24:47

@anonym: what is the current status of this issue? As the last notice was six months ago, it would be great to know the current state to decide how to go on.

#103 Updated by anonym 2019-12-16 12:43:08

The only remaining thing is this: https://bugzilla.mozilla.org/show_bug.cgi?id=1561542

The reviewer is hesitant (yay…) but raised a possibly interesting point. Let me go through his full comment:

Magnus wrote:
> Setting this pref opens up other problems though. You may have to set your (gmail etc) account to use “less secure applications”, and without that you won’t really be able to log in + no good indications of why.

Fair point, although of course still infinitely better than the current situation.

> Maybe when we have several options, we could have them listed, and let the user change them if required.

This is already the case: you can click “Manual config” and then select any of auth methods from the two (IMAP/POP + SMTP) dropdoxes, but IMHO this does not qualify as a solution at all. This is not the path we want to take for auto-configuration.

> And/or you could consider temporarily enabling the js during oauth.

Huh, yeah, maybe? This would give the ideal UX (e.g. GMail should work out of the box, without having to configure your account to accept “less secure applications”) while only exposing Thunderbird to JavaScript shipped from hosts the user already trusts their email to (FWIW). If we also lock down JavaScript a bit (disable JIT etc) I think I’d be pretty confident. OTOH I don’t think this should be default behavior — others that disable JavaScript might not appreciate this and would then want a always_allow_javascript_for_oauth pref.

Thoughts?

I also got an alternative idea for the “old” patch: instead of introducing yet another pref we could just check if JavaScript is disabled, and if so also disable OAuth2.

#104 Updated by intrigeri 2019-12-28 18:37:19

Hi,

> The only remaining thing is this: https://bugzilla.mozilla.org/show_bug.cgi?id=1561542

+ ensuring that our decisions from Feature #6156#note-93 are tracked somewhere: for example, it seems that back then we decided to drop a couple patches, but we still carry them and segfault had to refresh them for Thunderbird 68. Will you file tickets for these things? If not, I’m happy to do this with my FT lead hat on.

I’ve no idea about what is cheap or expensive to implement so some of my ideas below may be too wild. You tell me! :)

> The reviewer is hesitant (yay…) but raised a possibly interesting point. Let me go through his full comment:

tl;dr: I think having a pref that enables JS for OAuth2, even if it’s otherwise globally disabled, is the best lead we have. It requires further investigation but it would 1. avoid some UX nightmares (the current one, and the one the patch replaces it with); 2. avoid having to write more code to mitigate these nightmares.

> Magnus wrote:
>> Setting this pref opens up other problems though. You may have to set your (gmail etc) account to use “less secure applications”, and without that you won’t really be able to log in + no good indications of why.

> Fair point, although of course still infinitely better than the current situation.

That’s a strong statement. I understand you’re saying that the resulting UX, in this situation, is infinitely better than the one u reported about when filing that ticket. Is this what you meant?

I did not test myself but what Magnus writes suggests to me that, from a user’s perspective, it’s essentially just as bad: at the end of the day, you can’t read your email and you have no idea what to do to fix that. So I don’t really understand what’s “infinitely better”. Maybe the path to failure is less painful? Anyway, this may not matter much, see below.

>> Maybe when we have several options, we could have them listed, and let the user change them if required.

> This is already the case: you can click “Manual config” and then select any of auth methods from the two (IMAP/POP + SMTP) dropdoxes, but IMHO this does not qualify as a solution at all. This is not the path we want to take for auto-configuration.

I don’t understand what the “several options” are in this context.

It looks like there are two different use cases:

• Folks who have never used their GMail account in Tails before: they’ll have to log into their GMail account in a web browser, and do $stuff there, in order to get auto-config working. If we don’t tell them they need to do that at the right time, auto-configuration does not seem to be an accurate description of their UX :/
• Folks who have already configured their GMail account for “less secure applications”: in this case, I understand your patch does the right thing.

Problem is, Thunderbird can’t tell in which situation it is without trying, right?
Here as well, it may not matter much, see below.

>> And/or you could consider temporarily enabling the js during oauth.

> Huh, yeah, maybe? This would give the ideal UX (e.g. GMail should work out of the box, without having to configure your account to accept “less secure applications”) while only exposing Thunderbird to JavaScript shipped from hosts the user already trusts their email to (FWIW). If we also lock down JavaScript a bit (disable JIT etc) I think I’d be pretty confident.

IIRC, Torbirdy disabled JavaScript to protect against tracking and various kinds of infoleaks. These reasons certainly apply to HTML email and feeds coming from random places, but indeed, it could be that they don’t apply in a context (OAuth2) where the user is logging in anyway. IMO this requires a more detailed analysis, but I’m optimistic, and I encourage you to dive deeper in this direction :)

> OTOH I don’t think this should be default behavior — others that disable JavaScript might not appreciate this and would then want a always_allow_javascript_for_oauth pref.

Indeed, having JavaScript running when one has explicitly set the non-default pref("javascript.enabled", false); feels wrong.

> I also got an alternative idea for the “old” patch:

I don’t know what’s the “old” patch, so bear with me if I got the following wrong.

> instead of introducing yet another pref we could just check if JavaScript is disabled, and if so also disable OAuth2.

I understand it makes thing a bit more automatic outside of Tails (one still has to tweak their GMail account, but at least that’s one less pref they have to learn about and to set), which seems like a very good idea to me! Software discovering how to behave by itself seems better, in principle, than adding prefs :) But I still prefer the idea of making OAuth2 Just Work™ even when JS is disabled globally.

In the context of Tails, it would have no impact (neither good nor bad) on the UX, right?