Feature #8243

Support meek bridges

Added by intrigeri 2014-11-08 09:08:21 . Updated 2020-04-24 18:21:04 .

Status:
In Progress
Priority:
Elevated
Assignee:
johanbluecreek
Category:
Tor configuration
Target version:
Start date:
2014-11-08
Due date:
% Done:

100%

Feature Branch:
https://gitlab.com/johanbluecreek/tails.git:feature/8243-meek
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

(sajolida is watching this ticket)


Subtasks


Related issues

Related to Tails - Feature #7980: Support obfs4 bridges Resolved 2014-10-01
Related to Tails - Feature #7909: Support ScrambleSuit bridges Rejected 2014-09-17
Related to Tails - Feature #8287: Explain what is required to have new pluggable transports Rejected 2015-01-03
Related to Tails - Bug #8775: Tor Launcher should be able to resolve DNS host names for the proxy configuration Resolved 2015-01-22
Related to Tails - Bug #14555: Migrate to Tor Launcher compatible with Firefox ESR60 Resolved 2017-08-30
Related to Tails - Feature #15144: Consider migrating from Tor Launcher to anon-connection-wizard Confirmed 2018-01-03
Related to Tails - Feature #8825: Provide default bridges Confirmed 2015-01-30
Related to Tails - Feature #15197: Upgrade to Tor Browser 7.5 Resolved 2018-01-18
Related to Tails - Feature #15331: Support automatic bridge retrieval in Tor Launcher (Moat) In Progress 2017-12-12
Related to Tails - Feature #5461: Persistence preset: Tor configuration Confirmed
Has duplicate Tails - Bug #15836: Meek-lite Duplicate 2018-08-24
Has duplicate Tails - Feature #15972: Adding meek bridges to Tails Duplicate 2018-09-23

History

#1 Updated by intrigeri 2014-11-08 09:08:32

#2 Updated by intrigeri 2014-11-08 09:08:43

#3 Updated by intrigeri 2015-02-10 11:50:36

  • Target version set to Tails_1.4

Given the feedback we got on tor-talk (“Which PTs shall we prioritize for inclusion in Tails?” thread), this should be our top-priority in this area, along with Feature #7980 (obfs4).

Now, we haven’t reached a practically usable conclusion in the “keeping up with pluggable transports by using TBB’s Tor and tor-launcher” thread, so it’s not clear how meek support should be implemented.

#4 Updated by BitingBird 2015-05-09 02:45:04

  • Target version changed from Tails_1.4 to Tails_1.5

#5 Updated by BitingBird 2015-07-06 13:26:35

  • Target version deleted (Tails_1.5)

#6 Updated by intrigeri 2015-07-07 00:27:11

> Target version deleted (Tails_1.5)

(As per monthly meeting.)

#7 Updated by xiaolan 2016-03-29 23:16:53

intrigeri wrote:
> Given the feedback we got on tor-talk (“Which PTs shall we prioritize for inclusion in Tails?” thread), this should be our top-priority in this area, along with Feature #7980 (obfs4).
>
> Now, we haven’t reached a practically usable conclusion in the “keeping up with pluggable transports by using TBB’s Tor and tor-launcher” thread, so it’s not clear how meek support should be implemented.

meek in the Tor Browser works in China, but seems Tails doesn’t support it ..
if Tails supports meek it will be really great and I can test it in China..

https://trac.torproject.org/projects/tor/wiki/doc/meek

Thank you :)

#8 Updated by emmapeel 2017-04-21 07:03:07

Another user has asked today in #tails for meek in Tails because is the only option in China currently.

#9 Updated by Anonymous 2017-06-29 14:42:10

Two years ago intrigeri wrote that this should be our next top priority in this area.

What are the next tasks?

Upstream Tor bug: https://trac.torproject.org/projects/tor/ticket/13160 where Ximin says that he wants https://trac.torproject.org/projects/tor/ticket/12716 fixed before uploading the meek packages to Debian. However, there seems to be some confusion, see https://trac.torproject.org/projects/tor/ticket/12716#comment:32

I can ask Ximin about the current state of things to see how realistic this is.

#10 Updated by Anonymous 2017-06-29 14:42:49

  • Type of work changed from Code to Wait

For now, we simply have to wait.

#11 Updated by Anonymous 2017-06-30 14:25:21

  • related to Feature #8287: Explain what is required to have new pluggable transports added

#12 Updated by intrigeri 2017-09-14 12:54:41

intrigeri wrote:
> Now, we haven’t reached a practically usable conclusion in the “keeping up with pluggable transports by using TBB’s Tor and tor-launcher” thread, so it’s not clear how meek support should be implemented.

After having a look at the status of the packaging, taking a step back to look at the big picture, and learning about the fact that obfs4proxy can act as a simplistic meek client, I’m starting to lean towards not blocking on meek-client being in Debian, or at least not assuming we just have to wait and it’ll happen (that sounded realistic a couple years ago, but nobody is actively working on it anymore). So our options are: either we do that meek packaging work ourselves (seems hard but perhaps focussing on meek-client and forgetting the reflector etc. makes it easier), or we use obfs4proxy’s meek support, or we use the meek client shipped with Tor Browser.

I’ve tried obfs4proxy’s meek client support on Tails 3.1. In the Greeter I told Tails I need bridges, in Tor Launcher I’ve pasted the Tor Browser’s default meek bridges (replacing “meek” with “meek_lite” otherwise it won’t work), it fails as expected so I’ve set ClientTransportPlugin obfs3,obfs4,meek_lite exec /usr/bin/obfs4proxy managed in /etc/tor/torrc, restarted tor, clicked “Reconfigure” in Tor Launcher, and I see tor is starting obfs4proxy and delegating to it the task of talking meek protocol. But there’s a catch 22: connecting to the meek bridge requires working DNS… and we torify DNS resolution. This has been discussed already on Bug #8775. Manually adding currently working IP addresses for the 2 default meek bridges fronting domains to /etc/hosts fixes that:

93.184.221.200 ajax.aspnetcdn.com
52.222.172.206 a0.awsstatic.com

… and I’m connected to Tor via meek, without installing any special software :) Now, two problems remain:

  1. I have no idea if that’s enough to work in China because obfs4proxy’s minimal meek client does not normalize TLS signatures.
  2. I don’t know if hardcoding IP addresses in the ISO at Tails release time has any chance to work in the real world. If we’re willing to give it a try, we should first check what pieces of Tails would break when the IP changes (software that does proper socks5h won’t be affected, but things that go through libc/NSS will) and whether that’s acceptable. Or it could be time to look into using a different DNS resolver for tor/obfs4proxy. See Bug #8775 that’s specifically about this problem.

#13 Updated by intrigeri 2017-09-14 12:54:52

  • related to Bug #8775: Tor Launcher should be able to resolve DNS host names for the proxy configuration added

#14 Updated by intrigeri 2017-09-14 12:57:41

  • Type of work changed from Wait to Research

#15 Updated by iry 2017-09-16 06:51:13

Hi intigeri!

Thank you so much for testing these out! This will be a significant improvement on the censorship circumvention in Tails!

intrigeri wrote:
> So our options are: either we do that meek packaging work ourselves (seems hard but perhaps focussing on meek-client and forgetting the reflector etc. makes it easier), or we use obfs4proxy’s meek support, or we use the meek client shipped with Tor Browser.
>

Just a reminder: it seems the first two options do not have meek-reflector support on which new Tor launcher relies?

isis said in Tor ticket (https://trac.torproject.org/projects/tor/ticket/22871#comment:7):
> I also just remembered that you actually can’t do a POST /meek/* to BridgeDB unless you go through the meek reflector, because of the way the TLS termination is handled. Also FYI, this distributor relies on getting the client’s IP address in an X-Forwarded-For header from the meek reflector. We could consider setting up the same moat API as its own separate distributor for clients which can’t use meek, but that should be a new ticket. (Also, I’d prefer that they be separate distributors, since there’s a possibility that we may need to allocate differently, or treat different automated bridge distribution clients differently, e.g. different rate limiting, in the future.)

> I’ve tried obfs4proxy’s meek client support on Tails 3.1. In the Greeter I told Tails I need bridges, in Tor Launcher I’ve pasted the Tor Browser’s default meek bridges (replacing “meek” with “meek_lite” otherwise it won’t work), it fails as expected so I’ve set ClientTransportPlugin obfs3,obfs4,meek_lite exec /usr/bin/obfs4proxy managed in /etc/tor/torrc, restarted tor, clicked “Reconfigure” in Tor Launcher, and I see tor is starting obfs4proxy and delegating to it the task of talking meek protocol. But there’s a catch 22: connecting to the meek bridge requires working DNS… and we torify DNS resolution. This has been discussed already on Bug #8775. Manually adding currently working IP addresses for the 2 default meek bridges fronting domains to /etc/hosts fixes that:
>
> […]
>

That is awesome! I tested in Debian8 and it worked well, too: https://forums.whonix.org/t/censorship-circumvention-tor-pluggable-transports/2601/3

> … and I’m connected to Tor via meek, without installing any special software :) Now, two problems remain:
>
> # I have no idea if that’s enough to work in China because obfs4proxy’s minimal meek client does not normalize TLS signatures.

Good news: I tested using meek bridges via obfs4proxy under different ISPs in China, and this approach worked pretty well in those environment. However, we need more feedback from users in China to know how well this approach work.

This is also described in the link above, btw.

> # I don’t know if hardcoding IP addresses in the ISO at Tails release time has any chance to work in the real world. If we’re willing to give it a try, we should first check what pieces of Tails would break when the IP changes (software that does proper socks5h won’t be affected, but things that go through libc/NSS will) and whether that’s acceptable. Or it could be time to look into using a different DNS resolver for tor/obfs4proxy. See Bug #8775 that’s specifically about this problem.

That is an interesting problem, too. I am looking forward to seeing how Tails is going to solve it.

#16 Updated by yawning 2017-09-21 04:50:15

intrigeri wrote:
> So our options are: either we do that meek packaging work ourselves (seems hard but perhaps focussing on meek-client and forgetting the reflector etc. makes it easier), or we use obfs4proxy’s meek support, or we use the meek client shipped with Tor Browser.

Unless dcf changed something when I wasn’t paying attention, both helper-less meek-client and obfs4proxy’s meek_lite act near identically, because I cribbed off the former to implement the latter.

> # I have no idea if that’s enough to work in China because obfs4proxy’s minimal meek client does not normalize TLS signatures.

Currently this doesn’t appear to matter much, or at least, if it does, then Nathan hasn’t told me (Orbot uses meek_lite).

> # I don’t know if hardcoding IP addresses in the ISO at Tails release time has any chance to work in the real world. If we’re willing to give it a try, we should first check what pieces of Tails would break when the IP changes (software that does proper socks5h won’t be affected, but things that go through libc/NSS will) and whether that’s acceptable. Or it could be time to look into using a different DNS resolver for tor/obfs4proxy. See Bug #8775 that’s specifically about this problem.

It’s not immediately obvious to me if Azure does any sort of DNS based load balancing. AWS appears to be doing so.

#17 Updated by intrigeri 2017-11-19 14:08:34

iry wrote:
> > # I don’t know if hardcoding IP addresses in the ISO at Tails release time has any chance to work in the real world. If we’re willing to give it a try, we should first check what pieces of Tails would break when the IP changes (software that does proper socks5h won’t be affected, but things that go through libc/NSS will) and whether that’s acceptable. Or it could be time to look into using a different DNS resolver for tor/obfs4proxy. See Bug #8775 that’s specifically about this problem.
>
> That is an interesting problem, too. I am looking forward to seeing how Tails is going to solve it.

I’ve updated Bug #8775 that now is ready to be implemented by anyone interested. iry, do you know how Whonix solved it?

#18 Updated by iry 2017-11-20 00:36:08

intrigeri wrote:
> I’ve updated Bug #8775 that now is ready to be implemented by anyone interested. iry, do you know how Whonix solved it?

That is an exciting news to hear! Thank you so much for your work, intrigeri!

Whonix solved the problem by “allow[ing] Tor do resolve DNS using clearnet with your usual DNS settings that any clearnet VM would be using”. In Whonix-Gateway, Tor is running under debian-tor user which is the only user that is granted clearnet DNS access. I am not sure if this also satisfies the adversary mode of Tails.

The code of implementation by Patrick and other development discussion can be found here: https://forums.whonix.org/t/censorship-circumvention-tor-pluggable-transports/2601/14

Please let me know if there is anything I can help with!

#19 Updated by intrigeri 2017-12-12 08:33:18

  • related to Bug #14555: Migrate to Tor Launcher compatible with Firefox ESR60 added

#20 Updated by anonym 2017-12-29 19:42:07

iry suggested a “not so elegant” (his words!) solution: we ship a hosts file in Tails that contain the resolutions of the needed domain names. :)

#21 Updated by intrigeri 2017-12-30 07:21:22

> iry suggested a “not so elegant” (his words!) solution: we ship a hosts file in Tails that contain the resolutions of the needed domain names. :)

See the earlier discussion about this idea on this very ticket: it might work for Azure but probably not for AWS.

#22 Updated by iry 2017-12-30 09:26:59

intrigeri wrote:
> > iry suggested a “not so elegant” (his words!) solution: we ship a hosts file in Tails that contain the resolutions of the needed domain names. :)
>
> See the earlier discussion about this idea on this very ticket: it might work for Azure but probably not for AWS.

AWS is using DNS based load balancing, which means the IP address of the domain name we get may change all the time; however, I doubt this does not necessarily mean the older IP addresses we got does not work at all. (Maybe the performance is not the best, but it may still work.)

Hardcoding /etc/hosts is only a temporary approach which helps to mitigate the severe problem that Tails hardly works in China.

Anonym said he would keep trying this approach: https://labs.riseup.net/code/issues/8775#note-9 which is the elegant implementation.

#23 Updated by iry 2018-01-03 16:09:39

  • related to Feature #15144: Consider migrating from Tor Launcher to anon-connection-wizard added

#24 Updated by anonym 2018-01-18 20:50:54

  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri
  • Target version set to Tails_3.6
  • % Done changed from 0 to 40
  • QA Check set to Ready for QA
  • Feature Branch set to feature/8243-meek

The feature branch implements this ticket and its blocker, Bug #8775. Since you’ve already had an initial look I don’t think I have to say much about it, and I think you’ll be pretty happy with its current state, except:

  • there’s no design doc changes or automated test suite updates yet. I’ll wait until Feature #15197 is merged in order to save me some work.
  • commit:f5cbdd2d9008b79901ce5114d613a499b609697b enables Tor’s default Meek bridges in our Tor Launcher. My understanding is that it is the stealthiest pluggable transport (in practice, among the ones we support at least) despite requiring no user input, so it feels sane to ship them since users otherwise will be manually inputting the same text any way (and probably screwing up the “meek_lite” part). In fact, AFAIK there is no user-friendly way to learn about meek bridges. OTOH, my worry is that Tails might overload the meek bridges, since it is the only type of bridge we support. Thoughts?

So, I’m not asking for a merge, just for input on the above question, and a review and manual testing!

#25 Updated by intrigeri 2018-01-19 11:07:55

anonym wrote:
> * commit:f5cbdd2d9008b79901ce5114d613a499b609697b enables Tor’s default Meek bridges in our Tor Launcher. My understanding is that it is the stealthiest pluggable transport (in practice, among the ones we support at least) despite requiring no user input, so it feels sane to ship them since users otherwise will be manually inputting the same text any way (and probably screwing up the “meek_lite” part). In fact, AFAIK there is no user-friendly way to learn about meek bridges. OTOH, my worry is that Tails might overload the meek bridges, since it is the only type of bridge we support. Thoughts?

I have no idea how Tor Launcher looks like when only 1 type of default bridges is enabled => in order to understand if and how this is related to our reasoning and decision on Feature #8825, I’ll need to see this thing in action => I’ll build an ISO and will try it out.

> a review

Looks great to me modulo:

  • Let’s please not introduce new non-extended regexps. Sorry for being like a broken record.
  • The “Remove TBB’s default bridges. If we want them, we want it in Tor Launcher’s own profile” comment is confusing: it seems to conflict with the fact we’re enabling the default meek bridges in /etc/xul-ext/tor-launcher.js. Or I misunderstood, which proves it’s confusing.
  • If you don’t know about grep’s --line-regexp, I find it useful to move complexity out of regexps, which can be interesting given the definition of ^ and $ varies accross the regexp languages (beginning/end of string vs. beginning/end of line).
  • I suspect one of your commits fixes Bug #7903 (untested) => if I’m guessing right, please mark that ticket as RfQA to and consider rewriting history to encode this in the corresponding commit.

> and manual testing!

Not there yet.

#26 Updated by intrigeri 2018-01-19 11:09:11

#27 Updated by intrigeri 2018-01-19 11:11:37

And also: since this is being retargeted to 3.6, please merge devel into your branch so we can do QA with systemd 236 (that Tails 3.6 will get if this branch is merged) instead of systemd 234 (that current builds of this branch get). Better do that before you bother running the full test suite.

#28 Updated by intrigeri 2018-01-19 12:41:40

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Ready for QA to Dev Needed

intrigeri wrote:
> anonym wrote:
> > * commit:f5cbdd2d9008b79901ce5114d613a499b609697b enables Tor’s default Meek bridges in our Tor Launcher. My understanding is that it is the stealthiest pluggable transport (in practice, among the ones we support at least) despite requiring no user input, so it feels sane to ship them since users otherwise will be manually inputting the same text any way (and probably screwing up the “meek_lite” part). In fact, AFAIK there is no user-friendly way to learn about meek bridges. OTOH, my worry is that Tails might overload the meek bridges, since it is the only type of bridge we support. Thoughts?
>
> I have no idea how Tor Launcher looks like when only 1 type of default bridges is enabled => in order to understand if and how this is related to our reasoning and decision on Feature #8825, I’ll need to see this thing in action => I’ll build an ISO and will try it out.

I’ve tried it and ouch, that’s a tough one. I think I agree with your description of the problem. I’m not sure about what conclusion we can/should draw and how. To make sure we’re on the same page let me phrase how I see things using my own words:

  • On the one hand, the reasoning that lead us (Feature #8825) to not propose default Tor bridges applies here even stronger: back then we were considering proposing all default bridges from TBB, which would have spread the load somewhat. So I don’t think we (you and I) can dismiss that alone and implement something that’s totally against something we’ve decided at the project scale.
  • On the other hand, your UX argument makes sense: it would feel stupid to say “we now support meek but to use them, you need to locate the corresponding bridge addresses in the Tor Browser Git repo or some hidden wiki page; and then every time you start Tails, you’ll need to type them correctly and of course, don’t forget to replace ”meek" with “meek_lite”; what, wasn’t that obvious?".

So I propose we do this:

  1. Once the other known issues are resolved, merge this without the default meek bridges. Don’t announce this as a new feature except in changelog. Rationale: the code is basically ready, let’s not let it bitrot. It might even be useful to have this code around, merged and tested for moat support, even without the default set of bridges; we’ll see, I have no idea how moat support will look like.
  2. Wait for moat support in Tor Launcher to be ready for us to test it. Rationale: it might make this discussion moot (I wonder if the default bridges will still be a thing in TB once moat is there; and even if they’re not a thing anymore, for Tails users the benefits of having default bridges available becomes almost null once there’s a nice UI that lets them fetch their own bridges automatically).
  3. If moat integration in Tor Launcher does not magically make this problem disappear, go back to the drawing board (i.e. reopen the project-wide discussion on Feature #8825 and/or seek advice from UX people).

I think this plan only makes sense if we are strongly committed to make step 2 real. It might be that we have to (and then it’s FT core work), it might be that we don’t have to (and then we need to do it as volunteers). Either way, count me in but not alone.

#29 Updated by Anonymous 2018-01-19 16:34:16

#30 Updated by anonym 2018-01-26 11:44:29

  • Type of work changed from Research to Code

intrigeri wrote:
> anonym wrote:
> > […] default Meek bridges […]

Let’s continue this discussion on Feature #8825.

> > a review
>
> Looks great to me modulo:
>
> * Let’s please not introduce new non-extended regexps. Sorry for being like a broken record.

Fixed in feature/8825-default-meek-bridges.

> * The “Remove TBB’s default bridges. If we want them, we want it in Tor Launcher’s own profile” comment is confusing: it seems to conflict with the fact we’re enabling the default meek bridges in /etc/xul-ext/tor-launcher.js. Or I misunderstood, which proves it’s confusing.

commit:53131b0a30cb09e8eaf1368cc9f7f02509312eb1 should make it clearer.

> * If you don’t know about grep’s --line-regexp, I find it useful to move complexity out of regexps, which can be interesting given the definition of ^ and $ varies accross the regexp languages (beginning/end of string vs. beginning/end of line).

I know about it, but personally prefer ^$ over some tool-specific option.

> * I suspect one of your commits fixes Bug #7903 (untested) => if I’m guessing right, please mark that ticket as RfQA to and consider rewriting history to encode this in the corresponding commit.

Actually it doesn’t: notice how /etc/resolv-over-clearnet.conf is copied into the chroot. A bind-mount should solve it, but might get tricky to get right given the mount namespace. I’ll have a quick look.

#31 Updated by anonym 2018-01-26 12:31:32

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

anonym wrote:
> > * I suspect one of your commits fixes Bug #7903 (untested) => if I’m guessing right, please mark that ticket as RfQA to and consider rewriting history to encode this in the corresponding commit.
>
> Actually it doesn’t: notice how /etc/resolv-over-clearnet.conf is copied into the chroot. A bind-mount should solve it, but might get tricky to get right given the mount namespace. I’ll have a quick look.

Never mind the part about “mount namespaces”, that’s for tor only. So the fix is a trivial bind-mount, with the “hard” thing being how to deal with that mount vs try_cleanup_browser_chroot(). findmnt to the rescue! Yup, the problem Bug #7903 wants to document a workaround for is now solved!

I have rewritten the history of feature/8243-meek, but I’ve only rebased on devel, reordered commits and squashed commits via fixup. You have reviewed up to and including commit:a993d3eb30f88bf0a7a085db7e63653979846502.

#32 Updated by intrigeri 2018-01-30 08:06:25

  • Assignee changed from intrigeri to bertagaz

#33 Updated by intrigeri 2018-02-20 08:23:25

  • related to Feature #15331: Support automatic bridge retrieval in Tor Launcher (Moat) added

#34 Updated by intrigeri 2018-02-20 08:33:15

anonym, even though we’re not going to announce this feature anywhere but in the changelog, as a preparatory step for Feature #15331, may I suggest you call for testing of this half-baked feature (with suitable disclaimers) on -testers@?

#35 Updated by bertagaz 2018-02-21 12:48:50

  • Assignee changed from bertagaz to anonym
  • QA Check changed from Ready for QA to Info Needed

This branch automated test suite runs always end up with a failure of the “Starting the Unsafe Browser without a network connection results in a complaint about no DNS server being configured” scenario, in Jenkins and at home. It does not seem related to Bug #8775 though. Maybe Bug #7903? Did I miss something?

#36 Updated by bertagaz 2018-02-22 20:19:58

I’ve been trying to use meek_lite PT as explained in Feature #8243#note-12, but no success so far. I had to do some iptables ACCEPT’ing + apparomoring but it still fails to contact the authorities. I’ll go on testing a bit more. Meek newbie here.

#37 Updated by bertagaz 2018-02-23 15:10:24

I’ve done further testing and stumbled on something that’s been raised here before:

intrigeri wrote:
> And also: since this is being retargeted to 3.6, please merge devel into your branch so we can do QA with systemd 236 (that Tails 3.6 will get if this branch is merged) instead of systemd 234 (that current builds of this branch get). Better do that before you bother running the full test suite.

Actually it seems we ship systemd 232 at the moment, and that the BindReadOnlyPaths directive was introduced in systemd 233. So we need to upgrade it for this branch to work.

#38 Updated by bertagaz 2018-02-23 17:17:15

bertagaz wrote:
> I’ve done further testing and stumbled on something that’s been raised here before:
>
> intrigeri wrote:
> > And also: since this is being retargeted to 3.6, please merge devel into your branch so we can do QA with systemd 236 (that Tails 3.6 will get if this branch is merged) instead of systemd 234 (that current builds of this branch get). Better do that before you bother running the full test suite.
>
> Actually it seems we ship systemd 232 at the moment, and that the BindReadOnlyPaths directive was introduced in systemd 233. So we need to upgrade it for this branch to work.

Pushed a commit installing systemd from stretch-backports. Builds fine and manual test confirms it fixes the issue. I can use meek bridges! :)

Let see what Jenkins says about shipping this systemd version. Meanwhile I’ll play a bit more with this system too.

#39 Updated by anonym 2018-02-24 00:03:31

  • Assignee changed from anonym to bertagaz
  • QA Check changed from Info Needed to Ready for QA

bertagaz wrote:
> Pushed a commit installing systemd from stretch-backports.

Crap! The plan was for me to fix that before assigning it for review to you. I’m very sorry for the time wasted on debugging this! :/

Reassigning it back to you then! :)

#40 Updated by bertagaz 2018-02-24 10:33:07

  • Assignee changed from bertagaz to anonym
  • QA Check changed from Ready for QA to Info Needed

bertagaz wrote:
> This branch automated test suite runs always end up with a failure of the “Starting the Unsafe Browser without a network connection results in a complaint about no DNS server being configured” scenario, in Jenkins and at home. It does not seem related to Bug #8775 though. Maybe Bug #7903? Did I miss something?

You did not reply about this though. What’s the plan? I guess it’s expected that this test now fails (should have think twice before commenting) and we should just remove it?

#41 Updated by anonym 2018-02-24 12:50:47

  • Assignee changed from anonym to bertagaz
  • % Done changed from 40 to 50
  • QA Check changed from Info Needed to Ready for QA

bertagaz wrote:
> bertagaz wrote:
> > This branch automated test suite runs always end up with a failure of the “Starting the Unsafe Browser without a network connection results in a complaint about no DNS server being configured” scenario, in Jenkins and at home. It does not seem related to Bug #8775 though. Maybe Bug #7903? Did I miss something?
>
> You did not reply about this though. What’s the plan? I guess it’s expected that this test now fails (should have think twice before commenting) and we should just remove it?

Gah! So it is expected because the error we look for has moved to after the Unsafe Browser verification. I updated the test for this, but cannot find the change (I even tried a bit using git reflog). I suspect I have done some work on this branch that has been lost… any way, now I think the branch should be actually ready for you to look at and possibly merge. Sorry for the messy review! :(

#42 Updated by bertagaz 2018-02-24 16:40:02

  • % Done changed from 50 to 90
  • QA Check changed from Ready for QA to Pass

anonym wrote:
> Gah! So it is expected because the error we look for has moved to after the Unsafe Browser verification. I updated the test for this, but cannot find the change (I even tried a bit using git reflog). I suspect I have done some work on this branch that has been lost… any way, now I think the branch should be actually ready for you to look at and possibly merge. Sorry for the messy review! :(

That’s fine. I’ve done more tests, and finished the code reviewing. I’m ok with it now, I think we’re good to release that in 3.6 so I’ll merge it, yay!

#43 Updated by bertagaz 2018-02-24 19:00:20

  • Status changed from In Progress to Fix committed
  • % Done changed from 90 to 100

Applied in changeset commit:4bd6d7b4e61e34dcc384b15f2322b7130fce6fc9.

#44 Updated by bertagaz 2018-02-24 19:02:31

  • Assignee deleted (bertagaz)

And so is merged!

#45 Updated by intrigeri 2018-02-26 11:23:06

  • Status changed from Fix committed to In Progress
  • Assignee set to anonym
  • % Done changed from 100 to 90
  • QA Check changed from Pass to Dev Needed

Ouch, while reviewing ..origin/devel I just noticed that commit:bf317f15289f3c3beb966a9967d58f96debd0cd4 is incomplete: it only pins a subset of the binary packages built from src:systemd. E.g. in the .packages file I see libudev1:amd64 232-25+deb9u2. I really don’t think that mixing 2 different versions of systemd is a good idea.

Also, I believe the corresponding comment might be incorrect: we need systemd >= v233, not systemd > v233. Finally, commenting out that text with # is not supported. I’m surprised it seemingly works and I’d rather see us 1. be consistent with the rest of this file; 2. only use documented syntax instead of relying on luck.

#46 Updated by intrigeri 2018-02-26 11:23:27

  • Priority changed from Normal to Elevated

#47 Updated by anonym 2018-02-26 11:36:36

  • QA Check changed from Dev Needed to Ready for QA
  • Feature Branch changed from feature/8243-meek to feature/8243-meek feature/8243-meek+force-all-tests

intrigeri wrote:
> Ouch, while reviewing ..origin/devel I just noticed that commit:bf317f15289f3c3beb966a9967d58f96debd0cd4 is incomplete: it only pins a subset of the binary packages built from src:systemd.

Good catch!

E.g. in the .packages file I see libudev1:amd64 232-25+deb9u2. I really don’t think that mixing 2 different versions of systemd is a good idea.

Agreed! Fixed! And I also pushed a +force-all-tests branch so we get to see the results of a full run given the udev bump.

> Also, I believe the corresponding comment might be incorrect: we need systemd >= v233, not systemd > v233. Finally, commenting out that text with # is not supported. I’m surprised it seemingly works and I’d rather see us 1. be consistent with the rest of this file; 2. only use documented syntax instead of relying on luck.

Fixed!

#48 Updated by intrigeri 2018-02-26 12:02:23

LGTM at commit:2b5f771bd1ee64d52b6d27187b78e6e9ffbf015e :)

#49 Updated by anonym 2018-02-26 19:33:01

  • QA Check changed from Ready for QA to Dev Needed

We have one run of each branch:

Notably, the only scenario that failed in the normal (only @fragile) run was Boot with a hardware clock set way in the past and make sure that Tails sets the clock to the source date, which also failed in the force-all-tests run. It seems like the udev bump might have caused issues with at least the RTC used by our test suite VM. I will retry locally to get some more clarity.

#50 Updated by bertagaz 2018-02-27 13:59:15

anonym wrote:
> We have one run of each branch:
>
> * https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-8243-meek/1/
> * https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-8243-meek-force-all-tests/1/
>
> Notably, the only scenario that failed in the normal (only @fragile) run was Boot with a hardware clock set way in the past and make sure that Tails sets the clock to the source date, which also failed in the force-all-tests run. It seems like the udev bump might have caused issues with at least the RTC used by our test suite VM. I will retry locally to get some more clarity.

Indeed this test seems to always fail.

It seems systemd has its own way to set the clock for PID 1 when RTC is too old (see https://github.com/systemd/systemd/commit/021dd87bc055a5bfb2dcef83fc868fe24648b959).

It uses systemd’ build time to set the clock. Having a look at what time is set in the failed scenario (Sun Jan 28 15:58:39 UTC 2018.), it seems to correspond to systemd’s release/build date, so I think that’s the explanation. Not sure what to do about that though.

#51 Updated by anonym 2018-02-27 14:13:37

bertagaz wrote:
> anonym wrote:
> > We have one run of each branch:
> >
> > * https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-8243-meek/1/
> > * https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-8243-meek-force-all-tests/1/
> >
> > Notably, the only scenario that failed in the normal (only @fragile) run was Boot with a hardware clock set way in the past and make sure that Tails sets the clock to the source date, which also failed in the force-all-tests run. It seems like the udev bump might have caused issues with at least the RTC used by our test suite VM. I will retry locally to get some more clarity.
>
> Indeed this test seems to always fail.
>
> It seems systemd has its own way to set the clock for PID 1 when RTC is too old (see https://github.com/systemd/systemd/commit/021dd87bc055a5bfb2dcef83fc868fe24648b959).
>
> It uses systemd’ build time to set the clock. Having a look at what time is set in the failed scenario (Sun Jan 28 15:58:39 UTC 2018.), it seems to correspond to systemd’s release/build date, so I think that’s the explanation. Not sure what to do about that though.

Cheers! What a perfect timing, I was just investigating this and had came to the conclusion that the systemd bump probably introduced something like this. I’ll come up with something!

#52 Updated by intrigeri 2018-02-27 14:20:39

bertagaz wrote:
> It uses systemd’ build time to set the clock. Having a look at what time is set in the failed scenario (Sun Jan 28 15:58:39 UTC 2018.), it seems to correspond to systemd’s release/build date, so I think that’s the explanation. Not sure what to do about that though.

FTR I see this exact same failure, with that exact same time, on a test suite run for an ISO built from a branch built on top of current devel. So it seems this regression was already introduced there and is not caused by the follow-up fixes anonym is working on. That commit first appeared in systemd v229 so we’re only been “lucky” not to be hit by that test suite bug earlier.

#53 Updated by anonym 2018-02-27 14:57:39

  • Assignee changed from anonym to bertagaz
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:
> bertagaz wrote:
> > It uses systemd’ build time to set the clock. Having a look at what time is set in the failed scenario (Sun Jan 28 15:58:39 UTC 2018.), it seems to correspond to systemd’s release/build date, so I think that’s the explanation. Not sure what to do about that though.
>
> FTR I see this exact same failure, with that exact same time, on a test suite run for an ISO built from a branch built on top of current devel. So it seems this regression was already introduced there and is not caused by the follow-up fixes anonym is working on. That commit first appeared in systemd v229 so we’re only been “lucky” not to be hit by that test suite bug earlier.

Indeed! I sneaked the fix into this branch any way. :)

#54 Updated by bertagaz 2018-02-28 13:37:18

  • Status changed from In Progress to Fix committed
  • Assignee deleted (bertagaz)
  • % Done changed from 90 to 100
  • QA Check changed from Ready for QA to Pass

anonym wrote:
> intrigeri wrote:
> > FTR I see this exact same failure, with that exact same time, on a test suite run for an ISO built from a branch built on top of current devel. So it seems this regression was already introduced there and is not caused by the follow-up fixes anonym is working on. That commit first appeared in systemd v229 so we’re only been “lucky” not to be hit by that test suite bug earlier.
>
> Indeed! I sneaked the fix into this branch any way. :)

Works fine so it’s merged into devel. Let’s time this time we’re good. :)

#55 Updated by bertagaz 2018-03-14 11:06:27

  • Status changed from Fix committed to Resolved

#56 Updated by mercedes508 2018-08-24 15:46:08

#57 Updated by mercedes508 2018-10-15 17:53:20

#58 Updated by emmapeel 2019-06-11 08:29:08

  • Status changed from Resolved to Confirmed
  • Target version deleted (Tails_3.6)

after some discussion on XMPP, we don’t see this issue as solved. reopening thus.

#59 Updated by intrigeri 2019-08-08 11:15:06

  • related to Feature #5461: Persistence preset: Tor configuration added

#60 Updated by anonym 2019-08-11 20:43:59

  • Status changed from Confirmed to In Progress

Applied in changeset commit:tails|e4115249b2bdf26690d39f6cb30252f079d948db.

#61 Updated by intrigeri 2019-08-11 20:51:01

  • Status changed from In Progress to Confirmed

#62 Updated by intrigeri 2019-09-03 09:54:41

It’s worth noting that we should try to have our meek client have a close enough fingerprint to Tor Browser’s: otherwise, censors can more easily block our meek connections; and Tails users that choose meek bridges can be singled out and tracked (especially true as it’s currently so hard to use meek in Tails, I doubt you need more than one hand to count these users).

So far, we barely tried: we’ve been using a fully different client (obfs4proxy’s meek_lite, while Tor Browser 8.5 used the full-blown meek client, which uses Firefox for its network communication and especially TLS). But the situation is changing — starting with Tor Browser 9, they also use obfs4proxy’s meek_lite: https://trac.torproject.org/projects/tor/ticket/29430.

This is good because it’s closer to what we do. But if I got it right, they could do it because the new version of obfs4proxy they’re using uses its own, forked uTLS, that can emulate closely enough Firefox’ fingerprint: https://trac.torproject.org/projects/tor/ticket/29077, https://trac.torproject.org/projects/tor/ticket/29627. There seems to be little chances Debian can match this any time soon, so if we want our meek client to have the same fingerprint as Tor Browser’s, likely we’ll need to use the binary of obfs4proxy bundled with Tor Browser instead of the one from Debian.

#63 Updated by johanbluecreek 2020-02-06 17:29:56

  • Feature Branch changed from feature/8243-meek feature/8243-meek+force-all-tests to https://gitlab.com/johanbluecreek/tails.git:feature/8243-meek

This branch is based on wip/feature/15331-moat, which was trying to use Moat. Moat, that is selecting “Request a bridge from torproject.org” in configuring bridges part of the Tor Launcher, does not work for Tails, as that requires internet connection to receive a captcha and the get the bridges.

Instead, this branch gives support for Meek. As suggested by @intrigeri , it makes use of the meek_lite that is a part of the Tor Browser’s obfs4proxy. The debian package obfs4proxy is removed, and the radio-button suggesting Moat could work is removed.

The branch has been built and tested. The bridges works fine, which is controlled by comparing the fingerprint reported for the circuits listed by clicking on the Onion-applet, once tor has come up, and the fingerprint of the line added to /etc/tor/torrc for Meek. I also “simulated” a Chinese user where the clock would have been misconfigured, simply by `date -s +8hours` before the Tor Launcher was started to configure the bridges. This test also worked and time was synced and bridged circuits were established.

#64 Updated by intrigeri 2020-02-07 08:50:23

Hi @johanbluecreek,

Ooh yeah!

What’s the next step here?

If it’s “someone should push this to our CI and review the branch”, please set “Status” to “Needs Validation” :)

#65 Updated by johanbluecreek 2020-02-07 09:09:52

  • Status changed from Confirmed to Needs Validation

intrigeri wrote:
> What’s the next step here?
>
> If it’s “someone should push this to our CI and review the branch”, please set “Status” to “Needs Validation” :)

Sorry, yeah; push to CI and review. I’m not sure what to set the “% Done” to, but switched “Status”.

#66 Updated by anonym 2020-02-07 09:33:30

  • Status changed from Needs Validation to In Progress
  • Assignee set to johanbluecreek

johanbluecreek wrote:
> […] and the radio-button suggesting Moat could work is removed.

This will break our automated test suite’s bridge scenarios. They all use this step:

When /^I configure some (\w+) pluggable transports in Tor Launcher$/ do |bridge_type|
  @screen.wait_and_click('TorLauncherConfigureButton.png', 10)
  @screen.wait_and_click('TorLauncherBridgeCheckbox.png', 10)
  @screen.wait_and_click('TorLauncherBridgeList.png', 10)
[...]


You can find the corresponding images in features/images. This step assumes that clicking the “Tor is censored” checkbox will make the “custom bridges” text area visible, but with your branch it will require an extra click, something like:

When /^I configure some (\w+) pluggable transports in Tor Launcher$/ do |bridge_type|
  @screen.wait_and_click('TorLauncherConfigureButton.png', 10)
  @screen.wait_and_click('TorLauncherBridgeCheckbox.png', 10)
  @screen.wait_and_click('TorLauncherBridgeCustomRadio.png', 10)
  @screen.wait_and_click('TorLauncherBridgeList.png', 10)
[...]


where TorLauncherBridgeCustomRadio.png is an image of the radio button and label saying “Provide a bridge I know”. I don’t think there is much of a point pushing this to our infra before this is fixed.

Are you up for this? Otherwise I could do this part.

Also, before we can merge this our bridge docs (wiki/src/doc/first_steps/startup_options/bridge_mode.mdwn) needs to be updated to reflect that the default choice after selecting “Tor is censored” now is Meek (not custom bridges), and doesn’t need further configuration. And I guess we need some explanations/comparisons of the options? A job for @sajolida, I think! (?)

However, this made me start thinking. This change in Tails probably will start driving a lot more users to the Meek bridges, which could be a problem. Vanilla Tor Launcher, after all, has bundled obfs4 bridges as the default, not Meek. I think we (I volunteer) should ask tor-dev@ what they think about this, in particular if they think the Meek bridges can handle this. Do we have some sort of data about bridge usage in Tails? I guess we have some from Whisperback reports, but I wonder how representative that is.

Worst case, if we cannot use Meek as the default, maybe we just have to give up on our opinion that “public bridges == bad” and provide the bundled obfs4 bridges as default as well, i.e. exactly the same as in vanilla Tor Launcher.

@intrigeri, what do you think?

#67 Updated by sajolida 2020-03-18 02:32:47

I’m very happy to see support for meek bridges move forward but I’ll refrain from preparing updates to the doc until the code is ready.

@anonym: If you reassigned this ticket to johanbluecreek for the updates to the test suite, I’d suggest not blocking on them anymore (6 weeks have passed).

#68 Updated by johanbluecreek 2020-03-18 10:34:49

sajolida wrote:
> @anonym: If you reassigned this ticket to johanbluecreek for the updates to the test suite, I’d suggest not blocking on them anymore (6 weeks have passed).

My bad. I forgot to push and tell you about those updates. Updated branch on my gitlab has a fix for the test suite. The meek support requires an extra radio-button click to be able to run the test for the step of manually adding bridges, and that has been added to this branch.

#69 Updated by anonym 2020-03-18 13:43:37

  • Assignee changed from johanbluecreek to anonym

sajolida wrote:
> I’m very happy to see support for meek bridges move forward but I’ll refrain from preparing updates to the doc until the code is ready.

Cool! I think this whole thing also is blocked by my fears in Feature #8243#note-66 about overloading the Meek bridges. I’m gonna ask the Tor devs what they think, but I want to provide them some sort of ballpark figure; see your inbox, @sajolida! :)

johanbluecreek wrote:
> sajolida wrote:
> > @anonym: If you reassigned this ticket to johanbluecreek for the updates to the test suite, I’d suggest not blocking on them anymore (6 weeks have passed).
>
> My bad. I forgot to push and tell you about those updates. Updated branch on my gitlab has a fix for the test suite. The meek support requires an extra radio-button click to be able to run the test for the step of manually adding bridges, and that has been added to this branch.

I’ve pulled and pushed to our CI, so let’s see what Jenkins thinks! :) I’ll report back here.

I assigned the ticket to myself since I’m the blocker for moving this forward.

#70 Updated by anonym 2020-03-24 11:43:16

  • Assignee changed from anonym to johanbluecreek

Jenkins is happy! Out of 4 attempts, none had any related failures!

However, I have consulted tor-dev@ about the Meek bridge capacity issue, and the answer is that its already overloaded. This thread has the details: https://lists.torproject.org/pipermail/tor-dev/2020-March/014189.html

So, in order to add Meek support we need to add another default that is ~equally convenient. Our options are:

  • a default set of bridges (e.g. the one shipped in Tor Browser) which we want to avoid due to our threat model.
  • Moat, which we so far have ignored since it requires clearnet access.

So I’d like us to investigate Moat support and consider adding it as the default. I think all we have to do is to let the tor-launcher user access to whatever CDN Meek is hosted (if you are confused: Moat fetches the bridges using Meek as the transport! :)). There’s also a captcha involved, but I hope the image is forwarded through Moat and not served via something like recaptcha.google.com so we have to make a firewall exception for that as well.

@johanbluecreek, are you interested in digging into this?

#71 Updated by intrigeri 2020-03-24 13:14:16

> So I’d like us to investigate Moat support

FTR, prior art: Feature #15331

#72 Updated by johanbluecreek 2020-03-26 10:41:26

> Jenkins is happy! Out of 4 attempts, none had any related failures!

Cool!

> So I’d like us to investigate Moat support [—-]
> @johanbluecreek, are you interested in digging into this?

Absolutely. I will start looking.

>> So I’d like us to investigate Moat support
> FTR, prior art: Feature #15331

Thanks!

#75 Updated by sajolida 2020-04-24 18:21:04

  • Description updated