Tails

#2 Updated by sajolida 2014-12-12 17:39:45

• Assignee set to sajolida

I found two libraries to do SOCKS with Ruby:

• socksify (https://rubygems.org/gems/socksify) which is not in Debian.
• em-socksify (https://rubygems.org/gems/em-socksify) which relies on the EventMachine network communication engine which. Both are available on Jessie but not on Wheezy.

So for the time being, I’ll give a try at socksify.

#3 Updated by sajolida 2014-12-12 23:38:23

• Status changed from Confirmed to In Progress
• Assignee changed from sajolida to intrigeri
• QA Check set to Ready for QA

Use simple HTTP proxy usually involves caching and this is no good in our situation.

So I implemented this in commit 08d3364 using socksify optionnally.

#4 Updated by intrigeri 2014-12-13 08:58:04

• Assignee changed from intrigeri to sajolida
• QA Check changed from Ready for QA to Info Needed

Having to rely on a 3rd-party library just to add SOCKS support for HTTP requests feels, well, wow. I see two alternatives:

• Using Ruby binding around Curl, e.g. ruby-curb or ruby-ethon (both are in Jessie but not in Wheezy). Requiring Jessie feels better on the mid-term than requiring a 3rd-party library. I could look into a backport if needed.
• Worst case, if Ruby support for our needs in Debian is that poor, then we could turn the http_request function into a wrapper around the curl or wget binary, whichever better supports what http_request does.

Now, it depends on who wants to run this script in which environment: do we want to support running it in Tails/Wheezy? Or is it OK to require Jessie?

#6 Updated by sajolida 2014-12-14 18:28:40

• Assignee changed from sajolida to intrigeri

First, I feel the need to give more background to the work I did on check-mirrors the other day. As you know already, I don’t code really often because I’m not good at it, so it becomes quite painful, and I prefer doing things that are more interesting and more successfully to me.

Still, the work I did on Friday fulfills two important objectives to me:

- Being able to publish that code. To hopefully stop being the only one working on it. I’d rather have this taken over by someone else than continue patching it myself wilding forever. The good thing about it is that it is really self-contained. This is now made possible by 801b6eb.

That said, replacing http_request and wget by some curl binding is a good idea and would probably simplify the code greatly. That’s now Feature #8437.

- Being able for me, because I’m at the same time managing the pool of mirrors, to check mirrors from my own system (Tails as of now). This is made possible by this crappy SOCKS support. Also note that the code as of now doesn’t require this external library if you don’t want to use SOCKS. For example, this code can be run on our server without that library and work as usual.

So, my main priority now is to solve Feature #7667. If it is not acceptable for you to have my fix for Feature #8054 in the master branch, then I can put it somewhere else and apply it only locally. I really don’t mind.

#9 Updated by intrigeri 2014-12-22 09:06:45

• Assignee deleted (intrigeri)
• QA Check changed from Info Needed to Dev Needed

#10 Updated by BitingBird 2015-01-04 18:52:38

• Category changed from 214 to Infrastructure
• Affected tool set to check-mirrors

#13 Updated by sajolida 2015-03-01 18:13:32

• Assignee set to sajolida
• Type of work changed from Code to Wait

:)

So I subscribed myself to Tor#8137 and this is now a Wait (unless I do Feature #8437 first).

#14 Updated by sajolida 2015-04-05 12:59:33

• blocks Feature #7667: Create a public repo for check-mirrors added

#17 Updated by intrigeri 2015-07-07 14:41:32

• Type of work changed from Wait to Code

Sadly, even with torsocks 2.1 (built locally as part of Feature #9549) and setting AllowOutboundLocalhost 1 in torsocks.conf, and disabling the netfilter firewall, this still doesn’t work: I suspect that AllowOutboundLocalhost 1 only lets TCP connections to the loopback iface go through, while our full-featured DNS resolver is listening on 127.0.0.2:53 over UDP:

$ torsocks ruby ./check-mirrors.rb --fast --debug --dns 127.0.0.2
/usr/lib/ruby/1.9.1/resolv.rb:761:in `initialize': Operation not permitted - socket(2) - udp (Errno::EPERM)
    from /usr/lib/ruby/1.9.1/resolv.rb:761:in `new'
    from /usr/lib/ruby/1.9.1/resolv.rb:761:in `initialize'
    from /usr/lib/ruby/1.9.1/resolv.rb:537:in `new'
    from /usr/lib/ruby/1.9.1/resolv.rb:537:in `make_udp_requester'
    from /usr/lib/ruby/1.9.1/resolv.rb:495:in `each_resource'
    from /usr/lib/ruby/1.9.1/resolv.rb:391:in `each_address'
    from /usr/lib/ruby/1.9.1/resolv.rb:379:in `getaddresses'
    from ./check-mirrors.rb:290:in `<main>'

So, it seems that we’re back to square one. I see a few solutions:

1. add an option to torsocks to allow even UDP connections to localhost (perhaps the existing one can be extended to do that, its documentation doesn’t really pretend it’s blocking UDP so far);
2. only use torsocks, from within check-mirrors, for connections to the Internet, and then we can use 127.0.0.2:53 over UDP for DNS resolution;
3. add proper SOCKS support to check-mirrors, use it for connections to the Internet, and then we can use 127.0.0.2:53 over UDP for DNS resolution.

#19 Updated by sajolida 2015-08-11 07:57:31

• QA Check deleted (Dev Needed)
• Type of work changed from Code to Wait

Done in https://trac.torproject.org/projects/tor/ticket/16765.

#23 Updated by intrigeri 2016-09-10 08:07:24

• related to Bug #11736: torsocks complains when used with Monkeysign, but works anyway added

#24 Updated by intrigeri 2016-10-26 09:51:09

• Type of work changed from Wait to Code

I’ve uploaded torsocks 2.2 to jessie-backports, so the nightly builds from our stable and devel branch should have it in 6-24 hours. Then you can test if AllowOutboundLocalhost 2 is sufficient.

#25 Updated by intrigeri 2017-05-28 12:48:05

• Subject changed from Add support for SOCKS proxy to check-mirror to Add support for SOCKS proxy to check-mirrors
• QA Check set to Ready for QA

If I get the history of this ticket right, this should now be a solved problem, so next things to do are 1. verify that it’s really solved with current torsocks; 2. perhaps some custom SOCKS code should be removed?; 3. move on to Feature #7667 so we can eventually get some more help on check-mirrors :)

#26 Updated by sajolida 2017-05-29 16:37:57

• Target version set to Tails_3.1

Exactly, so I’ll put this back on my radar.

It would feel good to have other people help with this script.

#27 Updated by intrigeri 2017-08-12 20:46:15

• Target version changed from Tails_3.1 to Tails_3.2

#28 Updated by anonym 2017-09-28 18:29:21

• Target version changed from Tails_3.2 to Tails_3.3

#29 Updated by sajolida 2017-11-14 13:23:49

• Target version changed from Tails_3.3 to Tails_3.5

#30 Updated by Anonymous 2018-01-19 15:25:13

• blocked by deleted (Feature #7667: Create a public repo for check-mirrors)

#31 Updated by Anonymous 2018-01-19 15:25:20

• related to Feature #7667: Create a public repo for check-mirrors added

#32 Updated by anonym 2018-01-23 19:52:30

• Target version changed from Tails_3.5 to Tails_3.6

#33 Updated by sajolida 2018-01-31 14:05:43

• Target version deleted (Tails_3.6)

#34 Updated by sajolida 2018-04-21 15:31:45

• Assignee changed from sajolida to intrigeri
• Feature Branch set to check-mirrors.git:8054/torsocks-support

I tested again check-mirrors with torsocks in Tails 3.6.2. Running the same tests as in Feature #8054#note-11:

• torsocks ruby check-mirrors.rb --debug --fast works fine for the JSON pool but only checks 1 IP for the DNS pool.
• torsocks host dl.amnesia.boum.org 127.0.0.2 returns ;; connection timed out; no servers could be reached. I think that this service on 127.0.0.2 is not available anymore in Tails 3.6.2 and I think I remember it was the service that would allow resolving multiple IPs.

But we’re also planning to move on having only 1 IP in the DNS pool, see Bug #12833 and https://tails.boum.org/blueprint/HTTP_mirror_pool/#index6h2. So it’s no big deal (especially for this torsocks support feature).

I’m removing the code we had for the --torify option in 982b9f0.

Please have a look.

I’m also unparenting Feature #8437 from this ticket since it’s not needed to solve the problem we had here (“being able to run check-mirrors from Tails”).

I was also doubting whether removing the option for the --dns option as well which might not be useful anymore…

#35 Updated by intrigeri 2018-04-27 07:49:29

• Assignee changed from intrigeri to sajolida
• QA Check changed from Ready for QA to Pass

sajolida wrote:
> But we’re also planning to move on having only 1 IP in the DNS pool, see Bug #12833 and https://tails.boum.org/blueprint/HTTP_mirror_pool/#index6h2.

FYI the plan is rather to ditch the IP-based round-robin mirror pool.

> So it’s no big deal (especially for this torsocks support feature).

Agreed.

> I’m removing the code we had for the --torify option in 982b9f0.
>
> Please have a look.

Looks good to me, feel free to merge.

> I was also doubting whether removing the option for the --dns option as well which might not be useful anymore…

Indeed, let’s remove it.

#37 Updated by sajolida 2018-04-27 10:38:42

• Assignee changed from sajolida to intrigeri
• QA Check changed from Pass to Ready for QA

#38 Updated by intrigeri 2018-04-30 11:38:21

• Assignee changed from intrigeri to sajolida
• QA Check changed from Ready for QA to Pass

sajolida wrote:
> >> But we’re also planning to move on having only 1 IP in the DNS pool, see Bug #12833 and https://tails.boum.org/blueprint/HTTP_mirror_pool/#index6h2.
> >
> > FYI the plan is rather to ditch the IP-based round-robin mirror pool.
>
> Indeed: “DNS pool of 1” ~= “ditching the DNS pool” :)

Sure. It just won’t use any DNS trick (and won’t be a pool) anymore. Anyway, glad you apparently got it right despite the confusing wording :)

> >> I was also doubting whether removing the option for the --dns option as well which might not be useful anymore…
> >
> > Indeed, let’s remove it.
>
> I pushed fc2a933 in the same branch. Please have a look again.

Looks good. Feel free to merge whenever you want.

#39 Updated by sajolida 2018-04-30 16:56:47

• Status changed from In Progress to Resolved
• Assignee deleted (sajolida)
• QA Check deleted (Pass)

Merged!