Tails

#7 Updated by anonym 2019-05-29 22:57:05

Note to self: I never actually successfully configured anything with Tor Launcher, I’ve just seen it started. Trying to configure anything actually just says there are problems with the control port. In the journal I see:

stem.ProtocolError: GETINFO response didn't have an OK status:
Command filtered

so I probably just have to fix something in the onion-grater filter.

#8 Updated by anonym 2019-05-29 23:09:31

NOTE TO SELF: These results were from my test setup, not Tails, so this problem has never occurred in Tails.

Eh, from what I can tell, when I press “Connect” in Tor Launcher, onion-grater tries to connect to itself and (naturally) fails:

pgrep onion-grater
23458

[... normal stuff, where the tor-launcher filter is loaded and there's some back-and-forth ...]
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None) connected: loaded filter: None
Final rules:
commands: {}
events: {}
restrict-stream-events: false

/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): -> PROTOCOLINFO 1
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 250-PROTOCOLINFO 1
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 250-AUTH METHODS=NULL
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 250-VERSION Tor="0.3.5.8"
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 250 OK
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): -> AUTHENTICATE
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 250 OK
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): -> SETEVENTS
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 250 OK
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): -> GETCONF __owningcontrollerprocess
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): command filtered: GETCONF __owningcontrollerprocess
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 510 Command filtered
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): -> GETINFO version
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): command filtered: GETINFO version
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 510 Command filtered
/usr/bin/python3.5 (pid: 23458, user: root, port: 59812, filter: None) disconnected: client quit
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 59812)
Traceback (most recent call last):
  File "/usr/lib/python3.5/socketserver.py", line 625, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python3.5/socketserver.py", line 354, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.5/socketserver.py", line 681, in __init__
    self.handle()
  File "./onion-grater", line 629, in handle
    self.controller = self.connect_to_real_control_port()
  File "./onion-grater", line 569, in connect_to_real_control_port
    controller = stem.connection.connect(control_socket=global_args.control_socket_path)
  File "/usr/lib/python3/dist-packages/stem/connection.py", line 287, in connect
    return _connect_auth(control_connection, password, password_prompt, chroot_path, controller)
  File "/usr/lib/python3/dist-packages/stem/connection.py", line 371, in _connect_auth
    return controller(control_socket, is_authenticated = True)
  File "/usr/lib/python3/dist-packages/stem/control.py", line 1041, in __init__
    self.add_event_listener(_sighup_listener, EventType.SIGNAL)
  File "/usr/lib/python3/dist-packages/stem/control.py", line 2997, in add_event_listener
    if event_type and (self.get_version() < event_type._VERSION_ADDED):
  File "/usr/lib/python3/dist-packages/stem/control.py", line 454, in wrapped
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/stem/control.py", line 1235, in get_version
    version = stem.version.Version(self.get_info('version'))
  File "/usr/lib/python3/dist-packages/stem/control.py", line 454, in wrapped
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/stem/control.py", line 1163, in get_info
    stem.response.convert('GETINFO', response)
  File "/usr/lib/python3/dist-packages/stem/response/__init__.py", line 132, in convert
    message._parse_message(**kwargs)
  File "/usr/lib/python3/dist-packages/stem/response/getinfo.py", line 40, in _parse_message
    raise stem.ProtocolError("GETINFO response didn't have an OK status:\n%s" % self)
stem.ProtocolError: GETINFO response didn't have an OK status:
Command filtered

How/why does it connect to itself

#9 Updated by anonym 2019-05-30 09:38:06

I think I have stumbled upon a bug in stem (or onion-grater?) that I had to workaround with commit:f74cec4d7fc44d9b9eec37090990e1868ba592db so we can get some test results. That commit should be reverted, or the code it adds should be made saner (e.g. don’t retry indefinitely) if we really need something like that.

Any way, as of now, everything seems to work on the surface level! \o/ Let’s see what Jenkins thinks of the non-fragile tests -- later I should repush with +force-all-tests@ and compare those tests too.

#11 Updated by anonym 2019-05-30 21:37:43

The first full test suite run went pretty well:

Failing Scenarios:
cucumber features/additional_software_packages.feature:69 # Scenario: Recovering in offline mode after Additional Software previously failed to upgrade and then succeed to upgrade when online
cucumber features/electrum.feature:15 # Scenario: Using a persistent Electrum configuration
cucumber features/tor_stream_isolation.feature:9 # Scenario: tails-security-check is using the Tails-specific SocksPort
cucumber features/totem.feature:50 # Scenario: Watching a WebM video over HTTPS

219 scenarios (4 failed, 215 passed)
1658 steps (3 failed, 1655 passed)
346m16.087s

No failure is related to the browser, so we’re pretty in great shape!

#12 Updated by anonym 2019-06-13 10:50:03

Updated to Tor Browser 9.0a2. What do you think, Jenkins?

#14 Updated by intrigeri 2019-08-08 11:42:21

The first esr68-based bundles are coming: https://lists.torproject.org/pipermail/tbb-dev/2019-August/001013.html

#19 Updated by intrigeri 2019-09-08 05:38:37

9.0a6, based on 68.1.0esr, is now available: https://blog.torproject.org/new-release-tor-browser-90a6

#23 Updated by intrigeri 2019-09-09 11:08:02

OK, the branch now builds & Tor Browser starts. There are a number of serious issues that are tracked in subtasks. I’ll take a look at the test suite results, will file more subtasks if needed, and then will give this ticket back to the collective pool, hoping that this clarification of what needs to be done will help us organize this work :)

#27 Updated by segfault 2019-09-15 15:11:03

Pushed a commit which should fix the branch to FTBFS since the uBlock 1.22.2+dfsg-1 release.

#28 Updated by intrigeri 2019-10-02 07:33:57

I see that 9.0a7 tarballs are now available and will upgrade the branch to this new alpha right away.

#29 Updated by intrigeri 2019-10-02 09:24:58

intrigeri wrote:
> I see that 9.0a7 tarballs are now available and will upgrade the branch to this new alpha right away.

Done, let’s see how the test suite feels about it :)

#30 Updated by sajolida 2019-10-05 20:05:07

This one is really nasty!

To open an URL, I typed it in LibreOffice Writer and right-clicked on it to open the link :)

#31 Updated by intrigeri 2019-10-06 04:03:10

Hi @sajolida,

> This one is really nasty!
> To open an URL, I typed it in LibreOffice Writer and right-clicked on it to open the link :)

Did you mean to comment on Bug #17121?

#33 Updated by sajolida 2019-10-07 21:11:27

Indeed!

#34 Updated by anonym 2019-10-08 08:31:15

Note that we still have commit:f74cec4d7fc44d9b9eec37090990e1868ba592db (see Feature #16356#note-9 and in particular the commit message for details) which was meant as a temporary workaround. Not understanding what’s going on here is possibly bad enough to warrant an investigation, but there is also the potential for very real issues if tor dies, since onion-grater will endlessly try to reconnect as fast as possible which could lead to performance/battery drain, and (I think?) even a “memory leak” by filling the journal.

At the very least I think we should rate limit the retries after a few tries (to not needlessly delay the common case, where it works on the first retry), e.g.:

<code class="diff">
--- a/config/chroot_local-includes/usr/local/lib/onion-grater
+++ b/config/chroot_local-includes/usr/local/lib/onion-grater
@@ -567,8 +567,12 @@ class FilteredControlPortProxyHandler(socketserver.StreamRequestHandler):

     def connect_to_real_control_port(self):
         controller = None
+        tries = 0
         while not controller:
+            if tries >= 3:
+                time.sleep(1)
             controller = stem.connection.connect(control_socket=global_args.control_socket_path)
+            tries += 1
         stem.connection.authenticate_cookie(controller, cookie_path=global_args.control_cookie_path)
         return controller
</code>

#35 Updated by intrigeri 2019-10-09 07:27:47

Hi @anonym,

> Note that we still have commit:f74cec4d7fc44d9b9eec37090990e1868ba592db (see Feature #16356#note-9 and in particular the commit message for details) which was meant as a temporary workaround. Not understanding what’s going on here is possibly bad enough to warrant an investigation,

Yes, quite possibly. I don’t know if we’ll get to the end of it by the time we want to close this very ticket, so I’d rather see this tracked in a dedicated ticket.

> (I think?) even a “memory leak” by filling the journal.

journald has an upper limit for how much memory it’ll use, but yes.

> At the very least I think we should rate limit the retries after a few tries (to not needlessly delay the common case, where it works on the first retry), e.g.:

Sounds good, please go ahead!

#36 Updated by intrigeri 2019-10-09 14:48:49

As discussed on XMPP with anonym and segfault today, I’ve merged this branch into devel so it’s part of 4.0~rc1.