Bug #17110

WhisperBack cannot send reports since Sep 10: expired (pinned) certificate

Added by numbat 2019-10-01 10:16:27 . Updated 2019-10-25 16:02:37 .

Status:
Resolved
Priority:
High
Assignee:
groente
Category:
Infrastructure
Target version:
Start date:
Due date:
% Done:

100%

Feature Branch:
bugfix/17110-whisperback-drop-tls;whisperback:bugfix/17110-whisperback-drop-tls
Type of work:
Code
Blueprint:

Starter:
Affected tool:
WhisperBack
Deliverable for:

Description

Whisperback can’t send bug reports. Users have reported this and we’ve noticed this first hand as well.

amnesia@amnesia:~$ whisperback
[WARNING] whisperback.py:150 __load_conf: Failed to load conf /home/amnesia/.whisperback/config.py
[WARNING] whisperback.py:150 __load_conf: Failed to load conf /home/amnesia/config.py
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/whisperBack/whisperback.py", line 245, in save_exception
    func(*args)
  File "/usr/lib/python3/dist-packages/whisperBack/mail.py", line 77, in send_message_tls
    (resp, reply) = smtp.starttls(context=ssl_context)
  File "/usr/lib/python3.5/smtplib.py", line 766, in starttls
    server_hostname=self._host)
  File "/usr/lib/python3.5/ssl.py", line 385, in wrap_socket
    _context=self)
  File "/usr/lib/python3.5/ssl.py", line 760, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

Subtasks


Related issues

Has duplicate Tails - Bug #17120: No WhisperBack reports since September 10 Duplicate
Blocks Tails - Feature #13242: Core work: Sysadmin (Maintain our already existing services) Confirmed 2017-06-29

History

#1 Updated by Anonymous 2019-10-01 14:44:02

  • Affected tool set to WhisperBack

#2 Updated by goupille 2019-10-01 16:39:44

  • Assignee set to intrigeri

#3 Updated by intrigeri 2019-10-02 06:47:12

  • Assignee changed from intrigeri to Sysadmins
  • Priority changed from Elevated to High
  • Target version set to Tails_4.0

#4 Updated by intrigeri 2019-10-03 17:31:40

  • has duplicate Bug #17120: No WhisperBack reports since September 10 added

#5 Updated by intrigeri 2019-10-03 17:32:52

  • blocks Feature #13242: Core work: Sysadmin (Maintain our already existing services) added

#6 Updated by intrigeri 2019-10-03 17:33:12

sajolida says on Bug #17120 that he received no WhisperBack report since September 10.

#7 Updated by groente 2019-10-03 17:52:39

  • Assignee changed from Sysadmins to intrigeri
  • Type of work changed from Sysadmin to Code

the ssl error seems to be given on a connection to the local exim running on tails instances, that’s a bit beyond the scope of sysadmin work…

#8 Updated by intrigeri 2019-10-03 18:40:52

> the ssl error seems to be given on a connection to the local exim running on tails instances, that’s a bit beyond the scope of sysadmin work…

@groente: thanks. I understand this is not a sysadmin problem so I’ll look into it.

FTR there’s no local Exim running in Tails: WhisperBack delivers email directly over SMTP to a Onion service (postfix-hidden) hosted on whisperback.lizard.

#9 Updated by intrigeri 2019-10-04 07:32:58

intrigeri wrote:
> groente: thanks. I understand this is not a sysadmin problem so I’ll look into it.

The bug comes from (an ancient part of our) infra, and leaks into Tails images in order to implement a very strict security setup:

  • On the sysadmin side, tails_secrets_whisperback.git:files/ssl/4mvq3pnvid3awjln.onion.pem, which is used by our WhisperBack SMTP relay (that runs on whisperback.lizard), indeed expired on September 10 ⇒ adding my fellow sysadmins to the loop.
  • A copy of that file lives in tails.git: source:config/chroot_local-includes/etc/whisperback/4mvq3pnvid3awjln.onion.pem, which WhisperBack pins when establishing TLS connections to the WhisperBack SMTP relay ⇒ oh shiiiiiit, there’s no way we fix that without releasing a new Tails.

To fix the problem in time for 4.0~rc1, I see these options:

  • Update the certificate (or replace the key pair + self-sign again). Can we have a self-signed cert with no expiration date at all?
  • Drop TLS for WhisperBack: OpenPGP-encrypted email over a Tor onion service should provide enough security, no? Optionally, upgrade the onion service to HSv3 to increase our confidence in the authentication and in the 2nd layer of encryption it provides.

The first option is cheaper to implement right now but it has a higher maintenance cost and will require doing extra work to ensure this problem never happens again.
The second option requires a bit more work right now but then we’re done.

Personally, I think TLS is quite overkill here, and I’m leaning towards the second option. groente, segfault, zen: any thoughts on this?

And then, depending on how we decide to fix the problem, we may or may not need to take extra action to ensure that the problem never comes back. For example, if we decide to keep TLS with an expiration date in that loop:

  • Apparently we have no check anywhere to alert us when this certificate will expire soon → we should add either a monitoring check for this on the infra side (do we already have checks for upcoming certs expiration that we could easily extend?), or add a test case to the Tails automated test suite (where we already check that various OpenPGP keys expire in more than 3 months).
  • Make the monitoring check for the WhisperBack SMTP relay, which still passes flawlessly (ahem), test something that better reflects the actual use case for this service. Granted, this wouldn’t have alerted us in time to fix the problem in Tails 3.16, but still, I would have preferred having a full month to fix the problem, instead of 6 days.

Post-mortem: this piece of infra was set up in September 2009, a few weeks after the first public release of amnesia. For some reason we wanted TLS on top of encrypting email with OpenPGP + the end-to-end authentication + encryption provided by Tor onion services. So we created a TLS key pair. The self-signed certificate was set to expire 10 years later, that is, last month. The project was way too immature back then to think about what would happen 10 years later and set up a process to update this certificate. Oh well :)

#10 Updated by intrigeri 2019-10-04 07:37:00

  • Subject changed from Whisperback - certificate verify failed to WhisperBack cannot send reports since Sep 10: expired (pinned) certificate
  • Status changed from Confirmed to In Progress

#11 Updated by intrigeri 2019-10-04 08:01:33

#12 Updated by anonym 2019-10-04 08:03:54

intrigeri wrote:
> The second option requires a bit more work right now but then we’re done.

Well, if we go with option 1 we will have a clear deadline for when we need to do option 2 (or update the cert yet again) so I think it is fine to do the cheaper option now and add clear plans for dealing with 2 later, but in good time before the next cert expiry.

> Personally, I think TLS is quite overkill here, and I’m leaning towards the second option.

100% agreed. If there is anything for us to improve in this area’s security story it is to update to v3 onions (v2 onions, among other things, depend on rsa1024 which is starting to get a bit too weak).

#13 Updated by segfault 2019-10-04 08:16:50

  • Assignee changed from intrigeri to segfault

#14 Updated by segfault 2019-10-04 09:20:01

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (segfault)
  • Feature Branch set to bugfix/17110-whisperback-drop-tls;whisperback:bugfix/17110-whisperback-drop-tls

#15 Updated by intrigeri 2019-10-04 09:55:24

  • Assignee set to intrigeri

LGTM at first glance.

Next steps:

  1. drop TLS server-side
  2. do a more proper code review and test
  3. if time allows, switch to HSv3 (requires changes on both client and server side + in our infra monitoring); else, file a ticket to do that later

#16 Updated by segfault 2019-10-04 11:41:47

Note that I tested this by copying the modified files to a running Tails and sending a test report. WhisperBack said that the report was sent successfully.

#17 Updated by intrigeri 2019-10-05 06:42:49

  • Status changed from Needs Validation to In Progress

Applied in changeset commit:tails|502a5ad2b0e874e32d970fbcfefa54b1b80505c2.

#18 Updated by intrigeri 2019-10-05 06:42:50

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset commit:tails|2bf52bc0d8d38947b46b5e59be9e0952da3b7e7a.

#19 Updated by intrigeri 2019-10-05 06:46:02

  • Status changed from Resolved to Needs Validation
  • Assignee changed from intrigeri to groente

> Next steps:

> # drop TLS server-side
> # do a more proper code review and test
> # if time allows, switch to HSv3 (requires changes on both client and server side + in our infra monitoring)

All done!

While we certainly could have postponed the migration to a HSv3, I figured that now is a good time to do it: the WhisperBack feedback loop is broken already, so we can introduce backwards-incompatible changes without having to deal with the complexity of a transition period, during which we would have to support both the old v2 Onion service and the new v3 one.

Dear groente, can you please review the Puppet code changes? They’ve all been done today and the most relevant ones are tagged with this ticket ID.

#20 Updated by intrigeri 2019-10-19 11:53:54

#21 Updated by intrigeri 2019-10-21 11:46:18

  • Target version changed from Tails_4.0 to Tails_4.1

#22 Updated by groente 2019-10-25 16:02:37

  • Status changed from Needs Validation to Resolved

looks good!