WhisperBack cannot send reports since Sep 10: expired (pinned) certificate
Whisperback can’t send bug reports. Users have reported this and we’ve noticed this first hand as well.
amnesia@amnesia:~$ whisperback [WARNING] whisperback.py:150 __load_conf: Failed to load conf /home/amnesia/.whisperback/config.py [WARNING] whisperback.py:150 __load_conf: Failed to load conf /home/amnesia/config.py Exception in thread Thread-6: Traceback (most recent call last): File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/usr/lib/python3.5/threading.py", line 862, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3/dist-packages/whisperBack/whisperback.py", line 245, in save_exception func(*args) File "/usr/lib/python3/dist-packages/whisperBack/mail.py", line 77, in send_message_tls (resp, reply) = smtp.starttls(context=ssl_context) File "/usr/lib/python3.5/smtplib.py", line 766, in starttls server_hostname=self._host) File "/usr/lib/python3.5/ssl.py", line 385, in wrap_socket _context=self) File "/usr/lib/python3.5/ssl.py", line 760, in __init__ self.do_handshake() File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake self._sslobj.do_handshake() File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)
#8 Updated by intrigeri 2019-10-03 18:40:52
> the ssl error seems to be given on a connection to the local exim running on tails instances, that’s a bit beyond the scope of sysadmin work…
@groente: thanks. I understand this is not a sysadmin problem so I’ll look into it.
FTR there’s no local Exim running in Tails: WhisperBack delivers email directly over SMTP to a Onion service (postfix-hidden) hosted on
#9 Updated by intrigeri 2019-10-04 07:32:58
> groente: thanks. I understand this is not a sysadmin problem so I’ll look into it.
The bug comes from (an ancient part of our) infra, and leaks into Tails images in order to implement a very strict security setup:
- On the sysadmin side,
tails_secrets_whisperback.git:files/ssl/4mvq3pnvid3awjln.onion.pem, which is used by our WhisperBack SMTP relay (that runs on
whisperback.lizard), indeed expired on September 10 ⇒ adding my fellow sysadmins to the loop.
- A copy of that file lives in
tails.git: source:config/chroot_local-includes/etc/whisperback/4mvq3pnvid3awjln.onion.pem, which WhisperBack pins when establishing TLS connections to the WhisperBack SMTP relay ⇒ oh shiiiiiit, there’s no way we fix that without releasing a new Tails.
To fix the problem in time for 4.0~rc1, I see these options:
- Update the certificate (or replace the key pair + self-sign again). Can we have a self-signed cert with no expiration date at all?
- Drop TLS for WhisperBack: OpenPGP-encrypted email over a Tor onion service should provide enough security, no? Optionally, upgrade the onion service to HSv3 to increase our confidence in the authentication and in the 2nd layer of encryption it provides.
The first option is cheaper to implement right now but it has a higher maintenance cost and will require doing extra work to ensure this problem never happens again.
The second option requires a bit more work right now but then we’re done.
Personally, I think TLS is quite overkill here, and I’m leaning towards the second option. groente, segfault, zen: any thoughts on this?
And then, depending on how we decide to fix the problem, we may or may not need to take extra action to ensure that the problem never comes back. For example, if we decide to keep TLS with an expiration date in that loop:
- Apparently we have no check anywhere to alert us when this certificate will expire soon → we should add either a monitoring check for this on the infra side (do we already have checks for upcoming certs expiration that we could easily extend?), or add a test case to the Tails automated test suite (where we already check that various OpenPGP keys expire in more than 3 months).
- Make the monitoring check for the WhisperBack SMTP relay, which still passes flawlessly (ahem), test something that better reflects the actual use case for this service. Granted, this wouldn’t have alerted us in time to fix the problem in Tails 3.16, but still, I would have preferred having a full month to fix the problem, instead of 6 days.
Post-mortem: this piece of infra was set up in September 2009, a few weeks after the first public release of amnesia. For some reason we wanted TLS on top of encrypting email with OpenPGP + the end-to-end authentication + encryption provided by Tor onion services. So we created a TLS key pair. The self-signed certificate was set to expire 10 years later, that is, last month. The project was way too immature back then to think about what would happen 10 years later and set up a process to update this certificate. Oh well :)
#12 Updated by anonym 2019-10-04 08:03:54
> The second option requires a bit more work right now but then we’re done.
Well, if we go with option 1 we will have a clear deadline for when we need to do option 2 (or update the cert yet again) so I think it is fine to do the cheaper option now and add clear plans for dealing with 2 later, but in good time before the next cert expiry.
> Personally, I think TLS is quite overkill here, and I’m leaning towards the second option.
100% agreed. If there is anything for us to improve in this area’s security story it is to update to v3 onions (v2 onions, among other things, depend on rsa1024 which is starting to get a bit too weak).
#15 Updated by intrigeri 2019-10-04 09:55:24
- Assignee set to intrigeri
LGTM at first glance.
- drop TLS server-side
- do a more proper code review and test
- if time allows, switch to HSv3 (requires changes on both client and server side + in our infra monitoring); else, file a ticket to do that later
#19 Updated by intrigeri 2019-10-05 06:46:02
- Status changed from Resolved to Needs Validation
- Assignee changed from intrigeri to groente
> Next steps:
> # drop TLS server-side
> # do a more proper code review and test
> # if time allows, switch to HSv3 (requires changes on both client and server side + in our infra monitoring)
While we certainly could have postponed the migration to a HSv3, I figured that now is a good time to do it: the WhisperBack feedback loop is broken already, so we can introduce backwards-incompatible changes without having to deal with the complexity of a transition period, during which we would have to support both the old v2 Onion service and the new v3 one.
Dear groente, can you please review the Puppet code changes? They’ve all been done today and the most relevant ones are tagged with this ticket ID.