Feature #9796

HTTPS mirrors

Added by sajolida 2015-07-23 10:53:01 . Updated 2019-03-07 15:29:05 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
Start date:
2017-06-21
Due date:
% Done:

100%

Feature Branch:
Type of work:
Code
Starter:
Affected tool:
Deliverable for:

Description

We should allow or force our mirrors to server ISO images through HTTPS. This would be a cheap protection from simple MitM and other network attacks as they have been reported to us already (<FBypOyHFqoU059kPZzuL1jfruC8kTPJa@tails.boum.org-schleuder>).

team: intrigeri, u


Subtasks

Bug #12834: Update our validation schema/script to reject JSON files that contain HTTP mirrors Resolved

100

Feature #12835: Adjust contribute/how/mirror to only support HTTPS Resolved

50

Bug #12837: Drop HTTP mirrors Resolved

100

Feature #13597: Adjust bin/validate-config to only allow HTTPS URLs Resolved

0


Related issues

Related to Tails - Feature #11623: Provide SHA-Checksum and HTTPS Download Rejected 2016-08-06
Related to Tails - Feature #9102: Get tails.boum.org on the Chrome HSTS preload list Resolved 2015-03-24
Related to Tails - Bug #12836: Adjust our Puppet code to only support HTTPS mirrors Confirmed 2017-06-21
Related to Tails - Bug #12833: Implement our masterplan about fallback DNS round-robin pool & HTTPS In Progress 2017-06-21

History

#1 Updated by intrigeri 2015-08-03 05:21:11

IMO, next step is to wait for Let’s Encrypt to go live.

#2 Updated by sajolida 2015-08-14 12:24:21

  • Target version set to 2017

#3 Updated by Dr_Whax 2016-08-20 12:33:08

  • Description updated
  • Assignee deleted (None)

#4 Updated by sajolida 2016-08-30 13:17:37

  • related to Feature #11623: Provide SHA-Checksum and HTTPS Download added

#5 Updated by sajolida 2016-08-30 14:13:52

Note that right now we’re forcing people to do a verification which is almost as strong as HTTPs, either through the Firefox extension, BitTorrent, or OpenPGP. So what would it bring to have HTTPS only mirrors? Security in depth? Anything else?

#6 Updated by Anonymous 2016-09-03 05:17:59

sajolida wrote:
> Note that right now we’re forcing people to do a verification which is almost as strong as HTTPs, either through the Firefox extension, BitTorrent, or OpenPGP. So what would it bring to have HTTPS only mirrors? Security in depth? Anything else?

Yes, if you don’t download over Tor, HTTPS can prevent your ISP/employer/etc to see what you’re downloading exactly. Although, indeed, the ISP/employer/etc can see the domain name connected to, probably the prior DNS query and could make a guess from the downloaded filesize. But I still find that quite a good additional layer of encryption.

#7 Updated by intrigeri 2016-10-14 19:35:57

  • blocks Feature #9102: Get tails.boum.org on the Chrome HSTS preload list added

#8 Updated by intrigeri 2016-12-06 17:18:23

  • blocked by deleted (Feature #9102: Get tails.boum.org on the Chrome HSTS preload list)

#9 Updated by intrigeri 2016-12-06 17:18:30

  • related to Feature #9102: Get tails.boum.org on the Chrome HSTS preload list added

#10 Updated by intrigeri 2016-12-06 17:37:40

u wrote:
> sajolida wrote:
> > Note that right now we’re forcing people to do a verification which is almost as strong as HTTPs, either through the Firefox extension, BitTorrent, or OpenPGP. So what would it bring to have HTTPS only mirrors? Security in depth? Anything else?
>
> Yes, if you don’t download over Tor, HTTPS can prevent your ISP/employer/etc to see what you’re downloading exactly. Although, indeed, the ISP/employer/etc can see the domain name connected to, probably the prior DNS query and could make a guess from the downloaded filesize. But I still find that quite a good additional layer of encryption.

Right.

Also, whatever post-download verification, be it done by DAVE, a BitTorrent client, or OpenPGP, does not protect against exploitation with malicious data of the code used for downloading.

Another reason is that dl.a.b.o is the only reason why the nice boum.org folks cannot have HSTS for their domain; it would be nice if we could stop causing this problem to them.

And finally: it’s 2016, soon 2017, and the days of plaintext HTTP are over; it must die (we can’t do that for the whole WWW ourselves, but at least we can stop contributing to the “it’s OK to download software over plaintext HTTP” mindset; mind you, not all users get the subtle difference between “it’s mostly OK since we verify post-download” and “it’s OK in all cases”).

None of these reasons, taken in isolation, would be enough to make me do this work. But all together, they certainly are motivating enough :)

#11 Updated by sajolida 2016-12-07 08:13:07

I’m convinced now :)

#12 Updated by Anonymous 2017-03-13 13:04:27

Next steps:

  • Wait for Chromium 47 to be released (should happen tomorrow) and to be updated in Debian.
  • Issue an email request to our current mirror operators asking them to provide HTTPS.
  • Modify the mirror pool dispatcher script used in the Tails Upgrader.
  • Modify the mirror pool dispatcher web JS script to allow only HTTPS.
  • Release a new version of DAVE using the updated mirror pool dispatcher script.

Anything else?

#13 Updated by Anonymous 2017-03-13 13:04:58

  • Type of work changed from Sysadmin to Code

#14 Updated by intrigeri 2017-03-13 14:45:32

> * Modify the mirror pool dispatcher script used in the Tails Upgrader.
> * Modify the mirror pool dispatcher web JS script to allow only HTTPS.

IMO it would be easier to ensure our mirror pool only contains HTTPS mirrors, i.e. “update our pre-commit hook in mirrors.git to ensure no non-TLS mirror is in the list”. And then we don’t need to modify the mirror pool dispatcher everywhere.

#15 Updated by Anonymous 2017-03-18 11:38:20

current state:

$ grep url_prefix mirrors.json  | grep https | wc -l
37
$ grep url_prefix mirrors.json  | grep "http:" | wc -l
13

=> So 25.5% of our 51 mirrors are currently using http only.

5 of the remaining 13 http mirrors are using *.dl.amnesia.boum.org - so if i get it right this should be fixed by HSTS?

So we can contact the remaining 8 and ask them to set up HTTPS.

#16 Updated by Anonymous 2017-04-08 11:25:17

I think we should set a deadline for the mirror operators because otherwise this will never happen. What do you think about June 1st? That’s in about 2 months. Is that enough time to set up HTTPS? Or should we say June 15th?

#17 Updated by Anonymous 2017-04-08 11:28:45

Here is a sample email text:

Hi,

you are currently hosting a Tails mirror. Thank you very much!

We plan to switch our mirror pool to HTTPS only in _July 2017_. 
Your mirror is one of 13 out of 50 Tails mirrors which do not currently provide TLS.

Are you able and willing to set up HTTPS for your Tails mirror? Let's Encrypt makes it pretty cheap and easy theses days.

Mirrors which do not provide HTTPS shall be removed from our mirror pool on _July 1st 2017_.

Thank you very much for helping Tails and hosting a mirror.

Obviously, the date may be different, I just try to push this forward a little bit. Please improve and comment.

#18 Updated by intrigeri 2017-04-08 13:10:21

> Here is a sample email text:

Looks good to me! Thanks for pushing this forward :)

> What do you think about June 1st? That’s in about 2 months. Is that enough time to set up HTTPS? Or should we say June 15th?

Two months, starting from the time when we send them the advance warning, sounds reasonable to me. Personally I won’t have time to work on the next steps (drop non-HTTPS mirrors, adjust our pre-commit hook) before June 20. But feel free to set the deadline earlier if you can handle it :)

#19 Updated by Anonymous 2017-04-12 11:04:27

I’ll start sending it out now, using June 20th as a deadline. Then we can see who handles it. I can certainly take over the dropping part.

#20 Updated by Anonymous 2017-04-12 11:11:04

mail sent.

#21 Updated by intrigeri 2017-04-12 11:19:02

> mail sent.

\o/

#22 Updated by sajolida 2017-04-12 15:57:59

Cool!

#23 Updated by intrigeri 2017-04-14 10:59:45

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

BTW, what’s the plan wrt. the fallback DNS round-robin pool?

#24 Updated by Anonymous 2017-04-27 10:32:27

We are now down to 6 out of 49 mirrors using HTTP only. One of them is deactivated since > 6 months, because it was unresponsive.

#25 Updated by intrigeri 2017-06-15 07:44:48

The deadline is now in 5 days, and according to bin/stats, among our 45 enabled mirrors enabled, only 4 don’t provide HTTPS :) Yeepee, congrats!

I suggest we complete the work on the regular mirror pool without blocking on the DNS round-robin one: stop accepting HTTP mirrors, adjust contribute/how/mirror + our Puppet code to only support HTTPS, drop HTTP mirrors, and update our validation schema/script to reject JSON files that contain HTTP mirrors. This way, most downloads will benefit from the improvement sooner. And once we’re there, we can start thinking about what to do wrt. the fallback DNS round-robin pool. Makes sense?

#26 Updated by Anonymous 2017-06-21 17:07:30

intrigeri wrote:
> The deadline is now in 5 days, and according to bin/stats, among our 45 enabled mirrors enabled, only 4 don’t provide HTTPS :) Yeepee, congrats!
>
> I suggest we complete the work on the regular mirror pool without blocking on the DNS round-robin one: stop accepting HTTP mirrors, adjust contribute/how/mirror + our Puppet code to only support HTTPS, drop HTTP mirrors, and update our validation schema/script to reject JSON files that contain HTTP mirrors. This way, most downloads will benefit from the improvement sooner. And once we’re there, we can start thinking about what to do wrt. the fallback DNS round-robin pool. Makes sense?

Sounds good to me. I’ll create tickets for each of these items.

#27 Updated by Anonymous 2017-06-21 18:32:11

  • Assignee set to intrigeri

Reassigning this main ticket to you as all subtickets also currently belong to you.

#28 Updated by BitingBird 2017-08-28 18:58:10

  • Target version changed from 2017 to 2018

#29 Updated by intrigeri 2018-08-17 18:16:40

  • Blueprint set to https://tails.boum.org/blueprint/HTTP_mirror_pool/#HTTPS

#30 Updated by nodens 2018-08-30 13:18:27

  • Target version deleted (2018)

#31 Updated by intrigeri 2019-03-07 15:27:50

  • related to Bug #12836: Adjust our Puppet code to only support HTTPS mirrors added

#32 Updated by intrigeri 2019-03-07 15:27:55

  • related to Bug #12833: Implement our masterplan about fallback DNS round-robin pool & HTTPS added

#33 Updated by intrigeri 2019-03-07 15:29:05

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)

We’ve solved this for the most common case (JS enabled) so I’ve unparented the remaining follow-up tasks and am closing this ticket.