#1 Updated by hans 2014-10-16 09:23:39

I should add, I’ve had good luck with these HTTPS Debian mirrors:

• mirrors.ece.ubc.ca
• mirror.cse.unsw.edu.au
• mirrors.kernel.org
• spout.ussg.indiana.edu

Also, there is more discussion on this topic here:

• https://lists.debian.org/debian-security/2014/07/msg00002.html
• https://lists.debian.org/debian-security/2014/09/msg00035.html

#3 Updated by hans 2014-10-17 08:41:57

Yeah, sorry, I knew that TAILS forces everything through Tor, I should have removed that part from this post (I’m boiling down discussions from elsewhere).

Given all of the proven fingerprinting approaches, like using the font list from the browser, I think its a safe assumption that the apt-get metadata will also provide similar fingerprinting opportunities. For basic TAILS use, this is all avoided since updates are done via ISO images. But once a user installs packages via apt-get, that changes.

So for example, TAILS does not use the default set of mirrors that a standard Debian install uses. It actually uses a large, customized set of apt sources, and it seems quite likely that the set of mirrors and the order in which they are run will provide a way to identify the system as TAILS.

As for identifying individual machines, apt-get sends a lot of metadata, like language that the system is using, which packages need updates, etc. On top of the set of mirrors used, there is potentially enough info there to fingerprint the individual machine.

So then the question is how the apt-get downloads map to different Tor circuits. If all of the traffic from a given apt-get session goes over a single Tor circuit, then the exit node, the mirror server, and any network observer that can see the traffic between those two can use that as the fingerprint.

#4 Updated by hans 2014-10-17 09:14:13

To expand on this, if TAILS fetched all of its apt sources (wheezy, backports, testing, etc) via HTTPS from the same mirror (e.g. mirrors.kernel.org), then the exit node and network observer could not really distinguish which distro was making the connection since it hosts many distro mirrors. I think this setup would also prevent the custom pattern of repos from being distinguished since it would just show as downloading some series of files, and those files could be packages, repo metadata, perl modules, source tarballs, etc.

#5 Updated by sajolida 2014-10-18 00:37:55

Regarding the set of custom APT sources, you said that this would allow
identifying the system as Tail.

First of all, we use circuit isolation by default for APT as explained
in our design document:

https://tails.boum.org/contribute/design/stream_isolation/

So all the connections to different APT sources will not be processed by
a single exit node.

Furthermore, it is already pretty hard to hide the fact that you are
using Tails in the first place. See
https://tails.boum.org/doc/about/fingerprint/ so this is not something
that we can guarantee. We would like to improve on this situation to
hide Tails usage on the local network which is much more problematic
than to the exit node. Note that your issue only addresses
fingerprinting by the exit node.

In a similar way, we do tons of networking activity through the Tor
network that can only be Tails networking activity (checking for
upgrade for example). So I don’t really understand what kind of concrete
and new attacks, being able to spot some APT traffic as coming from
Tails would allow… Could you elaborate on this?

#6 Updated by hans 2014-10-22 18:35:49

I didn’t realize that apt-get was setup to use separate Tor circuits for each host/port. That definitely helps quite a bit.

It looks to me that TAILS uses four hosts for apt sources:

1. ftp.us.debian.org (wheezy, wheezy-backports, wheezy-proposed-updates, testing, unstable, experimental)
2. security.debian.org (wheezy/updates)
3. deb.torproject.org (wheezy, sid)
4. deb.tails.boum.org (1.2)

So the Tor stream isolation would isolate the traffic to/from those hosts. That reduces the identification of TAILS, since Debian developers and pro users often include multiple Debian releases on a machine, like TAILS does. I think there will still be cases where there could be a recognizable, individual fingerprint of packages from looking at the traffic only on ftp.us.debian.org or security.debian.org. That fingerprint would be unique to the given instances of TAILS, based on extra packages that users have installed. It is not uncommon to receive lots of updates from each stable point release (7.1, 7.2, etc), that would probably be the area of highest risk of fingerprinting. Using an HTTPS connection to the mirror would reduce that risk.

It is also not uncommon to get regular updates of lots of packages from security.debian.org. security.debian.org is only available via HTTP, but some of the HTTPS mirrors also include wheezy/updates, so that could be put as a higher priority apt source. That makes apt-get get as many security updates via the HTTPS mirror, while only getting the newest ones from http://security.debian.org, i.e. the ones that have not synced to the mirrors yet.

#7 Updated by hans 2014-10-22 18:42:28

Another possibility is to use a specific mirror for each Debian section, i.e. something like:

deb https://mirrors.kernel.org/debian wheezy main
deb https://mirror.cse.unsw.edu.au/debian wheezy-backports main
deb https://mirrors.ece.ubc.ca/debian wheezy-proposed-updates main
deb https://spout.ussg.indiana.edu/debian testing main
deb https://mirrors.kernel.org/debian unstable main
deb http://ftp.us.debian.org/debian experimental main
deb http://security.debian.org/ wheezy/updates main

This reminds me, since the user can possibly install different versions of a package from different sections (wheezy, backports, experimental, etc.), that will generate more unique fingerprint information.

#9 Updated by hans 2014-10-26 18:18:49

I’ve always been a defensive thinker, so I’m particularly good at devising concrete attacks. In this case, I can see metadata that can provide strong fingerprinting opportunities. Plus, it turns out that it is straightforward and relatively low risk to fix this issue. I don’t think its high priority, but I also don’t think it should be ignored.

#10 Updated by sajolida 2014-10-28 17:21:06

To be more explicit, I’m not sure that being able to fingerprint APT
traffic coming from Tails on the exit node is a real threat.

First of all, a vast majority of Tails users won’t ever run apt-get.

Second, even if an exit node is able to identify apt traffic as coming
from Tails, the OpenPGP mechanisms of APT would prevent it from
providing rogue packages or malware. So remote exploit is not a risk in
that case I think.

In case a Tails user installs custom software, at most, the exit node
could tell which software packages are installed by a same Tails user.
But without knowing where this user since it comes through Tor and who
she is. Also note that if this user is installing those packages on a
regular basis she might have them stored in the persistent volume and
only download them if a new version appears. That limits the
possibilities of time correlation on those download.

Honestly, I think that all of that is no big deal.

#11 Updated by hans 2014-10-28 17:56:20

I agree that this only applies to the minority of people who are installing extra packages. But if that is a fully supported feature, it should be vetted as much as the rest of TAILS.

Here’s one very specific threat that only involves apt: using the fingerprint provided to the mirror to identify a specific computer, a QUANTUMINSERT-style attack could then be used to redirect that computer’s traffic intended for ftp.us.debian.org. Since TAILS uses HTTP and a static host as the main apt mirror, it makes it easy to track and run this kind of attack on. The software inserted could then check for identifying information on the local computer, then uninstall if it was the wrong target.

If there was a published update for apt, then this would make this attack much easier. There would be an delay from when the apt update is published to ftp.us.debian.org and the time that TAILS received the update. So when apt connects and asks for the update, the attacker knows that the machine is vulnerable.

This kind of attack could also be applied by someone who gains access to ftp.us.debian.org.

#12 Updated by sajolida 2014-10-29 19:13:45

APT packages are cryptographically signed by their developers using
OpenPGP. They are verified by the client before installation. So
delivering malware through rogue packages is not really an issue that
depends on whether HTTP or HTTPS in use to download them.

#13 Updated by hans 2014-10-30 19:38:37

I didn’t mean delivering a bad package, I meant exploiting a vulnerability in apt as it connects via HTTP.

#14 Updated by intrigeri 2014-10-31 19:24:39

Wow, what a discussion. It seems to have drifted from “metadata leakage” to “exploiting remote security issues in APT”, I see people not understanding each other, and I see too little threat modeling for my taste. I think we really need a summary. I’ll try to provide a not-too-biased one.

First of all, it seems that I should debunk some myths:

• “APT packages are cryptographically signed by their developers using OpenPGP. They are verified by the client before installation.” => yes and no. The OpenPGP signature created by whoever uploads a package to Debian is not visible by the end-user. What the end-user’s APT sees is cryptographic signatures of the index files, created by the Debian infrastructure.
• APT’s cryptographic code is not perfect. Very recently (read: in the last few weeks), quite a few issues of the “doesn’t properly verify signatures” were discovered. It wasn’t the first time, and I wouldn’t be surprised if there were more to come. So, some defense in depth can definitely make sense IMO.

Below, I’ll be assuming an attacker who:

• can’t forge arbitrary X.509 certificates
• may run Tor exit nodes
• may run Debian mirrors
• may sit in between Tor exit nodes and Debian mirrors

I’ll also assume that apt-transport-https correctly verifies X.509 certificate chains.

fingerprinting Tails users

apt-get update

• currently: probably leaks to the attacker that a Tails user is doing apt-get update using that exit node
• over HTTPS, minimal number of mirrors: basically the same, since the traffic patterns (size of the downloaded files, etc.) are easily fingerprintable, I guess
• over HTTPS, spread over a maximal number of mirrors: a bit more complicated, since circuit isolation makes it so we leak less metadata to attackers; but still, that’s someone using APT to talk to HTTPS Debian mirrors over Tor… that is, most likely a Tails user

=> not worth bothering; we’re not trying to protect against this attack anyway.

apt-get install pkg1 pkg2 pkg3 …

Here, the attacker wants to fingerprint a given Tails user who has configured “additional software packages”.

Below, I’ll assume that most packages come from the primary Debian archive (currently: ftp.us.d.o, Wheezy, main+contrib+non-free), and that the list of installed packages coming from this single archive is enough to identify a given Tails user’s list of “additional software packages”. It’s the worst case.

• currently: an attacker sitting somewhere in between ftp.us.d.o and the exit node used to reach it (inclusive) can probably notice whenever this user boots Tails; this can perhaps be used for confirmation attacks; however, this is somewhat mitigated by circuit isolation (if what the attacker wants to confirm is not publicly visible, then they also have to monitor the traffic they want to confirm on another side of the Internet)
• HTTPS, regardless of the number of mirrors: basically the same, since the traffic patterns (size of the downloaded files, etc.) are easily fingerprintable, I guess

=> not worth bothering

exploiting remote exploits in APT

• currently: the attacker can exploit the default HTTP transport, and APT code that handles the downloaded data
• over HTTPS, regardless of the number of mirrors: the attacker can’t exploit the default HTTP transport anymore; instead, they can now exploit apt-transport-https; and anyway, if they control one of the configured APT repositories, they can still exploit the APT code that handles the downloaded data

=> it could be worth using HTTPS

exploiting signature validation issues in APT

This applies to both apt-get update and apt-get install.

• currently: easy
• over HTTPS, regardless of the number of mirrors: the attacker needs to control at least mirror configured to be used in Tails

=> it could be worth using HTTPS

(Temporary) conclusion

• I see no serious fingerprinting attack that’s part of our threat model.
• If apt-transport-https’ code is at least as solid as APT’s default HTTP transport, then it could be worth using HTTPS to protect against security issues in APT.
• It doesn’t seem useful to spread the load over several mirrors (and taking the risk of desynchronization, and more maintenance work).

So, let’s say we do want to use HTTPS to connect to as many APT repositories as possible:

• Who verifies that the apt-transport-https code is solid enough?
• Is it compatible with using apt-cacher-ng on build boxes? Apparently apt-cacher-ng doesn’t support proxying HTTPS, so we probably have to use a different APT sources configuration at build time from the one configured at runtime. Doable, but needs more patches.
• Who writes the patches?

#15 Updated by aguelier@ruggedinbox.com 2014-11-08 00:17:31

These are general vulnerabilities that affect the entire clearnet. Facebook has rightly ascertained that the best way to fix it is by running a Tor hidden service and any other site that doesn’t want to be vulnerable in this way should really be doing the same (including Debian, the Tor project, Tails, etc).

#16 Updated by hans 2014-11-08 21:25:16

@intrigeri that was a good breakdown of the issues, I agree with your conclusions.

It would be good to be able to access a Debian mirror via a Tor Hidden Service, but there are no stable mirrors with .onion addresses that I could find. I run a security-only mirror on a hidden service, but that’s not something for TAILS to rely on, its meant only as a test to encourage mirror operators to set up .onion addresses on their mirrors.

So further discussion about reducing fingerprinting of apt traffic and Tor Hidden Services for Debian mirrors should happen elsewhere, and this thread should be focused on the points that intregeri outlined. I hope to have some time to look into apt-transport-https more in the not too distant future, but don’t wait on me.

#19 Updated by hans 2016-01-27 10:00:25

So weasel added a Tor Hidden Service as an official Debian mirror:
http://richardhartmann.de/blog/posts/2015/08/25-Tor-enabled_Debian_mirror_part_2/

So this could greatly improve the issues discussed here by using the onion address to provide end-to-end encryption. I think TAILS is already using Tor TransProxy, so it should just be a matter of making sure that apt is also forced over that, then no other software should be needed (e.g. apt-transport-tor or apt-transport-https)

#20 Updated by intrigeri 2016-01-27 10:39:10

> I think TAILS is already using Tor TransProxy,

FYI Tails uses apt-transport-tor.

#22 Updated by hans 2016-07-31 13:59:26

FYI, weasel just added an official Tor Onion Service mirror of the security archive too. If that is listed as an apt source before security.debian.org, then most security updates will come via the Tor Onion Service, with the classic HTTP mirror as a failsafe backup.

https://onion.debian.org/

#23 Updated by intrigeri 2016-10-15 15:13:45

> Is it compatible with using apt-cacher-ng on build boxes? Apparently apt-cacher-ng doesn’t support proxying HTTPS, so we probably have to use a different APT sources configuration at build time from the one configured at runtime. Doable, but needs more patches.

I’ve looked into it. apt-cacher-ng now supports passing HTTPS connections through, but of course it can’t cache the corresponding data, so indeed we cannot afford using apt-transport-https at build time. Now, we already reconfigure APT source at boot time (source:config/chroot_local-includes/lib/live/config/1500-reconfigure-APT) so we could as well enable HTTPS there.

#25 Updated by hans 2016-12-17 11:19:08

For the record, here is a real world exploit that using Tor and/or HTTPS for all apt sources would protect against:
https://www.debian.org/security/2016/dsa-3733