Tails

Dear Tails !

Short story

I installed the tails-installer on my Debian Stretch system. The package came from jessie-backports, as mentioned in the documentation. Then I installed Tails on a USB drive with the tails-installer. Then I booted Tails from the USB drive, and it failed. I turns out that the filesystem (/live/filesystem.squashfs) was corrupted.

I had a look at the code of the Tails installer. If I’m not mistaken, it unzips the ISO on the USB drive using 7zip, but afterwards there’s no check of the integrity of the data: the installer doesn’t read again the data from the USB drive to ensure it’s correct. It assumes a flawless hardware/software.

I think that part could be improved. Given the fact that the user might use any USB drive, of unknown age and quality, ensuring the integrity of the data that has been written wouldn’t hurt. I would even say that it’s a must.

Long Story

Here’s the whole story.

After booting my USB drive, Tails got stuck at the Greeter. I had just the “waiting” cursor spinning forever, and no Greeter window would appear, only an empty blue background. I rebooted and switched to the log view (pressing <Esc> during the boot), which unveiled some Squashfs errors:

SQUASHFS error: Unable to read...

Back on my development machine, I did some investigation. I mounted the original Tails ISO, mounted my USB drive, and compared the two files.

$ sudo mount /dev/sdb1 /mnt/usb/
$ sudo mount ~/Downloads/tails-amd64-3.0~beta3.iso /mnt/iso/
$ cmp /mnt/usb/live/filesystem.squashfs /mnt/iso/live/filesystem.squashfs
/mnt/usb/live/filesystem.squashfs iso/live/filesystem.squashfs differ: byte 682254337, line 2654053

Files differ ! They should be identical !

So, after that, I did more investigation, and I’m still in the process of doing more tests. To make it short: I can reproduce the bug (aka a corrupted file) almost 100% of the time, but only under particular conditions.

• I can only reproduce with one USB drive.
  • I tested 2 USB drives and on SD Card.
  • The guilty USB drive is a Kingston DataTraveler 100 G3 32GB.
  • It’s brand new.
  • It seems to be authentic according to the packaging (see Kingston Product Verification - Colour Shift.
  • It seems to be authentic according to the F3 tools.
• I can only reproduce when it’s mounted with the `flush` option.
  • The flush option appeared in 2005: https://lwn.net/Articles/165733/.
  • It is described in man mount, and is an option specific to the fat filesystem.
  • I didn’t see any explicit mention in the tails-installer code, so I believe this mount option is enabled by default by the underlying library (udisks2).

The data corrupted seems to be at a different offset in the file, but I didn’t investigate much on that yet. I’m still doing more tests before reaching out to the fat maintainer (no joke here).

Some thoughts

I don’t know how many people are impacted by that, probably very few. In the end I might just be unlucky enough to have a USB drive with a rare firmware bug. However, it’s well known that fake SD Cards and USB Drives abound (Bunnie’s blog is a reference on the matter). Plus, USB drives can have a hard life in backpacks, pockets, drawers and so on. So hardware failure happens.

Furthermore, this kind of failure are very vicious, because the end-user will just see a random part of the system not working, without knowing that the underlying cause is a read-error in the filesystem. Therefore, they end up firing inaccurate bug reports that make life hard for developers.

Because of that, I think that double-checking what’s written in the installer is worth it, both users and devs benefit from it.

About the flush option

About the `flush` mount option, I think it’s worth considering NOT using it anymore. Apart from the bug that is discussed here, I took time to measure the write speed during my tests. I ran my tests on 3 devices, and here’s what I observe regarding the write speed.

• USB3 drive: 6 Mib/s with flush, 11 Mib/s without (almost x2 !)
• USB2 drive: 1 MiB/s with flush, 3 MiB/s without (x3 !)
• MMC sdcard: huge variance, between 4 MiB/s to 8MiB/s, seems slightly quicker without flush

I don’t know whether there are official measures and numbers available out there.

I don’t know if this `flush` option is widely used and tested. If it’s the default from udisks2, I guess it’s widely used, but it’s just a guess.

I think it’s not needed in our use case, since the installer can explicitly sync the data after it’s written, and explicitly tells the user when the USB drive is safe to remove.

Testing

In case some people feel like validating their filesystem after installing it with the Tails installer, there’s an important detail to know. You must ALWAYS flush the caches before comparing files, to force the system to physically read the memory. If you don’t do that, the kernel will use the data that is still in RAM instead of reading from the device.

Flushing the caches is easy to do, just a one-liner.

echo 3 | sudo tee /proc/sys/vm/drop_caches

Run that before each cmp invokation, and you will be safe.

Config

My test config:

$ uname -a
Linux debian 4.9.0-2-amd64 #1 SMP Debian 4.9.18-1 (2017-03-30) x86_64 GNU/Linux

Devices that fail:

• USB3 32GB “DATATRAVELER 100 G3” from Kingston Technology.

Devices that have no problem:

• USB2 4GB “POP KEY” from Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.)
• MMC SDCARD 16GB “SAMSUNG 16 EVO”