Bug #12445

tails-installer should validate file integrity after unpacking the ISO on the sd-card

Added by arnaud 2017-04-13 11:28:47 . Updated 2018-10-09 10:22:11 .

Status:
Duplicate
Priority:
Low
Assignee:
Category:
Installation
Target version:
Start date:
2017-04-13
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
0
Affected tool:
Installer
Deliverable for:

Description

Dear Tails !

Short story

I installed the tails-installer on my Debian Stretch system. The package came from jessie-backports, as mentioned in the documentation. Then I installed Tails on a USB drive with the tails-installer. Then I booted Tails from the USB drive, and it failed. I turns out that the filesystem (/live/filesystem.squashfs) was corrupted.

I had a look at the code of the Tails installer. If I’m not mistaken, it unzips the ISO on the USB drive using 7zip, but afterwards there’s no check of the integrity of the data: the installer doesn’t read again the data from the USB drive to ensure it’s correct. It assumes a flawless hardware/software.

I think that part could be improved. Given the fact that the user might use any USB drive, of unknown age and quality, ensuring the integrity of the data that has been written wouldn’t hurt. I would even say that it’s a must.

Long Story

Here’s the whole story.

After booting my USB drive, Tails got stuck at the Greeter. I had just the “waiting” cursor spinning forever, and no Greeter window would appear, only an empty blue background. I rebooted and switched to the log view (pressing <Esc> during the boot), which unveiled some Squashfs errors:

SQUASHFS error: Unable to read...

Back on my development machine, I did some investigation. I mounted the original Tails ISO, mounted my USB drive, and compared the two files.

$ sudo mount /dev/sdb1 /mnt/usb/
$ sudo mount ~/Downloads/tails-amd64-3.0~beta3.iso /mnt/iso/
$ cmp /mnt/usb/live/filesystem.squashfs /mnt/iso/live/filesystem.squashfs
/mnt/usb/live/filesystem.squashfs iso/live/filesystem.squashfs differ: byte 682254337, line 2654053

Files differ ! They should be identical !

So, after that, I did more investigation, and I’m still in the process of doing more tests. To make it short: I can reproduce the bug (aka a corrupted file) almost 100% of the time, but only under particular conditions.

  • I can only reproduce with one USB drive.
    • I tested 2 USB drives and on SD Card.
    • The guilty USB drive is a Kingston DataTraveler 100 G3 32GB.
    • It’s brand new.
    • It seems to be authentic according to the packaging (see Kingston Product Verification - Colour Shift.
    • It seems to be authentic according to the F3 tools.
  • I can only reproduce when it’s mounted with the `flush` option.
    • The flush option appeared in 2005: https://lwn.net/Articles/165733/.
    • It is described in man mount, and is an option specific to the fat filesystem.
    • I didn’t see any explicit mention in the tails-installer code, so I believe this mount option is enabled by default by the underlying library (udisks2).

The data corrupted seems to be at a different offset in the file, but I didn’t investigate much on that yet. I’m still doing more tests before reaching out to the fat maintainer (no joke here).

Some thoughts

I don’t know how many people are impacted by that, probably very few. In the end I might just be unlucky enough to have a USB drive with a rare firmware bug. However, it’s well known that fake SD Cards and USB Drives abound (Bunnie’s blog is a reference on the matter). Plus, USB drives can have a hard life in backpacks, pockets, drawers and so on. So hardware failure happens.

Furthermore, this kind of failure are very vicious, because the end-user will just see a random part of the system not working, without knowing that the underlying cause is a read-error in the filesystem. Therefore, they end up firing inaccurate bug reports that make life hard for developers.

Because of that, I think that double-checking what’s written in the installer is worth it, both users and devs benefit from it.

About the flush option

About the `flush` mount option, I think it’s worth considering NOT using it anymore. Apart from the bug that is discussed here, I took time to measure the write speed during my tests. I ran my tests on 3 devices, and here’s what I observe regarding the write speed.

  • USB3 drive: 6 Mib/s with flush, 11 Mib/s without (almost x2 !)
  • USB2 drive: 1 MiB/s with flush, 3 MiB/s without (x3 !)
  • MMC sdcard: huge variance, between 4 MiB/s to 8MiB/s, seems slightly quicker without flush

I don’t know whether there are official measures and numbers available out there.

I don’t know if this `flush` option is widely used and tested. If it’s the default from udisks2, I guess it’s widely used, but it’s just a guess.

I think it’s not needed in our use case, since the installer can explicitly sync the data after it’s written, and explicitly tells the user when the USB drive is safe to remove.

Testing

In case some people feel like validating their filesystem after installing it with the Tails installer, there’s an important detail to know. You must ALWAYS flush the caches before comparing files, to force the system to physically read the memory. If you don’t do that, the kernel will use the data that is still in RAM instead of reading from the device.

Flushing the caches is easy to do, just a one-liner.

echo 3 | sudo tee /proc/sys/vm/drop_caches

Run that before each cmp invokation, and you will be safe.

Config

My test config:

$ uname -a
Linux debian 4.9.0-2-amd64 #1 SMP Debian 4.9.18-1 (2017-03-30) x86_64 GNU/Linux

Devices that fail:

  • USB3 32GB “DATATRAVELER 100 G3” from Kingston Technology.

Devices that have no problem:

  • USB2 4GB “POP KEY” from Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.)
  • MMC SDCARD 16GB “SAMSUNG 16 EVO”

Subtasks


Related issues

Related to Tails - Feature #15292: Distribute a USB image Resolved 2016-04-14 2019-01-29
Related to Tails - Feature #7496: Make it possible to verify the integrity of a Tails USB device In Progress 2014-07-06
Is duplicate of Tails - Bug #16030: SquashFS errors during boot lead to false-positives on graphics card error reports Confirmed 2018-10-03

History

#1 Updated by mercedes508 2017-04-17 12:06:24

  • Assignee set to anonym
  • Priority changed from Elevated to Normal

#2 Updated by Anonymous 2017-06-27 15:57:18

  • Subject changed from Tails-installer lacks validation after unpacking the ISO on the sd-card to tails-installer should validate file integrity after unpacking the ISO on the sd-card

#3 Updated by Anonymous 2017-06-27 15:57:30

  • Status changed from New to Confirmed
  • Priority changed from Normal to Low
  • Starter set to No

#4 Updated by Anonymous 2017-06-27 15:57:54

  • Affected tool set to Installer

#5 Updated by Anonymous 2018-01-15 16:43:27

@kurono: would you also like to take a look? I think it’s a good idea :)

#6 Updated by kurono 2018-01-15 18:28:33

  • Assignee changed from anonym to kurono
  • Target version set to Tails_3.8

#7 Updated by intrigeri 2018-06-26 16:27:56

  • Target version changed from Tails_3.8 to Tails_3.9

#8 Updated by kurono 2018-07-10 20:37:53

  • Target version deleted (Tails_3.9)

#9 Updated by intrigeri 2018-07-10 21:11:14

> I installed the tails-installer on my Debian Stretch system. The package came from jessie-backports, as mentioned in the documentation.

Argh, that’s because the doc is not up-to-date yet: Feature #14677 has the updated one. tl;dr: install from stretch-backports, that has something more recent since May.

> I had a look at the code of the Tails installer. If I’m not mistaken, it unzips the ISO on the USB drive using 7zip, but afterwards there’s no check of the integrity of the data: the installer doesn’t read again the data from the USB drive to ensure it’s correct. It assumes a flawless hardware/software.

I’m pretty sure you’re not mistaken.

> I think that part could be improved. Given the fact that the user might use any USB drive, of unknown age and quality, ensuring the integrity of the data that has been written wouldn’t hurt. I would even say that it’s a must.

The long story was a good read, thanks for the investigation and for the report!

I agree that some integrity check would save users, help desk and foundations team quite some time and headaches: we see too many unreproducible issues that I’ve often suspected might be caused by such issues, but getting a user to compute and send us checksums is tedious, not fun for anyone involved, and rarely works in the real world. If Tails can tell itself that it has no chance to work properly, then it should tell the user and not just behave in a weird way as it currently does.

Now, I have to clarify something about the “tails-installer should” part of the ticket description: once the “USB image” project is completed (likely by the end of January 2019), Tails Installer will be used by a tiny minority of Tails users: those who use the cloning feature to create new Tails USB sticks from a running one. Nobody else will ever see it as initial installation will 1. be done with Etcher (Windows, macOS) or with GNOME Disks (Linux); 2. produce a “final Tails” instead of an “intermediate” one.

So implementing this integrity check in Tails Installer itself might not be the best use of our time wrt. having more beneficial impact on users.

If Etcher already has an integrity verification feature, great: most users will already be covered and this ticket can be closed (unless we want to do something else, specifically for Linux users). Else, I think the only time we’ll be in a position to verify integrity will be first boot. IIRC live-boot already has a feature to do exactly that but I’m not 100% sure. For example, last time I checked the Fedora live image does an integrity check on boot. One problem is that it takes time and slowing down every boot that much probably conflicts with “Make it easier to switch between a Tails contextual identity and another identity outside of Tails” (https://tails.boum.org/blueprint/strategic_planning/). So we would need to flag the stick as having booted once successfully so that we don’t test integrity again on next boots. We already considered this option and decided it was acceptable (Feature #8864, that also has some implementation ideas).

#10 Updated by kurono 2018-07-13 16:00:34

  • Assignee deleted (kurono)

#11 Updated by Anonymous 2018-08-17 09:28:47

If we work on the USB image project, we should close this ticket.

#12 Updated by Anonymous 2018-08-17 09:29:03

#13 Updated by Anonymous 2018-08-18 13:47:15

  • related to Feature #7496: Make it possible to verify the integrity of a Tails USB device added

#14 Updated by intrigeri 2018-10-09 10:22:11

  • Status changed from Confirmed to Duplicate

Rejecting in favour of Bug #16030, because Tails Installer is mostly going away and fixing this in Tails Installer won’t address the underlying problem for people using Windows and Mac.

#15 Updated by intrigeri 2018-10-09 10:22:23

  • is duplicate of Bug #16030: SquashFS errors during boot lead to false-positives on graphics card error reports added