Feature #14605

Improve documentation on "Manually copying your persistent data to a new USB stick"

Added by garrettr 2017-09-06 19:56:20 . Updated 2019-10-03 20:18:23 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Persistence
Target version:
Tails_4.0
Start date:
2017-09-05
Due date:
% Done:

0%

Feature Branch:
Type of work:
End-user documentation
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

The closest thing to documentation on backing up a Tails persistent volume (that I have been able to find) is Manually copying your persistent data to a new USB stick. Following these instructions results in the creation of a bootable backup of a Tails drive with a persistent volume.

While effective, there are a couple of “gotcha’s” in this document that can lead to confusing outcomes:

  • In Create a new USB stick, Step 3 says to “Enable again on this new USB stick the persistence features of your choice.” Some users may not remember or understand which persistence features they have enabled, and may be confused as to what to do at this step. If they forget to enable a persistence feature on the backup, the corresponding data will not be backed up after following the subsequent instructions.
  • Manually copying the files with Nautilus and confusing and error-prone:
    • If a user has all of the persistence features enabled, they may try to select all or drag-and-drop everything from /media/amensia/TailsData/ into /live/persistence/TailsData_unlocked. Nautilus does not preserve file ownership and permissions when copying, so if they accidentally copy one of the files owned by the tails-persistence-setup user (e.g. persistence.conf or live-additional-software.conf), it will end up owned by the incorrect user on the backup drive, which prevents the Persistent folders from being symlinked correctly due to the permissions checks documented in the Security section of the Tails persistence design document.
      • I believe this is the root cause of bug reports where users report that the “persistent volume has disappeared, but I’m still prompted for password when booting Tails,” e.g. in this r/tails thread.
    • Nautilus prints numerous obscure warning messages to the Root Terminal, which undermines user confidence.
    • Asking users to click through various warning dialogs from Nautilus about merging/replacing/skipping files also undermines user confidence (“did I actually back up all of my important data?”).
  • There is no documented “restore” process that explains how users can recover their persistent data from the bootable backup created from this documentation.

I’ve been trying to develop a process for reliably backing up and restoring Tails persistent volumes, and I have a proposed modification to this document that I think might fix all of these issues:

  • Retain the Create a new USB stick section, but modify Step 3 to say it doesn’t matter which persistence features you enable on the backup (read on to understand why).
  • Retain the Mount the old persistent volume section without changes.
  • In the Copy your old files to the new persistent volume section, replace steps 2-11 with a single command, run in the Root Terminal: rsync -avh /media/amnesia/TailsData/ /live/persistence/TailsData_unlocked/.

There are several benefits to this change:

  • rsync -a will retain the ownership and permissions of the copied files. That allows us to copy persistence.conf without creating ownership issues that will bork the symlinking of persistent directories on the backup drive.
    • Since we are copying persistence.conf, there is no need for the user to manually configure the persistent settings on the destination to match the source, since the enabled persistent settings are completely described in persistence.conf.
  • rsync -v prints the files that are copied, which allows a cautious user to audit the backup and ensure that their important data was indeed copied to the backup volume.
  • In my testing, rsync does not print any confusing error or warning messages, unlike Nautilus.
  • It requires a user to use the CLI, which it seems the original document may have been trying to avoid, but was also unable to prevent: CLI interaction is required to launch Nautilus in Step 2, and then again to fix permissions in Step 11. This changes replaces two CLI interactions with one, which seems preferable in my view.
  • Dramatically simplifies the Copy your old files section, reducing it from 11 separate steps to just 2.
  • The restoration step, which is currently not documented at all, is trivial: just rsync with source and dest swapped.
  • Users can efficiently update their backups by repeating the steps in the Rescue your files from the old Tails USB stick section, adding the --delete flag to the invocation of rsync.

I am happy to contribute these changes, but first want to make sure they are viable. Would the Tails maintainers, especially those who have a deeper understanding of the Persistence feature than I, welcome this change? Are there any potential gotchas with my proposal that I’ve failed to recognize? As always, I appreciate your time and consideration.

:sajolida:

Subtasks

Related issues

Related to Tails - Bug #10391: Useless step in https://tails.boum.org/doc/first_steps/persistence/copy/ Rejected 2015-10-19
Related to Tails - Bug #14851: Instructions to backup Tails are wrong in terms of permission Rejected 2017-10-14
Related to Tails - Bug #15685: Test manually creating a disk image as a backup technique Resolved 2018-06-25
Related to Tails - Feature #13457: Test backup script by a2 Resolved 2017-07-12
Has duplicate Tails - Feature #14624: Document how to migrate persistence to a new USB stick or update the system partitions of an existing device Duplicate 2017-09-12
Has duplicate Tails - Feature #12214: Document a way to manually backup persistent data Duplicate 2017-02-06
Blocked by Tails - Bug #16789: "Unable to access" error when trying to open the Persistence of another Tails from the Files utility Resolved
Blocks Tails - Feature #16711: Core work 2019Q3 → 2019Q4: Technical writing Resolved 2016-01-08

History

#1 Updated by Anonymous 2017-09-07 09:16:47

  • Subject changed from "Manually copying your persistent data to a new USB stick" documentation simplification to Improve documentation on "Manually copying your persistent data to a new USB stick"

Adding sajolida & cbrownstein as watchers so they can read through your proposal and comment on it.

#2 Updated by sajolida 2017-09-12 09:23:03

  • related to Feature #14624: Document how to migrate persistence to a new USB stick or update the system partitions of an existing device added

#3 Updated by sajolida 2017-09-23 10:52:38

  • related to Feature #12214: Document a way to manually backup persistent data added

#4 Updated by sajolida 2017-09-23 10:53:05

  • related to Bug #10391: Useless step in https://tails.boum.org/doc/first_steps/persistence/copy/ added

#5 Updated by sajolida 2017-09-23 10:59:49

#6 Updated by sajolida 2017-09-23 11:01:00

  • Status changed from New to Confirmed

Hi Garrett! Thank you so much for write such a detailed proposal. We’ve had on our roadmap for a while to create a much better experience for backing up persistence, relying on a GUI tool instead of manual instructions. We actually submitted that for a grant this year but it got rejected, so it feels good to see that people like you care about backing up their Tails USB stick :)

Now, I have to admit that we are completely overwhelmed with work and desperately lacking resources, even for helping out with smaller changes like this. I really want to have a closer look at your proposal but I really don’t know when I’ll find the time to do that.

So please be patient, but this is so important it won’t silently fall into the cracks!

#7 Updated by sajolida 2017-10-02 18:17:03

  • blocked by deleted (Feature #13423: Core work 2017Q3: Technical writing)

#8 Updated by sajolida 2017-10-02 18:17:04

  • blocks Feature #14758: Core work 2017Q4 → 2018Q1: Technical writing added

#9 Updated by sajolida 2017-11-06 10:37:13

  • related to Bug #14851: Instructions to backup Tails are wrong in terms of permission added

#10 Updated by sajolida 2018-03-27 11:25:45

  • blocked by deleted (Feature #14758: Core work 2017Q4 → 2018Q1: Technical writing)

#11 Updated by sajolida 2018-03-27 11:25:57

  • blocks Feature #15411: Core work 2018Q2 → 2018Q3: Technical writing added

#12 Updated by sajolida 2018-03-27 11:26:01

I’m out of budget for this quarter.

#13 Updated by sajolida 2018-04-11 13:39:57

  • Assignee set to cbrownstein

Cody: I’m assiging this one to you as part of the big problem space “Persistent storage vs Backups”.
Please check which parts of it are the best to tackle first.

#14 Updated by sajolida 2018-06-25 17:14:34

  • related to Bug #15685: Test manually creating a disk image as a backup technique added

#15 Updated by sajolida 2018-08-04 10:00:40

  • Assignee changed from cbrownstein to sajolida
  • Target version set to Tails_3.9

Cody: if you don’t mind I’ll take over this problem space for some time since I’ve been working on Bug #15685 lately.

#16 Updated by sajolida 2018-08-13 13:59:58

  • Target version changed from Tails_3.9 to Tails_3.10.1

#17 Updated by Anonymous 2018-08-17 09:54:14

#18 Updated by sajolida 2018-09-11 18:02:44

  • blocked by deleted (Feature #15411: Core work 2018Q2 → 2018Q3: Technical writing)

#19 Updated by sajolida 2018-09-11 18:02:51

  • blocks Feature #15941: Core work 2018Q4 → 2019Q2: Technical writing added

#20 Updated by sajolida 2018-10-21 20:46:46

  • Target version changed from Tails_3.10.1 to Tails_3.11

#21 Updated by sajolida 2018-12-10 15:40:33

  • Target version changed from Tails_3.11 to Tails_3.12

#22 Updated by sajolida 2019-01-28 18:45:34

  • Target version changed from Tails_3.12 to Tails_3.13

#23 Updated by sajolida 2019-03-15 18:05:40

  • Target version changed from Tails_3.13 to Tails_3.14

#24 Updated by sajolida 2019-05-10 11:14:38

  • Target version deleted (Tails_3.14)

#25 Updated by sajolida 2019-06-07 17:03:40

  • blocked by Bug #16789: "Unable to access" error when trying to open the Persistence of another Tails from the Files utility added

#26 Updated by sajolida 2019-06-12 18:35:53

  • related to deleted (Feature #14624: Document how to migrate persistence to a new USB stick or update the system partitions of an existing device)

#27 Updated by sajolida 2019-06-12 18:35:55

  • has duplicate Feature #14624: Document how to migrate persistence to a new USB stick or update the system partitions of an existing device added

#28 Updated by sajolida 2019-07-18 16:54:43

  • blocked by deleted (Feature #15941: Core work 2018Q4 → 2019Q2: Technical writing)

#29 Updated by sajolida 2019-07-18 16:54:49

  • blocks Feature #16711: Core work 2019Q3 → 2019Q4: Technical writing added

#30 Updated by sajolida 2019-07-23 16:07:19

#31 Updated by sajolida 2019-09-03 12:55:18

  • Status changed from Confirmed to Needs Validation
  • Assignee changed from sajolida to cbrownstein
  • Target version set to Tails_3.17

Some months ago, I rewrote the whole page to do in the direction proposed by @garrettr. Thanks a lot for moving us in this direction! (despite our super slow pace…). It was then blocked by Bug #16789, which should now be solved in 3.16. I couldn’t test it in 3.16 because it’s not released just yet but it works already in 4.0~beta1.

Cody: Please have a look!

If more people want to have a look, you can read the Markdown source here:

https://git.tails.boum.org/tails/tree/wiki/src/doc/first_steps/persistence/copy.mdwn?h=doc/14605-backup

#32 Updated by sajolida 2019-09-03 13:33:44

  • related to deleted (Feature #12214: Document a way to manually backup persistent data)

#33 Updated by sajolida 2019-09-03 13:34:01

  • has duplicate Feature #12214: Document a way to manually backup persistent data added

#34 Updated by intrigeri 2019-09-12 14:25:05

  • Target version changed from Tails_3.17 to Tails_4.0

#35 Updated by cbrownstein 2019-09-26 21:17:44

  • Status changed from Needs Validation to In Progress
  • Assignee changed from cbrownstein to sajolida

I’ve pushed a bunch of changes to my branch:

https://0xacab.org/cbrownstein/tails/commits/doc/14605-backup

I’m distinguishing between Tails and Tails USB stick; Tails as referring to the OS and Tails USB stick as referring to the physical USB stick on which Tails is installed.

N.B. Under “Create or update your backup,” step 7, since the command is being executed in a Root Terminal, the command prompt is # rather than $.

I don’t know if that’s significant. If it is significant: do you want to use your CSS expertise to fix it or should I try to fix it?

#36 Updated by sajolida 2019-09-28 01:30:40

  • Status changed from In Progress to Resolved
  • Assignee deleted (sajolida)

Thanks for all the smart fixes!

I worked around the prompt issue with d9d62fc7ce. We might do something better some day in Feature #15120.

Same as Feature #15718#note-18: I merged it locally and will push it soon.

#37 Updated by sajolida 2019-09-30 17:50:07

  • Description updated

@garrettr: Hey Garrett, sorry for the huge delay but this is finally live:

https://tails.boum.org/doc/first_steps/persistence/copy/

How does it look to you?

#38 Updated by cbrownstein 2019-10-01 00:06:23

@sajolida: I pushed a very small change (74179afe) to my branch. Please merge it when you have a chance. Thanks!

#39 Updated by sajolida 2019-10-03 20:18:23

Sure, done!