Tails

#1 Updated by intrigeri 2017-06-15 06:24:14

sajolida: this seems simple enough for me to handle myself, and then let you review. What do you think?

#5 Updated by sajolida 2017-09-11 18:50:50

Background info: We already changed the size of the system partition back in February 2017 when we switched from 1.5 GB to 2.5 GB (either 1.3 or 1.4). Note that IUK were introduced in 0.22. Back then we didn’t write anything fancy to help people transition and it seems like the transition was quite smooth.

Still, since then our user base has doubled :)

I couldn’t refrain from doing stats on the usage of persistence and found something surprising:

	2015Q1	2015Q1	2017Q2	2017Q2
dvd	22	17%	30	7%
dd	6	4%	15	3%
usb+persist	45	35%	154	38%
usb-persist	35	27%	180	44%
toram	3	2%	1	0%
qemu	5	3%	5	1%
vbox	8	6%	11	2%
vmware	1	0%	1	0%
total	126		402

NB: I don’t know how to distinguish USB sticks installed with UUI or Tails Installer, but only the ones installed with dd.

So the usage of USB has increased (4+35+27=) from 66% in 2015Q1 to 85% in 2017Q2 but the share of these with persistence has actually decreased from (45÷(6+45+35)) 52% in 2015Q1 to 44% in 2017Q2 (154÷(15+154+180)).

This data comes from WhisperBack report so it’s very limited and probably very biased.

Back to the topic, in Feature #12707 we identified that the migration will only be problematic for people trying to do “Clone & upgrade” from a USB stick that has more data in its system partition that can fit on the destination USB stick.

But the objective of this migration is to be beneficial to regular users by allowing them to avoid too frequent manual upgrades. And if we really want people to do that, they will need to understand why to bother and need guidance through the process. This guidance could be pointed to from Tails Installer if we detect a too small system partition (< 4 GB) and from the release notes, maybe from other places (Tails Upgrader?).

I imagined what would be the step of a user going through this. I imagined that the user could want to either:

• A. Take this as an opportunity to migrated to a bigger USB stick (“I was stuck with a smallish USB stick, now they are cheaper and it’s a good time to change.”).
• B. Keep the same USB stick and make its system partition bigger (“I bought a very big and fancy USB stick for my Tails and want to make the most out of it!”).

We should point to the /doc/first_steps/persistence/copy to rescue the files.

Note that everybody going through this will have to go on the command line (see the end of /doc/first_steps/persistence/copy).

We should advertise A as slightly easier than B (one copy of persistent files instead of two).

1. Proactive users

People might want to be proactive and go through this before facing a manual upgrade. This could be advertised in the release notes, maybe in Tails Upgrader, etc.

• A. Buy a new and bigger USB stick
  • 1. Clone old on new
  • 2. Restart on new
  • 3. Configure persistence on new
  • 4. Restart on new
  • 5. Rescue files from old (fix perms on the terminal)

• B. Extend the system partition on the same USB stick
  • 1. Clone old on intermediary
  • 2. Restart on intermediary
    • Intermediary must have enough space for all persistent data!
  • 3. Configure persistence on intermediary
  • 4. Restart on intermediary
  • 5. Copy files to intermediary
  • 6. Clone intermediary on old
  • 7. Restart on old
  • 8. Configure persistence on old
  • 9. Restart on old
  • 10. Rescue files from intermediary (fix perms on the terminal)

1. Reactive users

When doing a manual upgrade, we could detect too small system partitions and advertise the migration from Tails Installer during step #6 (“Upgrade by cloning”). Actually, before people do the actual upgrade and replace it with:

• A. Buy a new and bigger USB stick (the same as for “proactive” actually…)
  • 1. Clone intermediary on new
  • 2. Restart on new
  • 3. Configure persistence on new
  • 4. Restart on new
  • 5. Copy files to new (fix perms on the terminal)

• B. Extend the system partition on the same USB stick (pretty much the same as for “proactive” starting on step 3)
  • 3. Configure persistence on intermediary
    • Intermediary must have enough space for all persistent data!
  • 4. Restart on intermediary
  • 5. Copy files to intermediary
  • 6. Clone intermediary on old
  • 7. Restart on old
  • 8. Configure persistence on old
  • 9. Restart on old
  • 10. Rescue files from intermediary (fix perms on the terminal)

I don’t think it’s realistic to have detailed instructions for all this but it would be good to explain the workflow of each scenario. To make things a bit simpler we could try to have both the “proactive” and “reactive” scenarios on a single page (“/upgrade/system_partition”?) and point /upgrade/tails to them.

Last but not least, the error case when the source system partition has more data than what is possible to write on the target system partition (see Feature #12707), could be explained in /upgrade/clone which is the scenario where people clone from their friends.

To finish with, I’m really overwhelmed with work right now and don’t know how realistic doing all this is but I needed to do this analysis to understand better what we’re talking about. I’d like your opinion on what you think is worth doing in time for 3.2. Note that we could also do this work after 3.2 even though I’m not sure that postponing will bring me much more availability.

#6 Updated by intrigeri 2017-09-12 07:09:42

> Background info: We already changed the size of the system partition back in February 2017 when we switched from 1.5 GB to 2.5 GB (either 1.3 or 1.4).

I think this date is incorrect: my research suggests we instead did that in Tails 0.22 (December, 2013) by upgrading liveusb-creator to 3.11.6-24 that contains commit fea8d4d921fb115d806d882975d64aa2f8e61e59 (the first version of liveusb-creator that bumps to 2500 MiB is 3.11.6-20 which was never included in a Tails release).

> I couldn’t refrain from doing stats on the usage of persistence and found something surprising:

Thanks! :)

> | | 2015Q1 | 2015Q1 | 2017Q2 | 2017Q2 |
> | dvd | 22 | 17% | 30 | 7% |
> | dd | 6 | 4% | 15 | 3% |
> | usb+persist | 45 | 35% | 154 | 38% |
> | usb-persist | 35 | 27% | 180 | 44% |
> | toram | 3 | 2% | 1 | 0% |
> | qemu | 5 | 3% | 5 | 1% |
> | vbox | 8 | 6% | 11 | 2% |
> | vmware | 1 | 0% | 1 | 0% |
> | total | 126 | | 402 | |

> So the usage of USB has increased (4+35+27=) from 66% in 2015Q1 to 85% in 2017Q2 but the share of these with persistence has actually decreased from (45÷(6+45+35)) 52% in> 2015Q1 to 44% in 2017Q2 (154÷(15+154+180)).

I don’t find it particularly surprising: one can also read these numbers as “60% of those who want Tails without persistence migrated from DVD to USB”. There are plenty of good reasons to make this move without starting to use persistence, so why not? :)

> Note that everybody going through this will have to go on the command line (see the end of /doc/first_steps/persistence/copy).

Right. By the way we’re sometimes told this documentation has issues, see e.g. the ticket we got about it last week (I think help desk reassigned it to you so it’s probably on your radar).

> To finish with, I’m really overwhelmed with work right now and don’t know how realistic doing all this is but I needed to do this analysis to understand better what we’re talking about. I’d like your opinion on what you think is worth doing in time for 3.2. Note that we could also do this work after 3.2 even though I’m not sure that postponing will bring me much more availability.

I think we can, and should, postpone this migration doc: the primary goal of Feature #12705 is to improve things for new users / newly installed devices as soon as possible, and with the lowest cost possible, as an intermediary, incremental improvement until we tackle Feature #11679. We will get this even without documenting any migration procedure. As long as Bug #14622 is addressed in due time, this should make any currently supported use case worse than it currently is. Given we now have a sprint scheduled for Feature #11679 in February, I think we should:

• neither invest lots of time into this migration doc ourselves now, unless we’re very confident the output of this work will be reused for the post-Feature #11679 migration, which seems highly unclear to me at this point;
• nor strongly encourage users to migrate to a bigger system partition now: if we do that and then force them to migrate to yet another partition layout a few months later, many people will be rightfully pissed off;

So to conclude, let’s explicitly limit the scope of Feature #12705 and subtasks to new USB sticks (which is why we want to keep supporting upgrading older, smaller devices), and forget about migrations for now.

Now, I totally agree that our current “migrate my persistence data to a new device” story is less than ideal, which is a problem in many situations already regardless of Feature #12705; we will have to face this when we have a plan for Feature #11679, so I’m confident we won’t forget about it. Still, you put lots of good thought in the comment I’m replying to, so perhaps it’s worth capturing on a dedicated ticket?

#8 Updated by sajolida 2017-09-12 09:26:11

>> Note that everybody going through this will have to go on the
>> command line (see the end of /doc/first_steps/persistence/copy).
>
> Right. By the way we’re sometimes told this documentation has issues,
> see e.g. the ticket we got about it last week (I think help desk
> reassigned it to you so it’s probably on your radar).

Indeed, I forgot to mention that but it was another concern I had…

> So to conclude, let’s explicitly limit the scope of Feature #12705 and
> subtasks to new USB sticks (which is why we want to keep supporting
> upgrading older, smaller devices), and forget about migrations for
> now.

Full ack on all this :) And this will be a big relief for me until 3.2!

> Still, you put lots of good thought in the comment I’m
> replying to, so perhaps it’s worth capturing on a dedicated ticket?

Done in Feature #14624.

#12 Updated by anonym 2017-09-15 17:38:36

intrigeri wrote:
> Please review git diff origin/devel...origin/feature/12705-bump-system-partition-size -- wiki and if happy, close as resolved but don’t merge (the branch merge is tracked on the parent ticket).

Now this is easiest review with:

git diff 48354fb5fb5f63f054823a8bd04c008e7b07f92e...origin/feature/12705-bump-system-partition-size -- wiki

#16 Updated by intrigeri 2017-09-18 06:37:02

> Perfect!

/me blushes :)