Feature #11627

Consider updating the default system partition's size

Added by intrigeri 2016-08-10 01:58:03 . Updated 2017-06-30 06:20:51 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Installation
Target version:
Start date:
2016-08-10
Due date:
% Done:

100%

Feature Branch:
Type of work:
Discuss
Blueprint:

Starter:
Affected tool:
Installer
Deliverable for:

Description

At least one recent IUK has been very big (Tor Browser + Icedove + LibreOffice + Linux kernel) and forced lots of users to go through manual upgrades. I’ll check if the estimates that lead us to pick 2.5GB match the actual size of IUKs we’ve produced. If they don’t (and in practice people are forced to do manual upgrades more often than we would like), then we could consider growing the system partition again, until Feature #7499 or Feature #11131 saves the day.


Subtasks


Related issues

Related to Tails - Feature #7499: Extend the upgrader to allow full (self) upgrade Confirmed 2014-07-06
Related to Tails - Feature #11131: Endless automatic upgrades Rejected 2015-01-05
Related to Tails - Bug #11628: Error message on not-enough-free-space in Tails Upgrader is confusing Rejected 2016-08-10
Related to Tails - Feature #11679: Rethink the installation process and upgrade process Resolved 2016-08-20
Related to Tails - Feature #5301: Backup system for the Persistence Confirmed 2015-01-27
Related to Tails - Bug #11857: Provide IUK to go from version n-2 to version n Resolved 2016-10-02
Related to Tails - Feature #12214: Document a way to manually backup persistent data Duplicate 2017-02-06
Related to Tails - Feature #12705: Update the size of the system partition to >= 4 GiB Resolved 2017-06-15
Blocks Tails - Feature #12431: UX core work 2017Q2 Resolved 2017-04-07

History

#1 Updated by intrigeri 2016-08-10 01:58:57

  • related to Feature #7499: Extend the upgrader to allow full (self) upgrade added

#2 Updated by intrigeri 2016-08-10 01:59:04

#3 Updated by sajolida 2016-08-10 03:39:35

  • related to Bug #11628: Error message on not-enough-free-space in Tails Upgrader is confusing added

#4 Updated by intrigeri 2016-09-21 07:31:47

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10
  • Type of work changed from Research to Discuss

Now that we have IUKs that are 300-400MB large, a 1GB ISO, 2.5GB system partitions, and Tails Upgrader needs 3 times the size of the IUK to install it, according to my back-of-the-envelope calculation (1000 + 350*2 + 350*3 > 2500 > 1000 + 350 + 350*3) one can go through two incremental upgrades, but not three. So one has to do a full upgrade every 3 releases. Given we release ~12 times a year, this means that one has to do a full upgrade 4 times a year. Many users need help to do a full upgrade, so that’s too much for them and in the end I’ve seen people keep using an outdated Tails for a while, even though they would have upgraded if an incremental update had been possible.

If we bumped the system partition size to 4GB, then one could do 6 incremental upgrades (1000 + 350*5 + 350*3 = 3800 < 4000), and then a full upgrade would be needed only twice a year. I think we should do that, and extend the message displayed by Tails Upgrader when there’s not enough room left, to suggest reinstalling to get a bigger system partition (if it’s smaller than the current default size).

But then, 4GB USB sticks would not be able to host a persistent volume; we could still allow it, but warn about it at installation time; or we could simply require a 8GB stick for everyone who does an initial installation (can one purchase a 4GB stick nowadays in the closest supermarket?).

Note that there are chances that Feature #7499 and Feature #11131 will require more space for the system partition(s) at some point anyway, at least if they don’t fully deprecate our current incremental upgrade system.

#5 Updated by intrigeri 2016-09-21 07:36:20

I won’t be able to attend the October monthly meeting, so it would be nice if a couple other people checked beforehand if some more input from me is required.

#6 Updated by sajolida 2016-09-21 11:13:58

I’m sending you in private some stats on the size of storage devices gathered from WhisperBack (over the last year). They are quite dirty because they would require rounding up the sizes and filtering out which device are actually used to start Tails to get something more relevant.

It’s the output of internal:stats/whisperback_scripts/storage_size.sh. Feel free to write something better and I’ll run it.

But for the time being, I summed up in a spreadsheet the number of occurences of something close to usual USB sizes and the results sounds plausible:

128     151     11%
64       78      6%
32      151     11%
16      352     25%
8       416     30%
4       193     14%
2        48      3%
1        13      1%
       1402 

The most popular size is 8 GB, which is, in my experience, the smallest and cheapest USB sticks you can get nowadays in western Europe. Then 16 GB for people who want a bit more space. And the 4 GB for people still using sticks from 1-2 years ago.

Other than that, how complicated would it be to create system partitions of 4 GB on sticks of 8 GB or more, and system partitions of 2.5 on sticks of 4 GB?

Or to have the size of the system partition calculated based on the size of the USB stick, so it automatically gets bigger by default as USB stick get bigger on the market?

#7 Updated by emmapeel 2016-09-21 11:30:34

sajolida wrote:
>
> Or to have the size of the system partition calculated based on the size of the USB stick, so it automatically gets bigger by default as USB stick get bigger on the market?

I like this!

#8 Updated by intrigeri 2016-09-22 02:38:11

> I’m sending you in private some stats on the size of storage devices […]

Thanks!

> Other than that, how complicated would it be to create system partitions of 4 GB on sticks of 8 GB or more, and system partitions of 2.5 on sticks of 4 GB?

It should be pretty easy (not to say I think it’s a good idea: I did not think about it yet).

> Or to have the size of the system partition calculated based on the size of the USB stick, so it automatically gets bigger by default as USB stick get bigger on the market?

Same (once one has a good algorithm).

#9 Updated by intrigeri 2016-09-22 08:14:26

> Other than that, how complicated would it be to create system partitions of 4 GB on sticks of 8 GB or more, and system partitions of 2.5 on sticks of 4 GB?
> Or to have the size of the system partition calculated based on the size of the USB stick, so it automatically gets bigger by default as USB stick get bigger on the market?

I see one problem with this approach: “Upgrade by cloning” would not be guaranteed to work anymore, since one could not e.g. upgrade a stick with a 2.5GB system partition, using one with a system partition that has more than 2.5GB used. Granted, my proposal has the same problem, at least during some transition period, if we start mixing old sticks (2.5GB system partition) with newer ones (e.g. 4GB system partition). I don’t remember us handling this in any way back when we bumped the size to 2.5GB, by the way. In all cases, this can be addressed by adding a check in Tails Installer that compares the size of the destination partition with the size of the data to copy. And in the end, the only difference between all these approaches is how common it would be for users to hit this painful situation, and for how long (transition period with a potentially long tail vs. forever).

All this reminds me of some ideas that have been suggested in the past, like letting the user choose how they want to allocate storage, between having more space for incremental updates, and having more space for persistence. Historically, I’ve been reluctant at the idea of adding one more “what should I do?” step to the installation process (even with sane defaults, users will wonder if they need to change the default settings, and this will cause confusion and more user support work). FTR I still am reluctant but I guess I could live with it; now, this requires quite some UX design work to find a good way to expose this choice to the user, based e.g. on IUK size estimates and release frequency.

#10 Updated by sajolida 2016-09-24 05:57:07

Good catch.

And you’re right, the difference is about how common the problem would
be but also maybe how easy it would be to transmit to the user. Let’s
say we have three different cases:

  • All new sticks get 4GB: then trying to upgrade an old stick with a
    new one could give an error saying “*Due to changes in Tails 2.7, it’s
    impossible to clone and upgrade the USB stick TOSHIBA TransMemory (4.0
    GB). You can upgrade from ISO instead.*”.
  • Sticks of 4GB get 2.5GB and all others get 8GB: then trying to
    upgrade a 4GB stick with a 8GB one could give an error saying “**The USB
    stick TOSHIBA TransMemory (4 GB) is too small to be upgraded from the
    USB stick UFD 2.0 Silicon-Power8G (8 GB). You can upgrade from ISO
    instead.”. But the problem would also apply to “Install by cloning”
    which is probably much more frequent but here we should probably not let
    people install on 4GB sticks anymore.
  • **Sticks of 4GB get 2.5GB, sticks of 8GB get 4GB, sticks of 16GB get
    6GB: then it’s like in the previous case except that the problem with
    “Install by cloning” might happen for example when installing a 16GB
    stick from a 32GB stick and this should be possible.

All-in-all, this doesn’t seem crazy to explain to people.

Also, I don’t remember how things went the last time we bumped the
system partition (and thus the requirements for USB sticks) but if we
want to say that 4GB sticks are not supported anymore, then it would be
good to give some extra love to people who will have to move their data
from a 4GB stick to another one. This might actually be a reason for
people to stick with smaller sticks for long: it’s a pain to migrate all
your data and setup to a different stick. I’m not sure what this could
imply but /doc/first_steps/persistence/copy is bit hardcore…

Regarding letting the user choose, I’m confident that we could find a
way of:

  • Explaining this option in a understandable way. For example as a
    thread off between how long you want to be able to do automatic upgrades
    (“during 6 months…”) and how big you want your persistent volume to be.
  • Moving this option a bit out of the way and relying on a sane
    default for people who won’t bother.

I’m more concerned about the fact that this would be more work put into
Tails Installer while we still haven’t clarified its future. Still, if
someone is interested in coding this without waiting for Feature #11679, I don’t
mind helping with the UX. But if we were to improve a bit Tails
Installer as part of this transition maybe I’d find it more interesting
to work on an option to copy all the data of the persistent volume;
which would solve the transition issues I raised in my previous point
and be a very basic backup solution that could allow us to take it
easier on Feature #5301.

#11 Updated by sajolida 2016-09-24 05:58:53

  • related to Feature #11679: Rethink the installation process and upgrade process added

#12 Updated by sajolida 2016-09-24 05:58:54

  • related to Feature #5301: Backup system for the Persistence added

#13 Updated by intrigeri 2016-09-25 03:05:40

> And you’re right, the difference is about how common the problem would be but also maybe how easy it would be to transmit to the user. Let’s say we have three different cases:

Just to make it explicit: in all the following scenarios, the “it’s impossible to […]” errors would only be needed when the source system partition is already full enough of upgrades to not fit on the destination system partition, and not in the general case.

> All-in-all, this doesn’t seem crazy to explain to people.

Cool :)

> Also, I don’t remember how things went the last time we bumped the system partition (and thus the requirements for USB sticks) but if we want to say that 4GB sticks are not supported anymore, then it would be good to give some extra love to people who will have to move their data from a 4GB stick to another one. This might actually be a reason for people to stick with smaller sticks for long: it’s a pain to migrate all your data and setup to a different stick. I’m not sure what this could imply but /doc/first_steps/persistence/copy is bit hardcore…

Agreed.

> Regarding letting the user choose, I’m confident that we could find a way of:

Cool as well! This is reassuring.

> I’m more concerned about the fact that this would be more work put into Tails Installer while we still haven’t clarified its future.

I don’t really follow you on this one, so probably I’m missing something: the way I get it, the part of Tails Installer’s future that is unclear is its life on non-Tails platforms; but regardless of if/how we change the installation process on non-Tails platforms, and if/how we change the partition table layout, we should keep the “Upgrade by cloning” option available in Tails itself, no? And then, basically all the reasoning that we are doing on this ticket should remain valid for the foreseeable time, no?

> Still, if someone is interested in coding this without waiting for Feature #11679, I don’t mind helping with the UX. But if we were to improve a bit Tails Installer as part of this transition maybe I’d find it more interesting to work on an option to copy all the data of the persistent volume; which would solve the transition issues I raised in my previous point and be a very basic backup solution that could allow us to take it easier on Feature #5301.

I agree!

There’s just a part that I don’t understand: more interesting than what would it be?

#14 Updated by sajolida 2016-09-25 10:41:55

> Just to make it explicit: in all the following scenarios, the “it’s
> impossible to […]” errors would only be needed when the source
> system partition is already full enough of upgrades to not fit on the
> destination system partition, and not in the general case.

Yes, I had this in mind as well.

>> I’m more concerned about the fact that this would be more work put
>> into Tails Installer while we still haven’t clarified its future.
>
> I don’t really follow you on this one, so probably I’m missing
> something: the way I get it, the part of Tails Installer’s future
> that is unclear is its life on non-Tails platforms; but regardless of
> if/how we change the installation process on non-Tails platforms, and
> if/how we change the partition table layout, we should keep the
> “Upgrade by cloning” option available in Tails itself, no? And then,
> basically all the reasoning that we are doing on this ticket should
> remain valid for the foreseeable time, no?

I didn’t consider this.

First, if we decide to replace Tails Installer by something else on
every other platform, and if this “something else” allows for automatic
upgrades and persistence, then we might ask ourselves whether it’s worth
maintaining Tails Installer inside Tails if it’s “only” for cloning. Or
at least that’s not something I remember we decided.

Second, I thought that when we decided to pause our future plans for
Tails Installer, this implied pretty much everything that we could feel
as “our duty” to fix (to me, regardless of whether the feature is useful
inside or outside Tails because I admit that I didn’t consider that we
could keep Tails Installer inside Tails).

Third, the feature that you are proposing to add here (“being able to
choose the portion of the USB stick allocated to the system partition”)
is used at installation time (and not upgrade time). That’s why I
associated it with the future of Tails installation.

But if you were referring only to “bumping the system partitions size”
and “providing useful messages when upgrade by cloning is impossible”,
then I agree that this work could be done in Tails Installer even if its
future is uncertain (as part of its maintenance if we decide that the
current system partition size is really too small nowadays).

>> Still, if someone is interested in coding this without waiting for
>> Feature #11679, I don’t mind helping with the UX. But if we were to improve
>> a bit Tails Installer as part of this transition maybe I’d find it
>> more interesting to work on an option to copy all the data of the
>> persistent volume; which would solve the transition issues I raised
>> in my previous point and be a very basic backup solution that could
>> allow us to take it easier on Feature #5301.
>
> I agree!
>
> There’s just a part that I don’t understand: more interesting _than
> what_ would it be?

… “an option to copy all the data of the persistent volume” more
interesting than “being able to choose the portion of the USB stick
allocated to the system partition”. More interesting to me at least
because probably helpful to more people and in more use cases and easier
to design in terms of UX (and Gtk code). Now I guess that it would be
more work on the back end :)

#15 Updated by intrigeri 2016-09-25 11:10:09

> But if you were referring only to “bumping the system partitions size” and “providing useful messages when upgrade by cloning is impossible”, then I agree that this work could be done in Tails Installer even if its future is uncertain (as part of its maintenance if we decide that the current system partition size is really too small nowadays).

We can definitely agree on this! :)

(So I won’t debate the details of your argumentation even though I’m not exactly on the same page.)

(For the record I personally never proposed making the size of the system partition configurable or automatically adjusted to the device size. I was merely trying to factor in proposals and ideas I’ve heard in the wild, even if I had my doubts about them.)

>> There’s just a part that I don’t understand: more interesting than what would it be?

> … “an option to copy all the data of the persistent volume” more interesting than “being able to choose the portion of the USB stick allocated to the system partition”.

Fully agreed.

#16 Updated by sajolida 2016-10-02 17:50:30

  • related to Bug #11857: Provide IUK to go from version n-2 to version n added

#17 Updated by intrigeri 2016-12-11 08:39:22

  • Target version changed from Tails_2.9.1 to Tails_2.11

#18 Updated by intrigeri 2017-02-10 16:43:02

  • Assignee changed from intrigeri to sajolida

Data points, with updated figures (a 1100 MiB ISO and 400 MiB IUKs) and taking into account that the last upgrade lasts until the next release:

  • 2.5 GiB system partition => 0 or 1 upgrade => full upgrade every 1-2 months
  • 4 GiB system partition => 4 upgrades => full upgrade every 5 months
  • 6 GiB system partition => 9 upgrades => full upgrade every 10 months
  • 8 GiB system partition => 14 upgrades => full upgrade every 15 months

So as we can see the current situation is pretty bad: as users have been reminding me very regularly recently, it almost brings us back to the painful pre-incremental-upgrades days. I’d like to go ahead with the cheapest solution we can find, as soon as possible.

After re-reading this ticket, I propose we:

  • document that a 8 GiB stick is required
  • bump the default system partition size for newly installed Tails to 4 GiB on 8 GiB sticks, and to 8 GiB on sticks of 16 GiB or larger
  • refuse installing a new Tails on a stick smaller than 8 GiB
  • allow upgrading (by cloning) a legacy stick with a smaller (1.5 or 2.5 GiB) system partition when feasible; note that feasibility is a function of the size of data currently present on the source USB stick, and not of the size of its system partition
  • provide useful messages when upgrade by cloning is impossible; we can mention the several available options, along with their pros/cons, i.e. “upgrade from ISO” (easier but has to be done every time) and “migrate to a new USB stick with a larger system partition and copy your persistent data” (painful, but then you’re done with it and get incremental upgrades for a while)
  • ignore the “copy persistent data” feature for now, for several reasons:
    • it requires substantial design + code work, which would delay too much fixing the basic problem that everybody currently has
    • it’s unclear whether this solution would have a future, or would be somewhat wasted time in a year or two
    • most Tails users don’t use persistence yet, and delaying the fix to the upgrade problem wouldn’t be nice to them
    • regarding users with persistence enabled on a small USB stick: first, nothing in what I’m proposing makes things worse for them — it’s “just” that it helps them only a tiny bit; they still have the two options that already exist, the only change is that they will be told they have these two options, instead of only one: going through a manual upgrade (as most users have to do these days, so really they’re still in a bad place), or migrating to a bigger USB stick (as documented; I agree it’s a bit hardcore, but it’s a one-time thing, while manual full upgrade has to be done regularly)
  • follow-up bonus, if time allows: have Tails Upgrader mention the “migrate to a new USB stick with a larger system partition” too when incremental upgrade is impossible due to a too small system partition; would be nice, but I don’t want to block on that; it can be handled on Bug #11628.

Works for you?

#19 Updated by anonym 2017-03-09 14:00:30

  • Target version changed from Tails_2.11 to Tails_2.12

#20 Updated by sajolida 2017-04-07 08:34:23

#21 Updated by sajolida 2017-04-18 19:12:29

  • Target version changed from Tails_2.12 to Tails_3.0

#22 Updated by intrigeri 2017-05-20 06:31:04

It’s now too late to implement this in time for 3.0, but still: it would be nice to make a decision during this time frame, as the current UX is really problematic.

#23 Updated by sajolida 2017-05-23 22:19:55

Tonight I did more stats. See [internal]stats/whisperback_scripts/usb_size.rb.

In 2016 we had 10% of 4 GB USB sticks:

    528 Not running from USB
    264 8
    262 16
    135 4
    115 32
     37 64
     18 128
      6 60
      6 124
      2 36
      2 120
      1 68
      1 536
      1 320
      1 136

In 2017 we have so far we have 8% of 4 GB USB sticks:

    136 Not running from USB
    107 16
    105 8
     87 32
     55 64
     48 4
     17 128
      7 124
      2 196
      1 60
      1 252
      1 240

#24 Updated by sajolida 2017-05-25 19:32:47

  • related to Feature #12214: Document a way to manually backup persistent data added

#25 Updated by sajolida 2017-05-25 19:33:10

  • Assignee changed from sajolida to intrigeri
  • QA Check set to Info Needed

I read the whole ticket through again and I’m fine with your plan.

Anything else?

#26 Updated by intrigeri 2017-05-26 05:15:19

  • Target version changed from Tails_3.0 to Tails_3.1
  • % Done changed from 10 to 80
  • QA Check deleted (Info Needed)

Thanks! I’ll file tickets about implementing this plan then, before closing this ticket.

#27 Updated by intrigeri 2017-06-05 13:20:27

My goal for 3.1 is limited to filing the corresponding tickets and ensure we know who’s going to do what and when.

#28 Updated by intrigeri 2017-06-15 06:32:08

  • related to Feature #12705: Update the size of the system partition to >= 4 GiB added

#29 Updated by intrigeri 2017-06-15 06:32:38

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 80 to 100

Tickets created: Feature #12705 and subtasks.

#30 Updated by intrigeri 2017-06-30 06:20:51

  • Target version changed from Tails_3.1 to Tails_3.0.1