Bug #10239

Tails Upgrader sometimes make device unbootable

Added by emmapeel 2015-09-23 11:11:48 . Updated 2016-01-03 14:32:34 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Target version:
Tails_2.0
Start date:
2015-09-23
Due date:
% Done:

100%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Upgrader
Deliverable for:

Description

It looks like the script that triggers the automatic upgrade is failing to see when there is not enough space available on the Tails partition.

Description of the error:
————————————-

Instead of receiving the message ‘Not enough memory available’, the script tries to write the new download, and it ends up with a corrupted boot folder.

What should happen instead:
—————————————-
User should be advised to Update manually instead:

https://tails.boum.org/doc/first_steps/upgrade/#manual

Subtasks

Related issues

Related to Tails - Bug #10235: Document how to fix a Tails device that got broken for too many automatic updates Resolved 2015-09-23
Related to Tails - Bug #10347: Troubleshooting for automatic upgrades that fail to restart Resolved 2015-10-07
Has duplicate Tails - Bug #10292: Tails 1.6 regression - hangs on Begin: Running /scripts/init-premount Duplicate 2015-09-27

History

#1 Updated by emmapeel 2015-09-23 11:12:04

  • related to Bug #10235: Document how to fix a Tails device that got broken for too many automatic updates added

#2 Updated by intrigeri 2015-09-25 02:42:14

  • Assignee set to emmapeel
  • Priority changed from Normal to Elevated
  • QA Check changed from Dev Needed to Info Needed

Do you have more info about space available before the upgrade?

#3 Updated by intrigeri 2015-09-25 02:59:58

Actually, I need the same debugging info as https://tails.boum.org/doc/upgrade/error/install/.

#4 Updated by intrigeri 2015-09-25 03:11:08

FTR I have been reported one issue with very similar symptoms after a 1.5->1.5.1 automatic upgrade, and disk space was totally fine. I’m wondering if we’ve maybe jumped to conclusions a bit too fast regarding the disk space thing. I’m wondering if we might have some missing umount or sync in the reboot process.

#5 Updated by emmapeel 2015-09-27 06:25:03

I am afraid I still haven’t got the debugging info.

Users I helped commented on:

Doing many automatic upgrades previously
Fixing the problem with a manual upgrade

I keep asking for the debuging info. I think I will try to recreate this problem today installing Tails 1.2.1 and updating from there. I had one similar report for Tails 1.5.1 but I cannot find it :S

#6 Updated by goupille 2015-10-01 13:00:20

we’ve got a user that get the exact same error message trying to install Tails 1.6 on a brand new USB stick (16 GB Verbatim Store’n Go V3) with Tails installer (using “clone and install”)

#7 Updated by intrigeri 2015-10-01 18:38:44

> we’ve got a user that get the exact same error message trying to install Tails 1.6 on
> a brand new USB stick (16 GB Verbatim Store’n Go V3) with Tails installer (using
> “clone and install”)

This seems to be unrelated. May you please file a dedicated ticket about it?

#8 Updated by goupille 2015-10-02 03:17:01

ok, I’m waiting for more information from the user before opening a new ticket… I figured that, given the doubts about what causes the issue, and the similar error message, it worthed to be said here.

#9 Updated by intrigeri 2015-10-02 03:24:12

> I figured that, given the doubts about what causes the issue, and the similar error message, it worthed to be said here.

Sure :)

#10 Updated by intrigeri 2015-10-03 03:26:48

Let’s say the problem is “Tails Upgrader sometimes corrupts the device to the point it does not boot anymore”. We still don’t know whether this bug is caused by lack of storage space. I think the easiest way to prove or disprove that hypothesis would be to install Tails 1.5.1 on a USB stick with Tails Installer, then artificially make it space-constrained by creating a large file in the system partition, and then trying to go through the automatic upgrade and reproduce the problem. One likely will have to test different file sizes and bisect to find a reproducer (if any).

emmapeel, do you feel comfortable trying that? Do you think you can do it by October 15, or to find other people at frontdesk to do so?

If one can’t find a good correlation then I think we should look elsewhere. E.g. one could mount stuff with the “sync” option in remount_liveos_rw, and maybe call “sync” manually in the run method in Tails/IUK/Install.pm. If someone finds a reliable way to reproduce the bug, I’m curious to see if any of these hacks help at all.

#11 Updated by intrigeri 2015-10-03 03:28:07

  • Subject changed from Tails 'Automatic upgrade' script fails to realise there is not enough space on device to Tails Upgrader sometimes make device unbootable
  • Status changed from New to Confirmed

(It seems that this has been reported multiple times, seen it on Reddit too, so that’s confirmed.)

#12 Updated by intrigeri 2015-10-03 03:28:23

  • has duplicate Bug #10292: Tails 1.6 regression - hangs on Begin: Running /scripts/init-premount added

#13 Updated by anonym 2015-10-07 06:49:17

I just had a thought: so we require 3x the size of the compressed IUK of free disk space on the boot partition. Hence, if the IUK is compressed so well that it extracts to > 3x the compressed size (also taking the kernel/initrd swap into account), there’s an issue. Do we do any other check, later for the actual uncompressed size vs free disk space?

Maybe our UDF:s should record the exact amount of space needed (i.e. uncompressed squashfs file + difference of the potential kernel/initrd swap etc), and then we check against that?

#14 Updated by intrigeri 2015-10-12 08:36:40

  • Assignee changed from emmapeel to anonym

intrigeri wrote:
> Let’s say the problem is “Tails Upgrader sometimes corrupts the device to the point it does not boot anymore”. We still don’t know whether this bug is caused by lack of storage space. I think the easiest way to prove or disprove that hypothesis would be to install Tails 1.5.1 on a USB stick with Tails Installer, then artificially make it space-constrained by creating a large file in the system partition, and then trying to go through the automatic upgrade and reproduce the problem. One likely will have to test different file sizes and bisect to find a reproducer (if any).
>
> emmapeel, do you feel comfortable trying that? Do you think you can do it by October 15, or to find other people at frontdesk to do so?

Looks like emmapeel / frontdesk won’t do that. anonym, may you please take it as part of the time allocated for fixing regressions?

#15 Updated by anonym 2015-10-12 09:01:27

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Info Needed to Dev Needed

intrigeri wrote:
> intrigeri wrote:
> > Let’s say the problem is “Tails Upgrader sometimes corrupts the device to the point it does not boot anymore”. We still don’t know whether this bug is caused by lack of storage space. I think the easiest way to prove or disprove that hypothesis would be to install Tails 1.5.1 on a USB stick with Tails Installer, then artificially make it space-constrained by creating a large file in the system partition, and then trying to go through the automatic upgrade and reproduce the problem. One likely will have to test different file sizes and bisect to find a reproducer (if any).
> >
> > emmapeel, do you feel comfortable trying that? Do you think you can do it by October 15, or to find other people at frontdesk to do so?
>
> Looks like emmapeel / frontdesk won’t do that. anonym, may you please take it as part of the time allocated for fixing regressions?

I actually did this weeks ago and told emmapeel about it. From my tests, Tails Upgrader correctly shows the error if < 3 times the IUK’s size of disk space is available.

Also my comment Bug #10239#note-13 seems invalid. For instance, the 1.5.1_to_1.6 IUK is 106 MiB compressed and 109 MiB decompressed, so the 3x requirement is definitely enough (well, overkill :)).

#16 Updated by intrigeri 2015-10-12 09:19:33

Hi,

(Cc’ing frontdesk since there’s a question for them at the bottom, and emmapeel didn’t answer my most recent question, so it seems we need to get other frontdesk people on board.)

> I actually did this weeks ago and told emmapeel about it. From my tests, Tails Upgrader correctly shows the error if < 3 times the IUK’s size of disk space is available.

OK, so the initial hasty guesses that lead to the branch for Bug #10235 to be merged were wrong, and the documentation that was merged into master is incorrect. anonym, emmapeel, may you please fix this somehow?

Now, that’s too bad: if that test was buggy, it would probably be easy to fix, but now I’ve no idea why we suddenly have this problem on 1.51.5.1 and 1.5.1>1.6, while we’ve never seen it before (?). So:

  • Dear frontdesk, can you confirm we didn’t hear about such problems earlier? Please reply on the ticket.
  • In any case, we should add the sync hacks I suggested on Bug #10239#note-10. I can provide an initial untested patch but I won’t have time to test it in time for 1.7, so that’ll land onto our dear RM’s bugfixing plate. The thing is, to be able to check whether it fixes the problem or not, we need a reproducer, and nobody could provide any yet.

Cheers!

#17 Updated by intrigeri 2015-10-12 09:41:29

Also, now that the “fails to detect when there’s not enough disk space is available” option was discarded, the only piece of debugging info I from frontdesk (apart of the other question I’ve asked today) is, after such a failed upgrade:

  • the checksum of the 1.6.iuk file;
  • the directory listing of the live directory on the system partiton.

#18 Updated by intrigeri 2015-10-12 09:45:00

  • Assignee changed from intrigeri to anonym
  • % Done changed from 0 to 10
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:
> In any case, we should add the sync hacks I suggested on Bug #10239#note-10. I can provide an initial untested patch but I won’t have time to test it in time for 1.7, so that’ll land onto our dear RM’s bugfixing plate. The thing is, to be able to check whether it fixes the problem or not, we need a reproducer, and nobody could provide any yet.

That’s now bugfix/10239-more-sync in the upgrader’s Git repo. Given the info I have I don’t see what else I can do for now => please review and merge (I suggest using config/chroot_local-patches/ to save time now, and only do the upstream release + package update thing for 1.7~rc1), but likely we should not close this ticket before we see if it improves things (that is, once people upgrade to 1.8).

#19 Updated by anonym 2015-10-26 13:03:18

  • QA Check deleted (Ready for QA)

intrigeri wrote:
> intrigeri wrote:
> > In any case, we should add the sync hacks I suggested on Bug #10239#note-10. I can provide an initial untested patch but I won’t have time to test it in time for 1.7, so that’ll land onto our dear RM’s bugfixing plate. The thing is, to be able to check whether it fixes the problem or not, we need a reproducer, and nobody could provide any yet.
>
> That’s now bugfix/10239-more-sync in the upgrader’s Git repo. Given the info I have I don’t see what else I can do for now => please review and merge (I suggest using config/chroot_local-patches/ to save time now, and only do the upstream release + package update thing for 1.7~rc1), but likely we should not close this ticket before we see if it improves things (that is, once people upgrade to 1.8).

I’ve merged that into the tails-iuk shipped in Tails 1.7~rc1. What now? I guess we’ll just wait an see…

#20 Updated by anonym 2015-11-23 04:38:39

  • Target version changed from Tails_1.7 to Tails_1.8

#21 Updated by intrigeri 2015-12-05 14:06:48

  • Assignee changed from anonym to sajolida
  • QA Check set to Info Needed

After the post-1.6 (?) frenzy, were there many such reports again? We (at least I) didn’t hear from frontdesk nor from other support people since a while.

#22 Updated by sajolida 2015-12-07 08:13:04

  • Assignee changed from sajolida to mercedes508

Assigning to frontdesk for more info. I’m not following their communications enough anymore.

#23 Updated by emmapeel 2015-12-19 03:11:47

After the release of Tails 1.8 there were no reports about this.

Some users received the ‘not enough space on the disk to do an automatic upgrade’ and wrote to frontdesk scared because their disk was big enough and they were used to upgrade before, but they didn’t got a broken Tails.

#24 Updated by intrigeri 2015-12-20 02:20:07

  • Assignee changed from mercedes508 to anonym
  • Target version changed from Tails_1.8 to Tails_2.0
  • QA Check changed from Info Needed to Ready for QA

> After the release of Tails 1.8 there were no reports about this.

Thanks!

anonym, I suggest we close this bug as resolved (and we have another ticket that tracks the huge performance regression introduced by what seems to be a fix for this one). OK?

#25 Updated by anonym 2016-01-03 14:32:34

  • Status changed from Confirmed to Resolved
  • Assignee deleted (anonym)
  • % Done changed from 10 to 100
  • QA Check changed from Ready for QA to Pass

intrigeri wrote:
> anonym, I suggest we close this bug as resolved (and we have another ticket that tracks the huge performance regression introduced by what seems to be a fix for this one). OK?

ACK.

#26 Updated by intrigeri 2016-01-11 15:35:35

  • related to Bug #10347: Troubleshooting for automatic upgrades that fail to restart added