Bug #15557

Document workaround for partially applied automatic upgrade

Added by intrigeri 2018-04-30 10:03:15 . Updated 2019-07-31 11:26:57 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Start date:
2018-04-30
Due date:
% Done:

100%

Feature Branch:
doc/15557-upgrade-to-fix-upgrade
Type of work:
End-user documentation
Blueprint:

Starter:
Affected tool:
Upgrader
Deliverable for:

Description

As we see on Bug #14754 sometimes an automated upgrade is partially applied and as a result the upgraded system behaves weirdly. That is occuring a lot recently according to help desk, but I’ve found no way to fix it yet (there’s a tentative fix ready in Git, that will be used first during the 3.9→3.10 upgrade but I’m not counting on it too much).

So IMO we should document a workaround: assuming I’ve upgraded to 3.6.2 and my Tails does not work anymore, I should add a line that contains “3.6.2.squashfs” (without the quotes) at the end of live/Tails.module (at the root of the Tails system partition). Adjust the version number depending on what version one is upgrading to, e.g. if users have the same problem after upgrading to 3.7, they should add a line that contains “3.7.squashfs” instead. Presumably this must be done from a non-Tails system (or from another, working Tails).


Subtasks


Related issues

Related to Tails - Bug #10347: Troubleshooting for automatic upgrades that fail to restart Resolved 2015-10-07
Related to Tails - Bug #14754: Partially applied incremental upgrades cause all kinds of trouble Confirmed 2019-03-21
Related to Tails - Bug #11839: After automatic upgrade from Tails 2.5 to 2.6, keyboard and mouse stop working Rejected 2016-09-24
Blocks Tails - Feature #15411: Core work 2018Q2 → 2018Q3: Technical writing Resolved 2018-03-14
Blocks Tails - Feature #16711: Core work 2019Q3 → 2019Q4: Technical writing Resolved 2016-01-08

History

#1 Updated by intrigeri 2018-04-30 10:03:23

  • blocks Feature #15411: Core work 2018Q2 → 2018Q3: Technical writing added

#2 Updated by intrigeri 2018-04-30 10:03:50

  • Priority changed from Normal to Elevated

(Help desk say lots of users are affected.)

#3 Updated by sajolida 2018-04-30 16:47:42

  • Description updated
  • Assignee changed from sajolida to cbrownstein

Cody: Do you think you can work on it this week?

I’ll be super busy with the user testing of additional software :/

#4 Updated by sajolida 2018-05-04 18:56:05

Definitely related to Bug #10347. So I’m assigning it to you as well.

This one might be tough, so tell me if you need help!

#5 Updated by sajolida 2018-05-04 18:59:47

  • related to Bug #10347: Troubleshooting for automatic upgrades that fail to restart added

#6 Updated by bertagaz 2018-05-10 11:09:33

  • Target version changed from Tails_3.7 to Tails_3.8

#7 Updated by cbrownstein 2018-06-17 08:56:04

  • Assignee changed from cbrownstein to intrigeri
  • QA Check set to Info Needed

intrigeri wrote:
> As we see on Feature #14574 sometimes an automated upgrade is partially applied and as a result the upgraded system behaves weirdly.

I’m trying to understand the symptoms of a partially-applied automatic upgrade. Can “behaves weirdly” be more definite? Feature #14574 appears to be unrelated to this ticket. Is there a correct ticket I can reference?

> So IMO we should document a workaround: assuming I’ve upgraded to 3.6.2 and my Tails does not work anymore, I should add a line that contains “3.6.2.squashfs” (without the quotes) at the end of live/Tails.module (at the root of the Tails system partition). Adjust the version number depending on what version one is upgrading to, e.g. if users have the same problem after upgrading to 3.7, they should add a line that contains “3.7.squashfs” instead. Presumably this must be done from a non-Tails system (or from another, working Tails).

What can the user expect after this is done?

#8 Updated by intrigeri 2018-06-18 06:23:34

  • Description updated

#9 Updated by intrigeri 2018-06-18 06:23:48

  • related to Bug #14754: Partially applied incremental upgrades cause all kinds of trouble added

#10 Updated by intrigeri 2018-06-18 06:26:47

  • related to Bug #11839: After automatic upgrade from Tails 2.5 to 2.6, keyboard and mouse stop working added

#11 Updated by intrigeri 2018-06-18 06:34:10

  • Assignee changed from intrigeri to cbrownstein
  • QA Check changed from Info Needed to Dev Needed

> I’m trying to understand the symptoms of a partially-applied automatic upgrade. Can “behaves weirdly” be more definite? Feature #14574 appears to be unrelated to this ticket. Is there a correct ticket I can reference?

Sorry, typo. The correct ticket is Bug #14754.

tl;dr: symptoms are “some hardware that would usually work fine in Tails is not working anymore”.

When an automatic upgrade is only partially applied, in some (as we’ve seen on Bug #14754 and Bug #11839), the kernel and initramfs are upgraded but the SquashFS either has no kernel modules at all for the new kernel version. So drivers that are in the initramfs can be loaded but drivers that should be in the SquashFS cannot. The consequence is that some hardware won’t work. For example, user reports teach us that input devices are affected; it is not surprising technically, and it’s understandable that it’s the first problem users notice because without working input they can’t log in and notice how other stuff is broken, but I expect that other areas of hardware support are affected as well, starting with network devices. Furthermore other aspects of Tails, besides hardware support, will be broken, e.g. networking (can’t work without the kernel modules needed by our firewall), but well, in practice, who cares given their input devices and network adapters don’t work anyway.

If you need more info, don’t hesitate to ask :)

>> So IMO we should document a workaround: assuming I’ve upgraded to 3.6.2 and my Tails does not work anymore, I should add a line that contains “3.6.2.squashfs” (without the quotes) at the end of live/Tails.module (at the root of the Tails system partition). Adjust the version number depending on what version one is upgrading to, e.g. if users have the same problem after upgrading to 3.7, they should add a line that contains “3.7.squashfs” instead. Presumably this must be done from a non-Tails system (or from another, working Tails).

> What can the user expect after this is done?

Their Tails should now be fully upgraded (respectively to 3.6.2 and 3.7 in the above two examples) so any weird/broken behaviour described above should go away.

#12 Updated by intrigeri 2018-06-26 16:28:06

  • Target version changed from Tails_3.8 to Tails_3.9

#13 Updated by cbrownstein 2018-07-14 19:06:35

  • Assignee changed from cbrownstein to sajolida
  • QA Check changed from Dev Needed to Ready for QA

I’ve pushed a branch for review:

https://0xacab.org/cbrownstein/tails/commits/doc/15557-partial-automatic-upgrade

#14 Updated by sajolida 2018-07-16 18:20:49

  • Assignee changed from sajolida to cbrownstein
  • QA Check changed from Ready for QA to Dev Needed

Thanks for the draft!

  • Please split your two first sentences into a separate paragraph. It’s your intro. When writing for the web, it’s fine (and actually recommended) to do super short paragraph for scanability.
  • The system partition is flagged as hidden so it won’t appear in Files in Tails and Debian and will probably be hard to mount on Windows and macOS. I suggest instructing people to do that through GNOME Disks (yeah, it’s a pain but that’s the best I can think of). We could say that this is possible either from Tails or from Debian/Ubuntu.
  • Regarding “For security reasons, you should not mount your persistent partition if you have one!”, if the system on which you apply the workaround is untrusted, the same could apply to the system partition: an untrusted system do insert some malicious code in the Tails system from the system partition. So I wouldn’t write anything specific for the persistence partition. I’m not even sure it’s worth specifiying anything as people on Debian/Ubuntu probably used their system to install Tails in the first place.
  • Please format the instructions and a numbered list of steps. Once we added GNOME Disks to the dance that will really become necessary.
  • I would try to apply the command line formatting that we have, see Feature #15120, to describe the line that needs to be added to the file. For example, for the placeholder part of the line (“Replace x.x.”).

#15 Updated by cbrownstein 2018-07-16 22:50:34

  • Assignee changed from cbrownstein to sajolida
  • QA Check changed from Dev Needed to Info Needed

> * Please split your two first sentences into a separate paragraph. It’s your intro. When writing for the web, it’s fine (and actually recommended) to do super short paragraph for scanability.

Done.

> * The system partition is flagged as hidden so it won’t appear in Files in Tails and Debian and will probably be hard to mount on Windows and macOS. I suggest instructing people to do that through GNOME Disks (yeah, it’s a pain but that’s the best I can think of). We could say that this is possible either from Tails or from Debian/Ubuntu.

Any suggestions as to how detailed the GNOME Disks instructions should be? I’ll also have to add instructions for what to do after the partition is mounted.

> * Regarding “For security reasons, you should not mount your persistent partition if you have one!”, if the system on which you apply the workaround is untrusted, the same could apply to the system partition: an untrusted system do insert some malicious code in the Tails system from the system partition. So I wouldn’t write anything specific for the persistence partition. I’m not even sure it’s worth specifiying anything as people on Debian/Ubuntu probably used their system to install Tails in the first place.

I’m deleting the entire sentence for the reasons you’ve stated.

> * Please format the instructions and a numbered list of steps. Once we added GNOME Disks to the dance that will really become necessary.

Will do.

> * I would try to apply the command line formatting that we have, see Feature #15120, to describe the line that needs to be added to the file. For example, for the placeholder part of the line (“Replace x.x.”).

Will do.

#16 Updated by sajolida 2018-07-26 18:12:07

  • Assignee changed from sajolida to cbrownstein
  • QA Check changed from Info Needed to Dev Needed

We rediscussed this last week and Cody should have all the info to come up with a new version.

#17 Updated by cbrownstein 2018-08-02 23:51:48

  • Assignee changed from cbrownstein to sajolida
  • QA Check changed from Dev Needed to Ready for QA

I’ve rewritten the instructions to include usage of GNOME Disks.

#18 Updated by sajolida 2018-08-04 11:38:32

  • Assignee changed from sajolida to cbrownstein

Cool! I think it’s the right level of details and the problem is defined as good as possible. “Behaves weirdly” still sound very vague but that’s the best we can do I think.

I pushed a bunch of small fixes on doc/15557-partial-automatic-upgrade. Please have a look and then I can merge.

I didn’t test any of this but from some details in your instructions I understand that you did test that :)

#19 Updated by cbrownstein 2018-08-05 04:01:43

  • Assignee changed from cbrownstein to sajolida

“Plug your Tails USB stick that behaves weirdly.”

Can we change that sentence? The first verb definition of plug that comes to my mind is, “to stop, make tight, or secure by inserting a plug.”[1] Obviously that is not the definition of plug that we mean.

I suggest: “Plug in your Tails USB stick that behaves weirdly.” I have made that change in changeset eefb8e3afa. That is consistent with existing documentation, e.g., install instructions.

Other than that, my changes are small.

[1] https://www.merriam-webster.com/dictionary/plug

#20 Updated by cbrownstein 2018-08-06 14:50:20

  • Status changed from Confirmed to Resolved
  • % Done changed from 0 to 100

Applied in changeset commit:c7177be690840206bf9606b2625730c3eda9b5a7.

#21 Updated by intrigeri 2018-08-07 07:26:36

  • Status changed from Resolved to In Progress

(Apparently this was not merged but a buggy “Closes” in a commit message mislead Redmine into closing this ticket.)

#22 Updated by sajolida 2018-08-11 08:37:48

  • Status changed from In Progress to Resolved
  • Assignee deleted (sajolida)
  • QA Check deleted (Ready for QA)

I definitely missed ‘in’ in ‘plug in’. Thanks for spotting that!

I merged your branch and this ticket is now solved.

Yeah!

#23 Updated by sajolida 2019-06-14 11:38:59

@intrigeri: By the way, would doing a manual upgrade also fix the problem? If so, we should document this instead of /support/known_issues#index21h2 which requires another Linux system anyway.

If I understood correctly, please reopen this ticket and assign it back to me.

#24 Updated by intrigeri 2019-06-15 09:27:07

  • Status changed from Resolved to Confirmed
  • Assignee set to sajolida
  • Target version deleted (Tails_3.9)

Yes, it would :)

#25 Updated by sajolida 2019-07-09 09:38:31

  • Target version set to Tails_3.16

#26 Updated by sajolida 2019-07-09 09:38:45

  • blocks Feature #16711: Core work 2019Q3 → 2019Q4: Technical writing added

#27 Updated by sajolida 2019-07-30 20:02:49

  • Assignee changed from sajolida to intrigeri
  • Priority changed from Elevated to Normal
  • Feature Branch set to doc/15557-upgrade-to-fix-upgrade

Then please have a look at this branch!

#28 Updated by intrigeri 2019-07-30 21:50:52

  • Status changed from Confirmed to Needs Validation

#29 Updated by intrigeri 2019-07-30 21:56:23

  • Status changed from Needs Validation to In Progress

Applied in changeset commit:tails|260f00803b4e757a5967df3e4ec6d9531801f634.

#30 Updated by intrigeri 2019-07-30 21:59:20

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to sajolida

Merged and added commit:bbe9f9bc9c5e4aefddac7f1fab880f1026042faa on top, please review.

#31 Updated by intrigeri 2019-07-30 23:15:22

  • Status changed from Needs Validation to In Progress

Also, “your Tails might […] fails to start” seems buggy grammar to me. Same for “Behaves weirdly”.

#32 Updated by sajolida 2019-07-31 11:26:18

Oops! Fixed in rb8201b7748.

#33 Updated by sajolida 2019-07-31 11:26:57

  • Status changed from In Progress to Resolved
  • Assignee deleted (sajolida)