Bug #10976

persistence.conf lost, recoverable by reconfiguring

Added by emmapeel 2016-01-19 09:13:46 . Updated 2020-02-23 10:37:00 .

Status:
In Progress
Priority:
Elevated
Assignee:
Category:
Persistence
Target version:
Start date:
2019-02-12
Due date:
% Done:

100%

Feature Branch:
Type of work:
Research
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Once in a while this happens since many Tails, maybe the persistence.conf file gets broken.

But since Tails 1.8.2 many of such reports have been made, most of the times only rebooting normally (no emergency shutdown, nothing)

The Persistence is easily recovered by reconfiguring it again, with same passphrase.

I guess I should add a note on known issues, and that is how I will resolve this ticket. But I wanted the devs to know that (I think) something on 1.8.2 made the persistence.conf a little bit more fragile.

Not sure what will happen with Tails 2.0, maybe this dissapears.


Subtasks

Bug #16461: Backup persistence.conf before modifying it in t-p-s Resolved

100

Bug #16568: Make writing persistence.conf.bak more robust Resolved

100


Related issues

Related to Tails - Feature #14544: Spend software developer time on smallish UX improvements In Progress 2018-08-31
Related to Tails - Bug #15598: Explain loss of persistence.conf in known issues Resolved 2018-05-09
Related to Tails - Bug #17112: Persistence config files "insecure" backup get emptied in some situations Resolved
Related to Tails - Bug #17116: Improve lost persistence.conf instructions on known issues page Resolved
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by intrigeri 2016-01-19 13:28:44

  • Assignee set to emmapeel
  • QA Check set to Info Needed

> Once in a while this happens since many Tails, maybe the persistence.conf file gets broken.

> But since Tails 1.8.2 many of such reports have been made, most of the times only rebooting normally (no emergency shutdown, nothing)

> The Persistence is easily recovered by reconfiguring it again, with same passphrase.

I’m not sure what you mean with “persistence lost”. The fact it can be recovered leads me to think that the persistent data is not lost, simply not set up / mounted properly, probably due to a corrupted live-persistence.conf file, and:

  • the user is still proposed to unlock their persistent volume in the Greeter;
  • the data is still available in /live/persistence/TailsData_unlocked once unlocked.

This would be consistent with the fact that re-creating a proper live-persistence.conf by re-configuring the persistence makes it possible to have the persistent data accessible via usual paths again.

Right?

If I got it right, two things:

  • I’ve heard similar reports regularly for years, so I don’t think this problem is new; now, I totally believe you that it apparently happens much more often recently.
  • what we need is the debugging info from a session in which the persistence seems to be lost, and the content of live-persistence.conf in that same session. It’s just too sad to always hear about this problem, but never have seen the info that would help us understand what’s happening.

Cheers!

#2 Updated by emmapeel 2016-01-26 20:12:50

intrigeri wrote:
> > Once in a while this happens since many Tails, maybe the persistence.conf file gets broken.
>
> > But since Tails 1.8.2 many of such reports have been made, most of the times only rebooting normally (no emergency shutdown, nothing)
>
> > The Persistence is easily recovered by reconfiguring it again, with same passphrase.
>
> I’m not sure what you mean with “persistence lost”. The fact it can be recovered leads me to think that the persistent data is not lost, simply not set up / mounted properly, probably due to a corrupted live-persistence.conf file

Exactly!
>
> * the user is still proposed to unlock their persistent volume in the Greeter;
yes

> * the data is still available in /live/persistence/TailsData_unlocked once unlocked.
yes
>
> This would be consistent with the fact that re-creating a proper live-persistence.conf by re-configuring the persistence makes it possible to have the persistent data accessible via usual paths again.
>
> Right?
>
> If I got it right, two things:
>
> * I’ve heard similar reports regularly for years, so I don’t think this problem is new; now, I totally believe you that it apparently happens much more often recently.

Yes, I think this last month there were more.
> * what we need is the debugging info from a session in which the persistence seems to be lost, and the content of live-persistence.conf in that same session. It’s just too sad to always hear about this problem, but never have seen the info that would help us understand what’s happening.
>
Ok, will try to get this information.

#3 Updated by sajolida 2016-02-23 11:37:34

I experienced this myself in c7425c8012ba2f65e0b9876d11a573f7.

#4 Updated by intrigeri 2016-03-13 10:58:13

  • Category set to Persistence
  • Assignee changed from emmapeel to sajolida
  • QA Check deleted (Info Needed)

sajolida wrote:
> I experienced this myself in c7425c8012ba2f65e0b9876d11a573f7.

And in there, searching for “ext4” I find:

COMMAND=/usr/local/sbin/live-persist --read-write --log-file=/var/log/live-persist
activate /dev/mapper/TailsData_unlocked
Feb 23 10:31:54 amnesia kernel: EXT4-fs (dm-0): warning: mounting fs with errors, running e2fsck is recommended
Feb 23 10:31:54 amnesia kernel: EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
Feb 23 10:31:54 amnesia kernel: EXT4-fs (dm-0): re-mounted. Opts: data=ordered,acl

So, for some reason that filesystem was corrupted (possibly filling the FS as you say you did didn’t help ext4 being robust, I dunno, that might be a place where we can improve UX), which may have various interesting user-visible consequences such as files disappearing. I doubt that anything Tails -specific leads to such FS corruption, but one never knows. I’m not quite sure what we should do at this point.

#5 Updated by sajolida 2016-03-14 12:38:04

  • Assignee deleted (sajolida)

Me neither.

#6 Updated by adamantium 2016-10-19 00:02:13

> intrigeri wrote:
> > sajolida wrote:
> >
> > So, for some reason that filesystem was corrupted (possibly filling the FS as you say you did didn’t help ext4 being robust, I dunno, that might be a place where we can improve UX), which may have various interesting user-visible consequences such as files disappearing. I doubt that anything Tails -specific leads to such FS corruption, but one never knows. I’m not quite sure what we should do at this point.

I just experienced this problem after being forced to do a manual upgrade from Tails 2.5 to 2.6 (insufficient space on Tails system partion for automatic upgrade). Data is still there at /live/persistence/TailsData_unlocked, but no Persistent folder under the Places menu.

I am going to backup all the data present there, but it would be helpful if some documentation on how to fix the “Missing Persistent Folder” were present in places other users with the same problem might look. The documentation should address the following questions a user might have:

1. When one opens “configure persistent volume” and sees only Personal Data highlighted with the green checkmark, I experience the question: "If I select (or don’t select) these options I think I originally selected when I originally configured Tails umpteen versions ago, will the selection/nonselection cause this wizard to reset/delete my existing files, saved passwords, xmpp user accounts, gpg keys/rings, wifi passwords, etc to a default empty unconfigured state?

2. Even if I copy the contents of /live/persistence/TailsData_unlocked to a burned DVD or external usb hard drive, and the Tails persistence wizard does something I don’t like, will I be able to truly restore my data/configs in a meaningful way that allows me to just start using Tails like I always have?

3. Is it possible for me to completely clone this Tails USB drive (perfect bit for bit backup of both the Tails system AND TailsData encrypted partition), perhaps with clonezilla or a clonezilla image?

Troubleshooting documentation needs to be written for this missing (but not deleted) persistence files/directories issue. Perhaps even a tutorial that explains EXACTLY what to do and what NOT to do in the Tails persistence config wizard. We’ve all experienced some kind of data loss at some time. Not so fun. I’d really like to spare fellow Tails users from the frustration if not nightmare. A good addition to Tails documentation or other destination someone might look at when having the same problem would be a considerate action to Tails users.

#7 Updated by sajolida 2016-11-01 12:41:59

> I just experienced this problem after being forced to do a manual
> upgrade from Tails 2.5 to 2.6 (insufficient space on Tails system
> partion for automatic upgrade).

Did this happen on the first boot after the manual upgrade? or after
some time?

> I am going to backup all the data present there

Do you still have the content of persistence.conf in your backups? It’s
a file at the root of the persistent volume (also available from
/live/persistence/TailsData_unlocked/persistence.conf).

> but it would be
> helpful if some documentation on how to fix the “Missing Persistent
> Folder” were present in places other users with the same problem
> might look. The documentation should address the following questions
> a user might have:

Yeap, and thanks for all your very relevant suggestions. This is a very
painful situation and we need to fix that.

Documentation could be one way but we’re still gathering information to
see if we could actually prevent this from happening.

If we can’t fix this for real, I’m also interested in thinking a bit
more about the user scenario: how would someone goes from the problem
happening to them, to reading and applying this troubleshooting
documentation? Because if we don’t work on that, only few people might
apply the doc and likely after already being very confused, scared,
angry, and all.

For example, if the problem is a missing persistence.conf, could this be
detected and some user interface triggered when unlocking the persistence?

#8 Updated by adamantium 2016-11-03 20:23:05

First boot after the manual upgrade.

I checked the persistence.conf, and the file existed, but was completely blank.

Good news is that my backup was unnecessary, as running the persistence configuration wizard recreated a successfully fleshed out persistence.conf. Though it was tedious to create a new tails usb and manually copy the persistence related folders and apply the reassign ownership command, it was successful, and built my confidence in the manual persistence backup process. We might want to create an automatic persistence backup tool to add to the useability and robustness of Tails. Once Tails has more backup/restore functionality plus robust torified VOIP, I see increased adoption as a primary OS.

I like the idea of detecting a blank or corrupted persistence.conf file with a “repair” button for a user to click or possibly “backup, then repair”, right there in the Tails Greeter, then enter the persistence password, then check the existing persistence related folders, and recreate a proper persistence.conf.

#9 Updated by elouann 2016-11-03 21:10:42

> I checked the persistence.conf, and the file existed, but was completely blank.

I had the same experience, but persistence.conf simply disappeared in my case

> We might want to create an automatic persistence backup tool to add to the useability and robustness of Tails.

I agree. Some requirements and tests was published, do you want to have a look?
See https://tails.boum.org/blueprint/backups/

#10 Updated by sajolida 2016-11-05 18:59:00

  • Assignee set to intrigeri

> First boot after the manual upgrade.
>
> I checked the persistence.conf, and the file existed, but was
> completely blank.

That’s interesting… I’m not a file system expert but it looks like:

  • A weird coincidence to have it happen on the very first boot after the
    manual upgrade.
  • A weird file system corruption to empty a file instead of removing it.

I’ll let intrigeri add more knowledge here…

> We might want to create an automatic persistence backup tool to add
> to the useability and robustness of Tails. Once Tails has more
> backup/restore functionality plus robust torified VOIP, I see
> increased adoption as a primary OS.

That’s in our roadmap, hopefully for next year. I will most probably
conduct some user research to understand better what people need and
what would work for them before we pick up amongst the different option
that we have. If you’re interested in giving your input on this, and are
fine sharing an email with us, could you send me your address at
sajolida@pimienta.org.

> I like the idea of detecting a blank or corrupted persistence.conf
> file with a “repair” button for a user to click or possibly “backup,
> then repair”, right there in the Tails Greeter, then enter the
> persistence password, then check the existing persistence related
> folders, and recreate a proper persistence.conf.

Yeap. I thought about something like this as well. We could maybe even
always replace an empty or inexistent persistence.conf with its backup
without prompting the user.

#11 Updated by intrigeri 2016-11-11 09:32:43

  • Target version set to Tails_2.9.1

(Looks like I should have a look at it in a somewhat timely manner.)

#12 Updated by adamantium 2016-11-12 03:27:09

sajolida wrote:
> > I checked the persistence.conf, and the file existed, but was
> > completely blank.
>
> That’s interesting… I’m not a file system expert but it looks like:
>
> * A weird coincidence to have it happen on the very first boot after the
> manual upgrade.
>
> * A weird file system corruption to empty a file instead of removing it.

I agree that it is strange for a file as critical as persistence.conf to be either blanked or deleted. I didn’t think this file was touchable without root access (setting password during Tails greeter, etc.). Any speculation as to what could cause this suspicious result?

> That’s in our roadmap, hopefully for next year. I will most probably
> conduct some user research to understand better what people need and
> what would work for them before we pick up amongst the different option
> that we have. If you’re interested in giving your input on this, and are
> fine sharing an email with us, could you send me your address at
> sajolida@pimienta.org.

Look for an email from me soon.

> Yeap. I thought about something like this as well. We could maybe even
> always replace an empty or inexistent persistence.conf with its backup
> without prompting the user.

My only concerns here would be:

  • Does the “self-healing” function we might add create a possible vulnerability in Tails? Ex: would it be possible for a malicious agent to copy a malicious code/script/executable/startup_executable to a folder in /live/Persistence/TailsData_unlocked, then find a way to delete/blank/corrupt persistence.conf, and then upon next boot have our self-heal function recreate a persistence.conf that then executes the rogue code every startup?
  • What occasionally produces the blanked/deleted persistence.conf file in the first place? Have we been hacked? [My intent is NOT to be alarmist or offensive. My understanding of things is far above the average user, but not remotely to the level of sajolida and intrigeri. Truly I have extreme respect for the team that makes Tails happen.]

#13 Updated by Mango 2016-11-13 10:31:39

It seems like I currently have this problem after a couple of boot times since I upgraded to latest update. The persistence storage disppaears if you type in passphrase during login.

I thought I had lost the data but I login into a different TailsOS Usb and mount the the USB with the “missing” persistence sotrage. Typed in password and I was able to mount and access those data files.

I checked the persistence.conf file and it shows 0kb. Is their a temporary fix to this problem where it would restore the missing persistence link under ‘places’?

#14 Updated by sajolida 2016-11-13 12:17:29

I’m sorry about that :( Please backup your data and try to reconfigure
the persistent volume from Applications → Tails → Configure persistent
volume.

#15 Updated by intrigeri 2016-12-11 08:39:13

  • Target version changed from Tails_2.9.1 to Tails 2.10

#16 Updated by intrigeri 2017-01-09 18:20:18

  • Subject changed from Persistence lost, recoverable by reconfiguring to persistence.conf lost, recoverable by reconfiguring
  • Assignee changed from intrigeri to emmapeel
  • Target version deleted (Tails 2.10)
  • QA Check set to Info Needed

I’ve just re-read this entire ticket and sadly, “what we need is the debugging info from a session in which the persistence seems to be lost” still holds true.

Regarding when this problem happens immediately after a manual upgrade: I really don’t understand. The manual upgrade process does not unlock the persistent filesystem, and it doesn’t even touch that partition, so I don’t see how it could empty persistence.conf in there. In this case even more than in the general case, I’ll need debugging info to try and understand what’s happening.

So I’m adding help desk people as watchers, and hopefully, one of these days, a user will send a bug report during the session that’s affected by the problem, and our amazing help desk will forward the debugging info to me. And when you get such a bug report, please ask the OP:

  • the output of df -h;
  • the output of ls -l /live/persistence/*_unlocked/*.conf* (I’m interested in the mtime, not just in the size);
  • the output of ls -l /live/persistence/;
  • the output of getfacl /live/persistence/*_unlocked /live/persistence/*_unlocked/*.conf*;
  • whether they have reconfigured their persistence, in any way, during the previous Tails session (goal: pinpoint a bug in the persistence setup software);
  • whether they have done a clean shutdown to end the previous Tails session (i.e. via the GNOME top-right menu).

#17 Updated by sajolida 2017-01-26 19:27:59

https://twitter.com/seancc317/status/822966170278051841

Jan 22
sean collins-cruz
@seancc317

@Tails_live what can we do if persistent folder is gone after update? Am I totally screwed?

Seems to be the same and also happened after an upgrade.

#18 Updated by emmapeel 2018-02-15 09:18:15

  • Assignee changed from emmapeel to sajolida

I found out about a group of users that were resorting to a very painful workaround:

WHen this happened to them, they will start another Tails from scratch and copy the contents, instead of just recofinguring the persistent.conf file. Ouch!

Couldn’t we at least add some documentation somewhere while we wait for this information?

#19 Updated by sajolida 2018-02-16 21:25:43

  • blocks Feature #14758: Core work 2017Q4 → 2018Q1: Technical writing added

#20 Updated by adamantium 2018-02-19 23:02:15

I recently had the “blanked 0k persistence.conf” phenomena happen for me again. This was maybe a few boots after I manually upgraded to Tails 3.5. I successfully backed up and ran the configure persistence wizard and got it back, same as last time. There might have been some lack of proper shutdown (just pull the usb out of the computer and wait for the auto “security” shutdown). I can’t remember the details, and I don’t have a debug log (sorry sajolida and intrigeri). I also don’t know how to create or find such debug logs.

Do we have any more ideas on the possibility of detecting whether persistent features are present and self-healing by automatically recreating a new persistence.conf? Any further speculation on what could cause a blanked persistence.conf? This is truly mystifying.

#21 Updated by sajolida 2018-03-27 11:27:15

  • blocked by deleted (Feature #14758: Core work 2017Q4 → 2018Q1: Technical writing)

#22 Updated by sajolida 2018-03-27 11:27:22

  • blocks Feature #15411: Core work 2018Q2 → 2018Q3: Technical writing added

#23 Updated by sajolida 2018-03-27 11:27:27

I’m out of budget for this quarter.

#24 Updated by sajolida 2018-04-11 13:40:13

  • Assignee changed from sajolida to cbrownstein

Cody: I’m assiging this one to you as part of the big problem space “Persistent storage vs Backups”.
Please check which parts of it are the best to tackle first.

#25 Updated by intrigeri 2018-04-27 13:21:00

adamantium wrote:
> I recently had the “blanked 0k persistence.conf” phenomena happen for me again. This was maybe a few boots after I manually upgraded to Tails 3.5. I successfully backed up and ran the configure persistence wizard and got it back, same as last time.

See Bug #10976#note-16 :)

#26 Updated by intrigeri 2018-04-30 11:18:02

I got one report from help desk, sent on Apr 2, about this with some useful debugging info. tl;dr:

  • the persistent FS has plenty of free space
  • there’s no *.conf.insecure_disabled config file
  • all permissions and ACLs are good
  • /live/persistence/TailsData_unlocked was last modified on Apr 1 at 06:20; same for persistence.conf; so something definitely happened at that time
  • persistence.conf is empty
  • live-additional-software.conf was last modified on Apr 1 at 08:06 and is empty too
  • The user says they didn’t reconfigure their persistence nor use the emergency shutdown during the last session the persistent volume was working normally.

I’ve not found anything in our code, nor in live-boot, that touches persistence.conf except Tails Persistence Setup and live-persist (and the latter can only rename it to persistence.conf.insecure_disabled if permissions are wrong). So I fail to understand how this can happen without user intervention :/

#27 Updated by sajolida 2018-06-03 16:21:29

  • Assignee changed from cbrownstein to intrigeri
  • QA Check deleted (Info Needed)

How hard would it be possible to have a backup of persistence.conf, detect when the bug happens, and restore automatically from that?

  • persistence.conf.bak would get updated every time persistence.conf is modified by the user interface.
  • When unlocking the persistence, if persistence.conf is empty or unexisting we could compare the two files and copy back persistence.conf.bak onto persistence.conf.

This would automate what we are asking users to do when the bug happens…

#28 Updated by sajolida 2018-06-03 16:22:36

  • blocked by deleted (Feature #15411: Core work 2018Q2 → 2018Q3: Technical writing)

#29 Updated by sajolida 2018-06-03 16:44:40

I rephrased Bug #10976#note-27 after posting it in order to be more neutral. Sorry!

#30 Updated by intrigeri 2018-06-03 17:16:33

  • Target version set to Tails_3.10.1
  • QA Check set to Info Needed

Thanks for pushing me to think about workarounds to what looks more and more like an ext4 bug, possibly triggered by user actions that users (understandably) don’t expect will trigger any issue and don’t think of as potentially related.

I’m happy look into this at some point but it would not be realistic to expect this to happen before September. Meanwhile, whoever sees this happen (in particular Tails contributors) should send me a WhisperBack report + the debug info I’ve requested 1 year ago on this ticket, both from the very first session during which the problem is detected. Ideally, attach a timeline of what happened in the days before the bug, i.e. when this Tails was booted with persistence enabled, when it was upgraded, etc. (and technical users can look at that debugging info and from the files mtimes, they’ll guess what part of the timeline I’m mostly interested in: what happened around these mtimes :) This might help identify and fix the root cause before I spend time looking for a workaround.

And BTW I was wrong: we do have a piece of code that can, if buggy, trigger that: migrate_icedove_to_thunderbird() modifies persistence.conf. It’s been there for a year and it would take me a while to understand if the sed-based code can cause any issue, so perhaps a first step would be to drop that code.

#31 Updated by emmapeel 2018-06-27 07:52:09

It is great to try to fix this issue for good, but I humbly ask again for a mention in the known issues page or some note, as I think this is very scary users and potentially leads to lost information, see for example:

https://twitter.com/Vyshakcool95/status/1010193002923376641

#32 Updated by intrigeri 2018-06-27 08:41:08

> It is great to try to fix this issue for good, but I humbly ask again for a mention in the known issues page or some note,

Agreed: see the Bug #15598 sub-task.

#33 Updated by emmapeel 2018-06-27 09:10:19

Sorry, I had overlooked that issue. Great!

#34 Updated by sajolida 2018-06-27 11:45:29

emmapeel: Don’t hesitate to ping cbrownstein on behalf of front desk on Bug #15598.

#35 Updated by intrigeri 2018-06-30 16:44:55

  • Status changed from Confirmed to In Progress

Applied in changeset commit:e0e1b7b5f4b8256f988c3eda58b048041d6a8a5f.

#36 Updated by intrigeri 2018-08-23 05:40:20

Dear help desk, 3.9~rc1 removes the code I (vaguely hopefully) suspect was the culprit. But it also adds code that modifies the permissions on live-additional-software.conf on upgrades from 3.8 or older. So please keep an eye on this and tell me whether such issues happen again when running 3.9~rc1 or newer. The timing of the problem happening vs. upgrade to 3.9~rc1 or newer matters so please try to collect this info.

#37 Updated by sajolida 2018-08-31 11:49:13

  • related to Feature #14544: Spend software developer time on smallish UX improvements added

#38 Updated by intrigeri 2018-09-11 10:04:47

  • Assignee changed from intrigeri to goupille

intrigeri wrote:
> Dear help desk, 3.9~rc1 removes the code I (vaguely hopefully) suspect was the culprit. But it also adds code that modifies the permissions on live-additional-software.conf on upgrades from 3.8 or older. So please keep an eye on this and tell me whether such issues happen again when running 3.9~rc1 or newer. The timing of the problem happening vs. upgrade to 3.9~rc1 or newer matters so please try to collect this info.

Same question for upgrades to 3.9.

#39 Updated by goupille 2018-09-15 16:59:13

  • Assignee changed from goupille to intrigeri

at least one user is affected with tails 3.9 (I’m forwarding the bug report to you)

#40 Updated by goupille 2018-09-15 17:04:33

goupille wrote:
> at least two different users are affected with tails 3.9 (I’m forwarding a bug report to you)

#41 Updated by intrigeri 2018-09-18 08:58:42

I’ve analyzed report 2abf0655f6cc39a34db3b78ab6ba6a6c:

  • The persistent filesystem had not been unmounted cleanly. So data corruption is to be expected.
  • persistence.conf is not empty: it has /home/amnesia/Persistent. I don’t know if the user fixed it already themselves or not.
  • Apparently, timing is unrelated to upgrading Tails (or at least the user does not say so).
  • I don’t know if help desk tried to gather the info I’ve requested on 10976#note-16. I’ll try to do it myself.

#42 Updated by intrigeri 2018-10-20 11:06:26

  • Target version changed from Tails_3.10.1 to Tails_3.11

#43 Updated by goupille 2018-10-26 11:51:27

intrigeri wrote:

> * I don’t know if help desk tried to gather the info I’ve requested on 10976#note-16. I’ll try to do it myself.

I just resent you a report from another user with those informations (ticket number in the subject)

#44 Updated by pablonatalino 2018-10-27 16:59:38

I experienced this myself now.

#45 Updated by intrigeri 2018-10-31 10:23:08

Analyzed 4dc80ae26f8e3397fbe1437e1e1b232f: empty config file, mtime is ~20 minutes before the user sent their bug report. Permissions look OK, no disabled config file, no disk space issue.

Same for “Upgrade problems PLEASE HELP” except mtime is 2 days before help desk forwarded me the report (I was not told when the report was sent).

#46 Updated by sajolida 2018-11-02 05:36:18

Do I understand correctly and the fix proposed in Bug #10976#note-36 was not enough to solve the issue?

If so, should we consider building resiliency mechanisms even though we don’t know what is the root cause of the problem, like proposed in Bug #10976#note-27?

#47 Updated by intrigeri 2018-11-05 14:39:03

  • QA Check changed from Info Needed to Dev Needed

> Do I understand correctly and the fix proposed in Bug #10976#note-36 was not enough to solve the issue?

Yes.

> If so, should we consider building resiliency mechanisms even though we don’t know what is the root cause of the problem, like proposed in Bug #10976#note-27?

Yes, that’s the plan I agreed with and why this ticket is assigned to me with this target version. I’m not happy I did not manage to get to it since mid-September but it’s not because I think it’s unimportant :/

#48 Updated by sajolida 2018-11-06 02:04:52

Thanks! And sorry if my request for clarification sounded pushy.

#49 Updated by intrigeri 2018-11-06 08:46:07

  • related to Bug #15598: Explain loss of persistence.conf in known issues added

#50 Updated by intrigeri 2018-11-06 08:46:19

  • Priority changed from Normal to Elevated

#51 Updated by intrigeri 2018-12-02 21:39:16

  • Target version changed from Tails_3.11 to Tails_3.12

Too busy next week so I’d rather keep my spare cycles to polish, if needed, the various branches of mine that are waiting to be reviewed.

#52 Updated by intrigeri 2019-01-14 19:23:25

  • Target version changed from Tails_3.12 to Tails_3.13

Missed 3.12 too :/

#53 Updated by intrigeri 2019-01-24 16:18:00

Note to myself, from chattr(1):

       When  a  file with the 'S' attribute set is modified, the changes are written syn‐
       chronously on the disk; this is equivalent to the 'sync' mount option applied to a
       subset of the files.

I think we should set this on our persistence config files.

#54 Updated by intrigeri 2019-02-12 18:57:22

intrigeri wrote:
> Note to myself, from chattr(1):
>
> […]
>
> I think we should set this on our persistence config files.

Done on my branch for Bug #16461, where I’m tracking the round of improvements I want to make in 3.13.

#55 Updated by intrigeri 2019-02-12 19:00:49

  • Assignee changed from intrigeri to mercedes508
  • Target version changed from Tails_3.13 to Tails_3.14
  • QA Check changed from Dev Needed to Info Needed

(Assuming that Bug #16461 lands in 3.13.)

Hi @mercedes508 :) I see you’ll do the majority of the help desk shifts that follow the 3.13 release. So, please report 4-5 weeks after the 3.13 release, i.e. around mid-April, if this is still a problem; then reassign to me. Thanks in advance!

#56 Updated by intrigeri 2019-02-14 08:50:33

FTR this didn’t make it to “hot topics for help desk” since 6+ months, which suggests that maybe one of the robustness improvements I’ve done already helped. Help desk tells me they think it still happens once in a while.

#57 Updated by intrigeri 2019-03-17 08:43:10

intrigeri wrote:
> (Assuming that Bug #16461 lands in 3.13.)
>
> Hi @mercedes508 :) I see you’ll do the majority of the help desk shifts that follow the 3.13 release. So, please report 4-5 weeks after the 3.13 release, i.e. around mid-April, if this is still a problem; then reassign to me. Thanks in advance!

The branch for Bug #16461 was merged so I confirm my above request.

And if users still experience this problem, once they run 3.13 there should be a backup persistence.conf.bak file next to the empty persistence.conf => please ask them to share the contents of the backup file and try copying it to persistence.conf, instead of re-configuring their persistence from scratch :)

#58 Updated by mercedes508 2019-03-28 13:52:28

2a38a41a59194d76966d26b25c35f8b2
66342e7c-e525-1997-664d-141d35b8798b
CAMV72DhA5tPP4Yq=v4T3nVDpj-12-kbdrdmiQYL66X6=JKJTYw

#59 Updated by goupille 2019-04-10 18:41:23

Bug report: 942e619be10a2bbdc2e48a6cd1858f08

#60 Updated by CyrilBrulebois 2019-05-23 21:23:21

  • Target version changed from Tails_3.14 to Tails_3.15

#61 Updated by intrigeri 2019-06-02 15:27:22

  • QA Check deleted (Info Needed)

#62 Updated by CyrilBrulebois 2019-07-10 10:33:57

  • Target version changed from Tails_3.15 to Tails_3.16

#63 Updated by intrigeri 2019-08-30 17:35:07

  • Assignee changed from mercedes508 to numbat

Here as well, I doubt mercedes508 will take care of this => reassigning to another help desk member picked at random.

#64 Updated by CyrilBrulebois 2019-09-05 00:05:30

  • Target version changed from Tails_3.16 to Tails_3.17

#65 Updated by intrigeri 2019-09-12 14:25:12

  • Target version changed from Tails_3.17 to Tails_4.0

#66 Updated by cbrownstein 2019-09-27 00:53:14

  • Assignee changed from numbat to intrigeri

I’m currently experiencing this bug. I’ve followed the instructions on the known issues page [1] but my persistent features and files are still missing.

[1]: https://tails.boum.org/support/known_issues/index.en.html#index14h2

N.B. My files and settings still exist in the persistent volume but they aren’t being loaded.

Below is the information I’ve collected per Bug #10976#note-16. I’m still experiencing the bug, so I can collect more information if additional information would be helpful.

> * the output of df -h;

root@amnesia:/live/persistence/TailsData_unlocked# df -h
Filesystem                      Size  Used Avail Use% Mounted on
udev                            3.9G     0  3.9G   0% /dev
tmpfs                           787M   18M  769M   3% /run
/dev/sdb1                       8.0G  1.3G  6.8G  16% /lib/live/mount/medium
/dev/loop0                      1.1G  1.1G     0 100% /lib/live/mount/rootfs/filesystem.squashfs
/dev/loop1                      128M  128M     0 100% /lib/live/mount/rootfs/3.15.squashfs
tmpfs                           3.9G   40M  3.9G   1% /lib/live/mount/overlay
aufs                            3.9G   40M  3.9G   1% /
tmpfs                           3.9G   23M  3.9G   1% /dev/shm
tmpfs                           5.0M  4.0K  5.0M   1% /run/lock
tmpfs                           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs                           3.9G     0  3.9G   0% /var/tmp
tmpfs                           3.9G   51M  3.8G   2% /run/initramfs
tmpfs                           3.9G   12K  3.9G   1% /tmp
tmpfs                           787M   20K  787M   1% /run/user/113
/dev/mapper/TailsData_unlocked   21G  1.2G   18G   7% /live/persistence/TailsData_unlocked
tmpfs                           787M   24K  787M   1% /run/user/1000

> * the output of ls -l /live/persistence/*_unlocked/*.conf* (I’m interested in the mtime, not just in the size);

root@amnesia:/live/persistence/TailsData_unlocked# ls -l /live/persistence/*_unlocked/*.conf*
-rw-r--r-- 1 tails-persistence-setup tails-persistence-setup   0 Aug 28 17:57 /live/persistence/TailsData_unlocked/live-additional-software.conf
-rw-r--r-- 1 tails-persistence-setup tails-persistence-setup   0 Aug 28 17:51 /live/persistence/TailsData_unlocked/live-additional-software.conf.insecure_disabled
-rw------- 1 tails-persistence-setup tails-persistence-setup   0 Aug 28 17:57 /live/persistence/TailsData_unlocked/persistence.conf
-rw------- 1 tails-persistence-setup tails-persistence-setup 514 Aug 29 00:55 /live/persistence/TailsData_unlocked/persistence.conf.bak
-rw------- 1 tails-persistence-setup tails-persistence-setup 514 Aug 29 00:55 /live/persistence/TailsData_unlocked/persistence.conf.insecure_disabled

> * the output of ls -l /live/persistence/;

root@amnesia:/live/persistence/TailsData_unlocked# ls -l /live/persistence/
total 4
drwxrwxr-x+ 14 amnesia amnesia 4096 Aug 28 17:57 TailsData_unlocked

> * the output of getfacl /live/persistence/*_unlocked /live/persistence/*_unlocked/*.conf*;

root@amnesia:/live/persistence/TailsData_unlocked# getfacl /live/persistence/*_unlocked /live/persistence/*_unlocked/*.conf*
getfacl: Removing leading '/' from absolute path names
# file: live/persistence/TailsData_unlocked
# owner: amnesia
# group: amnesia
user::rwx
user:tails-persistence-setup:rwx
group::rwx
mask::rwx
other::r-x

# file: live/persistence/TailsData_unlocked/live-additional-software.conf
# owner: tails-persistence-setup
# group: tails-persistence-setup
user::rw-
group::r--
other::r--

# file: live/persistence/TailsData_unlocked/live-additional-software.conf.insecure_disabled
# owner: tails-persistence-setup
# group: tails-persistence-setup
user::rw-
group::r--
other::r--

# file: live/persistence/TailsData_unlocked/persistence.conf
# owner: tails-persistence-setup
# group: tails-persistence-setup
user::rw-
group::---
other::---

# file: live/persistence/TailsData_unlocked/persistence.conf.bak
# owner: tails-persistence-setup
# group: tails-persistence-setup
user::rw-
group::---
other::---

# file: live/persistence/TailsData_unlocked/persistence.conf.insecure_disabled
# owner: tails-persistence-setup
# group: tails-persistence-setup
user::rw-
group::---
other::---

> * whether they have reconfigured their persistence, in any way, during the previous Tails session (goal: pinpoint a bug in the persistence setup software);

I added and removed additional software.

> * whether they have done a clean shutdown to end the previous Tails session (i.e. via the GNOME top-right menu).

Yes, clean shutdown every time I’ve used Tails.

Also, here is the contents of persistence.conf.bak:

root@amnesia:/live/persistence/TailsData_unlocked# cat persistence.conf.bak 
/home/amnesia/Persistent        source=Persistent
/home/amnesia/.mozilla/firefox/bookmarks        source=bookmarks
/etc/NetworkManager/system-connections        source=nm-system-connections
/var/cache/apt/archives        source=apt/cache
/var/lib/apt/lists        source=apt/lists
/etc/cups        source=cups-configuration
/home/amnesia/.thunderbird        source=thunderbird
/home/amnesia/.gnupg        source=gnupg
/home/amnesia/.electrum        source=electrum
/home/amnesia/.purple        source=pidgin
/home/amnesia/.ssh        source=openssh-client
/home/amnesia        source=dotfiles,link

Copying persistence.conf.bak to persistence.conf did nothing.

#67 Updated by intrigeri 2019-09-30 09:30:23

Note to myself: according to https://lwn.net/Articles/799807/, calling sync on the directory that contains persistence.conf may help.

#68 Updated by intrigeri 2019-09-30 09:49:15

Hi @cbrownstein,

> I’m currently experiencing this bug. I’ve followed the instructions on the known issues page [1] but my persistent features and files are still missing.

It seems to me that there’s a step missing in that doc: between steps 3 and 4, you must enable the persistence features you need.
You might want to file a tech writing ticket to fix this :)

>> * the output of ls -l /live/persistence/;

>

> root@amnesia:/live/persistence/TailsData_unlocked# ls -l /live/persistence/
> total 4
> drwxrwxr-x+ 14 amnesia amnesia 4096 Aug 28 17:57 TailsData_unlocked
> 

Ouch, that’s wrong, which is why your persistence.conf was renamed to /live/persistence/TailsData_unlocked/persistence.conf.insecure_disabled.
To fix this, you need to run sudo chown root:root /live/persistence/TailsData_unlocked.

Now, the question is: how was your persistent filesystem given to amnesia:amnesia?

So:

  • When was the list time your persistent volume worked as expected? Your comments suggest it was late August. Is this correct? Or is your hardware clock ~1 month stuck in the past?
  • Did you do anything special with this filesystem between the last time your Tails stick worked, and the time when it became buggy (late August according to your comment)?
  • If you did not: on my side, I’ll dive into our codebase to try & find which code might have messed this up.

>> * whether they have reconfigured their persistence, in any way, during the previous Tails session (goal: pinpoint a bug in the persistence setup software);

> I added and removed additional software.

Thanks, this is useful. I’ll look particularly at the Additional Software code paths that may trigger those insecure permissions.

Do you remember when was that previous Tails session, during which you added and removed Additional Software?

> Copying persistence.conf.bak to persistence.conf did nothing.

Please retry after having fixed the ownership of the root directory of your persistent volume, as suggested above :)

#69 Updated by intrigeri 2019-09-30 10:03:36

  • Assignee deleted (intrigeri)

Cody’s report suggests the problem is not in tails-persistence-setup, so any FT member should be in a position to work on this. I don’t know when I will be able to work on this so I’ll avoid cookie licking.

#70 Updated by intrigeri 2019-09-30 10:03:51

#71 Updated by intrigeri 2019-09-30 16:53:36

  • Priority changed from Normal to Elevated
  • Target version deleted (Tails_4.0)

(The target version was set 8 months ago for the help desk part of the job, i.e. collecting data.)

#72 Updated by cbrownstein 2019-09-30 18:02:02

> > I’m currently experiencing this bug. I’ve followed the instructions on the known issues page [1] but my persistent features and files are still missing.
>
> It seems to me that there’s a step missing in that doc: between steps 3 and 4, you must enable the persistence features you need.
> You might want to file a tech writing ticket to fix this :)

I will add that step to the doc. But, I did re-enable the persistent features that were enabled before my persistent volume stopped working.

> >> * the output of ls -l /live/persistence/;
>
> > […]
>
> Ouch, that’s wrong, which is why your persistence.conf was renamed to /live/persistence/TailsData_unlocked/persistence.conf.insecure_disabled.
> To fix this, you need to run sudo chown root:root /live/persistence/TailsData_unlocked.

I will follow your instruction and update this ticket with the result.

> Now, the question is: how was your persistent filesystem given to amnesia:amnesia?
>
> So:
>
> * When was the list time your persistent volume worked as expected? Your comments suggest it was late August. Is this correct? Or is your hardware clock ~1 month stuck in the past?

This is correct: late August.

> * Did you do anything special with this filesystem between the last time your Tails stick worked, and the time when it became buggy (late August according to your comment)?
> * If you did not: on my side, I’ll dive into our codebase to try & find which code might have messed this up.

I did not do anything special with the filesystem.

> >> * whether they have reconfigured their persistence, in any way, during the previous Tails session (goal: pinpoint a bug in the persistence setup software);
>
> > I added and removed additional software.
>
> Thanks, this is useful. I’ll look particularly at the Additional Software code paths that may trigger those insecure permissions.

I’m not sure if this matters but I was using apt (rather than synaptic or other tool) to install and remove packages.

> Do you remember when was that previous Tails session, during which you added and removed Additional Software?

I believe, but don’t remember with 100% certainty, that my persistent volume survived at least one reboot after installing packages. But, I believe my persistent volume was missing the session immediately after removing packages.

> > Copying persistence.conf.bak to persistence.conf did nothing.
>
> Please retry after having fixed the ownership of the root directory of your persistent volume, as suggested above :)

I will, and will update this ticket with the result.

#73 Updated by cbrownstein 2019-10-01 17:27:38

> >> * the output of ls -l /live/persistence/;
>
> > […]
>
> Ouch, that’s wrong, which is why your persistence.conf was renamed to /live/persistence/TailsData_unlocked/persistence.conf.insecure_disabled.
> To fix this, you need to run sudo chown root:root /live/persistence/TailsData_unlocked.

Changing the ownership of TailsData_unlocked then copying persistence.conf.bak to persistence.conf worked to fix the problem.

However, live-additional-software.conf is empty, meaning none of the additional software I added is being installed. live-additional-software.conf.insecure_disabled exists but is empty and useless.

At least the packages are still in apt/cache to help me refresh my memory as to what I had added.

#74 Updated by intrigeri 2019-10-02 06:31:36

  • related to Bug #17112: Persistence config files "insecure" backup get emptied in some situations added

#75 Updated by intrigeri 2019-10-02 06:33:07

> Changing the ownership of TailsData_unlocked then copying persistent.conf.bak to persistent.conf worked to fix the problem.

Great! I’ll let you do whatever is needed so that the tech writing team has this problem on their radar and fixes the doc at some point :)

> However, live-additional-software.conf is empty, meaning none of the additional software I added is being installed. live-additional-software.conf.insecure_disabled exists but is empty and useless.

Ouch. I think I understood why this happened ⇒ filed Bug #17112 to track it.

Thanks a lot for the detailed report!

#76 Updated by cbrownstein 2019-10-02 18:42:48

  • related to Bug #17116: Improve lost persistence.conf instructions on known issues page added

#77 Updated by intrigeri 2019-10-07 06:41:28

intrigeri wrote:
> Note to myself: according to https://lwn.net/Articles/799807/, calling sync on the directory that contains persistence.conf may help.

Done on the branch for Bug #15102.

#78 Updated by intrigeri 2020-02-23 10:37:00

@numbat, dear help desk, did you see this problem reported on Tails 4.x?