Tails

#2 Updated by emmapeel 2016-01-26 20:12:50

intrigeri wrote:
> > Once in a while this happens since many Tails, maybe the persistence.conf file gets broken.
>
> > But since Tails 1.8.2 many of such reports have been made, most of the times only rebooting normally (no emergency shutdown, nothing)
>
> > The Persistence is easily recovered by reconfiguring it again, with same passphrase.
>
> I’m not sure what you mean with “persistence lost”. The fact it can be recovered leads me to think that the persistent data is not lost, simply not set up / mounted properly, probably due to a corrupted live-persistence.conf file

Exactly!
>
> * the user is still proposed to unlock their persistent volume in the Greeter;
yes

> * the data is still available in /live/persistence/TailsData_unlocked once unlocked.
yes
>
> This would be consistent with the fact that re-creating a proper live-persistence.conf by re-configuring the persistence makes it possible to have the persistent data accessible via usual paths again.
>
> Right?
>
> If I got it right, two things:
>
> * I’ve heard similar reports regularly for years, so I don’t think this problem is new; now, I totally believe you that it apparently happens much more often recently.

Yes, I think this last month there were more.
> * what we need is the debugging info from a session in which the persistence seems to be lost, and the content of live-persistence.conf in that same session. It’s just too sad to always hear about this problem, but never have seen the info that would help us understand what’s happening.
>
Ok, will try to get this information.

#3 Updated by sajolida 2016-02-23 11:37:34

I experienced this myself in c7425c8012ba2f65e0b9876d11a573f7.

#6 Updated by adamantium 2016-10-19 00:02:13

> intrigeri wrote:
> > sajolida wrote:
> >
> > So, for some reason that filesystem was corrupted (possibly filling the FS as you say you did didn’t help ext4 being robust, I dunno, that might be a place where we can improve UX), which may have various interesting user-visible consequences such as files disappearing. I doubt that anything Tails -specific leads to such FS corruption, but one never knows. I’m not quite sure what we should do at this point.

I just experienced this problem after being forced to do a manual upgrade from Tails 2.5 to 2.6 (insufficient space on Tails system partion for automatic upgrade). Data is still there at /live/persistence/TailsData_unlocked, but no Persistent folder under the Places menu.

I am going to backup all the data present there, but it would be helpful if some documentation on how to fix the “Missing Persistent Folder” were present in places other users with the same problem might look. The documentation should address the following questions a user might have:

1. When one opens “configure persistent volume” and sees only Personal Data highlighted with the green checkmark, I experience the question: "If I select (or don’t select) these options I think I originally selected when I originally configured Tails umpteen versions ago, will the selection/nonselection cause this wizard to reset/delete my existing files, saved passwords, xmpp user accounts, gpg keys/rings, wifi passwords, etc to a default empty unconfigured state?

2. Even if I copy the contents of /live/persistence/TailsData_unlocked to a burned DVD or external usb hard drive, and the Tails persistence wizard does something I don’t like, will I be able to truly restore my data/configs in a meaningful way that allows me to just start using Tails like I always have?

3. Is it possible for me to completely clone this Tails USB drive (perfect bit for bit backup of both the Tails system AND TailsData encrypted partition), perhaps with clonezilla or a clonezilla image?

Troubleshooting documentation needs to be written for this missing (but not deleted) persistence files/directories issue. Perhaps even a tutorial that explains EXACTLY what to do and what NOT to do in the Tails persistence config wizard. We’ve all experienced some kind of data loss at some time. Not so fun. I’d really like to spare fellow Tails users from the frustration if not nightmare. A good addition to Tails documentation or other destination someone might look at when having the same problem would be a considerate action to Tails users.

#7 Updated by sajolida 2016-11-01 12:41:59

> I just experienced this problem after being forced to do a manual
> upgrade from Tails 2.5 to 2.6 (insufficient space on Tails system
> partion for automatic upgrade).

Did this happen on the first boot after the manual upgrade? or after
some time?

> I am going to backup all the data present there

Do you still have the content of persistence.conf in your backups? It’s
a file at the root of the persistent volume (also available from
/live/persistence/TailsData_unlocked/persistence.conf).

> but it would be
> helpful if some documentation on how to fix the “Missing Persistent
> Folder” were present in places other users with the same problem
> might look. The documentation should address the following questions
> a user might have:

Yeap, and thanks for all your very relevant suggestions. This is a very
painful situation and we need to fix that.

Documentation could be one way but we’re still gathering information to
see if we could actually prevent this from happening.

If we can’t fix this for real, I’m also interested in thinking a bit
more about the user scenario: how would someone goes from the problem
happening to them, to reading and applying this troubleshooting
documentation? Because if we don’t work on that, only few people might
apply the doc and likely after already being very confused, scared,
angry, and all.

For example, if the problem is a missing persistence.conf, could this be
detected and some user interface triggered when unlocking the persistence?

#8 Updated by adamantium 2016-11-03 20:23:05

First boot after the manual upgrade.

I checked the persistence.conf, and the file existed, but was completely blank.

Good news is that my backup was unnecessary, as running the persistence configuration wizard recreated a successfully fleshed out persistence.conf. Though it was tedious to create a new tails usb and manually copy the persistence related folders and apply the reassign ownership command, it was successful, and built my confidence in the manual persistence backup process. We might want to create an automatic persistence backup tool to add to the useability and robustness of Tails. Once Tails has more backup/restore functionality plus robust torified VOIP, I see increased adoption as a primary OS.

I like the idea of detecting a blank or corrupted persistence.conf file with a “repair” button for a user to click or possibly “backup, then repair”, right there in the Tails Greeter, then enter the persistence password, then check the existing persistence related folders, and recreate a proper persistence.conf.

#9 Updated by elouann 2016-11-03 21:10:42

> I checked the persistence.conf, and the file existed, but was completely blank.

I had the same experience, but persistence.conf simply disappeared in my case

> We might want to create an automatic persistence backup tool to add to the useability and robustness of Tails.

I agree. Some requirements and tests was published, do you want to have a look?
See https://tails.boum.org/blueprint/backups/

#12 Updated by adamantium 2016-11-12 03:27:09

sajolida wrote:
> > I checked the persistence.conf, and the file existed, but was
> > completely blank.
>
> That’s interesting… I’m not a file system expert but it looks like:
>
> * A weird coincidence to have it happen on the very first boot after the
> manual upgrade.
>
> * A weird file system corruption to empty a file instead of removing it.

I agree that it is strange for a file as critical as persistence.conf to be either blanked or deleted. I didn’t think this file was touchable without root access (setting password during Tails greeter, etc.). Any speculation as to what could cause this suspicious result?

> That’s in our roadmap, hopefully for next year. I will most probably
> conduct some user research to understand better what people need and
> what would work for them before we pick up amongst the different option
> that we have. If you’re interested in giving your input on this, and are
> fine sharing an email with us, could you send me your address at
> sajolida@pimienta.org.

Look for an email from me soon.

> Yeap. I thought about something like this as well. We could maybe even
> always replace an empty or inexistent persistence.conf with its backup
> without prompting the user.

My only concerns here would be:

• Does the “self-healing” function we might add create a possible vulnerability in Tails? Ex: would it be possible for a malicious agent to copy a malicious code/script/executable/startup_executable to a folder in /live/Persistence/TailsData_unlocked, then find a way to delete/blank/corrupt persistence.conf, and then upon next boot have our self-heal function recreate a persistence.conf that then executes the rogue code every startup?

• What occasionally produces the blanked/deleted persistence.conf file in the first place? Have we been hacked? [My intent is NOT to be alarmist or offensive. My understanding of things is far above the average user, but not remotely to the level of sajolida and intrigeri. Truly I have extreme respect for the team that makes Tails happen.]

#13 Updated by Mango 2016-11-13 10:31:39

It seems like I currently have this problem after a couple of boot times since I upgraded to latest update. The persistence storage disppaears if you type in passphrase during login.

I thought I had lost the data but I login into a different TailsOS Usb and mount the the USB with the “missing” persistence sotrage. Typed in password and I was able to mount and access those data files.

I checked the persistence.conf file and it shows 0kb. Is their a temporary fix to this problem where it would restore the missing persistence link under ‘places’?

#14 Updated by sajolida 2016-11-13 12:17:29

I’m sorry about that :( Please backup your data and try to reconfigure
the persistent volume from Applications → Tails → Configure persistent
volume.

#17 Updated by sajolida 2017-01-26 19:27:59

https://twitter.com/seancc317/status/822966170278051841

Jan 22
sean collins-cruz
@seancc317

@Tails_live what can we do if persistent folder is gone after update? Am I totally screwed?

Seems to be the same and also happened after an upgrade.

#20 Updated by adamantium 2018-02-19 23:02:15

I recently had the “blanked 0k persistence.conf” phenomena happen for me again. This was maybe a few boots after I manually upgraded to Tails 3.5. I successfully backed up and ran the configure persistence wizard and got it back, same as last time. There might have been some lack of proper shutdown (just pull the usb out of the computer and wait for the auto “security” shutdown). I can’t remember the details, and I don’t have a debug log (sorry sajolida and intrigeri). I also don’t know how to create or find such debug logs.

Do we have any more ideas on the possibility of detecting whether persistent features are present and self-healing by automatically recreating a new persistence.conf? Any further speculation on what could cause a blanked persistence.conf? This is truly mystifying.

#23 Updated by sajolida 2018-03-27 11:27:27

I’m out of budget for this quarter.

#25 Updated by intrigeri 2018-04-27 13:21:00

adamantium wrote:
> I recently had the “blanked 0k persistence.conf” phenomena happen for me again. This was maybe a few boots after I manually upgraded to Tails 3.5. I successfully backed up and ran the configure persistence wizard and got it back, same as last time.

See Bug #10976#note-16 :)

#26 Updated by intrigeri 2018-04-30 11:18:02

I got one report from help desk, sent on Apr 2, about this with some useful debugging info. tl;dr:

• the persistent FS has plenty of free space
• there’s no *.conf.insecure_disabled config file
• all permissions and ACLs are good
• /live/persistence/TailsData_unlocked was last modified on Apr 1 at 06:20; same for persistence.conf; so something definitely happened at that time
• persistence.conf is empty
• live-additional-software.conf was last modified on Apr 1 at 08:06 and is empty too
• The user says they didn’t reconfigure their persistence nor use the emergency shutdown during the last session the persistent volume was working normally.

I’ve not found anything in our code, nor in live-boot, that touches persistence.conf except Tails Persistence Setup and live-persist (and the latter can only rename it to persistence.conf.insecure_disabled if permissions are wrong). So I fail to understand how this can happen without user intervention :/

#29 Updated by sajolida 2018-06-03 16:44:40

I rephrased Bug #10976#note-27 after posting it in order to be more neutral. Sorry!

#31 Updated by emmapeel 2018-06-27 07:52:09

It is great to try to fix this issue for good, but I humbly ask again for a mention in the known issues page or some note, as I think this is very scary users and potentially leads to lost information, see for example:

https://twitter.com/Vyshakcool95/status/1010193002923376641

#32 Updated by intrigeri 2018-06-27 08:41:08

> It is great to try to fix this issue for good, but I humbly ask again for a mention in the known issues page or some note,

Agreed: see the Bug #15598 sub-task.

#33 Updated by emmapeel 2018-06-27 09:10:19

Sorry, I had overlooked that issue. Great!

#34 Updated by sajolida 2018-06-27 11:45:29

emmapeel: Don’t hesitate to ping cbrownstein on behalf of front desk on Bug #15598.

#36 Updated by intrigeri 2018-08-23 05:40:20

Dear help desk, 3.9~rc1 removes the code I (vaguely hopefully) suspect was the culprit. But it also adds code that modifies the permissions on live-additional-software.conf on upgrades from 3.8 or older. So please keep an eye on this and tell me whether such issues happen again when running 3.9~rc1 or newer. The timing of the problem happening vs. upgrade to 3.9~rc1 or newer matters so please try to collect this info.

#40 Updated by goupille 2018-09-15 17:04:33

goupille wrote:
> at least two different users are affected with tails 3.9 (I’m forwarding a bug report to you)

#41 Updated by intrigeri 2018-09-18 08:58:42

I’ve analyzed report 2abf0655f6cc39a34db3b78ab6ba6a6c:

• The persistent filesystem had not been unmounted cleanly. So data corruption is to be expected.
• persistence.conf is not empty: it has /home/amnesia/Persistent. I don’t know if the user fixed it already themselves or not.
• Apparently, timing is unrelated to upgrading Tails (or at least the user does not say so).
• I don’t know if help desk tried to gather the info I’ve requested on 10976#note-16. I’ll try to do it myself.

#43 Updated by goupille 2018-10-26 11:51:27

intrigeri wrote:

> * I don’t know if help desk tried to gather the info I’ve requested on 10976#note-16. I’ll try to do it myself.

I just resent you a report from another user with those informations (ticket number in the subject)

#44 Updated by pablonatalino 2018-10-27 16:59:38

I experienced this myself now.

#45 Updated by intrigeri 2018-10-31 10:23:08

Analyzed 4dc80ae26f8e3397fbe1437e1e1b232f: empty config file, mtime is ~20 minutes before the user sent their bug report. Permissions look OK, no disabled config file, no disk space issue.

Same for “Upgrade problems PLEASE HELP” except mtime is 2 days before help desk forwarded me the report (I was not told when the report was sent).

#46 Updated by sajolida 2018-11-02 05:36:18

Do I understand correctly and the fix proposed in Bug #10976#note-36 was not enough to solve the issue?

If so, should we consider building resiliency mechanisms even though we don’t know what is the root cause of the problem, like proposed in Bug #10976#note-27?

#48 Updated by sajolida 2018-11-06 02:04:52

Thanks! And sorry if my request for clarification sounded pushy.

#53 Updated by intrigeri 2019-01-24 16:18:00

Note to myself, from chattr(1):

       When  a  file with the 'S' attribute set is modified, the changes are written syn‐
       chronously on the disk; this is equivalent to the 'sync' mount option applied to a
       subset of the files.

I think we should set this on our persistence config files.

#54 Updated by intrigeri 2019-02-12 18:57:22

intrigeri wrote:
> Note to myself, from chattr(1):
>
> […]
>
> I think we should set this on our persistence config files.

Done on my branch for Bug #16461, where I’m tracking the round of improvements I want to make in 3.13.

#56 Updated by intrigeri 2019-02-14 08:50:33

FTR this didn’t make it to “hot topics for help desk” since 6+ months, which suggests that maybe one of the robustness improvements I’ve done already helped. Help desk tells me they think it still happens once in a while.

#57 Updated by intrigeri 2019-03-17 08:43:10

intrigeri wrote:
> (Assuming that Bug #16461 lands in 3.13.)
>
> Hi @mercedes508 :) I see you’ll do the majority of the help desk shifts that follow the 3.13 release. So, please report 4-5 weeks after the 3.13 release, i.e. around mid-April, if this is still a problem; then reassign to me. Thanks in advance!

The branch for Bug #16461 was merged so I confirm my above request.

And if users still experience this problem, once they run 3.13 there should be a backup persistence.conf.bak file next to the empty persistence.conf => please ask them to share the contents of the backup file and try copying it to persistence.conf, instead of re-configuring their persistence from scratch :)

#58 Updated by mercedes508 2019-03-28 13:52:28

2a38a41a59194d76966d26b25c35f8b2
66342e7c-e525-1997-664d-141d35b8798b
CAMV72DhA5tPP4Yq=v4T3nVDpj-12-kbdrdmiQYL66X6=JKJTYw

#59 Updated by goupille 2019-04-10 18:41:23

Bug report: 942e619be10a2bbdc2e48a6cd1858f08

#67 Updated by intrigeri 2019-09-30 09:30:23

Note to myself: according to https://lwn.net/Articles/799807/, calling sync on the directory that contains persistence.conf may help.

#68 Updated by intrigeri 2019-09-30 09:49:15

Hi @cbrownstein,

> I’m currently experiencing this bug. I’ve followed the instructions on the known issues page [1] but my persistent features and files are still missing.

It seems to me that there’s a step missing in that doc: between steps 3 and 4, you must enable the persistence features you need.
You might want to file a tech writing ticket to fix this :)

>> * the output of ls -l /live/persistence/;

>

> root@amnesia:/live/persistence/TailsData_unlocked# ls -l /live/persistence/
> total 4
> drwxrwxr-x+ 14 amnesia amnesia 4096 Aug 28 17:57 TailsData_unlocked
> 

Ouch, that’s wrong, which is why your persistence.conf was renamed to /live/persistence/TailsData_unlocked/persistence.conf.insecure_disabled.
To fix this, you need to run sudo chown root:root /live/persistence/TailsData_unlocked.

Now, the question is: how was your persistent filesystem given to amnesia:amnesia?

So:

• When was the list time your persistent volume worked as expected? Your comments suggest it was late August. Is this correct? Or is your hardware clock ~1 month stuck in the past?
• Did you do anything special with this filesystem between the last time your Tails stick worked, and the time when it became buggy (late August according to your comment)?
• If you did not: on my side, I’ll dive into our codebase to try & find which code might have messed this up.

>> * whether they have reconfigured their persistence, in any way, during the previous Tails session (goal: pinpoint a bug in the persistence setup software);

> I added and removed additional software.

Thanks, this is useful. I’ll look particularly at the Additional Software code paths that may trigger those insecure permissions.

Do you remember when was that previous Tails session, during which you added and removed Additional Software?

> Copying persistence.conf.bak to persistence.conf did nothing.

Please retry after having fixed the ownership of the root directory of your persistent volume, as suggested above :)

#72 Updated by cbrownstein 2019-09-30 18:02:02

> > I’m currently experiencing this bug. I’ve followed the instructions on the known issues page [1] but my persistent features and files are still missing.
>
> It seems to me that there’s a step missing in that doc: between steps 3 and 4, you must enable the persistence features you need.
> You might want to file a tech writing ticket to fix this :)

I will add that step to the doc. But, I did re-enable the persistent features that were enabled before my persistent volume stopped working.

> >> * the output of ls -l /live/persistence/;
>
> > […]
>
> Ouch, that’s wrong, which is why your persistence.conf was renamed to /live/persistence/TailsData_unlocked/persistence.conf.insecure_disabled.
> To fix this, you need to run sudo chown root:root /live/persistence/TailsData_unlocked.

I will follow your instruction and update this ticket with the result.

> Now, the question is: how was your persistent filesystem given to amnesia:amnesia?
>
> So:
>
> * When was the list time your persistent volume worked as expected? Your comments suggest it was late August. Is this correct? Or is your hardware clock ~1 month stuck in the past?

This is correct: late August.

> * Did you do anything special with this filesystem between the last time your Tails stick worked, and the time when it became buggy (late August according to your comment)?
> * If you did not: on my side, I’ll dive into our codebase to try & find which code might have messed this up.

I did not do anything special with the filesystem.

> >> * whether they have reconfigured their persistence, in any way, during the previous Tails session (goal: pinpoint a bug in the persistence setup software);
>
> > I added and removed additional software.
>
> Thanks, this is useful. I’ll look particularly at the Additional Software code paths that may trigger those insecure permissions.

I’m not sure if this matters but I was using apt (rather than synaptic or other tool) to install and remove packages.

> Do you remember when was that previous Tails session, during which you added and removed Additional Software?

I believe, but don’t remember with 100% certainty, that my persistent volume survived at least one reboot after installing packages. But, I believe my persistent volume was missing the session immediately after removing packages.

> > Copying persistence.conf.bak to persistence.conf did nothing.
>
> Please retry after having fixed the ownership of the root directory of your persistent volume, as suggested above :)

I will, and will update this ticket with the result.

#73 Updated by cbrownstein 2019-10-01 17:27:38

> >> * the output of ls -l /live/persistence/;
>
> > […]
>
> Ouch, that’s wrong, which is why your persistence.conf was renamed to /live/persistence/TailsData_unlocked/persistence.conf.insecure_disabled.
> To fix this, you need to run sudo chown root:root /live/persistence/TailsData_unlocked.

Changing the ownership of TailsData_unlocked then copying persistence.conf.bak to persistence.conf worked to fix the problem.

However, live-additional-software.conf is empty, meaning none of the additional software I added is being installed. live-additional-software.conf.insecure_disabled exists but is empty and useless.

At least the packages are still in apt/cache to help me refresh my memory as to what I had added.

#75 Updated by intrigeri 2019-10-02 06:33:07

> Changing the ownership of TailsData_unlocked then copying persistent.conf.bak to persistent.conf worked to fix the problem.

Great! I’ll let you do whatever is needed so that the tech writing team has this problem on their radar and fixes the doc at some point :)

> However, live-additional-software.conf is empty, meaning none of the additional software I added is being installed. live-additional-software.conf.insecure_disabled exists but is empty and useless.

Ouch. I think I understood why this happened ⇒ filed Bug #17112 to track it.

Thanks a lot for the detailed report!

#77 Updated by intrigeri 2019-10-07 06:41:28

intrigeri wrote:
> Note to myself: according to https://lwn.net/Articles/799807/, calling sync on the directory that contains persistence.conf may help.

Done on the branch for Bug #15102.

#78 Updated by intrigeri 2020-02-23 10:37:00

@numbat, dear help desk, did you see this problem reported on Tails 4.x?