Tails

#1 Updated by intrigeri 2018-11-22 11:52:53

• Assignee set to sajolida
• QA Check set to Info Needed

Does this still happen? If yes, I think this probably qualifies as SponsorW_2017_Internal.

#2 Updated by intrigeri 2018-11-22 14:18:07

• Status changed from New to Confirmed
• Assignee changed from sajolida to alant
• QA Check deleted (Info Needed)
• Deliverable for set to 319

Actually I’ve just reproduced it:

[INFO] Starting to install additional software...
[INFO] Found additional packages list.
[INFO] Will install the following packages: make ssmtp backupninja freeipmi-tools libtemplate-perl rdiff-backup libyaml-tiny-perl libdata-dumper-concise-perl
[INFO] Reading package lists...
[INFO] Building dependency tree...
[INFO] Reading state information...
[INFO] The following additional packages will be installed:
[INFO]   bsd-mailx dialog freeipmi-common libappconfig-perl libfreeipmi16
[INFO]   libgnutls-openssl27 libgnutls30 libipmiconsole2 libipmidetect0 librsync1
[INFO] Suggested packages:
[INFO]   debconf-utils duplicity hwinfo mdadm subversion trickle wodim
[INFO]   freeipmi-ipmidetect freeipmi-bmc-watchdog gnutls-bin libtemplate-perl-doc
[INFO]   libtemplate-plugin-gd-perl libtemplate-plugin-xml-perl make-doc
[INFO] Recommended packages:
[INFO]   libdevel-argnames-perl python-pylibacl python-pyxattr
[INFO] The following NEW packages will be installed:
[INFO]   backupninja bsd-mailx dialog freeipmi-common freeipmi-tools
[INFO]   libappconfig-perl libdata-dumper-concise-perl libfreeipmi16
[INFO]   libgnutls-openssl27 libipmiconsole2 libipmidetect0 librsync1
[INFO]   libtemplate-perl libyaml-tiny-perl make rdiff-backup ssmtp
[INFO] The following packages will be upgraded:
[INFO]   libgnutls30
[INFO] 1 upgraded, 17 newly installed, 0 to remove and 85 not upgraded.
[INFO] Need to get 1080 kB/4724 kB of archives.
[INFO] After this operation, 14.9 MB of additional disk space will be used.
[INFO] Err:1 tor+http://vwakviie2ienjx6t.onion/debian stretch/main amd64 libgnutls30 amd64 3.5.8-5+deb9u4
[INFO]   Could not connect to localhost:9050 (127.0.0.1). - connect (111: Connection refused)
[INFO] Err:2 tor+http://vwakviie2ienjx6t.onion/debian stretch/main amd64 libgnutls-openssl27 amd64 3.5.8-5+deb9u4
[INFO]   Unable to connect to localhost:9050:
[INFO] E: Failed to fetch tor+http://vwakviie2ienjx6t.onion/debian/pool/main/g/gnutls28/libgnutls30_3.5.8-5+deb9u4_amd64.deb  Could not connect to localhost:9050 (127.0.0.1). - connect (111: Connection refused)
[INFO] E: Failed to fetch tor+http://vwakviie2ienjx6t.onion/debian/pool/main/g/gnutls28/libgnutls-openssl27_3.5.8-5+deb9u4_amd64.deb  Unable to connect to localhost:9050:
[INFO] E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
[WARNING] apt-get exited with returncode 100
[WARNING] Warning: installation of make ssmtp backupninja freeipmi-tools libtemplate-perl rdiff-backup libyaml-tiny-perl libdata-dumper-concise-perl failed

I have no libgnutls*.deb in my persistent APT cache.

#4 Updated by alant 2018-12-09 15:25:31

• Assignee changed from alant to sajolida
• QA Check set to Info Needed

#5 Updated by intrigeri 2018-12-09 16:09:50

• Assignee changed from sajolida to alant

> Installation should not connect to the network as the files should be in the cache.

Right.

> I see 2 possibilities:

> - some package was added manually to live-additional-software.conf. In this case we can’t do anything

It’s clearly not the case for the failure I’ve reported myself, for two reasons:

• The info I’ve shared shows that the two packages APT fails to fetch from remote mirrors are pulled merely as dependencies of packages I have in my list (probably gnutls-bin), which proves that I did not add them myself to live-additional-software.conf. APT does not try to fetch from remote mirrors any package that’s in my list.
• I’m so excited by our work on Additional Software that I’ve switched to the new thing as soon as I could and have no plan to edit that file manually ever again :)

In any case, as long as we don’t document that one can edit this file by hand anymore (do we?), then it’s unsupported and I don’t think we should bother dealing with it… although one could possibly argue that it’s been the only documented way to do that for years, so perhaps one can’t blame users for trying to use it again.

> - the package lists were updated manually or outside tails-additionnal-software and thus bypassed its mechanism to rollback to the old one in case of failure.

Do you mean I might have run apt update myself? Sure, that’s possible. I understand it can cause my cached APT lists to reference versions of libgnutls30 and libgnutls-openssl27 that are not in the cache. And indeed, it breaks the rollback mechanism.

> I think wore information is needed to consider how to improve the situation, if possible.

Given the above clarifications, I think we now understand when exactly this problem happens and why, so it seems to me that the next step is to decide whether the cost/benefit of fixing this problem is worth it.

Regarding the benefit:

• I have no strong opinion wrt. how much we should support running apt update by hand. I could live with “folks who run stuff as root on the command line will deal with it”.
• It seems to me that this failure mode can be triggered merely by starting Synaptic at an unfortunate time, without installing any package (config/chroot_local-patches/synaptic-update-at-startup.diff). It would be good to define “at an unfortunate time” more precisely. Depending on how often such unfortunate times happen, this bug could be anything between “yeah, some day, maybe, whatever” and a deal breaker. I’d rather not do the research myself but at first glance, I strongly suspect that it has more chances to affect users who keep their Tails running for extended periods.

Regarding the cost: you tell us :)

#6 Updated by alant 2018-12-12 12:47:59

• Subject changed from Additional Software tries to install before Tor is ready to Additional Software install step sometimes tries to download packages

> Given the above clarifications, I think we now understand when exactly this problem happens and why, so it seems to me that the next step is to decide whether the cost/benefit of fixing this problem is worth it.
>
> Regarding the benefit:
>
> * I have no strong opinion wrt. how much we should support running apt update by hand. I could live with “folks who run stuff as root on the command line will deal with it”.

Let’s call that case A. I agree supporting it is not a priority.

> * It seems to me that this failure mode can be triggered merely by starting Synaptic at an unfortunate time, without installing any package (config/chroot_local-patches/synaptic-update-at-startup.diff). It would be good to define “at an unfortunate time” more precisely. Depending on how often such unfortunate times happen, this bug could be anything between “yeah, some day, maybe, whatever” and a deal breaker. I’d rather not do the research myself but at first glance, I strongly suspect that it has more chances to affect users who keep their Tails running for extended periods.
>
Let’s call that case B.

I see another possibility, case C: a new Tails release including new apt lists is installed. Then the right version of the packages are not in the cache. If user is not connected to the Internet/Tor is not ready when the Gnome session is ready, then tails-additional-software-install.service fails. As a consequence, tails-additional-software-update.path is never run and the problem is not solved until the user starts a session when already connected to Tor (= cable + fast connection).

This looks a lot more annoying.

> Regarding the cost: you tell us :)

I see several ways of fixing/mitigating the issue:

1) Run tails-additional-software-upgrade.service after tails-additional-software-install.service even is it has faild. That would fix all cases for users that connect to the Internet. They would just have to wait more for their software to be installed. Implementing it looks cheap.

2) Run something like tails-additional-software install as a APT::Update::Post-Invoke hook of apt. That would solve case A and B. Implementing it looks cheap.

3) Have a local APT repository with higher priority to support truly offline ASP, that would be upgraded when going online. This seems me to be the only full solution to full offline use. Researching its security issues would be required. Implementing it looks quite expansive.

I propose implementing 1 and 2. Thoughts?

#8 Updated by alant 2018-12-19 20:43:16

• Status changed from Confirmed to In Progress

Applied in changeset commit:tails|cd83f2935e2df7cc97c8b0595a39d876ca9e687e.

#9 Updated by alant 2018-12-21 15:41:14

• QA Check changed from Info Needed to Dev Needed
• Feature Branch set to bugfix/155957-asp-dl-failure

> I propose implementing 1 and 2.

This should be done in bugfix/155957-asp-dl-failure

#15 Updated by alant 2019-03-14 15:07:03

• Assignee changed from alant to segfault
• QA Check changed from Dev Needed to Ready for QA

#16 Updated by intrigeri 2019-03-14 18:04:44

• Assignee changed from segfault to intrigeri
• Target version set to Tails_3.13

@alant, I’ll assume this is a candidate for 3.13.

segfault has very little time for Tails this month so I doubt he can review this in time so I’ll take it.

#17 Updated by intrigeri 2019-03-14 18:21:59

• Feature Branch changed from bugfix/155957-asp-dl-failure to bugfix/15957-asp-dl-failure

I see two branches on Jenkins so I’ll assume the one that’s ready for QA is the one that builds there and has been updated today. I’ll delete the other one. @alant, please let me know if I got it wrong.

#20 Updated by intrigeri 2019-03-15 13:18:48

• Assignee changed from intrigeri to alant
• QA Check changed from Ready for QA to Info Needed

@alant, about commit:2019763b11a39d33f5e3a691ca44a8179db2b3b7, first, I’m not sure I understand correctly which one of the problems this ticket is about is solved by this commit. If it’s case A, then I don’t think it’s worth the trouble and the can of worms and additional complexity that my review below reveals. If this commit is indeed not essential to fixing the most important problem(s) this ticket is about, then let me know, and I’m happy to run the test suite again on a version of this branch that has this commit reverted. If this revert makes the branch pass the test suite, then I’m happy to merge it in 3.13.

So, about commit:2019763b11a39d33f5e3a691ca44a8179db2b3b7, it introduces a class of race conditions and some possibly unwanted behaviour. I’m a bit worried about the side-effects, such as:

• If I run apt update && apt install xyz, likely apt install has a good chance to fail, as install_additional_packages will be locking APT, right?
• Similarly, given we have made Synaptic run (essentially) apt update when it starts, there’s a chance that it races against install_additional_packages, no?
• In upgrade_additional_packages we also run (essentially) apt update, which with this commit will start /usr/local/sbin/tails-additional-software update-post. So my understanding is that apt_get_returncode = _launch_apt_get(["update"]) will block on APT update + update-post to complete and I don’t know how the error handling that follows will behave. And then, this same function will run install_additional_packages(upgrade_mode=True) again, which feels wrong, no? It’s possible that I’m totally confused and that something very different will happen. If that’s the case, the actionable item here is “make this less confusing to reviewers and to whoever will work on this code next” :)

This branch breaks the automatic test we have for https://tails.boum.org/blueprint/additional_software_packages/offline_mode/#incomplete-online-upgrade. The test case expects a notification that indicates failure to upgrade and does not see it. Looking at the Journal, the upgrade did fail and in the logs I see: “Warning: the update failed” and later “E: Problem executing scripts APT::Update::Post-Invoke ‘/usr/local/sbin/tails-additional-software update-post’”. But I see no notification nor call to tails-additional-software-notify. This might be a consequence of having upgrade_additional_packages essentially call itself with commit:2019763b11a39d33f5e3a691ca44a8179db2b3b7.

If we do keep commit:2019763b11a39d33f5e3a691ca44a8179db2b3b7:

• I find the update-post wording confusing: it suggests to me this operation will “do an update after X”. post-update would solve this.
• Perhaps APT::Update::Post-Invoke-Success would make more sense?

The other commits look good to me.

#22 Updated by intrigeri 2019-03-16 16:11:05

• Target version changed from Tails_3.13 to Tails_3.14

Hi!

alan wrote:
> intrigeri wrote:
>> alant, about commit:2019763b11a39d33f5e3a691ca44a8179db2b3b7, first, I'm not sure I understand correctly which one of the problems this ticket is about is solved by this commit. If it's case A, then I don't think it's worth the trouble and the can of worms and additional complexity that my review below reveals. If this commit is indeed not essential to fixing the most important problem(s) this ticket is about, then let me know, and I'm happy to run the test suite again on a version of this branch that has this commit reverted. If this revert makes the branch pass the test suite, then I'm happy to merge it in 3.13. >> > It's supposed to solve case A and B: there was a manual update (eiter through apt@ directly or by starting Synaptic) that caused cached APT lists to reference versions of $package that are not in the packace cache.

> Case C is solved without this commit.

Great, thank you!

> This is indeed totally wrong.

Glad I spotted it then :)

> I think it’s good to merge now a version of the branch that solve only case C and spend more time trying to fix case A and B.

FTR this is Bug #16566 and I’m on it.

> I don’t really know what to do with commits 949e2e8..8cdddc9 which are basically this with its fixes
> […]
> That was needed for update-post but is still an improvement. I suggest we deal with that later when thnking about case A and B.

Yes, let’s postpone this. If this ever gets submitted again:

• Please explain what’s the benefit and to whom: it’s not obvious to me why this matters.
• It would be sweet if you could clean up the history a little. Usually I don’t insist on clean Git history but this case is rather extreme in terms of signal/noise ratio.

#23 Updated by alant 2019-03-16 16:47:56

• Target version changed from Tails_3.14 to Tails_3.13

intrigeri wrote:
> > I think it’s good to merge now a version of the branch that solve only case C and spend more time trying to fix case A and B.
>
> FTR this is Bug #16566 and I’m on it.
>
Great thanks.

> > I don’t really know what to do with commits 949e2e8..8cdddc9 which are basically this with its fixes
> > […]
> > That was needed for update-post but is still an improvement. I suggest we deal with that later when thnking about case A and B.
>
> Yes, let’s postpone this. If this ever gets submitted again:
>
> * Please explain what’s the benefit and to whom: it’s not obvious to me why this matters.

See Bug #16567

> * It would be sweet if you could clean up the history a little. Usually I don’t insist on clean Git history but this case is rather extreme in terms of signal/noise ratio.

If it’s not messing up CI and so on, I’d be happy to rebase the branch. Actually I was tempted to do it but was worried to break somethink by pusing a forced update.

#24 Updated by intrigeri 2019-03-16 16:52:21

• Target version changed from Tails_3.13 to Tails_3.14

>> * It would be sweet if you could clean up the history a little. Usually I don’t insist on clean Git history but this case is rather extreme in terms of signal/noise ratio.

> If it’s not messing up CI and so on, I’d be happy to rebase the branch. Actually I was tempted to do it but was worried to break somethink by pusing a forced update.

Yeah, please rebase on stable once the branch for Bug #16566 is merged.

Re-setting target version to 3.14 as I doubt we’ll get anything ready and merged in time for 3.13. I’d like some week-end :)

#26 Updated by alant 2019-03-16 17:30:53

• Target version changed from Tails_3.14 to Tails_3.13

intrigeri:
> Sorry I was not explicit enough: this question was about “ASP: fork before notifying after failed upgrade”.
>
This enables to have the main ASP process running in the forground and blocking and then fork so that the notification is not blocking. It was necessary for post-update to avoid the race conditions you mentionned, but looks cleaner to me than forking right away.

#28 Updated by intrigeri 2019-03-16 17:45:38

• Target version changed from Tails_3.13 to Tails_3.14
• QA Check deleted (Info Needed)

#31 Updated by CyrilBrulebois 2019-05-23 21:23:28

• Target version changed from Tails_3.14 to Tails_3.15

#32 Updated by CyrilBrulebois 2019-07-10 10:34:08

• Target version changed from Tails_3.15 to Tails_3.16

#33 Updated by intrigeri 2019-08-30 17:31:35

• Target version deleted (Tails_3.16)

Hi!

We’ve set up an automated process to ask our fellow contributors to update some tickets of theirs, in order to:

• better reflect your plans;
• bring down your amount of work-in-progress to a sustainable level;
• encourage team work and increase the chances that someone finishes the work;
• avoid a human doing ticket triaging and asking you the same questions on each such ticket.

In particular, this process identifies:

• Stalled work-in-progress
• Reviews waiting for a long time

However, in the current state of things, this process is not able to notice those tickets when their Target version has been repeatedly postponed by our Release Managers. Therefore, the ticket triaging team decided on Feature #16545 to remove the Target version whenever in such cases, when it does not feel realistic. This is what I’m doing on this ticket.

You now have a few options, such as:

• Deassign yourself. That’s fine. If it really matters, someone else, possibly you, may pick it up later. Then, if this ticket is relevant for a Tails team, bring it to their attention; else, forget it and take care of yourself :)
• If you think you can realistically come back to it and finish the work in the next 6 months, say so on this ticket, for example by setting a suitable “Target version”. This will communicate your plans to the rest of the project and ensure the task pops up on your radar at a suitable time. Of course, you can still realize later that it is not going to work as planned, and revisit today’s choice.

Cheers!

#34 Updated by alant 2020-03-08 18:57:41

• Status changed from In Progress to Rejected
• Assignee deleted (alant)

See Bug #16567 for the remaining issue. Let’s close this long and complicated ticket.