Tails

#2 Updated by intrigeri 2017-02-14 18:49:03

> This would improve the UX when using APT since it will not be necessary to run apt update most of the time,

I’m curious where this “most of the time” comes from. Last time I checked, it was rather “during the 7-10 days after a given ISO was built”, i.e. in practice “during the 5-7 days after a given ISO was released”. Can you please clarify?

> as will Tails Server users (and segfault, since he fears the initial apt update, that can take minutes, will put off users).

Meta: I’d like the benchmark on Stretch, requested on Feature #12237, to be done before we take this argument into account.

#4 Updated by anonym 2017-02-15 11:14:52

anonym wrote:
> The feature branch does this, and the image size is increased by 32 MiB.

This was with gzip compression, I now realize. I guess it should be done again, with xz, but I’d suspect the impact to be similar when looking at the relative sizes.

#5 Updated by anonym 2017-02-15 11:21:38

intrigeri wrote:
> > This would improve the UX when using APT since it will not be necessary to run apt update most of the time,
>
> I’m curious where this “most of the time” comes from. Last time I checked, it was rather “during the 7-10 days after a given ISO was built”, i.e. in practice “during the 5-7 days after a given ISO was released”. Can you please clarify?

Since packages don’t change frequently in Debian stable (where users will install packages from by default) most package info in even old APT lists will remain valid, so those packages can be installed without an apt update.

> > as will Tails Server users (and segfault, since he fears the initial apt update, that can take minutes, will put off users).
>
> Meta: I’d like the benchmark on Stretch, requested on Feature #12237, to be done before we take this argument into account.

Done, at least for apt update only: Feature #12237#note-12

#6 Updated by intrigeri 2017-02-15 12:55:55

>> > This would improve the UX when using APT since it will not be necessary to run apt update most of the time,
>>
>> I’m curious where this “most of the time” comes from. Last time I checked, it was rather “during the 7-10 days after a given ISO was built”, i.e. in practice “during the 5-7 days after a given ISO was released”. Can you please clarify?

> Since packages don’t change frequently in Debian stable (where users will install packages from by default) most package info in even old APT lists will remain valid, so those packages can be installed without an apt update.

I still don’t get it, very sorry if I’m missing the obvious or if I’m reasoning based on wrong assumptions. I’ll try to clarify my assumptions here.

It seems to me that whether packages change frequently or not is orthogonal to the APT list expiration problem: what matters is the value of the Valid-Until field.

I (naively?) believe that one single expired APT list (e.g. testing/sid) was enough to prevent apt install from accepting to use any list at all (including those that are not expired yet). If I’m wrong, then right, one can still install packages from one given APT source as long as its lists are still valid, even if other lists have expired. Is this how it works?

#7 Updated by anonym 2017-02-15 13:26:23

intrigeri wrote:
> It seems to me that whether packages change frequently or not is orthogonal to the APT list expiration problem:

It is definitely not orthogonal, even if Valid-Until works like you assumed: if you try to install any package that has been updated and whose .deb has been cleaned up from the Debian APT mirrors, then you’ll have an error (trying to fetch an url pointing to an non-existing .deb) until you run apt update and get the url pointing to the new version.

> what matters is the value of the Valid-Until field.

I suspect it only matters when fetching the lists (to protect against replay attacks), i.e. only when running apt update, and not afterwards. I just booted a VM, saw via grep Valid-Until /var/lib/apt/lists/* that all lists were expired, but I could still successfully apt download cowsay, because its version hasn’t changed since I fetched the list some months ago, but I can not apt download icedove, because it has had a security update since then, and the old version referenced in my APT lists has been cleaned up. I also tried rm /var/cache/apt/{src,}pkgcache.bin but they were regenerated without issue indicating that the Valid-Until value is not relevant there either.

tl;dr: apt install will work as long as all packages’ versions it wants to install are still present on the Debian mirrors.

#8 Updated by intrigeri 2017-02-15 19:56:46

> It is definitely not orthogonal, even if […]

Indeed, thanks!

>> what matters is the value of the Valid-Until field.

> I suspect it only matters when fetching the lists (to protect against replay attacks), i.e. only when running apt update, and not afterwards. I just booted a VM, saw via grep Valid-Until /var/lib/apt/lists/* that all lists were expired, but I could still successfully apt download cowsay, because its version hasn’t changed since I fetched the list some months ago, but I can not apt download icedove, because it has had a security update since then, and the old version referenced in my APT lists has been cleaned up. I also tried rm /var/cache/apt/{src,}pkgcache.bin but they were regenerated without issue indicating that the Valid-Until value is not relevant there either.

Excellent!

> tl;dr: apt install will work as long as all packages’ versions it wants to install are still present on the Debian mirrors.

Woohoo, that’s very good news. Thanks a lot! :)

(I’ll assume that the “potential for running network services with known security issues” issue is taken into account on the Tails Server side, and won’t bother about it here.)

#9 Updated by sajolida 2017-03-04 20:41:33

I read through this quickly and won’t block on shipping them if you decide on doing so.

Still, I’d be very interested in seeing which fraction of our user base will actually make use that data.

Right now the only clue I have is the 9% of WB reports that have additional packages amongst the 48% of reports with persistence, so that’s 4% of the total. But of course that’s a bare minimum and other
users (people without persistence, people using the future GUI for additional software, people using the future Tails Server) would boost this up by some unknown factor.

#12 Updated by sajolida 2018-01-28 15:26:31

By the way, how would this relate to build reproducibility? :)

#14 Updated by sajolida 2018-02-05 14:05:29

Alan: I think the first step could be to ask release managers and reproducible build folks if that’s going to be a problem. Maybe intrigeri can do that as he’s part of our team as well.

So unless you want to give it more thought yourself first, maybe reassign it to him with “Info Needed”.

Another issue I just thought about is IUK: the full APT lists would change for each release and they would always go in the IUK.

#15 Updated by sajolida 2018-02-05 14:05:57

Also target 3.9 for that unless you want to do it in time for 3.6.

#16 Updated by intrigeri 2018-02-06 17:05:05

> Alan: I think the first step could be to ask release managers and reproducible build folks if that’s going to be a problem. Maybe intrigeri can do that as he’s part of our team as well.

• From a RM point of view, at first glance I see no obvious issue with this proposal.
• Regarding reproducibility: I don’t know, but Alan can test this himself (set a branch as Ready for QA and it will be tested for reproducibility in Jenkins :)
• I’ve already expressed my other concerns (of which the only relevant one is probably “potential for running network services with known security issues”) above.

So I think next steps are on Alan’s plate:

1. think about the security consequences of enabling users to install a package from stable, with security issues have been fixed since via stable-security; we don’t enable them to do that right now
2. if that is deemed acceptable, build a PoC and ensure it does not break other stuff such as reproducibility

#17 Updated by anonym 2018-02-06 17:49:12

sajolida wrote:
> Another issue I just thought about is IUK: the full APT lists would change for each release and they would always go in the IUK.

The lists are N MB compressed (from Feature #12238#note-1 it seems N ~= 32 MB), so the actual effect is that each IUK will become N MB larger than they are currently. It is equivalent to if we made running a fresh apt update as part of the Tails Upgrader (e.g. if we shipped the IUKs as Debian packages… :)), and this is only (partially) redundant for users that activate persistent APT lists, and users that never will use APT to install something.

#24 Updated by bertagaz 2018-03-05 09:45:14

Given one of the drawback is the growth of the IUK size, I was wondering if we could eventually ship the APT lists in the ISO (if that’s what is decided), but not in the IUKs.

I’m a bit concerned about reproducibility though.

#25 Updated by intrigeri 2018-03-05 10:43:55

> I do not see how it relates to this ticket

I think Alan merely meant that we don’t need the change this ticket is about for the ASP project. But if you want/need to work on it for other reasons, feel free to :)

#26 Updated by alant 2018-03-06 11:29:11

intrigeri wrote:
> > I do not see how it relates to this ticket
>
> I think Alan merely meant that we don’t need the change this ticket is about for the ASP project. But if you want/need to work on it for other reasons, feel free to :)

Exactly. We decided we don’t need that for ASP, it doesn’t mean it’s not cool, just that I’m not commited to make it happen withn 3 monthes.

#29 Updated by sajolida 2018-06-05 16:09:20

While working on Additional Software we proposed Feature #15584: wrapping apt and apt-get on interactive shells to download the APT lists if there are none yet.

This would prevent people from wondering what’s going on when trying APT which was an important issue when testing the Additional Software beta.

So once we have Feature #15584 we could reject this ticket (or seriously lower its priority).

#30 Updated by intrigeri 2018-06-05 16:25:39

> So once we have Feature #15584 we could reject this ticket (or seriously lower its priority).

… unless it’s needed for something else :)