Feature #12237

Reduce apt update time during first start of Tails Server

Added by segfault 2017-02-14 16:03:44 . Updated 2017-04-25 19:37:02 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Start date:
2017-02-14
Due date:
% Done:

100%

Feature Branch:
Type of work:
Research
Blueprint:

Starter:
Affected tool:
Server
Deliverable for:

Description

Currently this takes between 2 and 10 minutes, depending on the Tor circuit.


Subtasks


Related issues

Related to Tails - Feature #11539: Don't ship the testing and unstable APT sources Rejected 2016-06-20
Related to Tails - Feature #12238: Ship full APT lists in the live file system Confirmed 2017-02-14

History

#1 Updated by segfault 2017-02-14 16:04:01

#2 Updated by segfault 2017-02-14 16:05:01

  • related to Feature #11539: Don't ship the testing and unstable APT sources added

#3 Updated by segfault 2017-02-14 16:13:59

Ideas:

* Don’t ship the testing and unstable APT sources (Feature #11539)

* Would reduce update time down to 30s
* Could break stuff in corner cases (see https://labs.riseup.net/code/issues/11539#note-1)

* Only fetch the repositories we actually need for Tails Server

* See for example http://askubuntu.com/a/65250

* More?

#4 Updated by segfault 2017-02-14 17:47:28

  • related to Feature #5688: Tails Server: Self-hosted services behind Tails-powered onion services added

#5 Updated by intrigeri 2017-02-14 18:03:33

It should be much faster on Stretch, thanks to optimizations in APT.
I say let’s benchmark before we try & solve this :)

#6 Updated by segfault 2017-02-14 18:10:39

> It should be much faster on Stretch, thanks to optimizations in APT.
That would be great :)
> I say let’s benchmark before we try & solve this :)
Sure.

#7 Updated by anonym 2017-02-15 09:06:18

segfault wrote:
> Ideas:

  • See the commit 440b9cc73a Completely disable APT translations. from the feature/12238-ship-apt-lists Git branch (of Feature #12238). On Jessie this makes quite a difference.

#8 Updated by anonym 2017-02-15 09:06:33

  • related to Feature #12238: Ship full APT lists in the live file system added

#9 Updated by anonym 2017-02-15 10:02:54

BTW, I have observed a ~50% slowdown since we switched to the APT onion services.

#10 Updated by anonym 2017-02-15 10:47:41

We also have to take into account that feature/stretch has one Debian version less among the APT sources, namely stable, since it is based on testing. When feature/stretch is released, it will have the same number of Debian versions in its APT sources.

#11 Updated by anonym 2017-02-15 11:21:06

After completely disabling APT translations I benchmarked only the apt update time (running it 20 times) like this:

. /usr/local/lib/tails-shell-library/tor.sh && \
(\
  for x in $(seq 1 20); do \
    rm -fr /var/lib/apt/lists/*; \
    systemctl restart tor@default; \
    until tor_is_working; do sleep 1; done; \
    time apt update; done \
) &> apt.log

and got:

  • Tails 2.10 median time: 1m 15s (for 23.6 MB of data)
  • Tails 3.0~beta1 median time: 50s (for 15.2 MB of data)

Taking my previous comment (Feature #12237#note-11) into account (one less APT source in 3.0~beta1) these results seems consistent with there being no noticeable improvement for apt update, so I guess the optimizations talked about in Feature #12237#note-6 are only about the generation of /var/cache/apt/{src,}pkgcache.bin. I guess it’d be interesting to benchmark the time needed to generate those on stable vs stretch, with and without persistent APT lists, but I still suspect that the slowness is dominated by the fetches over Tor.

I did have several outliers at >= five minutes thanks to slow circuits, and previously I’ve seen even worse (> 10 minutes). We’ll never be able to avoid this unless we ship the lists, so if we don’t we absolutely must have better feedback about what is going on with apt update in Tails Server. But even apt itself doesn’t have very good feedback (beyond “stuff is still being written to the screen, various numbers are increasing”), and neither does Synaptic, so I’m not sure we can easily achieve that.

#12 Updated by intrigeri 2017-02-15 13:00:19

Thanks for this data!

> Taking my previous comment (Feature #12237#note-11) into account (one less APT source in 3.0~beta1) these results seems consistent with there being no noticeable improvement for apt update, so I guess the optimizations talked about in Feature #12237#note-6 are only about the generation of /var/cache/apt/{src,}pkgcache.bin.

This matches what I (vaguely) remember.

> I guess it’d be interesting to benchmark the time needed to generate those on stable vs stretch, with and without persistent APT lists, but I still suspect that the slowness is dominated by the fetches over Tor.

IIRC the results sajolida gave me a while ago (using real storage hardware) were in the 30-60 seconds ballpark to for the binary cache generation, so the “dominated by” guess isn’t obvious me, and I’d like to see it actually measured. I can do it if you prefer, e.g. if you feel that I’m being unnecessarily painful while insisting for benchmarking the entire operation, as perceived by the user, instead of apt update only.

#13 Updated by segfault 2017-02-15 22:08:26

> See the commit 440b9cc73a Completely disable APT translations. from the feature/12238-ship-apt-lists Git branch (of Feature #12238). On Jessie this makes quite a difference.
Awesome, thanks a lot! :)

#14 Updated by sajolida 2017-03-04 20:41:41

> IIRC the results sajolida gave me a while ago (using real storage
> hardware) were in the 30-60 seconds ballpark to for the binary cache
> generation, so the “dominated by” guess isn’t obvious me, and I’d
> like to see it actually measured. I can do it if you prefer, e.g. if
> you feel that I’m being unnecessarily painful while insisting for
> benchmarking the entire operation, as perceived by the user, instead
> of apt update only.

Yes, in my experience of additional software, even without downloading
anything, the binary cache generation is extremelly slow and long and
storing the download still make a very bad experience. We want to solve
this in Bug #9059 for the Additional software feature.

But I’m glad to see that Tails Server is affected by this problem as well :)

#15 Updated by segfault 2017-03-08 18:54:40

  • Affected tool set to Server

#16 Updated by segfault 2017-04-25 19:36:57

#17 Updated by segfault 2017-04-25 19:37:02

  • Status changed from Confirmed to Resolved
  • % Done changed from 0 to 100
  • Parent task set to Feature #12230

I merged feature/12238-ship-apt-lists from Feature #12238 and now installing services works without updating the apt lists :)