Tails

#3 Updated by segfault 2017-02-14 16:13:59

Ideas:

* Don’t ship the testing and unstable APT sources (Feature #11539)

* Would reduce update time down to 30s
* Could break stuff in corner cases (see https://labs.riseup.net/code/issues/11539#note-1)

* Only fetch the repositories we actually need for Tails Server

* See for example http://askubuntu.com/a/65250

* More?

#5 Updated by intrigeri 2017-02-14 18:03:33

It should be much faster on Stretch, thanks to optimizations in APT.
I say let’s benchmark before we try & solve this :)

#6 Updated by segfault 2017-02-14 18:10:39

> It should be much faster on Stretch, thanks to optimizations in APT.
That would be great :)
> I say let’s benchmark before we try & solve this :)
Sure.

#7 Updated by anonym 2017-02-15 09:06:18

segfault wrote:
> Ideas:

• See the commit 440b9cc73a Completely disable APT translations. from the feature/12238-ship-apt-lists Git branch (of Feature #12238). On Jessie this makes quite a difference.

#9 Updated by anonym 2017-02-15 10:02:54

BTW, I have observed a ~50% slowdown since we switched to the APT onion services.

#10 Updated by anonym 2017-02-15 10:47:41

We also have to take into account that feature/stretch has one Debian version less among the APT sources, namely stable, since it is based on testing. When feature/stretch is released, it will have the same number of Debian versions in its APT sources.

#11 Updated by anonym 2017-02-15 11:21:06

After completely disabling APT translations I benchmarked only the apt update time (running it 20 times) like this:

. /usr/local/lib/tails-shell-library/tor.sh && \
(\
  for x in $(seq 1 20); do \
    rm -fr /var/lib/apt/lists/*; \
    systemctl restart tor@default; \
    until tor_is_working; do sleep 1; done; \
    time apt update; done \
) &> apt.log

and got:

• Tails 2.10 median time: 1m 15s (for 23.6 MB of data)
• Tails 3.0~beta1 median time: 50s (for 15.2 MB of data)

Taking my previous comment (Feature #12237#note-11) into account (one less APT source in 3.0~beta1) these results seems consistent with there being no noticeable improvement for apt update, so I guess the optimizations talked about in Feature #12237#note-6 are only about the generation of /var/cache/apt/{src,}pkgcache.bin. I guess it’d be interesting to benchmark the time needed to generate those on stable vs stretch, with and without persistent APT lists, but I still suspect that the slowness is dominated by the fetches over Tor.

I did have several outliers at >= five minutes thanks to slow circuits, and previously I’ve seen even worse (> 10 minutes). We’ll never be able to avoid this unless we ship the lists, so if we don’t we absolutely must have better feedback about what is going on with apt update in Tails Server. But even apt itself doesn’t have very good feedback (beyond “stuff is still being written to the screen, various numbers are increasing”), and neither does Synaptic, so I’m not sure we can easily achieve that.

#12 Updated by intrigeri 2017-02-15 13:00:19

Thanks for this data!

> Taking my previous comment (Feature #12237#note-11) into account (one less APT source in 3.0~beta1) these results seems consistent with there being no noticeable improvement for apt update, so I guess the optimizations talked about in Feature #12237#note-6 are only about the generation of /var/cache/apt/{src,}pkgcache.bin.

This matches what I (vaguely) remember.

> I guess it’d be interesting to benchmark the time needed to generate those on stable vs stretch, with and without persistent APT lists, but I still suspect that the slowness is dominated by the fetches over Tor.

IIRC the results sajolida gave me a while ago (using real storage hardware) were in the 30-60 seconds ballpark to for the binary cache generation, so the “dominated by” guess isn’t obvious me, and I’d like to see it actually measured. I can do it if you prefer, e.g. if you feel that I’m being unnecessarily painful while insisting for benchmarking the entire operation, as perceived by the user, instead of apt update only.

#13 Updated by segfault 2017-02-15 22:08:26

> See the commit 440b9cc73a Completely disable APT translations. from the feature/12238-ship-apt-lists Git branch (of Feature #12238). On Jessie this makes quite a difference.
Awesome, thanks a lot! :)

#14 Updated by sajolida 2017-03-04 20:41:41

> IIRC the results sajolida gave me a while ago (using real storage
> hardware) were in the 30-60 seconds ballpark to for the binary cache
> generation, so the “dominated by” guess isn’t obvious me, and I’d
> like to see it actually measured. I can do it if you prefer, e.g. if
> you feel that I’m being unnecessarily painful while insisting for
> benchmarking the entire operation, as perceived by the user, instead
> of apt update only.

Yes, in my experience of additional software, even without downloading
anything, the binary cache generation is extremelly slow and long and
storing the download still make a very bad experience. We want to solve
this in Bug #9059 for the Additional software feature.

But I’m glad to see that Tails Server is affected by this problem as well :)