Tails

#3 Updated by intrigeri 2014-03-30 17:04:06

I’m unsure about this one:

• users report quite often mirrors that they perceive as being too slow (especially in the few days after a release), so loading them more is not necessarily a good idea
• more generally, we encourage people to use BitTorrent in order to alleviate the reliance on a relatively small pool of mirrors, and as a way to easily contribute bandwidth back once their download is finished
• in the current state of things, the BitTorrent download does not depend on the boum.org nameservers, so it should work even if boum.org is down; adding web seeding might change this (at least with broken BitTorrent clients, perhaps)
• I don’t remember anyone complaining about slowness when downloading over BitTorrent, quite the contrary

What do you think?

#4 Updated by geb 2014-03-30 18:08:32

I agree with your points.

Webseeding would not only allow to increase interest of using BitTorrent but also to add some load distribution mechanism between HTTP mirrors, as it would allows to download simultaneously through multiple servers and torrent seeders.

It would also allows to integrate mirrors that are unable to offer torrent seeding and/or to not offer sufficient performance if used alone. Using those servers would offer signifiant gain in term of performance and resilience.

Even if those servers would rely on boum.org domain name and infrastructure, their use would be completely optional as they are only used to complement BitTorrent Seeders. There is no evidence that adding web-seeding would cause issues with any client. This feature is now 5 to 6 years old.

Anyway these is no immediate need for it, nor to take a decision now. I can just be nice to consider it and to keep the idea for later.

#5 Updated by intrigeri 2014-03-30 18:19:03

> It would also allows to integrate mirrors that are unable to offer torrent seeding
> and/or to not offer sufficient performance if used alone. Using those servers would
> offer signifiant gain in term of performance and resilience.

Just to make sure I got your idea right: you’re suggesting to create
another round-robin DNS pool (e.g. webseeds.tails.boum.org), to which
we could add web servers that are too slow to be used as regular HTTP
mirrors, right?

> There is no evidence that adding web-seeding would cause issues with any client.

Point taken.

#6 Updated by geb 2014-03-30 18:45:43

intrigeri wrote:
> > It would also allows to integrate mirrors that are unable to offer torrent seeding
> > and/or to not offer sufficient performance if used alone. Using those servers would
> > offer signifiant gain in term of performance and resilience.
>
> Just to make sure I got your idea right: you’re suggesting to create
> another round-robin DNS pool (e.g. webseeds.tails.boum.org), to which
> we could add web servers that are too slow to be used as regular HTTP
> mirrors, right?
>

Yes and no:

- We can reuse the actual DNS pool or setup a new one, dedicaced to that.
- Or we can add DNS records for every/some mirrors. Thus, content can be download from multiple BitTorrent seeders, but also from multiple servers simultaneously.

I Think the last point would be the most interesting. The list of servers we use in another part of the question. But maybe would it be more interesting to leverage a bit more bandwidth constrained servers for web-seeding and to use a bit less fast servers that can offer fast download when used alone.

#7 Updated by intrigeri 2014-03-30 19:31:55

> - We can reuse the actual DNS pool or setup a new one, dedicaced to that.

If we reuse the current DNS pool for web seeding, then we’ll have to
raise the requirements for including a server in this pool, which
does not feel like an improvement to me. So, another pool seems
better, if we go this way (which does not seem the best anyway,
I agree).

> - Or we can add DNS records for every/some mirrors. Thus, content can be download
> from multiple BitTorrent seeders, but also from multiple servers simultaneously.

> I Think the last point would be the most interesting. The list of servers we use in
> another part of the question. But maybe would it be more interesting to leverage
> a bit more bandwidth constrained servers for web-seeding and to use a bit less fast
> servers that can offer fast download when used alone.

Fully agreed. So, we would have e.g. webseed1.tails.boum.org,
webseed2.tails.boum.org, etc., and one (or more) server(s) behind each
of these hostnames. I like it.

As I see it, the only problem left is now: how do we move a mirror
from web seed to regular HTTP mirror status, and the opposite.
Reliability and performance seem to be the most relevant metrics, but
this needs to be made more precise. If anyone is interested in
thinking this problem through, you’ll want to talk to sajolida
privately to learn about the data we already have, and how it might
need to be improved.

#8 Updated by geb 2014-03-30 20:22:42

intrigeri wrote:
> > - Or we can add DNS records for every/some mirrors. Thus, content can be download
> > from multiple BitTorrent seeders, but also from multiple servers simultaneously.
>
> > I Think the last point would be the most interesting. The list of servers we use in
> > another part of the question. But maybe would it be more interesting to leverage
> > a bit more bandwidth constrained servers for web-seeding and to use a bit less fast
> > servers that can offer fast download when used alone.
>
> Fully agreed. So, we would have e.g. webseed1.tails.boum.org,
> webseed2.tails.boum.org, etc., and one (or more) server(s) behind each
> of these hostnames. I like it.
>

Not sure of what is better. Having a kind of pool, like you describe or to list every server and relies on clients to do the load balancing. Have to think about it and ask some BitTorrent clients developpers what can be the better.

> As I see it, the only problem left is now: how do we move a mirror
> from web seed to regular HTTP mirror status, and the opposite.
> Reliability and performance seem to be the most relevant metrics, but
> this needs to be made more precise. If anyone is interested in
> thinking this problem through, you’ll want to talk to sajolida
> privately to learn about the data we already have, and how it might
> need to be improved.

Agreed.

#14 Updated by geb 2016-06-06 07:04:05

Hi,

One good thing with webseeding is that it will be used in parallel with Torrent download. So it will use less bandwidth than a full download. So maybe the mirrors that were not fast enough for dl.a.b.o would be good candidates for being added to a webseeding.dl.b.o.org pool.

However has we don’t have estimate of the trafic it will generate to those mirrors, it may be better to wait for those mirrors to receive less requests. It will obviously be the case as we increased a bit the number of mirrors, but we also keept the possibility of adding some weights in mirror selection, maybe should we wait next releases and increase weight of fast mirrors, to favor them, letting slower mirrors handle less requests + webseeding trafic.

(Side note: i dunno i) if it is interesting ii) how clients would react, but maybe we can also discuss about adding multiple http://webseeding.dl.b.o.org entries in the Torrent, to parallize requests with multiples webseeding hosts)

#15 Updated by Dr_Whax 2016-06-06 14:23:07

geb wrote:
> Hi,
>
> One good thing with webseeding is that it will be used in parallel with Torrent download. So it will use less bandwidth than a full download. So maybe the mirrors that were not fast enough for dl.a.b.o would be good candidates for being added to a webseeding.dl.b.o.org pool.

Is this true for every client?

>
> However has we don’t have estimate of the trafic it will generate to those mirrors, it may be better to wait for those mirrors to receive less requests. It will obviously be the case as we increased a bit the number of mirrors, but we also keept the possibility of adding some weights in mirror selection, maybe should we wait next releases and increase weight of fast mirrors, to favor them, letting slower mirrors handle less requests + webseeding trafic.

Are we talking about the “Too slow mirrors” on: https://tails.boum.org/blueprint/HTTP_mirror_pool/? I think using the smaller mirrors for that seems to make sense, since it’s for improving the resilience of receiving a Tails iso.

>
> (Side note: i dunno i) if it is interesting ii) how clients would react, but maybe we can also discuss about adding multiple http://webseeding.dl.b.o.org entries in the Torrent, to parallize requests with multiples webseeding hosts)

Seems i’ll have to run some experiments with:

- Creating a trial webseeding mirror pool.
- Trying out various clients on Linux/Mac (I don’t have a Windows machine)

Come and report back to see what people think.

#16 Updated by geb 2016-06-07 01:17:25

>> One good thing with webseeding is that it will be used in parallel with Torrent download. So it will use less bandwidth than a full download. So maybe the mirrors that were not fast enough for dl.a.b.o would be good candidates for being added to a webseeding.dl.b.o.org pool.
> Is this true for every client?

Every one i tested so far (Transmission, Azareus, BitTorrent, uTorrent), yeah.

> Are we talking about the “Too slow mirrors”

Yeah. Too slow mirrors would be the better candidates for handling webseeding for me, because it will be done in parallel / complement with the classical torrent download.

>> (Side note: i dunno i) if it is interesting ii) how clients would react, but maybe we can also discuss about adding multiple http://webseeding.dl.b.o.org entries in the Torrent, to parallize requests with multiples webseeding hosts)
> Seems i’ll have to run some experiments with:

If it is just for the side note, maybe can we keep that for future and don’t block on that ? Up to you :)

#17 Updated by Anonymous 2017-06-30 12:34:42

@drwhax: is this still relevant? what’s missing to make it happen? or should it be dropped?

#18 Updated by Dr_Whax 2017-07-25 21:27:27

Let me look into it worth investing time into this still.

#19 Updated by geb 2017-07-26 14:15:13

hi,

u wrote:
> @drwhax: is this still relevant? what’s missing to make it happen? or should it be dropped?

When we created this bug a few years ago, the situation was a bit different:

• there were only 10 or 20 mirrors.
• some of them where hosted into low end providers (OVH, online.net) known to do some shapping when you consume too much traffic.
• some of those mirrors were seing almost 10Mbs of trafic

We now have much more mirrors (thanks to the tails-mirrors team !), most of them hosted by big hosts like universities etc. So there is less presure on those mirrors.

I beleive it means that there is less need to work on BitTorrent distribution, expect if we want to increase its use, and if we have good candidate for that (mirrors on low end hosters).

What’s your opinion ?

#20 Updated by intrigeri 2017-07-26 17:48:56

> We now have much more mirrors (thanks to the tails-mirrors team !), most of them hosted by big hosts like universities etc. So there is less presure on those mirrors.

… and FWIW we have a mechanism to give weight to each mirror, so given a good data collection / feedback loop, we could send more clients to the big mirrors, which incidentally would make it less of a problem to have more smallish mirrors without harming UX.