Tails

#2 Updated by intrigeri 2017-04-05 17:19:53

> Let also assume that per year we have a delay where we host 2 baseboxes (when we update it), and maybe one other when we need to change the build system (as a safety in the estimation). Does it seem sound?

This seems to implicitly assume at least:

• that we’re going to host baseboxes in some central place (unclear given Feature #12409)
• that we’re going to bother garbage collecting old baseboxes: IIRC last time we discussed this, our conclusion was that it was not worth the effort

I suggest you write down your assumptions explicitly, so that we can ensure we update the estimates if/when these assumptions change, and more generally it’ll help understanding where our conclusions come from.

> Now we have to consider that we need to keep the same amount of partial APT snapshots than baseboxes. For now we don’t really have a way to estimate how much disk space it means, as we have no mechanism in the basebox build system to have a list of Debian packages needed for that. I’ll fill a ticket about that. In the meantime, I’ll do a basebox build with an empty APT cache so that we get an idea.

It seems that whey you write “partial APT snapshots”, you mean “tagged APT snapshots” (given you’re discussing the list of needed Debian packages). Now I’m lost: do we have plans to actually generate such snapshots? I see no such thing on the blueprint, and I don’t recall any discussion about it. The blueprint says we’re going to use frozen (but not tagged) APT snapshots, and that we’ll keep them around for 6 months. So if I’m remembering + reading things right, generating a list of packages installed when creating a basebox won’t teach us anything useful. Please clarify if I got some of this wrong.

#17 Updated by intrigeri 2017-06-01 06:43:46

(Relaxed relationship with related tickets: the research leading to the numbers we need might happen here, or on other tickets that are about specific issues. If the former, then this ticket would block the others; if the latter, then it’s the opposite. So let’s stick to “Related to” for now.)

#19 Updated by intrigeri 2017-06-01 09:07:02

Wrt. the storage space needed for APT snapshots used by the build boxes, I think the problem has been greatly simplified now that updating the basebox is part of our release process, and we use the same snapshots as in the ISO.

#22 Updated by bertagaz 2017-06-05 12:20:03

I’ve adapted the above numbers with what was decided on Bug #12574#note-6 regarding isobuilders root partition.

#23 Updated by bertagaz 2017-06-05 12:27:44

bertagaz wrote:
> h3. Artifacts
>
> Reproducible builds will add one ISO each time a build fails. Difficult to
> guess how often it will happen. Let’s consider 50% of the time worst case?
>
> -> Count number of base branches artifacts we keep now and multiply.
>
> h2. Will we need more isobuilders?
>
> -> Need to evaluate how much time it may delay the output we have, meaning
> having a look to the number of base branches builds in the past, or maybe
> assume we’ll want N more?

So here are the number of base branches builds (stable + devel + testing + feature/stretch) in the past:

month  : base_branches / total
---
2015-02: 86  / 263 
2015-03: 134 / 358 
2015-04: 92  / 351 
2015-05: 147 / 268 
2015-06: 91  / 248 
2015-07: 50  / 101 
2015-08: 92  / 195 
2015-09: 87  / 470 
2015-10: 106 / 935 
2015-11: 105 / 611 
2015-12: 117 / 809
2016-01: 154 / 757
2016-02: 126 / 603
2016-03: 89  / 839
2016-04: 85  / 779
2016-05: 143 / 1055
2016-06: 146 / 1022
2016-07: 126 / 1178
2016-08: 128 / 658
2016-09: 141 / 404
2016-10: 147 / 575
2016-11: 145 / 482
2016-12: 151 / 476
2017-01: 172 / 625
2017-02: 159 / 590
2017-03: 239 / 674
2017-04: 217 / 638
2017-05: 161 / 478

Some datapoints:

• Tails 2.0 was released end of 2016 January
• reproducible build jobs (at least for feature/stretch) were added end of 2016 November.
• porting to Stretch started on 2016 August
• the high peaks during 2016 May to July were mostly due to a lot of work happening in the test suite side, tagging a lot of them as fragile and fixing a lot of them.

#24 Updated by intrigeri 2017-06-14 19:12:42

> h3. /var/lib/jenkins

> artifacts 5G

What’s this about?

> h3. Artifacts

> Reproducible builds will add one ISO each time a build fails.

I’ll assume below you instead mean “each time a build is not reproducible”. Correct? Otherwise, please clarify as I don’t get it.

> Difficult to guess how often it will happen. Let’s consider 50% of the time worst case?

I seriously hope that we won’t be breaking reproducible builds that commonly. Let’s say 20%?

> -> Count number of base branches artifacts we keep now and multiply.

This seems to be based on the implicit assumption that we’re going to do reproducibility testing on all active branches: I agree we should, because we should not merge branches that break reproducibility (once our ISO is reproducible); but I don’t remember any ticket about this. Please create one if we have none, so we base on hardware cost estimates on actual plans :)

Once this is clarified, I agree with the proposed way of calculating this.

> h3. time-based APT snapshots

> Probably one or two to keep sometimes when we update the basebox in the middle of the 4 months. Size unknown for now, would need evaluation.

Err, we currently keep the snapshots used for major releases for 6 months, not 4:

6. Make it so the time-based APT repository snapshots are kept around
long enough, by bumping their `Valid-Until` to 6 months from now:
[[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]

Please check if that’s still relevant, or if $someone forgot to update release_process.mdwn when we decided to keep baseboxes for 4 months only.

> h2. Memory

> Current: 14.5G * 4

> We did not bump it a lot in the past, biggest one was due to Vagrant itself,

Here, we need to account for the additional memory we already had to allocate when switching to Vagrant: it’s been taken on our spare reserve and has already forced us to do some trade-offs, so it doesn’t come for free.

> but it should stabilize in the future.

I wonder what makes you think so. Every time we work on porting Tails to a new version of Debian, we have to bump the amount of RAM needed for in-memory builds. I suggest you look at this growth rate and take it into account.

> h2. Will we need more isobuilders?

> With the reproducibles builds we may have a lower output time,

I guess this again implicitly relies on the assumption that we’re going to do reproducibility testing on all active branches. Correct?

With this assumption in mind, well, “may” seems half-assed: we’re simply gonna build twice as many ISO images, so it’ll definitely make the feedback loop longer.

> -> Need to evaluate how much time it may delay the output we have, meaning having a look to the number of base branches builds in the past, or maybe assume we’ll want N more?

Do you mean “active” branches instead of “base” branches?

Anyway, I think we can assume that building twice as many ISO images without affecting the developer/RM experience too much roughly requires doubling the throughput of our CI for ISO builds. This can be done in various ways, by making builds faster and/or having more isobuilders. I propose you simply add this info to the blueprint for Bug #11680, and we don’t bother trying to find a more precise estimate here about the needed throughput increase. OK?

#26 Updated by intrigeri 2017-06-14 19:18:32

Also, it would be nice if this was done during the 3.1 cycle, if you can: this ticket is blocking Feature #11806 that has become somewhat urgent while we’ve allocated space needed here and there… including to handle the unplanned needs of the reproducible builds system. I didn’t look closely, but I think we won’t have enough space anymore to grow storage volumes to match our needs in a couple months, and I would dislike having to delete more isotesters/isobuilders again.

#28 Updated by bertagaz 2017-06-15 09:53:01

intrigeri wrote:
> > h3. /var/lib/jenkins
>
> > artifacts 5G
>
> What’s this about?

We need disk space in this partition to retrieve the two ISOs.

Now I agree that with the disk space we have to build the basebox, it should be enough without adding more for the artifacts. So let’s forget this.

> > h3. Artifacts
>
> > Reproducible builds will add one ISO each time a build fails.
>
> I’ll assume below you instead mean “each time a build is not reproducible”. Correct? Otherwise, please clarify as I don’t get it.

Yes, that’s what I meant.

>
> > Difficult to guess how often it will happen. Let’s consider 50% of the time worst case?
>
> I seriously hope that we won’t be breaking reproducible builds that commonly. Let’s say 20%?

As you wish. I don’t know how to estimate this right now.

> > -> Count number of base branches artifacts we keep now and multiply.
>
> This seems to be based on the implicit assumption that we’re going to do reproducibility testing on all active branches: I agree we should, because we should not merge branches that break reproducibility (once our ISO is reproducible); but I don’t remember any ticket about this. Please create one if we have none, so we base on hardware cost estimates on actual plans :)

Ok, in my mind, the assumption was: we try to reproduce base branches only, as we’ve discussed to set up reproducible jobs for this branches only for now. I’ve created Feature #12715 to discuss that and decide something.

> Once this is clarified, I agree with the proposed way of calculating this.
>
> > h3. time-based APT snapshots
>
> > Probably one or two to keep sometimes when we update the basebox in the middle of the 4 months. Size unknown for now, would need evaluation.
>
> Err, we currently keep the snapshots used for major releases for 6 months, not 4:
>
> […]
>
> Please check if that’s still relevant, or if $someone forgot to update release_process.mdwn when we decided to keep baseboxes for 4 months only.

Right, I think the release process is not up to date, as we’ve decided since then to bump this snapshots at every releases, so we though 4 months were enough.

> > h2. Memory
>
> > Current: 14.5G * 4
>
> > We did not bump it a lot in the past, biggest one was due to Vagrant itself,
>
> Here, we need to account for the additional memory we already had to allocate when switching to Vagrant: it’s been taken on our spare reserve and has already forced us to do some trade-offs, so it doesn’t come for free.
>
> > but it should stabilize in the future.
>
> I wonder what makes you think so. Every time we work on porting Tails to a new version of Debian, we have to bump the amount of RAM needed for in-memory builds. I suggest you look at this growth rate and take it into account.

Ack.

> > h2. Will we need more isobuilders?
>
> > With the reproducibles builds we may have a lower output time,
>
> I guess this again implicitly relies on the assumption that we’re going to do reproducibility testing on all active branches. Correct?

Nop, I was assuming we’ll reproduce base branches only.

> Anyway, I think we can assume that building twice as many ISO images without affecting the developer/RM experience too much roughly requires doubling the throughput of our CI for ISO builds. This can be done in various ways, by making builds faster and/or having more isobuilders. I propose you simply add this info to the blueprint for Bug #11680, and we don’t bother trying to find a more precise estimate here about the needed throughput increase. OK?

Let see how Feature #12715 goes.

> > So here are the number of base branches builds (stable + devel + testing + feature/stretch) in the past:
> I’m sorry I don’t understand how it’s relevant here. Can you please clarify?

That’s because I was considering reproduce base branches only.

> Also, it would be nice if this was done during the 3.1 cycle, if you can: this ticket is blocking Feature #11806 that has become somewhat urgent while we’ve allocated space needed here and there… including to handle the unplanned needs of the reproducible builds system. I didn’t look closely, but I think we won’t have enough space anymore to grow storage volumes to match our needs in a couple months, and I would dislike having to delete more isotesters/isobuilders again.

That was my intent.

#29 Updated by intrigeri 2017-06-19 11:37:34

bertagaz wrote:
> intrigeri wrote:
>> > h3. /var/lib/jenkins
>>
>> > artifacts 5G
>>
>> What’s this about?

> We need disk space in this partition to retrieve the two ISOs.

OK.

> Now I agree that with the disk space we have to build the basebox, it should be enough without adding more for the artifacts. So let’s forget this.

ACK.

Note: please move the current state of your thoughts to a blueprint, as having to read the entire ticket history + apply each incremental change to understand what’s the current proposal has already become too painful. But we can/should still discuss changes here :)

>> > h3. Artifacts
>>
>> > Reproducible builds will add one ISO each time a build fails.
>>
>> I’ll assume below you instead mean “each time a build is not reproducible”. Correct? Otherwise, please clarify as I don’t get it.

> Yes, that’s what I meant.

OK, good. Please update this in the proposal once it’s been moved to a blueprint.

>> > Difficult to guess how often it will happen. Let’s consider 50% of the time worst case?
>>
>> I seriously hope that we won’t be breaking reproducible builds that commonly. Let’s say 20%?

> As you wish. I don’t know how to estimate this right now.

If we wanted to do any kind of serious estimate, we could start by looking at the history of the Feature #5630 branch builds and see how often we’ve broken it by merging other branches. But I doubt it’s worth the hassle at this point.

>> > h3. time-based APT snapshots
>>
>> > Probably one or two to keep sometimes when we update the basebox in the middle of the 4 months. Size unknown for now, would need evaluation.
>>
>> Err, we currently keep the snapshots used for major releases for 6 months, not 4:
>>
>> […]
>>
>> Please check if that’s still relevant, or if $someone forgot to update release_process.mdwn when we decided to keep baseboxes for 4 months only.

> Right, I think the release process is not up to date, as we’ve decided since then to bump this snapshots at every releases, so we though 4 months were enough.

Then please ensure this is fixed => ticket

>> > h2. Memory
>>
>> > Current: 14.5G * 4
>>
>> > We did not bump it a lot in the past, biggest one was due to Vagrant itself,
>>
>> Here, we need to account for the additional memory we already had to allocate when switching to Vagrant: it’s been taken on our spare reserve and has already forced us to do some trade-offs, so it doesn’t come for free.
>>
>> > but it should stabilize in the future.
>>
>> I wonder what makes you think so. Every time we work on porting Tails to a new version of Debian, we have to bump the amount of RAM needed for in-memory builds. I suggest you look at this growth rate and take it into account.

> Ack.

I’ll let you update this proposal on the blueprint then :)

>> Also, it would be nice if this was done during the 3.1 cycle, if you can: this ticket is blocking Feature #11806 that has become somewhat urgent while we’ve allocated space needed here and there… including to handle the unplanned needs of the reproducible builds system. I didn’t look closely, but I think we won’t have enough space anymore to grow storage volumes to match our needs in a couple months, and I would dislike having to delete more isotesters/isobuilders again.

> That was my intent.

Great!

#34 Updated by intrigeri 2017-07-04 09:44:40

I’ve taken a look and improved the blueprint a bit, please have a look at my changes.

#36 Updated by bertagaz 2017-07-04 12:04:34

intrigeri wrote:
> I’ve taken a look and improved the blueprint a bit, please have a look at my changes.

Looks good. So the only remaining is APT snapshots. I wonder how realistic it is to settle on 40G for one time-based snapshot given that’s what the 2.12 one used (as shown in Bug #12725)? If we do, then as we stated on the blueprint that would mean 4 * 40G we’d have to keep during two release cycles.

#37 Updated by intrigeri 2017-07-05 08:41:38

> So the only remaining is APT snapshots. I wonder how realistic it is to settle on 40G for one time-based snapshot given that’s what the 2.12 one used (as
> shown in Bug #12725)?

Yes, but please apply to this number the ratio I’ve computed on Feature #12111, and apply a 1.3 ratio of top of that to account for the growth of the Debian archive.

> If we do, then as we stated on the blueprint that would mean 4 * 40G we’d have to keep during two release cycles.

The release says 3 extra snapshots, not 4, so perhaps I’m confused or not looking at the right place?
Once this is clarified, this sounds good to me (I’m too lazy to find the reasoning behind this “3” number, which is not on the blueprint, but I’ll trust you have copied the result correctly).

#40 Updated by intrigeri 2017-07-05 10:13:41

Deleted the “blocked by Feature #12715” relationship (as the current estimates are about the “worst” case situation) so we can close this ticket.

#43 Updated by bertagaz 2017-07-05 10:37:13

intrigeri wrote:
> Well, no: there was still the snapshots lifetime thing left to handle first, otherwise these estimates are pretty much off with the real world. I see no related ticket created since I’ve asked you to do so (in order to avoid forgetting this bit) so I assume this disappeared from your radar. Tracking this takes me more time / mental space / energy than handling it myself, so I went ahead: please review commit:945027e0d908a3120af873cd3f817ff1c10c5a31.

Oooch, yes I forgot that. Looks good to me, thanks for the fix.