Tails

#1 Updated by intrigeri 2016-09-19 03:40:57

• related to Feature #10851: Give lizard enough free storage to host our freezable APT repository added

#2 Updated by intrigeri 2016-09-19 03:41:37

• Description updated

#3 Updated by anonym 2016-11-25 10:56:55

• Target version changed from 284 to Tails 2.10

#4 Updated by intrigeri 2016-12-01 12:20:56

• related to Bug #12008: torbrowser archive disks are running out of disk space added

#5 Updated by intrigeri 2017-01-07 17:15:05

• Status changed from Confirmed to In Progress
• Assignee changed from intrigeri to bertagaz
• % Done changed from 0 to 20
• QA Check set to Ready for QA

Added a new, updated spreadsheet in sysadmin.git (storage-2017-ticket_11806.ods). My conclusion is that we should be good until the end of 2018, without purchasing any additional storage, if:

• we reclaim unused space (Bug #12119)
• we make use of 2/3 of the space currently unallocated in our PVs
• my estimates are mostly correct; in particular the ones about space needed for reproducible builds (Vagrant baseboxes) are rough guesses, because the current design is still stored on your hard drive only and I can’t read it (Feature #11988); 5GB should be enough to store a dozen baseboxes (and more likely 20); I don’t know if we want to create & keep more baseboxes than that in 2 years

Please review and reassign to me. We can finalize this once Feature #11988 is done.

#6 Updated by intrigeri 2017-01-07 17:16:17

• related to Bug #12119: Shrink lizard's jenkins-data LV added

#7 Updated by anonym 2017-01-24 20:48:51

• Target version changed from Tails 2.10 to Tails_2.11

#8 Updated by bertagaz 2017-03-08 10:38:06

• Target version changed from Tails_2.11 to Tails_2.12

#9 Updated by bertagaz 2017-03-08 13:24:40

• Priority changed from Normal to Elevated

#10 Updated by intrigeri 2017-03-26 10:18:24

• related to Feature #12111: APT snapshots: add arm64 architecture added

#11 Updated by intrigeri 2017-04-20 07:18:41

• Target version changed from Tails_2.12 to Tails_3.0~rc1

So, 2.12 wasn’t realistic, that’s fine. Do you think you can handle this during your next sysadmin shift (weeks 17-19)? Let’s make sure we’re not going to run out of disk space and avoid having to deal with such problems in a hurry :)

#12 Updated by intrigeri 2017-04-22 09:26:22

• related to Bug #12469: Resize ISO history and Tor Browser archive LVs added

#13 Updated by bertagaz 2017-05-21 16:03:11

• Target version changed from Tails_3.0~rc1 to Tails_3.0

#14 Updated by intrigeri 2017-05-24 07:04:05

• blocked by Feature #12002: Estimate hardware cost of reproducible builds in Jenkins added

#17 Updated by intrigeri 2017-05-25 09:28:46

• related to deleted (Feature #12111: APT snapshots: add arm64 architecture)

#18 Updated by intrigeri 2017-05-25 09:28:54

• blocks Feature #12111: APT snapshots: add arm64 architecture added

#19 Updated by intrigeri 2017-05-27 09:05:35

• Target version changed from Tails_3.0 to Tails_3.1

Blocked by Feature #12002 (that has no target version other than “2017”) and then in turn by some upcoming changes in our CI system that are in the early research phase, so postponing.

#20 Updated by intrigeri 2017-06-25 11:57:30

• Priority changed from Elevated to High

We’re short on disk space on several partitions, and can’t grow them as much as we would like as we need more storage and we’re blocking on Feature #12002 and then on Feature #11806 => bumping priority.

#21 Updated by intrigeri 2017-06-30 19:10:17

• blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#22 Updated by bertagaz 2017-07-01 14:55:19

• Assignee changed from bertagaz to intrigeri
• % Done changed from 20 to 30
• QA Check changed from Ready for QA to Info Needed

intrigeri wrote:
> We’re short on disk space on several partitions, and can’t grow them as much as we would like as we need more storage and we’re blocking on Feature #12002 and then on Feature #11806 => bumping priority.

Your estimates sound good so far for me. I’ve pushed a newer revision that:

• Add the growth with 4 new isobuilders root partitions to the equation. Not so much to count in the end, but I wanted to add this root partitions line.
• Add the isobuilders libvirt partitions (but that’s only 80G)
• Update from 200G to 400G to include the growth of the Jenkins partitions for 8 isobuilders in the “reproducible builds” line.
• Did not touch the “Jenkins data” line (artifacts), as we are currently using half only of it, and the 20% growth ratio we agreed on should not fill this partition that much.

Still missing the APT snapshots updates from Feature #12002, but worth having a look already if you wish so.

#24 Updated by intrigeri 2017-07-04 09:57:00

• Assignee changed from intrigeri to bertagaz
• % Done changed from 30 to 40
• QA Check deleted (Info Needed)

Reviewed, fixed/update/improved some stuff. Comments & questions:

• what’s “VM’s root” exactly? Is that only isobuilders or all VMs? Please clarify on the spreadsheet.
• in some places I was confused as the estimates for isobuilders there were not the same as what you wrote on the blueprint for Feature #12002, so please pick one set of estimates and apply it consistently to both that blueprint and the spreadsheet (probably the one that takes into account future growth as that is part of the cost of reproducible builds);
• I’ve added 100G of slack to take into account Piwik (see the corresponding thread).
• I’ve bumped a bit the APT snapshots estimate to better take Feature #12111 into account.

So indeed, at first glance we’re looking at adding about 2TB of storage. I don’t know if we can add 2 more drives though, and I would like to retire our two 5-yeard-old rotating hard-drives anyway, but let’s not bother speculating until we have the final number.

#25 Updated by bertagaz 2017-07-05 09:39:29

• Assignee changed from bertagaz to intrigeri
• QA Check set to Info Needed

intrigeri wrote:
> Reviewed, fixed/update/improved some stuff. Comments & questions:
>
> * what’s “VM’s root” exactly? Is that only isobuilders or all VMs? Please clarify on the spreadsheet.

I meant all the VMs, including the possibly 4 new isobuilders. Fixed in the file.

> * in some places I was confused as the estimates for isobuilders there were not the same as what you wrote on the blueprint for Feature #12002, so please pick one set of estimates and apply it consistently to both that blueprint and the spreadsheet (probably the one that takes into account future growth as that is part of the cost of reproducible builds);

Not sure which place you’re talking about in this file. Maybe the difference you see is because there I’ve counted 8 isobuilders in the estimates for 2018?

> * I’ve added 100G of slack to take into account Piwik (see the corresponding thread).
> * I’ve bumped a bit the APT snapshots estimate to better take Feature #12111 into account.

Good catch!

> So indeed, at first glance we’re looking at adding about 2TB of storage. I don’t know if we can add 2 more drives though, and I would like to retire our two 5-yeard-old rotating hard-drives anyway, but let’s not bother speculating until we have the final number.

The case we have is supposed to be able to handle two more disks, but we should ask. It looks like that in the end of 2018, we’ll have (for now) only 400G left, so I wonder if we shouldn’t just add two disk, and then think later about upgrading the old rotating one to get more disk space.

#26 Updated by intrigeri 2017-07-05 20:00:02

• blocks Bug #13425: Upgrade lizard's storage (2017 edition) added

#27 Updated by intrigeri 2017-07-06 13:36:01

• Status changed from In Progress to Resolved

>> * what’s “VM’s root” exactly? Is that only isobuilders or all VMs? Please clarify on the spreadsheet.

> I meant all the VMs, including the possibly 4 new isobuilders. Fixed in the file.

Thank you :)

>> * in some places I was confused as the estimates […]

> Not sure which place you’re talking about in this file. Maybe the difference you see is because there I’ve counted 8 isobuilders in the estimates for 2018?

Yes, probably. I think I might have written my last comment before having fully understood this.

> The case we have is supposed to be able to handle two more disks, but we should ask. It looks like that in the end of 2018, we’ll have (for now) only 400G left, so I wonder if we shouldn’t just add two disk, and then think later about upgrading the old rotating one to get more disk space.

Let’s discuss that on Bug #13425, which is about addressing the problem this ticket defines.

So I think we’re done here, woohoo!

Conclusion: adding 1.7 TB of storage should be enough to satisfy our needs until the end of 2018.

#28 Updated by intrigeri 2017-07-06 13:36:51

• Assignee deleted (intrigeri)
• % Done changed from 40 to 100
• QA Check changed from Info Needed to Pass

#29 Updated by intrigeri 2018-08-09 14:23:40

• related to Feature #15780: Update server storage planning needs for 2019-2020 added