Feature #9400

Evaluate future storage needs on lizard

Added by intrigeri 2015-05-14 09:17:18 . Updated 2015-06-09 16:07:39 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Infrastructure
Target version:
Tails_1.4.1
Start date:
2015-05-14
Due date:
% Done:

100%

Feature Branch:
Type of work:
Research
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

  • isotesterN VMs: at least 45GB/VM, and we want to set up 3 more such VMs; note that storage needs will increase once snapshots are used more intensively in the test suite, which is WIP (Bug #5571), so we should account for that
  • autobuilt ISOs: now that we have better estimates of how many branches we’re going to build, and how long we’re going to keep them (see proposed algorithm on Feature #6439), we should redo our disk storage needs estimates
  • Tor Browser archive: will current allocation be sufficient for, say, a year?
  • Tails ISO archive: will current allocation be sufficient for, say, a year?
  • freezable APT repo
  • bitcoind

Subtasks

Feature #9401: Evaluate peak working directory size during a full test suite run Resolved

100
Feature #9508: Evaluate freezable APT repo's storage needs Resolved

100

Related issues

Related to Tails - Feature #10243: Update system requirements for lizard failover Resolved 2015-09-25
Related to Tails - Bug #16025: jenkins-data-disk is running out of diskspace again Resolved 2018-10-03

History

#1 Updated by intrigeri 2015-05-30 09:24:14

  • Description updated

#2 Updated by intrigeri 2015-05-30 09:54:00

  • Description updated
  • Status changed from Confirmed to In Progress
  • Tor Browser archive:
    • we store 2 copies
    • current = 10GB (*2) allocated, 6GB (*2) used
    • growth = 2GB/month (we’ve imported 3 alphas of 4.5, so that’s perhaps an extreme case, but let’s plan for the worst)
    • in a year = 6+12*2 = 30GB; but we store two copies => 60GB i.e. +40GB
    • performance: can be slow (the subset that’s used for ISO builds is served by apt-cacher-ng)
  • ISO archive
    • we store 2 copies
    • current = 60GB (*2) allocated, 40GB (*2) used
    • growth = 1.2GB/month
    • in a year = 40+12*1.2 = 55GB; but we store two copies => 110GB, that fits into the currently allocated space
    • performance: can be slow (not public)
  • autobuilt ISOs
    • current = 100GB allocated
    • peak = (25 not-merged-yet topic branches + 5 main branches) * 15 builds of each = 450 ISOs = 500GB i.e. +400GB
    • performance: can be slow (limiting factor is NFS speed)

#3 Updated by intrigeri 2015-05-30 09:59:30

  • bitcoind
    • current = 52GB allocated, 43GB used
    • growth = roughly doubles each year
    • in a year = 90GB i.e. +40GB
    • performance: should be fast or on separate storage from services that must be fast themselves

#4 Updated by intrigeri 2015-05-30 19:20:01

  • freezable APT repo:
    • current = 0
    • in a year = 425GB, i.e. +425GB
    • performance: can be slow (the subset that’s used for ISO builds is served by apt-cacher-ng)

#5 Updated by intrigeri 2015-05-30 19:44:37

  • Assignee changed from intrigeri to bertagaz
  • QA Check set to Ready for QA

Only the info from Feature #9401 is now missing.

I think we should also move to faster storage:

  • apt-cacher-ng’s data (it’s hit by more and more systems, and we can’t allocate enough RAM to it so that e.g. data used by ISO builds would be in memory cache): 20GB
  • isobuilderN’s Jenkins partition (it contains the workspaces): 2*6GB
  • the puppetmaster’s data partition (it is hit by more and more systems): 5GB

So, summing up, in a year we’ll need, in addition to what we already have:

  • slow storage: (Tor Browser archive + ISO archive + autobuilt ISOs + freezable APT repo - bitcoind - acng data - isobuilderN’s Jenkins data - puppetmaster’s data) = 40 + 0 + 400 + 425 - 52 - 20 - 12 - 5 = 776GB => a couple 1TB disks would be fine; perhaps 2TB would be more future-proof if not crazy expensive
  • fast storage: (isotesterN * 3 + bitcoind + acng data + isobuilderN’s Jenkins data + puppetmaster’s data) = XXX * 3 + 90 + 20 + 12 + 5 = XXX * 3 + 127GB => might fit into a couple 256GB SSDs, if we compromise a bit here and there (e.g. isotesterN’s system partition can be on slow storage)

bertagaz, can you please have a look, even if we’re still lacking a value for XXX? As said elsewhere already, I think this blocks Feature #6196, so we’d better move quickly on this one.

#6 Updated by intrigeri 2015-06-03 14:55:35

So, we now know XXX: 4GB for the system + 2GB for Git checkout and ISO + 14GB for /tmp/TailsToaster = 20GB.

=> in a year our additional fast storage needs will be: (isotesterN * 3 + bitcoind + acng data + isobuilderN’s Jenkins data + puppetmaster’s data) = 20 * 3 + 90 + 20 + 12 + 5 = 187GB => will fit into a couple 256GB SSDs.

So, bertagaz: I’m proposing we by a couple 1 or 2TB rotating drives (depending on the price delta, and also perhaps some 2TB drives are faster due to higher density), and a couple 200-320GB SSDs (focussing on performance + reliability more than on size).

And when we get ourselves another box dedicated to running the test suite, we’ll see if we migrate the SSDs there or buy new ones. I’ll ask our beloved hardware expert what exact gear he recommends these days, which should give you a bit of time to comment before the order is passed.

#7 Updated by intrigeri 2015-06-03 16:06:22

intrigeri wrote:
> I’ll ask our beloved hardware expert what exact gear he recommends these days […]

Done.

#8 Updated by intrigeri 2015-06-05 20:23:58

Current best option seems to be 2*1TB Samsung 850EVO SSDs.

#9 Updated by intrigeri 2015-06-09 16:07:39

  • Status changed from In Progress to Resolved
  • Assignee deleted (bertagaz)
  • QA Check changed from Ready for QA to Pass

bertagaz privately ack’ed the current proposal. Next steps will be tracked in new, dedicated tickets.

#10 Updated by intrigeri 2015-06-09 16:09:27

  • blocks Feature #9551: Have more storage plugged into lizard added

#11 Updated by intrigeri 2015-06-11 22:06:18

  • blocked by deleted (Feature #9551: Have more storage plugged into lizard)

#12 Updated by intrigeri 2015-09-25 07:30:04

  • related to Feature #10243: Update system requirements for lizard failover added

#13 Updated by bertagaz 2018-10-03 09:36:18

  • related to Bug #16025: jenkins-data-disk is running out of diskspace again added