Bug #13526

apt-snapshots partition lacks disk space

Added by intrigeri 2017-07-27 11:24:44 . Updated 2017-08-12 22:16:34 .

Status:
Resolved
Priority:
Normal
Assignee:
intrigeri
Category:
Infrastructure
Target version:
Tails_3.2
Start date:
2017-07-27
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

On Bug #12725 we temporarily addressed the problem, but we’re back to a problematic situation: only 27GB available today. We still have a little bit of margin (75GB on SSDs, 50GB on rotating drives, better not use those for this service) to allocate more space though. I’ll check if we can do that without harming other ever-growing services.

Subtasks

Related issues

Related to Tails - Bug #13425: Upgrade lizard's storage (2017 edition) Resolved 2017-07-05
Related to Tails - Bug #12725: Sort out the apt-snapshots-disk partition situation on apt.lizard Resolved 2017-06-16
Related to Tails - Feature #13579: Cron script cleaning up the time-based apt-snapshots tmp dir Resolved 2017-08-04
Related to Tails - Feature #11912: Purge Jessie time-based APT snapshots Resolved 2016-11-13
Blocks Tails - Feature #13233: Core work 2017Q3: Sysadmin (Maintain our already existing services) Resolved 2017-06-29

History

#1 Updated by intrigeri 2017-07-27 11:24:54

  • blocks Feature #13233: Core work 2017Q3: Sysadmin (Maintain our already existing services) added

#2 Updated by intrigeri 2017-07-27 11:26:32

  • Description updated

#3 Updated by intrigeri 2017-07-27 13:05:56

  • Status changed from Confirmed to Resolved
  • % Done changed from 0 to 100

Apparently the only other storage volume that’ll require more space soon is the bitcoin-data one. So I went ahead and gave apt-snapshots 10 more GB. Let’s hope Bug #13425 happens before we hit too many similar issues at once.

#4 Updated by intrigeri 2017-07-27 13:06:08

  • related to Bug #13425: Upgrade lizard's storage (2017 edition) added

#5 Updated by intrigeri 2017-07-27 15:02:34

  • Status changed from Resolved to In Progress
  • % Done changed from 100 to 50

grml, that wasn’t enough.

#6 Updated by intrigeri 2017-07-27 15:08:58

  • % Done changed from 50 to 60
  • QA Check set to Ready for QA

Given it 5GB again. One of Feature #11912 and Bug #13425 now must happen soon, otherwise we’ll be in a really problematic situation.

#7 Updated by intrigeri 2017-07-27 15:44:12

  • Status changed from In Progress to Resolved
  • % Done changed from 60 to 100
  • QA Check deleted (Ready for QA)

This time it’s fixed. Let’s “hope” that either we’ll get more storage soon, because the two issues that prevent us from cleaning up our APT snapshots can’t be solved before Tails 3.2.

#8 Updated by intrigeri 2017-07-29 06:07:49

  • related to Bug #12725: Sort out the apt-snapshots-disk partition situation on apt.lizard added

#9 Updated by intrigeri 2017-07-29 06:18:50

  • Status changed from Resolved to In Progress
  • % Done changed from 100 to 50

And here we go again, only 38 GiB left => Icinga2 warning.

We should see quite some disk space reclaimed on August 21 (Feature #13178#note-14) so I’ll ignore this for now. I’ll keep an eye on it and will only allocate more space to that volume if it reaches critical levels before this cleanup happens. Rationale: I want to avoid our free space in PVs to itself reach critical levels.

Another (crazy?) way to reclaim a whole lot of disk space after the 3.1 release would be to do Feature #11738 during this cycle, merge it in time for 3.1, and then do Feature #11912 shortly after the 3.1 release. I’ll ask the RM if he would consider this option.

#10 Updated by groente 2017-08-04 11:36:58

Icinga warnings went critical today. I noticed 13GB worth of stale tmp directories in /srv/apt-snapshots/time-based/tmp , which have now been purged to get the critical back to warning atleast. Emptying this tmp dir should be part of the garbage collection procedure.

#11 Updated by groente 2017-08-04 14:29:07

Created https://labs.riseup.net/code/issues/13579 to deal with the tmp dir situation.

#12 Updated by intrigeri 2017-08-05 12:36:59

  • related to Feature #13579: Cron script cleaning up the time-based apt-snapshots tmp dir added

#13 Updated by intrigeri 2017-08-06 12:23:06

  • Target version changed from Tails_3.1 to Tails_3.2

#14 Updated by intrigeri 2017-08-12 22:16:34

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100

Solved via Feature #11912:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vdc        641G  449G  192G  71% /srv/apt-snapshots

#15 Updated by intrigeri 2017-08-12 22:17:06

  • related to Feature #11912: Purge Jessie time-based APT snapshots added