Tails

#4 Updated by intrigeri 2017-05-26 13:23:17

> Blocks Feature Feature #12576: Have Jenkins use basebox:clean_old instead of basebox:clean_all added

I don’t understand why, but perhaps it’s not important.

Let’s just fix it and I’ll shut up :)

#5 Updated by bertagaz 2017-05-26 13:48:51

intrigeri wrote:
> > Blocks Feature Feature #12576: Have Jenkins use basebox:clean_old instead of basebox:clean_all added
>
> I don’t understand why, but perhaps it’s not important.

Because at the moment we don’t have enough space to build a new basebox AND host several baseboxes in /var/lib/jenkins/.vagrant.d/, which will probably happen if we switch to basebox:clean_old.

#6 Updated by intrigeri 2017-05-26 14:11:46

> Because at the moment we don’t have enough space to build a new basebox AND host several baseboxes in /var/lib/jenkins/.vagrant.d/, which will probably happen if we switch to basebox:clean_old.

Wow, interesting! If that means we’re going to store multiple baseboxes both in /var/lib/libvirt/images and in /var/lib/jenkins/.vagrant.d, then perhaps it’s a problem ⇒ file a ticket about it and discuss potential solutions with anonym (before allocating disk space specifically to accommodate these two sets of mostly-duplicated data).

Meta: I have almost no clue how the whole thing works, so perhaps there’s a very good reason to do this, and then forget it, sorry.

#9 Updated by bertagaz 2017-05-27 14:08:48

intrigeri wrote:
> Wow, interesting! If that means we’re going to store multiple baseboxes both in /var/lib/libvirt/images and in /var/lib/jenkins/.vagrant.d, then perhaps it’s a problem ⇒ file a ticket about it and discuss potential solutions with anonym (before allocating disk space specifically to accommodate these two sets of mostly-duplicated data).

As I get it, we delete the baseboxes in /var/lib/libvirt/images through the use of the forcecleanup option. I think if we end up having several ones in this directory, it’s because of a build failure, that prevent rake from executing the related clean_up_builder_vms function in this case. I’ll track the builds to see if/why this situation happens in the next days.

#12 Updated by intrigeri 2017-05-28 10:34:00

I’m concerned we’ll become quickly confused (and it’ll be hard to maintain Redmine ticket semantics) if we discuss the clean up process of /var/lib/libvirt/images/ on a ticket that’s about something else entirely, i.e. /var/lib/jenkins, so please, anonym:

• address the “are we really going to store essentially the same data twice on each isobuilder?” question here;
• answer questions that are specific to the libvirt storage pool GC process on Bug #12599, rather than here.

Thanks :)

#13 Updated by anonym 2017-05-28 11:17:56

bertagaz wrote:
> bertagaz wrote:
> > intrigeri wrote:
> > > Wow, interesting! If that means we’re going to store multiple baseboxes both in /var/lib/libvirt/images and in /var/lib/jenkins/.vagrant.d, then perhaps it’s a problem ⇒ file a ticket about it and discuss potential solutions with anonym (before allocating disk space specifically to accommodate these two sets of mostly-duplicated data).
> >
> > As I get it, we delete the baseboxes in /var/lib/libvirt/images through the use of the forcecleanup option. I think if we end up having several ones in this directory, it’s because of a build failure, that prevent rake from executing the related clean_up_builder_vms function in this case. I’ll track the builds to see if/why this situation happens in the next days.
>
> Hmmm, assigning to anonym, because I just checked and there’s still at least one volume in the libvirt default storage after a build is finished (e.g. tails-builder-amd64-jessie-20170524-8cc1ccbade_vagrant_box_image_0.img).

Ah, that is right; the file

~/.vagrant.d/boxes/${BOX_NAME}/0/libvirt/box.img

is copied to

/var/lib/libvirt/images/"${BOX_NAME}_vagrant_box_image_0.img

by Vagrant whenever it sets up a domain using that base box (unless it already exists). This is redundant.

> And indeed I don’t see code in the Rakefile that takes care of removing such volumes, only the ones that have the same name than the vagrant VM.

There is code for removing such volume in clean_up_basebox(), so we only do it when removing base boxes.

> Is that expected?

Yes. I had underestimated how much of a problem using more disk space was, so I didn’t think this would matter.

> I guess we could simply remove it too, if we store it in ~/.vagrant.d/?

Yup, Vagrant will make the copy if needed. So I guess we can do this cleanup by default, since a ~800 MiB disk copy shouldn’t increase the build time too much for non-Jenkins users. Just to be sure we’re in sync, I implemented the change I believe to be safe (but I haven’t tested it!) on the feature/12599 branch (commit:69faba0c1d5e7b517616103f8e1c14528bdb55e8). What do you think?

#14 Updated by intrigeri 2017-05-28 14:44:31

> Yes. I had underestimated how much of a problem using more disk space was, so I didn’t think this would matter.

Meta: I’m constantly advocating against spending substantial engineering time on issues that can trivially be solved with a bit more hardware. This nagging from my part probably participated in creating the situation we’re trying to fix here. Now, there are caveats I want to clarify to try and fix the confusion I have perhaps created:

• Alas, our hardware doesn’t grow magically on-demand. So ideally, new requirements need to be roughly evaluated & communicated so whatever hardware purchase & installation is needed by a change can happen before we deploy stuff… and break bit of our infra (Feature #12002). But granted, often we go through several iterations, design changes, and it’s simply impossible to accurately evaluate the final requirements in advance. The only realistic solution I can think of, to avoid this chicken’n’egg problem, as long as we’re managing the bare metal our stuff runs on, is to do infra development in a setup that looks very much like the production one, but is not the production one.
• Sometimes only a little bit of software engineering effort is enough to avoid raising hardware requirements, and it’s worth spending this time instead of upgrading hardware (which has a cost in terms of sysadmin work). Seeing your fix for Bug #12599, I guess that ticket falls into this category :)

#24 Updated by bertagaz 2017-07-06 11:34:25

intrigeri wrote:
> We need these 100G for other matters (ever-growing data for services that existed before vagrant-libvirt), we’ve been struggling with disk space in a painful way since a couple months “thanks” to the timing of the vagrant-libvirt deployment vs. hardware planning, and I’ve grown tired of this situation already, so: no, sorry; I really don’t want to make things even worse. Let’s deal with the storage upgrade that the vagrant-libvirt stuff requires before allocating even more space to it.

Having a look at the spreadsheet I’m not sure to see which services will require that much data. I thought taking 30G and leaving 100G free was affordable while we purchase more HDDs. But I get you’re upset. Let’s wait then.