Bug #13302

/var/lib/libvirt/images sometimes gets filled on isobuilders, take 2

Added by intrigeri 2017-06-30 18:06:44 . Updated 2017-07-17 08:25:20 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Continuous Integration
Target version:
Start date:
2017-06-30
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:
289

Description

icinga2 just told me that this problem is not 100% solved, and just happened again on isobuilder1 during https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_devel/27/:

$ ls -l /var/lib/libvirt/images
total 2.8G
-rw------- 1 libvirt-qemu libvirt-qemu 1.5G Jun 30 16:46 tails-builder-amd64-jessie-20170609-daf786c121_default.img
-rwxr--r-- 1 libvirt-qemu libvirt-qemu 1.3G Jun 30 15:02 tails-builder-amd64-jessie-20170609-daf786c121_vagrant_box_image_0.img*
$ df /var/lib/libvirt/images
/dev/vdc    ext4      2.9G  2.8G     0 100% /var/lib/libvirt/images

Subtasks


Related issues

Related to Tails - Bug #12599: /var/lib/libvirt/images gets filled on isobuilders Resolved 2017-05-25
Blocks Tails - Bug #12618: Retrieving ISO build artifacts sometimes fails on Jenkins Resolved 2017-05-31

History

#1 Updated by intrigeri 2017-06-30 18:06:55

  • related to Bug #12599: /var/lib/libvirt/images gets filled on isobuilders added

#2 Updated by intrigeri 2017-07-05 17:58:02

  • Subject changed from /var/lib/libvirt/images gets filled on isobuilders, take 2 to /var/lib/libvirt/images sometimes gets filled on isobuilders, take 2

#3 Updated by bertagaz 2017-07-07 13:57:51

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 50

intrigeri wrote:
> icinga2 just told me that this problem is not 100% solved, and just happened again on isobuilder1 during https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_devel/27/:
>
> […]

I’ve growned the libvirt partitions of the isobuilders of 1G (to 4G), so we should be good by now. Let’s wait a few days to see.

#4 Updated by bertagaz 2017-07-07 13:58:40

  • blocks Bug #12618: Retrieving ISO build artifacts sometimes fails on Jenkins added

#5 Updated by intrigeri 2017-07-07 15:49:53

> I’ve growned the libvirt partitions of the isobuilders of 1G (to 4G), so we should be good by now. Let’s wait a few days to see.

In passing, I wonder why the previous size was enough most of the time, but sometimes not. I somewhat suspect the “sometimes not” means there’s a bug somewhere, and perhaps growing the partition won’t be enough. Anyway, I agree with what you did, and we’ll see :)

#6 Updated by bertagaz 2017-07-14 11:26:26

  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Ready for QA

intrigeri wrote:
> > I’ve growned the libvirt partitions of the isobuilders of 1G (to 4G), so we should be good by now. Let’s wait a few days to see.
>
> In passing, I wonder why the previous size was enough most of the time, but sometimes not. I somewhat suspect the “sometimes not” means there’s a bug somewhere, and perhaps growing the partition won’t be enough. Anyway, I agree with what you did, and we’ll see :)

There is maybe several reasons other than a bug: e.g building Stretch makes the snapshot grow a bit more than to build Jessie. Some changes in the basebox creation or provisioning happened too. We’ll see.

Anyway, icinga2 is happy since a week, we seems to be good now.

#7 Updated by intrigeri 2017-07-17 08:25:20

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 50 to 100
  • QA Check changed from Ready for QA to Pass

bertagaz wrote:
> > In passing, I wonder why the previous size was enough most of the time, but sometimes not. I somewhat suspect the “sometimes not” means there’s a bug somewhere, and perhaps growing the partition won’t be enough. Anyway, I agree with what you did, and we’ll see :)
>
> There is maybe several reasons other than a bug: e.g building Stretch makes the snapshot grow a bit more than to build Jessie. Some changes in the basebox creation or provisioning happened too. We’ll see.

None of this explains why this problem was happening seemingly randomly, but whatever, we’ll see.

> Anyway, icinga2 is happy since a week, we seems to be good now.

Great :)