Bug #12507

Vagrant: too little memory allocated for non-RAM builds

Added by vk 2017-05-04 10:16:50 . Updated 2019-04-08 13:35:28 .

Status:
Rejected
Priority:
Normal
Assignee:
vk
Category:
Build system
Target version:
Start date:
2017-05-04
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
1
Affected tool:
Deliverable for:

Description

Hi,

Would it be feasible to have a possibility to customize amount of RAM for build VM?
Right now it’s 512M for on-disk builds and 9ish G for RAM builds.
I can submit a patch which will add a possibility to specify this in `TAILS_BUILD_OPTIONS`, similar to `cpus=N`.

Also, I’d like to have ability to point vagrant-libvirt to a custom storage domain. Not everyone is using `default` one.
This can be done either from separate env variable, or via `TAILS_BUILD_OPTIONS`.
Can submit a patch for this as well.

Please let me know.


Subtasks


Related issues

Related to Tails - Feature #12533: Ability to point vagrant-libvirt to a custom storage pool Confirmed 2017-05-10

History

#1 Updated by intrigeri 2017-05-07 15:33:34

  • Assignee set to anonym

anonym, can you please handle the initial triaging of this ticket?

Also, maybe this should be split into 2 different tickets.

#2 Updated by anonym 2017-05-08 10:58:55

  • Assignee changed from anonym to vk
  • QA Check set to Info Needed

vk wrote:
> Would it be feasible to have a possibility to customize amount of RAM for build VM?

Sure, but I’m curious why you want this. I mean, currently you achieve the same by bumping the numbers in vagrant/lib/tails_build_settings.rb, and I cannot really come up with a scenario where that is not suitable. If your build fails because there’s too little memory around, that number should be bumped permanently in Git, not bumped temporarily. But I might have missed something! :)

> I can submit a patch which will add a possibility to specify this in `TAILS_BUILD_OPTIONS`, similar to `cpus=N`.

If you have a legit use case, sure! It might be a good time to rename the ram and noram options to rambuild and diskbuild, so ram=N can be used for this, although this would need some coordination with out sysadmins so they can adjust our infra accordingly.

> Also, I’d like to have ability to point vagrant-libvirt to a custom storage domain. Not everyone is using `default` one.
> This can be done either from separate env variable, or via `TAILS_BUILD_OPTIONS`.
> Can submit a patch for this as well.

This makes a lot of sense, but please file a new ticket and assign it to me. A patch would be nice, but there’s some significant rework going on so if you want to do it, please base your patch on the wip/11972-use-vagrant-in-jenkins Git branch. In any case, assign the new ticket to me.

#3 Updated by vk 2017-05-10 16:18:08

anonym wrote:
> vk wrote:
> > Would it be feasible to have a possibility to customize amount of RAM for build VM?
>
> If your build fails because there’s too little memory around, that number should be bumped permanently in Git, not bumped temporarily. But I might have missed something! :)

You did not miss anything, this is exactly my use case. With 512MB build was failing, adding 2G swap to the tails-builder VM did fix it; I’m testing with 2048MB / no swap now, will let you know if that’s enough.
Another option would be to just have swap partition added to the builder VM, not sure what would be the acceptable solution in your infra context.

> If you have a legit use case, sure! It might be a good time to rename the ram and noram options to rambuild and diskbuild, so ram=N can be used for this, although this would need some coordination with out sysadmins so they can adjust our infra accordingly.

If it will be decided to bump the VM_MEMORY_FOR_DISK_BUILDS in vagrant/lib/tails_build_settings.rb - I’ll take care about renaming the options too.

> This makes a lot of sense, but please file a new ticket and assign it to me. A patch would be nice, but there’s some significant rework going on so if you want to do it, please base your patch on the wip/11972-use-vagrant-in-jenkins Git branch. In any case, assign the new ticket to me.

Filed https://labs.riseup.net/code/issues/12533, assigned to you.

#4 Updated by emmapeel 2017-05-25 09:19:37

  • Status changed from New to Confirmed
  • Assignee deleted (vk)
  • QA Check changed from Info Needed to Dev Needed

#5 Updated by emmapeel 2017-05-25 09:19:59

  • Subject changed from Vagrant: ability to adjust RAM, ability to point vagrant-libvirt to custom storage domain to Vagrant: ability to adjust RAM

#6 Updated by intrigeri 2017-05-25 09:44:54

  • Tracker changed from Feature to Bug
  • Subject changed from Vagrant: ability to adjust RAM to Vagrant: too little memory allocated for non-RAM builds
  • Assignee set to vk
  • Type of work changed from Sysadmin to Code

(vk is working on it so they should remain the assignee.)

#7 Updated by anonym 2017-05-25 12:27:36

vk wrote:
> anonym wrote:
> > vk wrote:
> > > Would it be feasible to have a possibility to customize amount of RAM for build VM?
> >
> > If your build fails because there’s too little memory around, that number should be bumped permanently in Git, not bumped temporarily. But I might have missed something! :)
>
> You did not miss anything, this is exactly my use case. With 512MB build was failing, adding 2G swap to the tails-builder VM did fix it; I’m testing with 2048MB / no swap now, will let you know if that’s enough.
> Another option would be to just have swap partition added to the builder VM, not sure what would be the acceptable solution in your infra context.

Let’s bump the RAM instead of adding a swap partition since the RAM requirement still will be pretty low, and swap can be terribly slow if you don’t have an SSD.

> > If you have a legit use case, sure! It might be a good time to rename the ram and noram options to rambuild and diskbuild, so ram=N can be used for this, although this would need some coordination with out sysadmins so they can adjust our infra accordingly.
>
> If it will be decided to bump the VM_MEMORY_FOR_DISK_BUILDS in vagrant/lib/tails_build_settings.rb - I’ll take care about renaming the options too.

If disk builds fail due to too little RAM then VM_MEMORY_FOR_DISK_BUILDS definitely should be bumped, so yeah, it’s decided! :) So now this ticket is about how much to bump it.

However, note that bumping VM_MEMORY_FOR_DISK_BUILDS will bump VM_MEMORY_FOR_RAM_BUILDS as much, but RAM builds do work so this is undesirable. This means that BUILD_SPACE_REQUIREMENT is too big, and that’s very likely; we have been lazy and just bumped BUILD_SPACE_REQUIREMENT for the past years when it probably was the system that needed more RAM for its processes. So, you should decrease BUILD_SPACE_REQUIREMENT with the same amount you bump VM_MEMORY_FOR_DISK_BUILDS.

[Yeah, we never do disk builds, in case you wondered. :)]

Bonus points: it’d be cool if you did a disk build and monitored the disk usage very closesly, e.g. you do a rake vm:ssh and run something like:

while true; do
  df / | tail -n1 | awk '{ print $3 }'
  sleep 0.5
done >> /home/vagrant/disk-usage-monitoring.log


in parallel with the build. Then we can look for the highest value, subtract the first value (i.e. disk usage before the build started) and thus get the peak usage, which then could be used to find a better value for BUILD_SPACE_REQUIREMENT. Of course, we’ll want a higher value, to leave some marigin.

#8 Updated by Anonymous 2018-01-15 16:34:15

Ping?

#9 Updated by Anonymous 2018-08-17 09:03:50

  • related to Feature #12533: Ability to point vagrant-libvirt to a custom storage pool added

#10 Updated by Anonymous 2018-08-17 09:05:27

Are you still interested in working on this? If not, please deassign yourself from this ticket. It will make ticket tracking/triaging easier for us. Thanks.

#11 Updated by intrigeri 2019-04-08 13:35:28

  • Status changed from Confirmed to Rejected

@vk, please let us know if you’re still interested and I’ll reopen this ticket :)