Bug #15310

Tails builds occasionally fail because VM is out of memory

Added by segfault 2018-02-13 16:37:17 . Updated 2018-03-14 11:04:58 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Build system
Target version:
Tails_3.6
Start date:
2018-02-13
Due date:
% Done:

100%

Feature Branch:
bugfix/15310-bump-builder-memory bugfix/15310-bump-builder-memory-devel
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

It happened to me repeatedly in the last weeks that Tails builds failed with virtual memory exhausted: Cannot allocate memory. My workaround is to increase the VM memory size from 768 MiB to 1024 MiB (I did not test any other value). I would be glad if you could fix this in the main repo, so I don’t have to remember doing this on every branch I want to build.

Note: My branches are all based on stable, I don’t know if the same error also occurs on devel.

Subtasks

History

#1 Updated by intrigeri 2018-02-13 19:30:06

  • Category set to Build system
  • Target version set to Tails_3.7

> My workaround is to increase the VM memory size from 768 MiB to 1024 MiB

Could you please share how you do that exactly?

Possibly related: since (at least) mid-November my ISO builds fail; I did not manage to understand why by monitoring the VM’s Journal and kmsg. Since I’ve switched to noram they work just fine. I suspect that’s the same problem.

#2 Updated by segfault 2018-02-13 19:53:45

intrigeri wrote:
> > My workaround is to increase the VM memory size from 768 MiB to 1024 MiB
>
> Could you please share how you do that exactly?


diff --git a/vagrant/lib/tails_build_settings.rb b/vagrant/lib/tails_build_settings.rb
index c74d38e2b2..5fae352858 100644
--- a/vagrant/lib/tails_build_settings.rb
+++ b/vagrant/lib/tails_build_settings.rb
@@ -9,7 +9,7 @@ VIRTUAL_MACHINE_HOSTNAME = 'vagrant-stretch'
 BUILD_SPACE_REQUIREMENT = 12.25*1024

 # Virtual machine memory size for on-disk builds
-VM_MEMORY_FOR_DISK_BUILDS = 768
+VM_MEMORY_FOR_DISK_BUILDS = 1024

 # Virtual machine memory size for in-memory builds
 VM_MEMORY_FOR_RAM_BUILDS = VM_MEMORY_FOR_DISK_BUILDS + BUILD_SPACE_REQUIREMENT

#3 Updated by intrigeri 2018-02-14 08:18:48

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10
  • QA Check set to Ready for QA

intrigeri wrote:
> > My workaround is to increase the VM memory size from 768 MiB to 1024 MiB
>
> Could you please share how you do that exactly?
>
> Possibly related: since (at least) mid-November my ISO builds fail; I did not manage to understand why by monitoring the VM’s Journal and kmsg. Since I’ve switched to noram they work just fine. I suspect that’s the same problem.

Thanks segfault! So FTR my problem is different: even with segfault’s workaround applied (and rake basebox:clean_all to ensure I get a fresh VM) my build still fails unless I use noram. So let’s focus this ticket back to the problem segfault reported.

#4 Updated by intrigeri 2018-02-19 14:00:23

  • Target version changed from Tails_3.7 to Tails_3.6

#5 Updated by anonym 2018-02-22 16:12:16

  • % Done changed from 10 to 50
  • Feature Branch set to bugfix/15310-bump-builder-memory bugfix/15310-bump-builder-memory-devel

I decided to first try “moving” 256M of the requirement from BUILD_SPACE_REQUIREMENT to VM_MEMORY_FOR_DISK_BUILDS. If jenkins is happy, I’ll merge into stable (and devel, so I pushed a branch making sure it works in there as well).

#6 Updated by intrigeri 2018-02-23 07:43:39

> If jenkins is happy, I’ll merge

If I remember correctly that gzipcomp uses more memory, it would be good to also test with it (which Jenkins won’t do).

#7 Updated by intrigeri 2018-02-23 07:54:03

> If I remember correctly that gzipcomp uses more memory, it would be good to also test with it (which Jenkins won’t do).

That was stupid, ignore me. Sorry! (I always forget that VM_MEMORY_FOR_DISK_BUILDS is also used for RAM builds. Which might indicate that this variable could be better names :)

#8 Updated by anonym 2018-02-23 11:44:44

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Ready for QA to Info Needed

intrigeri wrote:
> > If I remember correctly that gzipcomp uses more memory, it would be good to also test with it (which Jenkins won’t do).
>
> That was stupid, ignore me. Sorry! (I always forget that VM_MEMORY_FOR_DISK_BUILDS is also used for RAM builds. Which might indicate that this variable could be better names :)

Does commit:d315b405499e561e8302ed02515a1372a5fcf1b2 address your concern?

#9 Updated by intrigeri 2018-02-24 12:04:38

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Info Needed to Ready for QA

> intrigeri wrote:
>> > If I remember correctly that gzipcomp uses more memory, it would be good to also test with it (which Jenkins won’t do).
>>
>> That was stupid, ignore me. Sorry! (I always forget that VM_MEMORY_FOR_DISK_BUILDS is also used for RAM builds. Which might indicate that this variable could be better names :)

> Does commit:d315b405499e561e8302ed02515a1372a5fcf1b2 address your concern?

Absolutely, thanks a lot!

#10 Updated by anonym 2018-02-26 12:34:30

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 50 to 100
  • QA Check changed from Ready for QA to Pass

Good! All builds succeeded on Jenkins, so I’m happy and have merged into stable (-> devel) and killed the -devel branch.

#11 Updated by bertagaz 2018-03-14 11:04:58

  • Status changed from Fix committed to Resolved