Tails

#3 Updated by intrigeri 2016-12-01 19:27:25

The process killed by the OOM killer is:

[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[  713]   109   713  1281356    27548     160        0             0 java

#6 Updated by intrigeri 2016-12-06 10:00:33

(“Whatever the reason, this bug simply doesn’t happen on isobuilder1 with vagrant-libvirt” might be a valid answer :)

#9 Updated by intrigeri 2016-12-18 10:19:20

intrigeri wrote:
> The only relevant changes I see immediately before this date are allocating a bit more huge pages on lizard, and giving each ISO builder VM a bit more RAM. I don’t see how given ISO builders more memory can trigger this problem.

vagrant/provision/assets/build-tails mounts a tmpfs with size=100%, so giving ISO builders more memory can actually affect this.

#11 Updated by intrigeri 2016-12-19 13:14:15

I’ve noticed that the “cached” value in the output of free sometimes stays very high, after a build has succeeded. During a build, this value grows with the amount of used space on the tmpfs mounted on /tmp/tails-build.*. So it looks like the way we umount this tmpfs (as part of trap cleanup EXIT, with the -l == lazy option) might sometimes not work, which leaves the memory used by that build occupied, which breaks the next builds.

Detailed analysis, on isobuilder2:

• Dec 18, https://jenkins.tails.boum.org/job/build_Tails_ISO_feature-stretch/101/
  • 15:10:37: build has started, tmpfs not mounted yet, 379 MB cached memory
  • 16:08:45: sudo umount -f -l /tmp/tails-build.EvA6qUle
  • 16:08:47: the tmpfs is not listed in /proc/mounts anymore, but then there’s still 7590 MB cached memory
  • 16:10:23: [WS-CLEANUP] Deleting project workspace…[WS-CLEANUP] done
  • 16:10:25: Finished: SUCCESS
  • 16:10:28: cached memory drops to 6386 MB
  • tl;dr: the tmpfs was not successfully unmounted, and the corresponding memory is never freed
• Dec 18, https://jenkins.tails.boum.org/job/build_Tails_ISO_feature-stretch-unfrozen/50/
  • 21:19:00: build starts, still more than 6 GB of cached memory
  • 21:46:06: 7179 MB cached memory, of which /tmp/tails-build.vhvkbSG0 uses 385 MB
  • 21:46:10: dpkg fails which fails the build
  • 21:46:10: perl -ni -E say $mountpoint if (($mountpoint) = ($_ =~ m{^(?:aufs|tmpfs|devpts-live|proc-live|sysfs-live) on (/tmp/tails-build(?:-tmpfs)?\.[/[:alnum:]]+)}))
  • 21:46:10: sudo umount -f -l /tmp/tails-build.vhvkbSG0
  • 21:46:11: the tmpfs is not listed in /proc/mounts anymore, there’s 6661 MB cached memory
  • 21:46:13: Finished: FAILURE
  • tl;dr: the tmpfs was successfully unmounted, and the corresponding memory freed
• then a couple builds fail before having mounted their tmpfs, and cached memory doesn’t change in the significative way
• Dec 19, https://jenkins.tails.boum.org/job/build_Tails_ISO_feature-5684-screen-locker/19/console
  • 04:00:05: 10147 MB cached memory, of which /tmp/tails-build.pCpXxptd uses 3.8 GB
  • 04:00:09: oom killer kills the (Jenkins) java process, FATAL: hudson.remoting.RequestAbortedException: java.io.EOFException
  • 04:00:15: 10302 MB cached memory, of which /tmp/tails-build.pCpXxptd uses 4.1 GB
  • 04:00:20: the tmpfs is not listed in /proc/mounts anymore, cached memory dropped to 6168 MB
  • tl;dr: the tmpfs was successfully unmounted, and the corresponding memory freed

I see the same pattern on:

• on isobuilder3: https://jenkins.tails.boum.org/job/build_Tails_ISO_feature-8230-greeter-revamp-phase1/39/ succeeds but leaves lots of cached memory occupied, and the next build (https://jenkins.tails.boum.org/job/build_Tails_ISO_feature-11293-network-manager-spoof-mac/33/) triggers the OOM killer
• on isobuilder4: https://jenkins.tails.boum.org/job/build_Tails_ISO_bugfix-11948-remove-keyserver-proxy-option-from-enigmail/30/ succeeds but leaves lots of cached memory occupied, and the next build (https://jenkins.tails.boum.org/job/build_Tails_ISO_bugfix-11831-nvidia-maxwell-firmware/32/) triggers the OOM killer

#12 Updated by intrigeri 2016-12-19 13:18:29

intrigeri wrote:
> bertagaz: I don’t understand anymore (sorry if I forgot!) why Feature #11972 would fix this breakage.

If the analysis I’ve posted in my previous comment is correct, then Feature #11972 will indeed fix this: the memory used by the tmpfs will be freed by shutting down the Vagrant level-2 VM. Still, we’ve been suffering from this breakage since more than a month, and there’s no ETA that I’m aware of on Feature #11972, so I’ll try to find a quick fix without waiting Feature #11972. I’ll try not spending too much time on it though.