Bug #17626: USB image test sometimes fails on Jenkins with "No space left"

Bug #17626

USB image test sometimes fails on Jenkins with "No space left"

Added by segfault 2020-04-16 14:00:05 . Updated 2020-04-25 05:40:08 .

Status:

Resolved

Priority:

Elevated

Assignee:

intrigeri

Category:

Continuous Integration

Target version:

Tails_4.6

Start date:

Due date:

% Done:

Feature Branch:

Type of work:

Sysadmin

Blueprint:

Starter:

Affected tool:

Deliverable for:

Description

See for example https://jenkins.tails.boum.org/job/test_Tails_ISO_devel/2039/

Error creating disk "usbimage" in "/tmp/TailsToaster/TailsToasterStorage". Need 7200 MiB but only 7695 MiB is available of which 500 MiB is reserved for other temporary files.
<false> is not true. (Test::Unit::AssertionFailedError)
./features/support/helpers/storage_helper.rb:99:in `create_new_disk'
./features/step_definitions/common_steps.rb:65:in `/^I (temporarily )?create an? (\d+) ([[:alpha:]]+) (?:([[:alpha:]]+) )?disk named "([^"]+)"$/'
features/usb_install.feature:128:in `And I create a 7200 MiB disk named "usbimage"'

Subtasks

Related issues

Blocks Tails - Feature #16209: Core work: Foundations Team

Confirmed

History

#1 Updated by segfault 2020-04-16 14:00:41

blocks Feature #16209: Core work: Foundations Team added

#2 Updated by segfault 2020-04-16 14:21:13

Feature Branch set to bugfix/17626-fix-no-space-left

I pushed a commit which prints the output of du -ah /tmp/TailsToaster when the error occurs, which should help us debug it.

#3 Updated by segfault 2020-04-16 14:24:40

Dumping intrigeri’s input on this issue from XMPP:

(05:01:40 PM) intrigeri: I’ve seen it in the past and back then I bumped “disk space” (aka. RAM because tmpfs, iirc), which was enough. Then it came back a few weeks ago, around the time we merged the ~~Bug #15460~~ branch. I did not investigate. I’ve asked my fellow RMs to check.
(05:02:48 PM) intrigeri: I don’t know if it indicates some regression in our test suite (possibly leftover files that are not cleaned up?) or an expected side-effect of ~~Bug #15460~~ (in which case we need to ask sysadmins to give more space in /tmp/TailsToaster).
(05:05:29 PM) intrigeri: I’m a bit surprised we did not spot this before the merge. Could very well be caused by another change, actually.
(05:09:25 PM) intrigeri: checking if it’s really tmpfs, I don’t remember what we did last time
(05:10:58 PM) intrigeri: yes, it’s a tmpfs with size=22910M. in other words: in this case we can’t assume “disk space is cheap” when working on the test suite. RAM is not very cheap and lizard is reaching its limits in this respect.
(05:11:57 PM) intrigeri: I guess I could bump it by 20M or somethig without risking much. We can keep growing the tmpfs without giving the VM more RAM, and some day we’ll get OOM, and we’ll cross that bridge once we’re there, I suppose. Would be fine by me.

#4 Updated by CyrilBrulebois 2020-04-22 09:45:14

Assignee set to Sysadmins
Priority changed from Normal to Elevated

Looking at the https://jenkins.tails.boum.org/view/RM/job/test_Tails_ISO_stable/2305/consoleFull log:

Need 7200 MiB but only 7675 MiB is available of which 500 MiB is reserved for other temporary files.

So I suppose we’ll need “20M or something” to be at least 25M as far as I can compute → I’ll ask for 30M to be on the safe side.

→ Hello fellow sysadmins!

Per previous comment, it seems our test_Tails_ISO_* Jenkins jobs are using a tmpfs defined with size=22910M; we would like to have it bumped by 30M since we’re (only slightly) running out of disk space. I haven’t spotted where that size is defined, feel free to let us know where that’s handled so that we can provide a patch next time such a tweak is needed, thanks!

Bumping severity as this makes all our testsuite runs fail (even if some extra failures are not uncommon).

#5 Updated by intrigeri 2020-04-22 16:27:37

I’ve seen cases where 45MB was missing.

#6 Updated by intrigeri 2020-04-22 16:31:56

segfault wrote:
> I pushed a commit which prints the output of du -ah /tmp/TailsToaster when the error occurs, which should help us debug it.

Thank you @segfault. I’ve merged this branch is I think it can be useful in the future to have such information earlier in the problem analysis and assigning process.

#7 Updated by intrigeri 2020-04-22 16:40:04

Category set to Continuous Integration
Assignee changed from Sysadmins to intrigeri
Target version set to Tails_4.6
Feature Branch deleted (~~bugfix/17626-fix-no-space-left~~)

Here are some logs thanks to segfault’s commit:

20:05:41   Scenario: Installing Tails with GNOME Disks from a USB image            # features/usb_install.feature:125
20:05:41     Given I have started Tails from DVD without network and logged in     # features/step_definitions/snapshots.rb:170
20:06:12     And I plug and mount a USB drive containing a Tails USB image         # features/step_definitions/usb.rb:206
20:06:13     And I create a 7200 MiB disk named "usbimage"                         # features/step_definitions/common_steps.rb:63
20:06:13       du -ah "/tmp/TailsToaster" | sort -hr | head -n20
20:06:13       15G  /tmp/TailsToaster
20:06:13       11G  /tmp/TailsToaster/TailsToasterStorage
20:06:13       9.3G /tmp/TailsToaster/TailsToasterStorage/__internal
20:06:13       2.0G /tmp/TailsToaster/tails-greeter-snapshot.memstate
20:06:13       2.0G /tmp/TailsToaster/no-network-logged-in-snapshot.memstate
20:06:13       1.2G /tmp/TailsToaster/TailsToasterStorage/uGfdYgQWsQ
20:06:13       455M /tmp/TailsToaster/.guestfs-0/appliance.d
20:06:13       455M /tmp/TailsToaster/.guestfs-0
20:06:13       448M /tmp/TailsToaster/.guestfs-0/appliance.d/root
20:06:13       60M  /tmp/TailsToaster/chutney-data/nodes.1587493632
20:06:13       60M  /tmp/TailsToaster/chutney-data
20:06:13       4.6M /tmp/TailsToaster/run-2020-04-21_18:27:04_+0000-git_475ba7c-7yNJNV
20:06:13       4.1M /tmp/TailsToaster/.guestfs-0/appliance.d/kernel
20:06:13       3.7M /tmp/TailsToaster/run-2020-04-21_18:27:04_+0000-git_475ba7c-7yNJNV/01:37:45_Writing_a_Tails_isohybrid_to_a_USB_drive_and_booting_it,_then_installing_Tails_on_top_of_it_using_Tails_Installer,_and_it_still_boots.mkv
20:06:13       3.0M /tmp/TailsToaster/.guestfs-0/appliance.d/initrd
20:06:13       1.7M /tmp/TailsToaster/chutney-data/nodes.1587493632/003auth
20:06:13       1.7M /tmp/TailsToaster/chutney-data/nodes.1587493632/002auth
20:06:13       1.7M /tmp/TailsToaster/chutney-data/nodes.1587493632/001auth
20:06:13       1.7M /tmp/TailsToaster/chutney-data/nodes.1587493632/000auth
20:06:13       1.5M /tmp/TailsToaster/chutney-data/nodes.1587493632/040obfs4
20:06:13       Error creating disk "usbimage" in "/tmp/TailsToaster/TailsToasterStorage". Need 7200 MiB but only 7649 MiB is available of which 500 MiB is reserved for other temporary files. (NoSpaceLeftError)
20:06:13       ./features/support/helpers/storage_helper.rb:103:in `create_new_disk'
20:06:13       ./features/step_definitions/common_steps.rb:66:in `/^I (temporarily )?create an? (\d+) ([[:alpha:]]+) (?:([[:alpha:]]+) )?disk named "([^"]+)"$/'
20:06:13       features/usb_install.feature:128:in `And I create a 7200 MiB disk named "usbimage"'

and:

19:55:47   Scenario: Installing Tails with GNOME Disks from a USB image            # features/usb_install.feature:125
19:55:47     Given I have started Tails from DVD without network and logged in     # features/step_definitions/snapshots.rb:170
19:56:18     And I plug and mount a USB drive containing a Tails USB image         # features/step_definitions/usb.rb:206
19:56:18     And I create a 7200 MiB disk named "usbimage"                         # features/step_definitions/common_steps.rb:63
19:56:18       du -ah "/tmp/TailsToaster" | sort -hr | head -n20
19:56:18       15G  /tmp/TailsToaster
19:56:18       11G  /tmp/TailsToaster/TailsToasterStorage
19:56:18       9.3G /tmp/TailsToaster/TailsToasterStorage/__internal
19:56:18       2.0G /tmp/TailsToaster/tails-greeter-snapshot.memstate
19:56:18       2.0G /tmp/TailsToaster/no-network-logged-in-snapshot.memstate
19:56:18       1.2G /tmp/TailsToaster/TailsToasterStorage/ckVSKzBAiQ
19:56:18       455M /tmp/TailsToaster/.guestfs-0/appliance.d
19:56:18       455M /tmp/TailsToaster/.guestfs-0
19:56:18       448M /tmp/TailsToaster/.guestfs-0/appliance.d/root
19:56:18       59M  /tmp/TailsToaster/chutney-data/nodes.1587407169
19:56:18       59M  /tmp/TailsToaster/chutney-data
19:56:18       4.1M /tmp/TailsToaster/.guestfs-0/appliance.d/kernel
19:56:18       3.0M /tmp/TailsToaster/.guestfs-0/appliance.d/initrd
19:56:18       1.7M /tmp/TailsToaster/chutney-data/nodes.1587407169/003auth
19:56:18       1.7M /tmp/TailsToaster/chutney-data/nodes.1587407169/002auth
19:56:18       1.7M /tmp/TailsToaster/chutney-data/nodes.1587407169/001auth
19:56:18       1.7M /tmp/TailsToaster/chutney-data/nodes.1587407169/000auth
19:56:18       1.5M /tmp/TailsToaster/chutney-data/nodes.1587407169/040obfs4
19:56:18       1.5M /tmp/TailsToaster/chutney-data/nodes.1587407169/039obfs4
19:56:18       1.5M /tmp/TailsToaster/chutney-data/nodes.1587407169/038obfs4
19:56:18       Error creating disk "usbimage" in "/tmp/TailsToaster/TailsToasterStorage". Need 7200 MiB but only 7685 MiB is available of which 500 MiB is reserved for other temporary files. (NoSpaceLeftError)
19:56:18       ./features/support/helpers/storage_helper.rb:103:in `create_new_disk'
19:56:18       ./features/step_definitions/common_steps.rb:66:in `/^I (temporarily )?create an? (\d+) ([[:alpha:]]+) (?:([[:alpha:]]+) )?disk named "([^"]+)"$/'
19:56:18       features/usb_install.feature:128:in `And I create a 7200 MiB disk named "usbimage"'

I see nothing in there that looks wrong, nor that would help me identify where the increased test suite disk space needs come from.

So yeah, I would say our best bet at this point is to give isotesters ~100MB more in /tmp/TailsToaster.

I’ll give it a try.

#8 Updated by intrigeri 2020-04-23 04:35:21

Type of work changed from Code to Sysadmin

#9 Updated by intrigeri 2020-04-23 04:47:41

Subject changed from USB image test sometimes fails with "No space left" to USB image test sometimes fails on Jenkins with "No space left"

#10 Updated by intrigeri 2020-04-25 05:40:08

Status changed from Confirmed to Resolved

The problem this issue is about did not happen on Jenkins since I deployed the tentative fix (checked with find /var/lib/jenkins/jobs/test_Tails_ISO_* -type f -name log -ctime -2 -exec grep 'Need 7200 MiB but only' '{}' \;).