Feature #10971

Try giving 6 vcpus to each isotester on lizard

Added by intrigeri 2016-01-18 15:27:42 . Updated 2016-01-26 12:04:31 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Tails_2.0
Start date:
2016-01-18
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

See Feature #9264#note-28. The goals are:

  1. see if this makes the test suite run faster, that is, if we could run the test suite more often on lizard v2;
  2. see if this increases global CPU usage consumption on the host (we’re currently wasting quite a lot of CPU cores that are idling).

Subtasks

Related issues

Related to Tails - Feature #6729: Bump the number of CPU cores the testing VM has Resolved 2014-02-19

History

#1 Updated by intrigeri 2016-01-18 15:35:34

  • Description updated
  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

Change applied.

#2 Updated by intrigeri 2016-01-18 16:22:43

#3 Updated by intrigeri 2016-01-19 13:15:00

  • % Done changed from 10 to 20

intrigeri wrote:
> # see if this makes the test suite run faster, that is, if we could run the test suite more often on lizard v2;

It doesn’t make the test suite run any faster, but:

  • we don’t run the tests that use emulated USB storage; for those ones, the QEMU run by isotesterN might benefit from more cores;
  • we still give only one vcpu to the TailsToaster VM (Feature #6729); I suspect that giving it 1-2 more vcpus might make a difference; I’m going to try this out on a branch (test/10971-more-CPUs-for-TailsToaster).

> # see if this increases global CPU usage consumption on the host (we’re currently wasting quite a lot of CPU cores that are idling).

It doesn’t, our CPUs are still strongly underused.

My current conclusion is that on lizard v2 hardware, we should actually run more than 4 isotesters.

I’d like to try this setup:

  • keep 2 isotesters with our standard config (3 vcpus, 20GiB of RAM for disk cache)
  • downgrade a bit 2 isotesters to 3 vcpus, 10GiB of RAM or so
  • add 2 more isotesters, with 3 vcpus and 10GiB of RAM or so

I hope that something like 6-8 ISO testers would help us get the maximum we can out of our current hardware, if giving each of them more vcpus is not a winning strategy.

#4 Updated by intrigeri 2016-01-19 15:03:44

  • related to Feature #6729: Bump the number of CPU cores the testing VM has added

#5 Updated by intrigeri 2016-01-19 15:06:06

FTR, right now (15:04 UTC) each isotester is running a test suite (this is uncommon these days since building an ISO takes almost as long as running the part of the test suite that’s not tagged as fragile), and each isobuilder is building an ISO, so it can be interesting to look at the Munin data around this time.

#6 Updated by intrigeri 2016-01-21 16:06:50

intrigeri wrote:
> * we still give only one vcpu to the TailsToaster VM (Feature #6729); I suspect that giving it 1-2 more vcpus might make a difference; I’m going to try this out on a branch (test/10971-more-CPUs-for-TailsToaster).

The first 3 runs each took between 60 and 68 minutes, while the typical run from devel takes between 74 and 80 minutes. It would be interesting to know the impact of adding a vcpu to TailsToaster on the tests that use emulated USB storage, so I’ve merged this change into bugfix/10720-installer-freezes-on-jenkins.

#7 Updated by intrigeri 2016-01-24 18:30:51

intrigeri wrote:
> intrigeri wrote:
> > * we still give only one vcpu to the TailsToaster VM (Feature #6729); I suspect that giving it 1-2 more vcpus might make a difference; I’m going to try this out on a branch (test/10971-more-CPUs-for-TailsToaster).
>
> The first 3 runs each took between 60 and 68 minutes, while the typical run from devel takes between 74 and 80 minutes.

… and the next 3 runs contradicted that (62-79 minutes).

> It would be interesting to know the impact of adding a vcpu to TailsToaster on the tests that use emulated USB storage, so I’ve merged this change into bugfix/10720-installer-freezes-on-jenkins.

This time I’ll wait to have enough data.

#8 Updated by intrigeri 2016-01-24 19:42:05

  • Target version changed from Tails_2.0 to Tails_2.2

#9 Updated by intrigeri 2016-01-25 14:40:50

  • Target version changed from Tails_2.2 to Tails_2.0

intrigeri wrote:
> My current conclusion is that on lizard v2 hardware, we should actually run more than 4 isotesters.

This other experiment is now tracked by Feature #10996.

#10 Updated by intrigeri 2016-01-26 11:00:24

  • % Done changed from 20 to 50

Sadly, I modified other settings in the meantime (allocating one more vcpu to TailsToaster on some branches, and running builds+tests from those branches manually quite a few times), so it’s not clear whether the load + iowait + runtime increase means that allocating more vcpus pushes our hardware to its limits somehow, or that I simply loaded the box more than usual.

Also, we still lack the most important information we would need, that is: could we run the entire test suite more often with these changes?

Still, I think it’s enough data to conclude that giving each isotester more vcpus doesn’t buy us much, if anything. I’m going to revert them to 3 vcpus, and will continue with Feature #10996.

#11 Updated by intrigeri 2016-01-26 12:04:31

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 50 to 100

Since 11:00 UTC today, all isotesters are back to 3 vcpus. I call this experiment done.