Tails

#1 Updated by intrigeri 2015-09-17 09:02:30

On my dev system, testing an ISO built from current stable, test suite from current stable branch as well:

• evince.feature, --view, no video capture:
  • 1 vcpu: 8:54, 9:10, 8:55
  • 2 vcpus: 8:21, 8:01, 8:02
• evince.feature, no --view, --capture enabled:
  • 1 vcpu: 9:26, 9:47, 10:18
  • 2 vcpus: 9:04, 9:26, 8:41

No idea if this would be useful for anyone else (e.g. we would have to benchmark this on isotesterN.lizard), but at least for me it would save quite some test suite runtime.

#2 Updated by intrigeri 2015-09-17 12:00:49

Implementation hints: see set_ram_size for how to dynamically patch the VM’s config. But in this case it should probably be done in the “a computer” step, depending on the configuration.

#3 Updated by kytv 2015-09-17 14:02:15

This probably isn’t “good” but it seems to work.

--- a/features/step_definitions/common_steps.rb
+++ b/features/step_definitions/common_steps.rb
@@ -77,6 +77,7 @@ end
 Given /^a computer$/ do
   @vm.destroy_and_undefine if @vm
   @vm = VM.new($virt, VM_XML_PATH, $vmnet, $vmstorage, DISPLAY)
+  @vm.set_number_of_cpus($config['NUMBER_OF_CPUS']) if $config['NUMBER_OF_CPUS']
 end

 Given /^the computer has (\d+) ([[:alpha:]]+) of RAM$/ do |size, unit|
diff --git a/features/support/helpers/vm_helper.rb b/features/support/helpers/vm_helper.rb
index d838130..0c219a5 100644
--- a/features/support/helpers/vm_helper.rb
+++ b/features/support/helpers/vm_helper.rb
@@ -293,6 +293,18 @@ class VM
     return list
   end

+  def set_number_of_cpus(num_of_cpus)
+    raise "The number of CPUs can only be set for inactive vms" if is_running?
+
+    cpus_on_host = File.read('/proc/cpuinfo').scan(/^processor\s*:/).size.to_i
+    num_of_cpus <= cpus_on_host ? use_cpus = num_of_cpus : use_cpus = cpus_on_host
+
+    domain_xml = REXML::Document.new(@domain.xml_desc)
+    domain_xml.elements['domain/vcpu'].text = use_cpus
+    domain_xml.elements['domain/vcpu'].attributes['placement'] = 'static'
+    update(domain_xml.to_s)
+  end
+
   def set_ram_size(size, unit = "KiB")
     raise "System memory can only be added to inactice vms" if is_running?
     domain_xml = REXML::Document.new(@domain.xml_desc)

#5 Updated by intrigeri 2016-01-19 13:15:29

test/10971-more-CPUs-for-TailsToaster gives 2 vcpus to TailsToaster. This will allow us to measure if it makes any difference on Jenkins.

#8 Updated by intrigeri 2016-02-26 18:27:41

The last 10 successful test suite runs (between Feb 14 and 26) on the devel branch took 56.6 minutes on average. The last 10 successful test suite runs (between Feb 16 and 26) on that branch took 48.2 minutes on average. That’s almost 15% less. Of course, this doesn’t tell us that our infrastructure would be able to run more tests, or run tests faster, if we gave all TailsToaster’s a 2nd CPU, but still: this encourages me to take this idea into account while working on Feature #11011 and Feature #11113.

#11 Updated by intrigeri 2016-02-26 18:33:36

I’ve merged this branch into experimental, and created https://jenkins.tails.boum.org/job/manual_test_Tails_ISO_experimental/ for easier benchmarking.

#16 Updated by intrigeri 2016-05-27 14:02:25

anonym, given one step of the test suite became fragile with Tor Browser 6.x unless you give 2 vcpus to TailsToaster, maybe you want to take this one (or the cheap version that only bumps the number of vcpus to 2) for 2.4?

#20 Updated by intrigeri 2016-06-04 13:02:53

> Also, I think it’s clear to us that we shouldn’t allow configuring the number of cores, since it clearly has an effect on tests. Instead, we should fix on some N number of cores, and N = 2 seems reasonable.

ACK, please retitle accordingly :)

#22 Updated by intrigeri 2016-07-30 15:37:54

https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_test-10971-more-cpus-for-tailstoaster-11588-10733/ (starting with build 9) tests with 2 vcpus and the virtual hardware changes brought for Bug #11588. It might be more relevant than https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_test-10971-more-cpus-for-tailstoaster/.

#23 Updated by intrigeri 2016-08-23 06:32:59

The test/10971-more-CPUs-for-TailsToaster+11588+10733 branch seems to fix the “timeout after clicking Login in the Greeter” issue that I see on my branches that re-enable USB tests:

• feature/from-intrigeri-for-2.6: 3 timeout after clicking Login / 22 runs
• test/11588-usb-on-jenkins+10733: 3 timeout after clicking Login / 31 runs
• test/10971-more-CPUs-for-TailsToaster+11588+10733: never seen that bug!
• stable, devel: seen it at least once recently (https://jenkins.tails.boum.org/view/RM/job/test_Tails_ISO_stable/391/)

So I’m now starting to thing that instead of trying to fix that robustness issue in the “1 CPU for TailsToaster” case, we should instead work on stabilizing the “2 CPU for TailsToaster” branch, that additionally saves 10% of the test suite runtime (with USB tests included) on Jenkins, and fixes weird stuff like Bug #10442.