Bug #9645

Create at least a second VM for testing ISO images

Added by bertagaz 2015-06-25 03:02:17 . Updated 2015-09-03 06:01:04 .

Status:
Resolved
Priority:
High
Assignee:
Category:
Continuous Integration
Target version:
Start date:
2015-06-25
Due date:
2015-07-15
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
0
Affected tool:
Deliverable for:
267

Description

To test our automated tests design, we’ll need something close to what our setup will look alike. For that, having several isotesters running will help.


Subtasks


Related issues

Blocks Tails - Feature #9486: Support running multiple instances of the test suite in parallel Resolved 2015-06-25

History

#1 Updated by bertagaz 2015-06-25 03:03:06

#2 Updated by bertagaz 2015-06-25 03:03:33

  • blocks #8668 added

#3 Updated by bertagaz 2015-06-25 03:04:04

  • blocked by Feature #9399: Extend lizard's storage capacity added

#4 Updated by intrigeri 2015-06-28 04:06:23

  • Subject changed from Create at least a second isotester to Create at least a second VM for testing ISO images

#5 Updated by bertagaz 2015-06-28 09:06:12

  • Target version changed from Tails_1.4.1 to Tails_1.5

#6 Updated by intrigeri 2015-07-12 02:56:40

  • blocks deleted (Feature #9399: Extend lizard's storage capacity)

#7 Updated by bertagaz 2015-07-17 09:34:35

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 50

Installed and started to configure the 3 new isotesters. Using virt-clone, it wasn’t much more work to do 1 or 3.

They are waiting for their first successful puppet agent run, it seems we’ll need to upgrade our Jenkins master and slaves to a newer version before, and it’s only in Sid at the moment.

#8 Updated by intrigeri 2015-07-17 10:01:40

> Installed and started to configure the 3 new isotesters.

Yay :)

> Using virt-clone, it wasn’t much more work to do 1 or 3.

Does virt-clone make e.g. machine-id, random seeds, and sshd / Puppet TLS key pairs unique?

#9 Updated by bertagaz 2015-07-17 11:02:53

intrigeri wrote:
> Does virt-clone make e.g. machine-id, random seeds, and sshd / Puppet TLS key pairs unique?

It doesn’t for the sshd / Puppet TLS part, but at least save time on the installation process of the base system. I changed this last bits myself. I didn’t think about checking the random seed and machine-id though. Will do.

#10 Updated by intrigeri 2015-07-18 00:56:37

> It doesn’t for the sshd / Puppet TLS part, but at least save time on the installation process of the base system. I changed this last bits myself. I didn’t think about checking the random seed and machine-id though. Will do.

OK, cool. Needless to say, please make sure that this alternate way of setting up a VM is documented :)
My only remaining concern is that it removes one of the rare opportunities to test deploying our manifests from scratch, but oh well, we should find better ways to do so anyway.

#11 Updated by bertagaz 2015-07-18 01:47:15

intrigeri wrote:
> > It doesn’t for the sshd / Puppet TLS part, but at least save time on the installation process of the base system. I changed this last bits myself. I didn’t think about checking the random seed and machine-id though. Will do.
>
> OK, cool. Needless to say, please make sure that this alternate way of setting up a VM is documented :)
> My only remaining concern is that it removes one of the rare opportunities to test deploying our manifests from scratch, but oh well, we should find better ways to do so anyway.

So that’s good, because I cloned the freshly installed isotester2, not isotester1, so they will, be tested from scratch. :)

#12 Updated by intrigeri 2015-08-02 08:01:15

  • blocks Feature #9486: Support running multiple instances of the test suite in parallel added

#13 Updated by intrigeri 2015-08-02 08:01:29

  • Due date set to 2015-07-15

#14 Updated by intrigeri 2015-08-02 12:34:58

  • Priority changed from Normal to High

#16 Updated by bertagaz 2015-08-05 05:35:31

  • Target version changed from Tails_1.5 to Tails_1.6

Postponing, even if it will be finished very soon.

#17 Updated by bertagaz 2015-08-20 01:39:17

  • blocked by Bug #10066: Python-otr removed from Debian testing added

#18 Updated by bertagaz 2015-08-20 01:55:48

  • blocks deleted (Bug #10066: Python-otr removed from Debian testing)

#19 Updated by bertagaz 2015-08-21 05:16:23

  • Assignee changed from bertagaz to intrigeri
  • % Done changed from 50 to 80
  • QA Check set to Ready for QA

Ok, I think I finished this, the 4 isotesters are up and configured and I’m currently running the test suite on all of them.

I did change their respective machine_id, installed haveged and done some scratch on the disk to feed their randomness, restarted them for their random seed to be different and then created new ssh and puppet keys (Tor wasn’t installed yet). Had to adapt a bit our manifests, and in the end, the install went fine.

So if it feels good enough to you, please close this ticket.

#20 Updated by bertagaz 2015-08-22 06:39:26

Forgot to add the relevant yaml config files, so I’ve created and configured some xmpp user accounts and created the needed files (local.yml, pidgin.yml, tor.yml, ssh.yml and sftp.yml) on each isotesters.

#21 Updated by bertagaz 2015-08-24 00:51:46

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Ready for QA to Dev Needed

Seems that the isotesters need a bit bigger rootfs. Will grow them.

#22 Updated by bertagaz 2015-08-24 06:23:52

  • Assignee changed from bertagaz to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

bertagaz wrote:
> Seems that the isotesters need a bit bigger rootfs. Will grow them.

Done.

#23 Updated by intrigeri 2015-08-25 04:13:35

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Ready for QA to Dev Needed

I’ve looked at the corresponding Puppet changes. Sounds good! Two questions:

Regarding the firewall changes: any reason not to restrict this to a specific set of ports?

Regarding the *.yml local config files: I see nothing about them in our Puppet stuff. Were they deployed by hand? IMO that stuff belongs to our tails_secrets_jenkins module, or similar.

#24 Updated by intrigeri 2015-08-26 06:02:47

  • Deliverable for set to 267

#25 Updated by bertagaz 2015-08-26 07:45:57

intrigeri wrote:
> I’ve looked at the corresponding Puppet changes. Sounds good! Two questions:
>
> Regarding the firewall changes: any reason not to restrict this to a specific set of ports?

That’s because one connected to the master 8080 port, the slave connects to it with a randomly chosen port. Excerpt from a slave log:

INFO: Locating server among [https://jenkins.tails.boum.org/, http://jenkins.lizard:8080/]
[...]
INFO: Connecting to jenkins.lizard:33967

> Regarding the *.yml local config files: I see nothing about them in our Puppet stuff. Were they deployed by hand? IMO that stuff belongs to our tails_secrets_jenkins module, or similar.

Right, good idea. I haven’t done it because the git wasn’t deployed by our puppet module. Will do that and put the files in /etc/TailsToaster/

#26 Updated by bertagaz 2015-08-27 09:24:55

  • Assignee changed from bertagaz to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

bertagaz wrote:
> intrigeri wrote:
> > Regarding the *.yml local config files: I see nothing about them in our Puppet stuff. Were they deployed by hand? IMO that stuff belongs to our tails_secrets_jenkins module, or similar.
>
> Right, good idea. I haven’t done it because the git wasn’t deployed by our puppet module. Will do that and put the files in /etc/TailsToaster/

Done, please review.

#27 Updated by intrigeri 2015-08-30 10:10:12

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Ready for QA to Dev Needed

> Done, please review.

Yay!

  • MAX_NEW_TOR_CIRCUIT_RETRIES: 50 was meant as a temporary fix. I’d rather not encode it in our config forever => please drop it.
  • Nitpicking: I find the TailsToaster_config and /etc/TailsToaster names misleading: at the moment these configuration files are not about the TailsToaster VM (the system under test), but instead they configure how the test suite (which controls that VM) works. This might change a bit in the future (e.g. when the test suite’s config supports specifying how many cores shall be given to the system under test), but not essentially. Granted, the default name of the temporary directory is misleading too, so well, let’s say that’s not a blocker.
  • Firewall rules: OK, makes sense; I pushed 46cdf04 on top.

Almost there :)

#28 Updated by bertagaz 2015-08-31 02:36:28

  • Assignee changed from bertagaz to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:
>
> * MAX_NEW_TOR_CIRCUIT_RETRIES: 50 was meant as a temporary fix. I’d rather not encode it in our config forever => please drop it.

Done and deployed. Didn’t know it was temporary.

> * Nitpicking: I find the TailsToaster_config and /etc/TailsToaster names misleading: at the moment these configuration files are not about the TailsToaster VM (the system under test), but instead they configure how the test suite (which controls that VM) works. This might change a bit in the future (e.g. when the test suite’s config supports specifying how many cores shall be given to the system under test), but not essentially. Granted, the default name of the temporary directory is misleading too, so well, let’s say that’s not a blocker.

Hmmm yes, get your point. In my mind, the TailsToaster name is not only the VM name, but also the codename of the software I thought about when POC’ing it, hence this situation. :)

> * Firewall rules: OK, makes sense; I pushed 46cdf04 on top.

Good catch, thanks!

#29 Updated by intrigeri 2015-09-01 00:53:49

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Ready for QA to Info Needed

>> * MAX_NEW_TOR_CIRCUIT_RETRIES: 50 was meant as a temporary fix. I’d rather not encode it in our config forever => please drop it.

> Done and deployed. Didn’t know it was temporary.

Double-checked, looks good!

> In my mind, the TailsToaster name is not only the VM name, but also the codename of the software I thought about when POC’ing it, hence this situation. :)

OK, thanks.

One last question: why do all isotesterN have 10GB allocated to /srv? (except cargo-culting what isotester1 had, which is probably no good reason in itself since that one had very specific requirements)

#30 Updated by bertagaz 2015-09-01 01:07:38

  • Assignee changed from bertagaz to intrigeri

intrigeri wrote:
>
> One last question: why do all isotesterN have 10GB allocated to /srv? (except cargo-culting what isotester1 had, which is probably no good reason in itself since that one had very specific requirements)

Hmmm, because I just copied isotester1’s conf? Shall I shorten that to something like 4GB or 5GB?

#31 Updated by intrigeri 2015-09-01 02:10:53

  • Assignee changed from intrigeri to bertagaz

> Hmmm, because I just copied isotester1’s conf? Shall I shorten that to something like 4GB or 5GB?

What do you plan to use that filesystem for? Jenkins workspace? If yes, then 1. it should probably be mounted somewhere more appropriate; 2. it needs to be large enough to contain a Git checkout, two ISO images (--iso and --old-iso), and that’s all, no?

#32 Updated by bertagaz 2015-09-01 03:22:43

  • Assignee changed from bertagaz to intrigeri

intrigeri wrote:
> > Hmmm, because I just copied isotester1’s conf? Shall I shorten that to something like 4GB or 5GB?
>
> What do you plan to use that filesystem for? Jenkins workspace?

Yes.

>If yes, then
>1. it should probably be mounted somewhere more appropriate;

We could mount them in /var/lib/jenkins/, but we can configure the slaves to use a subdirectory of /srv/ as the workspace.

I guess you prefer the former, as it won’t require a manual configuration of the slaves.

>2. it needs to be large enough to contain a Git checkout, two ISO images (--iso and --old-iso), and that’s all, no?

Yes, hence my proposal to reduce them to something like 4GB or 5GB. That would leave a little room just in case.

#33 Updated by intrigeri 2015-09-01 03:32:47

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Info Needed to Dev Needed

> We could mount them in /var/lib/jenkins/, but we can configure the slaves to use a subdirectory of /srv/ as the workspace.

> I guess you prefer the former, as it won’t require a manual configuration of the slaves.

Indeed, unless we have good reasons to use a non-default directory, let’s not :)

>>2. it needs to be large enough to contain a Git checkout, two ISO images (--iso and --old-iso), and that’s all, no?

> Yes, hence my proposal to reduce them to something like 4GB or 5GB. That would leave a little room just in case.

OK, let’s go for 4GB then!

#34 Updated by bertagaz 2015-09-03 06:01:04

  • Status changed from In Progress to Resolved
  • Assignee deleted (bertagaz)
  • % Done changed from 80 to 100
  • QA Check deleted (Dev Needed)

Closing this ticket, after having moved the /srv/ filesystem in /var/lib/jenkins for all isotesters but isotester1. Left the later untouched so that the test suite team can keep on playing with it. That was the last bit to do here.

I’ve created Feature #10158 to track this isotester1 specific case and apply the same change later.