Tails

#1 Updated by intrigeri 2016-04-15 08:08:35

• Assignee changed from bertagaz to anonym
• QA Check set to Info Needed

> a Git checkout at commit commit:5a63784d408e50924f3c38903fc28ae39bf7e0aa

IMO hard-coding the exact commit we want won’t scale, and we instead need to track a branch. So, what branch?

#2 Updated by anonym 2016-04-20 11:39:39

• Assignee changed from anonym to bertagaz

intrigeri wrote:
> > a Git checkout at commit commit:5a63784d408e50924f3c38903fc28ae39bf7e0aa
>
> IMO hard-coding the exact commit we want won’t scale, and we instead need to track a branch. So, what branch?

Ok, IIRC that was what someone suggested. The only branch upstream has is master.

Also, please see commit:ee566389e24a160b79b55590befd4456328c466d that adds instructions for setting this up. Is it getting too complex to have this setup working?

#3 Updated by anonym 2016-04-22 04:48:18

• Priority changed from Normal to Elevated

I’d like to get tests running as soon as possible so we can start gathering stats about how much of an improvement this is.

#5 Updated by bertagaz 2016-04-23 07:34:58

• Assignee changed from bertagaz to anonym

Ok, so I added a chutney checkout on all isotesters, cloning master branch of https://git.torproject.org/chutney.git. I also configured the test suite as explained.

First run failed with a ‘KeyError: ’include:bridge-obfs4.tmpl’, see https://jenkins.tails.boum.org/job/test_Tails_ISO_test-9521-chutney/10/console

What am I missing?

#6 Updated by bertagaz 2016-04-23 07:35:42

• Assignee changed from anonym to bertagaz

anonym wrote:
> I just realized that test/9521-chutney introduces Tor as a dependency for running the automated test suite, so that should be taken into account on the puppet side.

Tor is already installed everywhere. :)

#7 Updated by bertagaz 2016-04-23 07:58:50

• Assignee changed from bertagaz to anonym

bertagaz wrote:
> What am I missing?

Got it, missing the bridge-obfs4.tmpl file/link in the torrc_template/ directory of the chutney repo checkout. Silly me, RTFM!

That’s a bit troubling because there is no permanent checkout of the Tails Git repo on the isotesters. It’s cloned fresh for each test suite run in the Jenkins workspace, and then deleted.

I don’t think we should provide this file on the isotesters using puppet, or it would have to be upgraded each time it changes in Tails repo.

We could modify the test suite wrapper script on the isotesters so that it copies this file before each run, and delete it afterward to leave the chutney repo clone clean.

But I think this should probably be resolved at the test suite code level. The chutney instance it starts should be able to take any configuration we ship in our Tails repo. I can deploy the wrapper solution while it’s not implemented in the test suite code though.

#8 Updated by anonym 2016-04-23 08:54:44

• Assignee changed from anonym to bertagaz
• QA Check changed from Info Needed to Dev Needed

bertagaz wrote:
> We could modify the test suite wrapper script on the isotesters so that it copies this file before each run, and delete it afterward to leave the chutney repo clone clean.

Could we do that for now, as an ugly/hacky short-term solution given the long-term solution below? Something like:

if [ -d "${TAILS_GIT}"/features/chutney ]; then
    cp "${TAILS_GIT}"/features/chutney/*.tmpl /path/to/chutney-src/torrc_templates/
fi

Just so we can start gathering stats ASAP.

> But I think this should probably be resolved at the test suite code level. The chutney instance it starts should be able to take any configuration we ship in our Tails repo. I can deploy the wrapper solution while it’s not implemented in the test suite code though.

The long-term solution is Bug #11364 which indeed will allow us to solve this nicely inside the test suite’s code. I plan to get to it early May.

#9 Updated by bertagaz 2016-04-24 02:41:36

• Assignee changed from bertagaz to anonym
• QA Check changed from Dev Needed to Info Needed

anonym wrote:
> bertagaz wrote:
> > We could modify the test suite wrapper script on the isotesters so that it copies this file before each run, and delete it afterward to leave the chutney repo clone clean.
>
> Could we do that for now, as an ugly/hacky short-term solution given the long-term solution below? Something like:

I deployed something similar but also taking care of removing the files, otherwise the chutney clone repo would have dangling files preventing it to be updated.

> Just so we can start gathering stats ASAP.

Sadly it still fail, see this run in Jenkins. I wonder what’s happening. We’re running the test suite using sudo, maybe some necessary environment variables are removed by it? Did you see hat kind of errors already?

> The long-term solution is Bug #11364 which indeed will allow us to solve this nicely inside the test suite’s code. I plan to get to it early May.

\o/

#15 Updated by anonym 2016-04-24 13:46:48

• File do_not_use_system_torrc.patch added
• Assignee changed from anonym to bertagaz
• QA Check changed from Info Needed to Dev Needed

bertagaz wrote:
> Jenkins run n°17 seems to confirm that with no /etc/tor/torrc, chutney is starting fine. There are a lot of Tor processes running in this session.

I can confirm that Chutney will use /etc/tor/torrc at one point.

> I’m not sure now where this torrc sourcing comes from. From Tor itself, or from chutney?

I managed to track it down, and I’ll file an upstream bug. Until then, please try the attached patch to the Chutney’s you just deployed.

#17 Updated by bertagaz 2016-04-25 02:42:35

• Assignee changed from bertagaz to anonym
• QA Check changed from Dev Needed to Info Needed

Ok, I’ve added a chutney repo at git.tails.boum.org, with a feature/tails_test_suite branch containing your three patches. It’s deployed on the isotesters, and I’ve started a build, which should lead in a new test suite run if everything goes well. Finger crossed!

#18 Updated by bertagaz 2016-04-25 03:48:56

• % Done changed from 0 to 30
• QA Check deleted (Info Needed)
• Type of work changed from Sysadmin to Wait

So that seems to work quite well. Chutney started correctly, is using TMP_DIR/chutney-data and the Tor processes are listening on the correct IP.

I’m putting this ticket on hold with a Wait type of work. Once your patches are merged upstream and Bug #11364 is resolved, please re-assign it to me so that I’ll complete this ticket by removing all our workarounds.

#19 Updated by bertagaz 2016-04-25 03:49:42

• blocked by Bug #11364: Add option to Chutney for override torrc_templates directory. added

#22 Updated by intrigeri 2016-04-28 13:33:49

• Status changed from Confirmed to In Progress
• Assignee changed from anonym to bertagaz
• Type of work changed from Wait to Sysadmin

I see that vcsrepo { $chutney_srcdir is done as root: let’s please not do that. I think we consistently avoid it everywhere else, no?

#23 Updated by bertagaz 2016-05-06 04:24:10

• Assignee changed from bertagaz to anonym
• Type of work changed from Sysadmin to Wait

intrigeri wrote:
> I see that vcsrepo { $chutney_srcdir is done as root: let’s please not do that. I think we consistently avoid it everywhere else, no?

Right, done.

#25 Updated by anonym 2016-05-10 09:03:03

• Parent task deleted (Feature #9521)

The plan is to merge test/9521-chutney and close Feature #9521 before this ticket is fixed, so unparenting.

#26 Updated by anonym 2016-05-11 06:23:44

• related to Feature #9521: Use the chutney Tor network simulator in our test suite added

#27 Updated by intrigeri 2016-05-12 05:39:22

• blocks deleted (Bug #11364: Add option to Chutney for override torrc_templates directory.)

#28 Updated by intrigeri 2016-05-12 05:40:13

• Status changed from In Progress to Rejected
• Assignee deleted (anonym)
• % Done changed from 30 to 100

We use a Git submodule for chutney nowadays, so I’ve dropped the copy we had added to our isotesters.