Tails

#2 Updated by groente 2019-05-15 11:51:02

• Assignee changed from groente to hefee
• QA Check set to Info Needed

hefee wrote:
> The scripts for handling git->weblate also needs to have tests. As the scripts are working with a git repository as input, we need a test repository to construct testcases. I created https://salsa.debian.org/hefee/weblate-scripts-test.git for that purpose and added this as submodule to puppet-tails to run the tests successfully.
> IMO adding a submodule there is no issue, as this submodule is only needed for running the tests.

tbh, i think this is adding a bit much to what is still primarily a puppet repo. is it perphaps time to create a completely separate repo for the python development for weblate where you can run your tests and only deploy releases in puppet-tails?

#4 Updated by intrigeri 2019-06-02 15:09:51

• QA Check deleted (Info Needed)

(Preparing to drop the “QA Check” field as per “[Tails-dev] Proposal: Redmine workflow change”.)

#6 Updated by hefee 2019-06-27 17:14:46

• Subject changed from Add submodule in puppet-tails for weblate integration tests to Add tests in puppet-tails for Weblate integration scripts
• Description updated

The current plan is to have a python script that create the needed git repository every time you run the tests.

#7 Updated by hefee 2019-07-01 10:59:20

• File Vagrantfile added
• Assignee changed from hefee to intrigeri
• Feature Branch set to https://salsa.debian.org/hefee/puppet-tails/tree/hefee/master

I have now finally updated the tests to create an own repository for tests for the Weblate integration scripts.
see 670ab55d

I created a local VM to test (see attached Vagrant file):
rsync -a —delete . vagrant@192.168.XXX.XXX:puppet-tails/files/weblate/scripts/ && ssh vagrant@192.168.XXX.XXX “. venv/bin/activate && cd ~/puppet-tails/files/weblate/scripts && python3 -m nose”

#8 Updated by hefee 2019-07-01 10:59:45

• Status changed from Confirmed to Needs Validation

#9 Updated by intrigeri 2019-07-02 08:41:37

• Status changed from Needs Validation to In Progress
• Assignee changed from intrigeri to hefee

Hi!

> I have now finally updated the tests to create an own repository for tests for the Weblate integration scripts.
> see 670ab55d

Thanks, merged!

FTR I did not review the tests in detail yet: I just took a quick look to make sure we were not doing too scary things there (and we don’t :) I’ll review this some more as part of Feature #16326.

The only thing I’d like before we close this ticket is: see the test suite pass once; this depends on the Vagrant thing since I can’t install the test suite dependencies on my host system (security policy). So:

> I created a local VM to test (see attached Vagrant file):

Wrt. that Vagrantfile (feel free to ignore):

• I think we need to add it to the source tree, otherwise everyone who’s supposed to run these tests, and is subject to the same security policy as me, will have to maintain their own Vagrantfile; I’d rather share one :)
• Using Stretch would allow us to test something closer to the production setup (and in particular, po4a from Buster behaves quite differently in some cases).
• Any reason not to bump the weblate-2.19.1 version to what we currently run in production?
• I’ve tried using it. To get the data share working, I had to copy the Vagrantfile to the tests directory and run vagrant from there. Is this how it should be? This feels somewhat inconsistent with the rsync -a --delete . part of your instructions, no?
• I see rsync: mkdir "/home/vagrant/puppet-tails/files/weblate/scripts" failed: No such file or directory (2) when I try to run the rsync command you provided. Indeed, there’s no /home/vagrant/puppet-tails directory in the Vagrant system. Is a setup step missing somewhere?

#10 Updated by hefee 2019-07-03 08:24:21

• Status changed from In Progress to Needs Validation
• Assignee changed from hefee to intrigeri
• Feature Branch changed from https://salsa.debian.org/hefee/puppet-tails/tree/hefee/master to https://salsa.debian.org/hefee/puppet-tails/tree/hefee/16716-weblate-tests

intrigeri wrote:
> * I think we need to add it to the source tree, otherwise everyone who’s supposed to run these tests, and is subject to the same security policy as me, will have to maintain their own Vagrantfile; I’d rather share one :)

good point, lets ship one.

> * Using Stretch would allow us to test something closer to the production setup (and in particular, po4a from Buster behaves quite differently in some cases).

done

> * Any reason not to bump the weblate-2.19.1 version to what we currently run in production?

nope - Just I havn’t bumped the version in Vagrantfile. Maybe you have a good tip, how I can switch between users in the Vagrantfile, as more should be run as vagrant user and not as root. But I didn’t found good solutions.

> * I’ve tried using it. To get the data share working, I had to copy the Vagrantfile to the tests directory and run vagrant from there. Is this how it should be? This feels somewhat inconsistent with the rsync -a --delete . part of your instructions, no?

oh yeah - I made to much modifications to my VM. I updated the Vagrant file, that is working without any modifications.

> * I see rsync: mkdir "/home/vagrant/puppet-tails/files/weblate/scripts" failed: No such file or directory (2) when I try to run the rsync command you provided. Indeed, there’s no /home/vagrant/puppet-tails directory in the Vagrant system. Is a setup step missing somewhere?

this rsync is only needed if you want sync your local working directory to the VM. That is not necessary, if you “just” want to run the tests.

#11 Updated by intrigeri 2019-07-03 10:24:54

• Status changed from Needs Validation to In Progress
• Assignee changed from intrigeri to hefee

Hi!

> Maybe you have a good tip, how I can switch between users in the Vagrantfile, as more should be run as vagrant user and not as root. But I didn’t found good solutions.

I think the usual way one solves this in Vagrant-land is to move the non-root provisioning steps to a script and run it via sudo. Apart of the calls to apt, it looks like everything could be done as the vagrant user, right? I’ll let you decide if you want to polish as part of this ticket or not. One advantage of moving provisioning to an external script is that it makes the whole thing one step closer to be usable e.g. on GitLab CI, but that’s no game changer at this point.

> I updated the Vagrant file, that is working without any modifications.

Much better indeed, thanks! I’ve pushed some improvements to the doc. Then I’ve followed the instructions and when I run the tests I see this:

EE.
======================================================================
ERROR: Failure: OSError (DATA_DIR /usr/local/share/weblate/data/home is not writable!)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/usr/lib/python3/dist-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/usr/lib/python3/dist-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/usr/lib/python3/dist-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/usr/lib/python3.5/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/usr/lib/python3.5/imp.py", line 172, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 693, in _load
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 673, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/home/vagrant/puppet-tails/files/weblate/scripts/tests/test_update_weblate_components.py", line 13, in <module>
import tailsWeblate
File "/home/vagrant/puppet-tails/files/weblate/scripts/tailsWeblate.py", line 10, in <module>
application = get_wsgi_application()
File "/home/vagrant/venv/lib/python3.5/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application
django.setup(set_prefix=False)
File "/home/vagrant/venv/lib/python3.5/site-packages/django/__init__.py", line 22, in setup
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
File "/home/vagrant/venv/lib/python3.5/site-packages/django/conf/__init__.py", line 56, in __getattr__
self._setup(name)
File "/home/vagrant/venv/lib/python3.5/site-packages/django/conf/__init__.py", line 41, in _setup
self._wrapped = Settings(settings_module)
File "/home/vagrant/venv/lib/python3.5/site-packages/django/conf/__init__.py", line 110, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/usr/local/share/weblate/weblate/__init__.py", line 96, in <module>
check_data_writable()
File "/usr/local/share/weblate/weblate/trans/data.py", line 40, in check_data_writable
create_and_check_dir(data_dir('home'))
File "/usr/local/share/weblate/weblate/trans/data.py", line 33, in create_and_check_dir
'DATA_DIR {0} is not writable!'.format(path)
OSError: DATA_DIR /usr/local/share/weblate/data/home is not writable!
-------------------- >> begin captured logging << --------------------
PIL.Image: DEBUG: Importing BmpImagePlugin
PIL.Image: DEBUG: Importing BufrStubImagePlugin
PIL.Image: DEBUG: Importing CurImagePlugin
PIL.Image: DEBUG: Importing DcxImagePlugin
PIL.Image: DEBUG: Importing DdsImagePlugin
PIL.Image: DEBUG: Importing EpsImagePlugin
PIL.Image: DEBUG: Importing FitsStubImagePlugin
PIL.Image: DEBUG: Importing FliImagePlugin
PIL.Image: DEBUG: Importing FpxImagePlugin
PIL.Image: DEBUG: Importing FtexImagePlugin
PIL.Image: DEBUG: Importing GbrImagePlugin
PIL.Image: DEBUG: Importing GifImagePlugin
PIL.Image: DEBUG: Importing GribStubImagePlugin
PIL.Image: DEBUG: Importing Hdf5StubImagePlugin
PIL.Image: DEBUG: Importing IcnsImagePlugin
PIL.Image: DEBUG: Importing IcoImagePlugin
PIL.Image: DEBUG: Importing ImImagePlugin
PIL.Image: DEBUG: Importing ImtImagePlugin
PIL.Image: DEBUG: Importing IptcImagePlugin
PIL.Image: DEBUG: Importing JpegImagePlugin
PIL.Image: DEBUG: Importing Jpeg2KImagePlugin
PIL.Image: DEBUG: Importing McIdasImagePlugin
PIL.Image: DEBUG: Importing MicImagePlugin
PIL.Image: DEBUG: Importing MpegImagePlugin
PIL.Image: DEBUG: Importing MpoImagePlugin
PIL.Image: DEBUG: Importing MspImagePlugin
PIL.Image: DEBUG: Importing PalmImagePlugin
PIL.Image: DEBUG: Importing PcdImagePlugin
PIL.Image: DEBUG: Importing PcxImagePlugin
PIL.Image: DEBUG: Importing PdfImagePlugin
PIL.Image: DEBUG: Importing PixarImagePlugin
PIL.Image: DEBUG: Importing PngImagePlugin
PIL.Image: DEBUG: Importing PpmImagePlugin
PIL.Image: DEBUG: Importing PsdImagePlugin
PIL.Image: DEBUG: Importing SgiImagePlugin
PIL.Image: DEBUG: Importing SpiderImagePlugin
PIL.Image: DEBUG: Importing SunImagePlugin
PIL.Image: DEBUG: Importing TgaImagePlugin
PIL.Image: DEBUG: Importing TiffImagePlugin
PIL.Image: DEBUG: Importing WebPImagePlugin
PIL.Image: DEBUG: Importing WmfImagePlugin
PIL.Image: DEBUG: Importing XbmImagePlugin
PIL.Image: DEBUG: Importing XpmImagePlugin
PIL.Image: DEBUG: Importing XVThumbImagePlugin
weblate-vcs: DEBUG: weblate: git: git --version [retcode=0]
weblate-vcs: DEBUG: weblate: git: git review --version [retcode=1]
weblate-vcs: DEBUG: weblate: git: git svn --version [retcode=1]
weblate-vcs: DEBUG: weblate: git: git describe --always [retcode=0]
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: test suite for <class 'tests.test_update_weblate_git.TestUWG'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/nose/suite.py", line 210, in run
self.setUp()
File "/usr/lib/python3/dist-packages/nose/suite.py", line 293, in setUp
self.setupContext(ancestor)
File "/usr/lib/python3/dist-packages/nose/suite.py", line 316, in setupContext
try_run(context, names)
File "/usr/lib/python3/dist-packages/nose/util.py", line 471, in try_run
return func()
File "/home/vagrant/puppet-tails/files/weblate/scripts/tests/test_update_weblate_git.py", line 36, in setUpClass
createRepo(str(cls.repopath))
File "/home/vagrant/puppet-tails/files/weblate/scripts/tests/utils.py", line 34, in createRepo
shutil.copy(datapath/"a.fa.po.localchanges", os.path.join(basepath, "a.fa.po"))
File "/usr/lib/python3.5/shutil.py", line 241, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/lib/python3.5/shutil.py", line 103, in copyfile
if _samefile(src, dst):
File "/usr/lib/python3.5/shutil.py", line 88, in _samefile
return os.path.samefile(src, dst)
File "/usr/lib/python3.5/genericpath.py", line 90, in samefile
s1 = os.stat(f1)
TypeError: argument should be string, bytes or integer, not PosixPath
-------------------- >> begin captured logging << --------------------
git.cmd: DEBUG: Popen(['git', 'init'], cwd=/tmp/tmp8vhg33eq, universal_newlines=False, shell=None)
git.cmd: DEBUG: Popen(['git', 'cat-file', '--batch-check'], cwd=/tmp/tmp8vhg33eq, universal_newlines=False, shell=None)
git.cmd: DEBUG: Popen(['git', 'cat-file', '--batch'], cwd=/tmp/tmp8vhg33eq, universal_newlines=False, shell=None)
git.cmd: DEBUG: Popen(['git', 'tag', 'base-1', 'HEAD'], cwd=/tmp/tmp8vhg33eq, universal_newlines=False, shell=None)
git.cmd: DEBUG: Popen(['git', 'checkout', '-f', 'master'], cwd=/tmp/tmp8vhg33eq, universal_newlines=False, shell=None)
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 2 tests in 0.366s

FAILED (errors=2)

I think that’s the only blocker at this point. In any case, I’ve pushed the current state of things.

Finally, one nitpick: cloning the weblate Git repo takes a while here. Perhaps downloading the release tarball (without the full Git history) would work just as wellin this context?

#13 Updated by hefee 2019-07-08 22:09:27

• Status changed from In Progress to Needs Validation
• Assignee changed from hefee to intrigeri
• Feature Branch changed from https://salsa.debian.org/hefee/puppet-tails/tree/hefee/16716-weblate-tests to https://salsa.debian.org/hefee/puppet-tails/tree/hefee/16844-different-working-directory

Tests passes now on VM for me. (it is another branch, as I needed to fix them for the new script anyways.)

intrigeri wrote:
> Finally, one nitpick: cloning the weblate Git repo takes a while here. Perhaps downloading the release tarball (without the full Git history) would work just as wellin this context?

It is out-of-scope for me ATM and Weblate does not have signatures for there tarballs. That was the initial reason to use git directly. (https://github.com/WeblateOrg/weblate/issues/2868)

#14 Updated by intrigeri 2019-07-09 14:11:49

• Status changed from Needs Validation to Resolved

> Tests passes now on VM for me. (it is another branch, as I needed to fix them for the new script anyways.)

Cherry-picked and pushed!

> intrigeri wrote:
>> Finally, one nitpick: cloning the weblate Git repo takes a while here. Perhaps downloading the release tarball (without the full Git history) would work just as wellin this context?

> It is out-of-scope for me ATM and Weblate does not have signatures for there tarballs. That was the initial reason to use git directly. (https://github.com/WeblateOrg/weblate/issues/2868)

Fair enough!