Tails

#1 Updated by hefee 2019-08-23 17:33:49

happend 23.Aug 2019 11:53:31 until 12:52:30 (every 5min) UTC
than @intrigeri has restarted mariadb and it was quiet but on 15:43:03 UTC we see another failure.

Mostly the stacktrace points to /usr/local/share/weblate/manage.py, somtimes to our own scripts /var/lib/weblate/scripts/update_weblate_components.py

Does it make sense to consider to switch to PostgreSQL (Bug #16994)

#3 Updated by intrigeri 2019-08-24 07:05:41

> Mostly the stacktrace points to /usr/local/share/weblate/manage.py, somtimes to our own scripts /var/lib/weblate/scripts/update_weblate_components.py

I’ve added some logging to cron.sh so it’s easier to tell, by looking at the email date vs. timestamps in the log file, what exact operation we were running when it failed.

#4 Updated by intrigeri 2019-08-25 06:04:48

Today the problem was caused by commit_pending getting stuck on a specific resource. I had to follow the corresponding doc in translate-server.git to repair it, to the point that I had to delete all translation history for that resource (translation_id). Apparently this did not lose the current translation status of that file, but the latest changes were not committed.

So I’ve clicked “commit” on https://translate.tails.boum.org/projects/tails/install-mac-usb/es/ and it failed. Stacktrace:

2019-08-25 05:41:26,453 ERROR Internal Server Error: /commit/tails/install-mac-usb/es/
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/decorators.py", line 23, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/django/views/decorators/http.py", line 40, in inner
return func(request, *args, **kwargs)
File "/usr/local/share/weblate/weblate/trans/views/git.py", line 136, in commit_translation
return perform_commit(request, obj)
File "/usr/local/share/weblate/weblate/trans/views/git.py", line 68, in perform_commit
request,
File "/usr/local/share/weblate/weblate/trans/views/git.py", line 45, in execute_locked
result = call(*args, **kwargs)
File "/usr/local/share/weblate/weblate/trans/models/translation.py", line 434, in commit_pending
)[0]
File "/usr/local/lib/python3.5/dist-packages/django/db/models/query.py", line 289, in __getitem__
return list(qs)[0]
IndexError: list index out of range

Then I tried to repair this by uploading the PO file we currently have in Git (wiki/src/install/mac/usb.es.po) and it failed too:

2019-08-25 05:45:10,479 ERROR Handled exception IndexError: b'list index out of range'

And now, any attempt to translate that resource fails, as long as it would trigger a commit: if I set “Review state” to “Needs editing”, it works; if I set it to “Approved”, it fails with a strack trace that shows Weblate tries to commit the file.

I could successfully add a translation to another resource and commit it.

At least cron.sh seems to work again, but I’m clueless wrt. how to repair this broken resource.

@hefee, can we somehow reset Weblate for this resource, e.g. make it forget it ever existed, then re-create it, then upload the known-good PO file by hand?

#5 Updated by hefee 2019-08-26 07:34:31

intrigeri wrote:

Thanks a lot for digging into it.

> hefee, can we somehow reset Weblate for this resource, e.g. make it forget it ever existed, then re-create it, then upload the known-good PO file by hand?

sure we can ;D Let’s delete and readd it.

And here we are. My log for this action:

weblate@translate:~/scripts$ python3
Python 3.5.3 (default, Sep 27 2018, 17:25:39) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tailsWeblate
>>> sp = tailsWeblate.subProject('wiki/src/install/mac/usb.es.po')
>>> sp # to check if we have the correct component
<SubProject: Tails/wiki/src/install/mac/usb.*.po>
>>> sp.delete()
(12291, {'trans.Source': 56, 'screenshots.Screenshot_sources': 0, 'trans.WhiteboardMessage': 0, 'trans.Translation': 15, 'trans.SubProject': 1, 'trans.Unit': 840, 'trans.Change': 11377, 'trans.Compon
entList_components': 2})
>>> tailsWeblate.addComponent('wiki/src/install/mac/usb.es.po')
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.ar.po (ar) [1/15]
INFO tails/wikisrcinstallmacusbpo/ar: processing wiki/src/install/mac/usb.ar.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.ca.po (ca) [2/15]
INFO tails/wikisrcinstallmacusbpo/ca: processing wiki/src/install/mac/usb.ca.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.de.po (de) [3/15]
INFO tails/wikisrcinstallmacusbpo/de: processing wiki/src/install/mac/usb.de.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.es.po (es) [4/15]
INFO tails/wikisrcinstallmacusbpo/es: processing wiki/src/install/mac/usb.es.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.fa.po (fa) [5/15]
INFO tails/wikisrcinstallmacusbpo/fa: processing wiki/src/install/mac/usb.fa.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.fr.po (fr) [6/15]
INFO tails/wikisrcinstallmacusbpo/fr: processing wiki/src/install/mac/usb.fr.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.id.po (id) [7/15]
INFO tails/wikisrcinstallmacusbpo/id: processing wiki/src/install/mac/usb.id.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.it.po (it) [8/15]
INFO tails/wikisrcinstallmacusbpo/it: processing wiki/src/install/mac/usb.it.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.pl.po (pl) [9/15]
INFO tails/wikisrcinstallmacusbpo/pl: processing wiki/src/install/mac/usb.pl.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.pt.po (pt) [10/15]
INFO tails/wikisrcinstallmacusbpo/pt: processing wiki/src/install/mac/usb.pt.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.ru.po (ru) [11/15]
INFO tails/wikisrcinstallmacusbpo/ru: processing wiki/src/install/mac/usb.ru.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.sr_Latn.po (sr_Latn) [12/15]
INFO tails/wikisrcinstallmacusbpo/sr_Latn: processing wiki/src/install/mac/usb.sr_Latn.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.tr.po (tr) [13/15]
INFO tails/wikisrcinstallmacusbpo/tr: processing wiki/src/install/mac/usb.tr.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.zh.po (zh) [14/15]
INFO tails/wikisrcinstallmacusbpo/zh: processing wiki/src/install/mac/usb.zh.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.zh_TW.po (zh_TW) [15/15]
INFO tails/wikisrcinstallmacusbpo/zh_TW: processing wiki/src/install/mac/usb.zh_TW.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: updating completed
<SubProject: Tails/wiki/src/install/mac/usb.*.po>

-> python script out of this (if you are in the ~/scripts directory, otherwise the import fails):

<code class="python">

#!/usr/bin/env python3
import tailsWeblate

fpath = 'wiki/src/install/mac/usb.es.po' 
sp = tailsWeblate.subProject(fpath)
sp.delete()
sp = tailsWeblate.addComponent(fpath)
</code>

@intrigeri can you please install ipython3, this is a much more feature rich python shell with things like tab completion and integrated documentation, …

#6 Updated by intrigeri 2019-09-12 09:11:25

The problem this ticket is about did not happen again, so I propose we do this:

1. document the steps one needs to take to repair things if/when it happens again, based on hefee’s earlier comment here
2. close this ticket
3. if on the mid-term we notice that this problem happens again and too often, come back to it