Bug #16995

translate.lizard: cron.sh fail with 'Lock wait timeout exceeded; try restarting transaction'

Added by hefee 2019-08-23 17:29:41 . Updated 2019-09-17 12:42:25 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
Type of work:
Contributors documentation
Blueprint:

Starter:
Affected tool:
Translation Platform
Deliverable for:
309

Description

We see it quite frequently that our cron.sh fail with ‘Lock wait timeout exceeded; try restarting transaction’

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/mysql/base.py", line 101, in execute
    return self.cursor.execute(query, args)
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 226, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorvalue
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 217, in execute
    res = self._query(query)
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 378, in _query
    rowcount = self._do_query(q)
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 341, in _do_query
    db.query(q)
  File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 280, in query
    _mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

Subtasks


Related issues

Related to Tails - Bug #16994: Weblate database switch to PostgreSQL Resolved 2020-04-01 2020-04-01
Related to Tails - Bug #17063: Cleanup translate-server.git Resolved

History

#1 Updated by hefee 2019-08-23 17:33:49

happend 23.Aug 2019 11:53:31 until 12:52:30 (every 5min) UTC
than @intrigeri has restarted mariadb and it was quiet but on 15:43:03 UTC we see another failure.

Mostly the stacktrace points to /usr/local/share/weblate/manage.py, somtimes to our own scripts /var/lib/weblate/scripts/update_weblate_components.py

Does it make sense to consider to switch to PostgreSQL (Bug #16994)

#2 Updated by hefee 2019-08-23 17:34:06

  • related to Bug #16994: Weblate database switch to PostgreSQL added

#3 Updated by intrigeri 2019-08-24 07:05:41

> Mostly the stacktrace points to /usr/local/share/weblate/manage.py, somtimes to our own scripts /var/lib/weblate/scripts/update_weblate_components.py

I’ve added some logging to cron.sh so it’s easier to tell, by looking at the email date vs. timestamps in the log file, what exact operation we were running when it failed.

#4 Updated by intrigeri 2019-08-25 06:04:48

Today the problem was caused by commit_pending getting stuck on a specific resource. I had to follow the corresponding doc in translate-server.git to repair it, to the point that I had to delete all translation history for that resource (translation_id). Apparently this did not lose the current translation status of that file, but the latest changes were not committed.

So I’ve clicked “commit” on https://translate.tails.boum.org/projects/tails/install-mac-usb/es/ and it failed. Stacktrace:

2019-08-25 05:41:26,453 ERROR Internal Server Error: /commit/tails/install-mac-usb/es/
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/decorators.py", line 23, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/django/views/decorators/http.py", line 40, in inner
return func(request, *args, **kwargs)
File "/usr/local/share/weblate/weblate/trans/views/git.py", line 136, in commit_translation
return perform_commit(request, obj)
File "/usr/local/share/weblate/weblate/trans/views/git.py", line 68, in perform_commit
request,
File "/usr/local/share/weblate/weblate/trans/views/git.py", line 45, in execute_locked
result = call(*args, **kwargs)
File "/usr/local/share/weblate/weblate/trans/models/translation.py", line 434, in commit_pending
)[0]
File "/usr/local/lib/python3.5/dist-packages/django/db/models/query.py", line 289, in __getitem__
return list(qs)[0]
IndexError: list index out of range

Then I tried to repair this by uploading the PO file we currently have in Git (wiki/src/install/mac/usb.es.po) and it failed too:

2019-08-25 05:45:10,479 ERROR Handled exception IndexError: b'list index out of range'

And now, any attempt to translate that resource fails, as long as it would trigger a commit: if I set “Review state” to “Needs editing”, it works; if I set it to “Approved”, it fails with a strack trace that shows Weblate tries to commit the file.

I could successfully add a translation to another resource and commit it.

At least cron.sh seems to work again, but I’m clueless wrt. how to repair this broken resource.

@hefee, can we somehow reset Weblate for this resource, e.g. make it forget it ever existed, then re-create it, then upload the known-good PO file by hand?

#5 Updated by hefee 2019-08-26 07:34:31

intrigeri wrote:

Thanks a lot for digging into it.

> hefee, can we somehow reset Weblate for this resource, e.g. make it forget it ever existed, then re-create it, then upload the known-good PO file by hand?

sure we can ;D Let’s delete and readd it.

And here we are. My log for this action:


weblate@translate:~/scripts$ python3
Python 3.5.3 (default, Sep 27 2018, 17:25:39) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tailsWeblate
>>> sp = tailsWeblate.subProject('wiki/src/install/mac/usb.es.po')
>>> sp # to check if we have the correct component
<SubProject: Tails/wiki/src/install/mac/usb.*.po>
>>> sp.delete()
(12291, {'trans.Source': 56, 'screenshots.Screenshot_sources': 0, 'trans.WhiteboardMessage': 0, 'trans.Translation': 15, 'trans.SubProject': 1, 'trans.Unit': 840, 'trans.Change': 11377, 'trans.Compon
entList_components': 2})
>>> tailsWeblate.addComponent('wiki/src/install/mac/usb.es.po')
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.ar.po (ar) [1/15]
INFO tails/wikisrcinstallmacusbpo/ar: processing wiki/src/install/mac/usb.ar.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.ca.po (ca) [2/15]
INFO tails/wikisrcinstallmacusbpo/ca: processing wiki/src/install/mac/usb.ca.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.de.po (de) [3/15]
INFO tails/wikisrcinstallmacusbpo/de: processing wiki/src/install/mac/usb.de.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.es.po (es) [4/15]
INFO tails/wikisrcinstallmacusbpo/es: processing wiki/src/install/mac/usb.es.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.fa.po (fa) [5/15]
INFO tails/wikisrcinstallmacusbpo/fa: processing wiki/src/install/mac/usb.fa.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.fr.po (fr) [6/15]
INFO tails/wikisrcinstallmacusbpo/fr: processing wiki/src/install/mac/usb.fr.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.id.po (id) [7/15]
INFO tails/wikisrcinstallmacusbpo/id: processing wiki/src/install/mac/usb.id.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.it.po (it) [8/15]
INFO tails/wikisrcinstallmacusbpo/it: processing wiki/src/install/mac/usb.it.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.pl.po (pl) [9/15]
INFO tails/wikisrcinstallmacusbpo/pl: processing wiki/src/install/mac/usb.pl.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.pt.po (pt) [10/15]
INFO tails/wikisrcinstallmacusbpo/pt: processing wiki/src/install/mac/usb.pt.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.ru.po (ru) [11/15]
INFO tails/wikisrcinstallmacusbpo/ru: processing wiki/src/install/mac/usb.ru.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.sr_Latn.po (sr_Latn) [12/15]
INFO tails/wikisrcinstallmacusbpo/sr_Latn: processing wiki/src/install/mac/usb.sr_Latn.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.tr.po (tr) [13/15]
INFO tails/wikisrcinstallmacusbpo/tr: processing wiki/src/install/mac/usb.tr.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.zh.po (zh) [14/15]
INFO tails/wikisrcinstallmacusbpo/zh: processing wiki/src/install/mac/usb.zh.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: checking wiki/src/install/mac/usb.zh_TW.po (zh_TW) [15/15]
INFO tails/wikisrcinstallmacusbpo/zh_TW: processing wiki/src/install/mac/usb.zh_TW.po, revision has changed
INFO tails/wikisrcinstallmacusbpo: updating completed
<SubProject: Tails/wiki/src/install/mac/usb.*.po>

-> python script out of this (if you are in the ~/scripts directory, otherwise the import fails):

<code class="python">

#!/usr/bin/env python3
import tailsWeblate

fpath = 'wiki/src/install/mac/usb.es.po' 
sp = tailsWeblate.subProject(fpath)
sp.delete()
sp = tailsWeblate.addComponent(fpath)
</code>

@intrigeri can you please install ipython3, this is a much more feature rich python shell with things like tab completion and integrated documentation, …

#6 Updated by intrigeri 2019-09-12 09:11:25

The problem this ticket is about did not happen again, so I propose we do this:

  1. document the steps one needs to take to repair things if/when it happens again, based on hefee’s earlier comment here
  2. close this ticket
  3. if on the mid-term we notice that this problem happens again and too often, come back to it

#7 Updated by intrigeri 2019-09-12 09:11:40

  • Subject changed from tranlate.lizard: cron.sh fail with 'Lock wait timeout exceeded; try restarting transaction' to translate.lizard: cron.sh fail with 'Lock wait timeout exceeded; try restarting transaction'

#8 Updated by Anonymous 2019-09-13 09:30:17

  • Deliverable for set to Sponsor_L

intrigeri wrote:
> The problem this ticket is about did not happen again, so I propose we do this:
>
> # document the steps one needs to take to repair things if/when it happens again, based on hefee’s earlier comment here
> # close this ticket
> # if on the mid-term we notice that this problem happens again and too often, come back to it

Sounds good.

intrigeri drafts something in translate-server, hefee reviews it and moves it to the design doc because this information should be public and translate-server should disappear on the long term, there is no secret information in there.

#9 Updated by intrigeri 2019-09-13 10:21:55

  • Assignee set to intrigeri
  • Target version set to Tails_4.0

#10 Updated by intrigeri 2019-09-13 12:20:11

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to hefee
  • Parent task set to Feature #15086
  • Type of work changed from Sysadmin to Contributors documentation

u wrote:
> intrigeri drafts something in translate-server,

Done: 11b914f.

#11 Updated by hefee 2019-09-16 16:55:58

  • Assignee changed from hefee to intrigeri

intrigeri wrote:
> u wrote:
> > intrigeri drafts something in translate-server,
> Done: 11b914f.

- You don’t need to create a script, you can simply use also a interactive python shell like ipython3.

- the last command sp = tailsWeblate.addComponent(fpath) already uses the PO file that is committed in git, so no need to do step 10. I would suggest to update the PO file in git directly, why you suggest using Weblate upload feature?

<code class="diff">
--- a/sysadmin.mdwn
+++ b/sysadmin.mdwn
@@ -112,7 +112,7 @@ To fix that, one can delete the translation change that breaks stuff.
         delete from trans_change where translation_id = 9734;

 9. Make Weblate forget the broken component, then re-add it.
-   To do so, add a script in `~weblate/scripts/` based on this example:
+   To do so, either create a script in `~weblate/scripts/` based on this example:

         #!/usr/bin/env python3
         import tailsWeblate
@@ -121,5 +121,5 @@ To fix that, one can delete the translation change that breaks stuff.
         sp.delete()
         sp = tailsWeblate.addComponent(fpath)

-10. In the Weblate web interface, upload the last known good PO file
-    for this component from Git.
+  or start `ipython3` in the `~weblate/scripts/` to have a interactive Python
+  shell and enter the commands line by line.
</code>

#12 Updated by intrigeri 2019-09-17 10:33:51

  • Assignee changed from intrigeri to hefee

Thanks for the feedback! I’ve applied your patch and polished a bit.

#13 Updated by hefee 2019-09-17 12:41:54

  • related to Bug #17063: Cleanup translate-server.git added

#14 Updated by hefee 2019-09-17 12:42:25

  • Status changed from In Progress to Resolved
  • Assignee deleted (hefee)

I mark this as resolved, as the bug itself is now fixed/documented. The removal of translate-server.git is another task (Bug #17063).