Tails

#1 Updated by zen 2019-07-29 19:23:05

• Assignee set to intrigeri

#2 Updated by intrigeri 2019-08-01 14:56:44

• Target version set to Tails_3.16

#3 Updated by intrigeri 2019-08-01 15:13:01

• Status changed from Confirmed to Needs Validation
• Assignee changed from intrigeri to emmapeel

> Since some days weblate blocks very often, all components, each three minutes.

I believe this is a direct consequence of https://git-tails.immerda.ch/puppet-tails/commit/?id=254ff4641cee2f716d5e645366a9c6453a2c1aff (that we had to do to ensure that Weblate does not fiddle with the Git repo while we’re updating it; previously, we would occasionally lose work done on Weblate or erroneously include changes in the wrong commit).

> It makes it hard to work, as you are confronted by errors all the time and need to go back the page, sometimes you loose your suggestion, etc.

ACK. I’ve just experienced this myself (after clicking “Suggest”, my suggestion was lost). I agree it’s a huge pain.

To clarify, the trade-off is between this sort of problems and those caused by running the cronjob less often (e.g. one can waste time translating a string that disappeared/changed in the canonical Git repo, accepted translations take more time to go live, etc.).

We’ve been runing that cronjob every 5 minutes so far. I’ve just changed this so it now runs only every 30 minutes.

Please let me know how annoying this now is in practice, with this change applied. If needed, we can probably go a bit further and run it every hour only.

#4 Updated by emmapeel 2019-08-06 07:04:48

• Assignee changed from emmapeel to intrigeri

It is still very annoying.

#5 Updated by intrigeri 2019-08-06 10:41:02

• Assignee changed from intrigeri to emmapeel

> It is still very annoying.

Thanks for this quick feedback! I’ve made the cronjob run only every 4 hours. Good enough?

Note to myself, in case that’s still not good enough:

• It might be because the less often we run the cronjob, the longer it blocks Weblate.
• Ultimately, Weblate losing work while it’s locked is a Weblate bug. We could check whether newer versions improve this (after we’ve upgraded) and if not, report a bug upstream.
• If this can’t be resolved on the Weblate side, we could try mitigating the problem by making our code that is guarded by this lock run faster: profile it to find where the most time is spent, look for low-hanging optimization fruits.

#6 Updated by emmapeel 2019-08-07 09:24:58

• Assignee changed from emmapeel to intrigeri

now it seems is blocked since last night

#9 Updated by intrigeri 2019-08-07 10:44:30

• Assignee changed from intrigeri to emmapeel

>> now it seems is blocked since last night

> I’m investigating further.

Fixed and documented (in translate-server.git) how I did it.

It turned out to be unrelated to this ticket ⇒ back to your plate :)

#14 Updated by intrigeri 2019-08-19 05:49:43

• Status changed from Needs Validation to In Progress
• Assignee changed from emmapeel to hefee

Hi @hefee!

>> Except “Update Weblate components in Weblate repo”, no? This operation should not happen while Weblate is itself modifying its Git repo, and it needs locking, right?

> Well the updating Weblate components in Weblate repo don’t need locking, because we simply bring the Weblate components in line with the files.

FTR it seems that this is incorrect: this script does modify the repo with (the Python equivalent of) git pull. But it does not matter because as you nicely explained below, this script locks the repo itself:

> Weblate itself is using a own lock file to communicate, that something changes the git repository. That lock mechanism is already used by update_weblate_components.py for the complete operation (https://git-tails.immerda.ch/puppet-tails/commit/?id=adb28d5c960206d534a1b1836f4e6d657b96cbf0). That means as long as the script is running nothing else can change the git repository.

Great! In passing, do you have any idea what’s the UX for translators while the repo is locked (as opposed to locking translations with lock_translation)?

> I see two options to test:
> 1. remove the lock completely, run the cron every */5 and look if we spot issues
> 2. only lock the update_weblate_components.py, makes a measurement how long the lock exists and than run the cron more often if possible

> Me personally would give option 1 a try.

Yeah, let’s do this!
Let’s coordinate on XMPP and do this at a well-chosen time, when we can keep an eye on things continuously for a few hours and revert if anything goes wrong.
I should be around every day this week except Tuesday, Wednesday, and Sunday.

#15 Updated by hefee 2019-08-20 19:27:32

• Assignee changed from hefee to intrigeri

intrigeri wrote:
> Hi hefee! > > >> Except "Update Weblate components in Weblate repo", no? This operation should not happen while Weblate is itself modifying its Git repo, and it needs locking, right? > > > Well the updating Weblate components in Weblate repo don't need locking, because we simply bring the Weblate components in line with the files. > > FTR it seems that this is incorrect: this script does modify the repo with (the Python equivalent of) git pull. But it does not matter because as you nicely explained below, this script locks the repo itself: > > > Weblate itself is using a own lock file to communicate, that something changes the git repository. That lock mechanism is already used by update_weblate_components.py for the complete operation ("$":https://git-tails.immerda.ch/puppet-tails/commit/?id=adb28d5c960206d534a1b1836f4e6d657b96cbf0). That means as long as the script is running nothing else can change the git repository. > > Great! In passing, do you have any idea what's the UX for translators while the repo is locked (as opposed to locking translations with lock_translation@)?

Nope I have no idea, but the circumstances, when Weblate will interact with the repo are not that often. And the filelock has a Timeout of 120secs. So I expect, that those operations may take longer. Additionally it is normal, to run cronjobs, that lock the repo in background like commit_pending and also if several users trigger a write event, than this lock mechanism takes care that the repo don’t blow up.

> > I see two options to test:
> > 1. remove the lock completely, run the cron every */5 and look if we spot issues
> > 2. only lock the update_weblate_components.py, makes a measurement how long the lock exists and than run the cron more often if possible
>
> > Me personally would give option 1 a try.
>
> Yeah, let’s do this!

Geat.

> Let’s coordinate on XMPP and do this at a well-chosen time, when we can keep an eye on things continuously for a few hours and revert if anything goes wrong.
> I should be around every day this week except Tuesday, Wednesday, and Sunday.

Let’s do it on Thursday 10am CEST?

#18 Updated by intrigeri 2019-08-22 10:26:32

• Status changed from In Progress to Needs Validation
• Assignee changed from intrigeri to emmapeel

We’ve removed the lock around 08:17 UTC today. Please let us know if this problem is solved :)

#19 Updated by CyrilBrulebois 2019-09-05 00:05:42

• Target version changed from Tails_3.16 to Tails_3.17

#20 Updated by intrigeri 2019-09-12 09:08:25

• Status changed from Needs Validation to Resolved

intrigeri wrote:
> We’ve removed the lock around 08:17 UTC today. Please let us know if this problem is solved :)

Three weeks later without any new breakage report, I’ll optimistically assume that this problem is gone. Please reopen if this problem happens again :)

#21 Updated by intrigeri 2019-09-12 14:27:48

• Target version changed from Tails_3.17 to Tails_4.0