Tails

#1 Updated by muri 2015-09-30 08:15:10

the weblate documentation says (https://docs.weblate.org/en/latest/admin/translating.html):

_You might however want to have more eyes on the translation and require more people to accept them. This can be achieved by suggestion voting. You can enable this on Component configuration configuration by Suggestion voting and Autoaccept suggestions.
_
maybe this feature can help to implement some kind of ‘four-eyes principle’

#2 Updated by Anonymous 2015-09-30 11:10:06

hi muri!

yes, we’ve activated this feature and for now, only trusted users have access to the platform.

right now, we’re having a farsi translation sprint and eventually we can merge only farsi translations when it’s done.

but we need to work out a general strategy. for example what happens when somebody translates into german using the platform?

#3 Updated by muri 2015-09-30 12:01:21

ah, oke. you mean the merge into master.
i thought the question “How do we verify?” referred to how one weblate user can verify the translation of another weblate user and how this translation gets marked as done, like the [pull] on tails-l10n …

#4 Updated by sajolida 2015-10-03 14:20:02

During the October meeting we realized that this was highly dependent on the way we could implement a review process inside Weblate which was something in the original list of requirements (“MUST: provide user roles (admin, reviewer, translator”). We created a ticket to investigate this: Feature #10331.

#9 Updated by intrigeri 2015-11-05 10:52:18

Looks like this may need some preliminary research / proposal work before it’s ready to be discussed, so I would set Type of work = Research. Just my 2 cts :)

#10 Updated by muri 2015-11-07 07:42:25

so, i set up my own weblate instanz and tried to figure out some kind of workflow (both in weblate and mergin the results):

so, first of all, in weblate every file to be translated is a component and every component consists of strings. as mentioned above, there is a feature called suggestion voting, which means that a one doesn’t simply translate a string, but one suggests a translation. as soon as N users voted for that suggestion, that translation is used. ‘translation is used’ means, that it is commited, together with all the other translations that can be used (enough users voted on). this commit can be done by hand (if one user has the permissions to do so) or by cron job. then the changes are in the local repository of this specific component.
then the changes can be pushed to a remote repository (by hand, if one has the permission, or automatically when comitted).
this remote repository can be set specific to the component (which in weblate means ‘file’).

i haven’t found anything about branches in weblate. to me it seems that if we use one repository for translations, all the commits of translated content ends up in that repository. so the person merging these commits into the tails repo would have to look over all of them at once or look at the commits that affect one specific file (like a the branches we use at the moment).
there is a pre-commit-script and a post-commit-script option available in weblate, but i don’t know yet if that could be used to group commits together into a branch (if that is even needed?). i think would have to look into git to do so…

i think the notification )if needed) could be done once a day by a commit hook to tails-l10n?

#15 Updated by intrigeri 2015-12-01 14:27:32

> So, how could we prevent committing to the Git PO files for non-activated languages? Assigning tentatively to u…

It seems to me that Weblate has to store, somehow, PO files and their history for languages that are work-in-progress. I guess it totally makes sense that they are stored in Git.

So perhaps the question is: how do we avoid that our main/official Git repo continuously grows substantially due to translations that may, or may not, be completed some day? I think this calls for some indirection layer being set up between our official Git repo and the one Weblate uses. I guess that what we merge from Weblate should perhaps be a filtered version of the branch managed by Weblate, with history rewritten to exclude PO files that are not enabled on our website. But then merging back from the official master branch into Weblate’s one will introduce duplicate commits. Can we do better than this?

(I have a kinda off-topic comment to add here, that is I think very much related to this problem: PO files for languages that we don’t build won’t be updated automatically by ikiwiki when the corresponding master page is modified (new/modified strings). Similarly, when we introduce new translatable web pages, our ikiwiki won’t create the corresponding PO files for languages it does not know about. Is this something that is addressed by our current Weblate setup, or that is tracked somewhere? (Sorry for the naive question, I didn’t follow the whole thing closely.) If it’s not addressed yet: I think this problem also calls for some indirection layer being set up between our official Git repo and the one Weblate uses; in other words, someone/something may have to do the ikiwiki refresh with a slightly different ikiwiki.setup (with all languages known to Weblate enabled), so that all those PO files are created/updated as needed. The result of this refresh will probably be stored in Git and Weblate will be fed with it, or something, right?)

#16 Updated by sajolida 2016-01-04 10:55:47

Here are my personal interpretation from the discussion we had at 32C3:

We want to have:

• Controlled versions of both the website changed through ikiwiki
  and the translations changed through the translation platform
  because they might be changed in parallel.
• Both languages that are activated on the website and languages
  that are not.

Current limitations to solve:

• Ikiwiki should update the PO files of non-activated languages
  when the website changes but ignore them when building the
  website.
• Weblate should only commit reviewed changes.
• Weblate should automatically merge the PO files updated by
  ikiwiki when the website changes.

Then anybody with commit permissions on the main repo, can merge
the weblate repo into the main repo at any time.

#19 Updated by intrigeri 2016-02-13 02:36:03

> We want to have:

ACK.

> Current limitations to solve:

Wow, I see it was Christmas time :) But it’s not anymore, and anyway Christmas is a lie, so I’m afraid I’ll have to take the role of turning a very hopeful wishlist into something less nice, that we can perhaps make happen this year. It might imply a bit more work on our side, but only stuff many of us know how to do, and so much less waiting and hoping for years that IMO it’s worth it.

> * Ikiwiki should update the PO files of non-activated languages when the website changes but ignore them when building the website.

Adding these two requirements to the set of current ikiwiki semantics would make it self-contradictory, so no, this is not a realistic option. And IMO it would be a very bad design decision to change these semantics to accomodate for “real source pages that we build” vs. “half-read source pages that we don’t build but still we modify them and commit them to Git”. There are simply too many places in ikiwiki that would need to learn about this subtle difference, which IMO is too much complexity just to accommodate a rare use case that can be solved differently. Upstream might think differently (especially if provided patches), but don’t count on me to help sell them this idea, because if I were them, I would not buy it :)

Looks like as often, splitting a hard problem gives us to easy ones.

Regarding the “update the PO files” aspect: it seems quite easy to solve in an ad-hoc way on the weblate (or whatever web translation platform) side: simply ikiwiki --refresh (with a larger set of po_slave_languages — sorry for the “slave” work, I feel stupid having chosen than 7 years ago) after merging from the official master branch. So there’s no need to hardcode this custom workflow into ikiwiki itself, which I think is good news. And it also means that anyone moderately skilled at platform and scripting stuff can do it, instead of requiring someone who feels like patching ikiwiki, which is probably very good news. The only downside I see is that our official master branch will cary outdated PO files for languages that are not enabled on our website yet. I think we can live with it, for most practical purposes. And if we can’t, well, we’ll need a bot that merges from origin/master, does the aforementioned special refresh, and pushes back there (a bit more infra, but no rocket science) — and maybe a script run at freeze time, before we send the call for translations, could be good enough.

Regarding the “ignore them” aspect: looks like a simple matter of adding, to the list of excludes, any \.LL\.po that we want ikiwiki to ignore, e.g. something like (untested):

--- a/ikiwiki.setup
+++ b/ikiwiki.setup
@@ -105,7 +105,7 @@ ENV: {}
 # regexp of normally excluded files to include
include: '^\.htaccess$'
 # regexp of files that should be skipped
-exclude: '(^blueprint\/.*|^contribute\/how\/promote\/material\/.*|\/discussion\..*|\/Discussion\..*)'
+exclude: '(^blueprint\/.*|^contribute\/how\/promote\/material\/.*|\/discussion\..*|\/Discussion\..*|\.it\.po$)'
 # specifies the characters that are allowed in source filenames
wiki_file_chars: '-[:alnum:]+/._~'
 # allow symlinks in the path leading to the srcdir (potentially insecure)

… would totally ignore all Italian PO files.

> * Weblate should only commit reviewed changes.

This feels like an awkward, or unrealistic, expectation to me. But this is perhaps because I know close to nothing about Weblate. Where is it supposed to store changesets that have not been reviewed yet, then? Does it have any storage backend aside of Git, where it would store not-yet-reviewed changesets, and be able to merge them with newly pulled stuff later on? Or perhaps you mean it should maintain two branches, one for reviewed changes, and one for not-yet-reviewed changes? Really, I probably don’t get it.

If Weblate doesn’t provide this exact feature yet (that seems hard to implement), I suggest we look for other ways to get what we need. Otherwise I’m concerned we’ll fall into our usual self-inflicted trap of requiring something perfect, and instead having to live with the current crap we have, while some middle-ground could have been just fine.

> * Weblate should automatically merge the PO files updated by ikiwiki when the website changes.

This is already encoded in the “automatic pull from main Git repo” MUST on our translation platform spec, so it should not be an issue since Weblate is supposed to implement that spec. And if it doesn’t, well, this should be easy to script I guess.

> Then anybody with commit permissions on the main repo, can merge the weblate repo into the main repo at any time.

Indeed, this would be the ideal outcome! :)

#20 Updated by muri 2016-02-13 14:52:41

intrigeri wrote:

> This feels like an awkward, or unrealistic, expectation to me. But this is perhaps because I know close to nothing about Weblate. Where is it supposed to store
> changesets that have not been reviewed yet, then? Does it have any storage backend aside of Git, where it would store not-yet-reviewed changesets, and be able to
> merge them with newly pulled stuff later on? Or perhaps you mean it should maintain two branches, one for reviewed changes, and one for not-yet-reviewed changes?
> Really, I probably don’t get it.

part of the answer to this (the multiple stages a repository resp. a file in a repository goes through) should be more clear as soon as i get to do Feature #10802

#21 Updated by sajolida 2016-02-14 17:20:20

> Wow, I see it was Christmas time :)

Please avoid sarcasm. This is how far we went on this day with 5-6
people around the table.

But I’m glad you’re building up on our analysis :)

> Regarding the “update the PO files” aspect: it seems quite easy to
> solve in an ad-hoc way on the weblate (or whatever web translation
> platform) side: simply ikiwiki --refresh (with a larger set of
> po_slave_languages — sorry for the “slave” work

So this would be having an ikiwiki instance somewhere dedicated to
updating the PO files of non-activated languages, instead of doing this
on the production website. Smart solution!

Note that this is no way contradictory with my initial phrasing in

#23 Updated by sajolida 2016-02-21 10:39:01

In Feature #10802#note-5, muri explains that we could have translations pending for review not written to disk (maybe there only in the database until then).

#24 Updated by emmapeel 2016-02-29 12:03:09

intrigeri wrote:
> Is this something that is addressed by our current Weblate setup, or that is tracked somewhere? (Sorry for the naive question, I didn’t follow the whole thing closely.) If it’s not addressed yet: I think this problem also calls for some indirection layer being set up between our official Git repo and the one Weblate uses; in other words, someone/something may have to do the ikiwiki refresh with a slightly different ikiwiki.setup (with all languages known to Weblate enabled), so that all those PO files are created/updated as needed. The result of this refresh will probably be stored in Git and Weblate will be fed with it, or something, right?)

Yes, the same that when the new language is added, with ikiwiki.setup

#29 Updated by sajolida 2016-03-19 20:44:19

Actually, that’s unrelated to Feature #9183 since the content built on Weblate’s ikiwiki will be the same as origin/master.

#32 Updated by intrigeri 2016-03-25 22:35:09

> Reassign to u who’s assigned to the “Translation web platform” in our roadmap.

Thank you, I’m glad to not be responsible for the next step of this work. Still: u, we can discuss this next time we meet if you want my input, as a fun way of side-tracking :)