Feature #15086: Translation platform: Write design documentation

#1 Updated by Anonymous 2018-06-12 13:15:59

Blueprint set to https://tails.boum.org/blueprint/translation_platform

#2 Updated by Anonymous 2018-06-12 13:22:29

Status changed from Confirmed to In Progress

#3 Updated by intrigeri 2018-06-26 16:27:50

Target version changed from Tails_3.8 to Tails_3.9

#4 Updated by Anonymous 2018-09-03 17:40:42

Target version changed from Tails_3.9 to Tails_3.11

#5 Updated by Anonymous 2018-11-02 19:50:45

Target version changed from Tails_3.11 to Tails_3.12

I don’t see this happening as early as 3.11, but we’ll work on this together with hefee.

#6 Updated by anonym 2019-01-30 11:59:13

Target version changed from Tails_3.12 to Tails_3.13

#7 Updated by Anonymous 2019-03-12 14:48:44

Subject changed from Write design documentation for other projects to reuse our work to Translation platform: Write design documentation for other projects to reuse our work

#8 Updated by Anonymous 2019-03-18 15:55:39

Target version changed from Tails_3.13 to Tails_3.16

I’d rather do that once this has been implemented :)

#9 Updated by Anonymous 2019-07-05 16:17:06

Assignee set to hefee
Affected tool set to Translation Platform

#10 Updated by Anonymous 2019-07-05 16:17:27

Status changed from In Progress to Confirmed

#11 Updated by Anonymous 2019-07-09 16:20:16

Subject changed from Translation platform: Write design documentation for other projects to reuse our work to Translation platform: Write design documentation

Some information that could go into translate-server.git for future reference:

>> - clean the VM because files are shattered in various & unpredictible
>> locations

@intrigeri wrote:

I just did what I felt was good enough for these 2 things:

- Move logs to /var/log

- Turn class parameters into constants in all cases where we’ve never
supported setting them to a non-default value (and have no reason
to ever do so).

ulrike: I believe this fixes:

intrigeri:
> I’d like to broaden the scope a bit
> to also include: “make all scripts and configuration files managed by
> Puppet honor the parameters of the tails::weblate class” (currently we
> pass parameters to this class, honor them in some places, but
> hard-code their default value in various scripts, so if someone ever
> passes non-default values for these parameters, well, things won’t
> work™).

- Use the remaining class parameters in templates instead of
hard-coding their default value.

The end result is definitely not perfect but improving things further
would start having a pretty high cost/benefit ratio so I don’t think
it’s worth the effort

#12 Updated by Anonymous 2019-07-09 16:31:36

blocks ~~Feature #15089~~: Write public technical report about the Translation platform added

#13 Updated by intrigeri 2019-07-10 07:53:34

Description updated

#14 Updated by Anonymous 2019-07-19 16:46:08

@hefee: I think a good start is to merge our blueprint with all information from translate-server.git that can be made public.

#15 Updated by intrigeri 2019-08-15 08:34:33

Description updated
Type of work changed from Communicate to Contributors documentation

#16 Updated by hefee 2019-08-16 22:19:36

Status changed from Confirmed to Needs Validation
Assignee deleted (~~hefee~~)
Feature Branch set to hefee/15086-desgin-documentation

I updated now the blueprint and added some paragraphs about the scripts that we are running.

Additionally I scanned through the translation-server.git and removed not needed stuff and updated the remaining files.

Do you see more stuff that should end up in the blueprint?

The README.mdwn is another (older) approch to describe the scripts, what do you find more readable?

#17 Updated by CyrilBrulebois 2019-09-05 00:05:28

Target version changed from Tails_3.16 to Tails_3.17

#18 Updated by intrigeri 2019-09-12 14:25:08

Target version changed from Tails_3.17 to Tails_4.0

#20 Updated by Anonymous 2019-09-18 10:11:15

I’m on it!

#21 Updated by Anonymous 2019-09-19 12:53:26

Status changed from Needs Validation to In Progress
Assignee set to hefee
Feature Branch deleted (~~hefee/15086-desgin-documentation~~)

Hi hefee!

I’ve done a first review and tried to rewrite some parts that I found hard to understand. Can you please go through the document and improve the parts that I marked with “XXX”? Then reassign to me again for review please.

Thank you!

Also, you can do this directly in the master branch. Blueprints are editable for everyone, so no need to create a branch.

#22 Updated by Anonymous 2019-09-19 12:58:34

PS I have not deleted your branch (hefee/15086-desgin-documentation). Can you do this yourself please?

#23 Updated by Anonymous 2019-09-19 13:50:22

Some other things I’ve noticed:

I’ve started to cleanup the translate-server.git repository with information that is either obsolete or should live in our public documentation.
There are some files left though, can you please move them to a public place, after having deleted the sections that are outdated? You might want to coordinate this with intrigeri who might have some ideas on where to store this information instead.

Concerned files:

- sysadmin.mdwn → Not sure where this contents could go?

- weblate.mdwn
- tmserver.mdwn (I think this one can be deleted)

Also, I’ve noticed that there is a documentation in contribute/l10n_tricks that explains how to add a new language to weblate. I’ve now linked to this documentation from the blueprint. But I’m not sure that l10_tricks.mdwn is the right place for this documentation. I’ve also not checked this documentation for accuracy. Can you please do this as part of the design documentation?

#24 Updated by Anonymous 2019-09-26 13:01:34

hefee wrote:
> Additionally I scanned through the translation-server.git and removed not needed stuff and updated the remaining files.

As said, I did this again, and there is still stuff that needs to be moved somewhere else.

> Do you see more stuff that should end up in the blueprint?
>
> The README.mdwn is another (older) approch to describe the scripts, what do you find more readable?

I think we can get rid of the README now.

Please note that if I want to re-review the doc + have intrigeri do a tech review + write the call for translation before Oct 22nd, we have not that much time left to finish this work.

#25 Updated by hefee 2019-09-29 22:45:58

related to ~~Bug #17063~~: Cleanup translate-server.git added

#26 Updated by hefee 2019-09-29 22:58:38

Assignee deleted (~~hefee~~)

u wrote:
> I’ve started to cleanup the translate-server.git repository with information that is either obsolete or should live in our public documentation.
> There are some files left though, can you please move them to a public place, after having deleted the sections that are outdated? You might want to coordinate this with intrigeri who might have some ideas on where to store this information instead.

Done. As removing all the files are not necessary the part of this issue. I created a new issue ~~Bug #17063~~, to separate the discussions. In the end it is only `sysadmin.mdwn` is left.

>
> Concerned files:
>
> - sysadmin.mdwn → Not sure where this contents could go?
→ move discussion to ~~Bug #17063~~.

> - weblate.mdwn
→ moved to blueprint

> - tmserver.mdwn (I think this one can be deleted)
→ moved to blueprint
>
> Also, I’ve noticed that there is a documentation in contribute/l10n_tricks that explains how to add a new language to weblate. I’ve now linked to this documentation from the blueprint. But I’m not sure that l10_tricks.mdwn is the right place for this documentation. I’ve also not checked this documentation for accuracy. Can you please do this as part of the design documentation?

This part was written by intrigeri and I reviewed it.

> I have not deleted your branch (hefee/15086-desgin-documentation). Can you do this yourself please?

done

> I’ve done a first review and tried to rewrite some parts that I found hard to understand. Can you please go through the document and improve the parts that I marked with “XXX”? Then reassign to me again for review please.

I gone through the document and improved the parts you marked with “XXX”.

> Also, you can do this directly in the master branch. Blueprints are editable for everyone, so no need to create a branch.

Well at the state, when I created the branch, I had no commit access to the master branch, so I needed to create a branch as far I know. Now this is different, as I got full commit access to tails.git.

#27 Updated by Anonymous 2019-09-30 18:01:24

Status changed from In Progress to Needs Validation
Assignee set to intrigeri

Thanks! I’ve reviewed the new version and will now let intrigeri add any bits that we might have overlooked.

@hefee: blueprints are world editable via the website.

#28 Updated by Anonymous 2019-09-30 18:01:52

Parent task deleted (~~~~Feature #15085~~~~)

unparenting

#29 Updated by Anonymous 2019-09-30 18:02:27

Parent task set to ~~Feature #15085~~

Reverting

#30 Updated by intrigeri 2019-10-01 05:53:46

Thank you both. I’ll try to make time this week for this review but I’m very busy with urgent+important things that have to be done in time for 4.0, so I can’t promise I’ll be able to work on this on short notice ⇒ it may have to wait a few weeks. So at this point, I unfortunately can’t tell you anything more precise, that could help you schedule time for the next steps (e.g. acting on my review if needed, moving this to the final location, sending the link wherever it makes sense, etc.).

#31 Updated by Anonymous 2019-10-01 09:27:23

I don’t think this is a blocker.

But, in order to get closure rather sooner than later on this project on my side, instead of doing back and forth reviews between the three of us (“acting on your review”) I would instead ask you to please implement your fixes right away, and only send it back to hefee or myself if there is something you cannot answer or act upon yourself.

#32 Updated by intrigeri 2019-10-01 10:59:12

> But, in order to get closure rather sooner than later on this project on my side, instead of doing back and forth reviews between the three of us (“acting on your review”) I would instead ask you to please implement your fixes right away, and only send it back to hefee or myself if there is something you cannot answer or act upon yourself.

Sure thing! This will be smoother for me as well :)

#33 Updated by Anonymous 2019-10-01 14:44:51

> Sure thing! This will be smoother for me as well :)
Great :)

#34 Updated by intrigeri 2019-10-03 07:21:43

Status changed from Needs Validation to In Progress

Applied in changeset commit:tails|161b3436732e950ae3c84897bc7db0a7d3a3fac3.

#35 Updated by intrigeri 2019-10-03 07:23:36

Assignee changed from intrigeri to hefee

Hi!

I’ve reviewed this piece of doc. First, congrats: this work is amazing! By reading it, I understood a few things that were previously not 100% clear to me.

As agreed beforehand, I’ve pushed a number of (mostly minor) improvements: commit:161b3436732e950ae3c84897bc7db0a7d3a3fac3. I’ll let you review them if you’d like to.

There were a few things I was not sure about so I’ve not acted on them myself. I’ll split them between questions directed particularly to hefee, and questions that also affect u.

For hefee and u:

There’s a “XXX” left. It seems to me that this comment is correct and the script should be renamed. I’m not sure how important this is and given we’re likely running out of budget, I would be fine with leaving things as-is.
The bits about Docker are outdated, no? I propose we leave them in the blueprint when we’ll move the rest of it elsewhere.
I think I would move the specification (“Choosing a translation web platform”) to a dedicated sub-page. If that’s fine by you, I volunteer to do it myself.
More generally, this blueprint mixes rather different kinds of information together. I think it’s been fine so far, as a single place to write all this made this work simpler; but I don’t think it would be good enough to simply move it to contribute/design as-is. I proposed a rough plan on ~~Bug #17063#note-6~~. Should I take care of splitting the blueprint and moving pieces to contribute/*, or would you folks like us to first discuss together what should go where, or would one of you want to take care of thinking this through by themselves, or something else?

For hefee:

The blueprint claims that touch and changing mtime triggers a Git commit, which is incorrect AFAICT: Git does not care about such filesystem metadata. So the explanation of why we need to avoid creating useless merge commits seems buggy to me. hefee, if you agree, could you please fix it?
I don’t understand the logical “then” relationship there “Weblate’s Git repository is not bare. Hence […]”. Even if it were a bare repo, we would have to merge its changes, no?
Structure-wise, it’s not clear to me why “Machine translation” is in its own section instead of being part of “Setup of our translation platform”. For example, I don’t understand why this section is different in nature from the one about the staging website, which is in “Setup of our translation platform”. I would move it there, if you agree.

#36 Updated by Anonymous 2019-10-03 09:47:18

intrigeri wrote:

> I’ve reviewed this piece of doc. First, congrats: this work is amazing! By reading it, I understood a few things that were previously not 100% clear to me.

Cool!

> As agreed beforehand, I’ve pushed a number of (mostly minor) improvements: commit:161b3436732e950ae3c84897bc7db0a7d3a3fac3. I’ll let you review them if you’d like to.
>
> There were a few things I was not sure about so I’ve not acted on them myself. I’ll split them between questions directed particularly to hefee, and questions that also affect u.
>
> For hefee and u:
>
> * There’s a “XXX” left. It seems to me that this comment is correct and the script should be renamed. I’m not sure how important this is and given we’re likely running out of budget, I would be fine with leaving things as-is.

I put the XXX, so I cannot be the person replying to this question :)

> * The bits about Docker are outdated, no? I propose we leave them in the blueprint when we’ll move the rest of it elsewhere.

> * I think I would move the specification (“Choosing a translation web platform”) to a dedicated sub-page. If that’s fine by you, I volunteer to do it myself.

@intrigeri: That’s a very good idea → please go ahead.

> * More generally, this blueprint mixes rather different kinds of information together. I think it’s been fine so far, as a single place to write all this made this work simpler; but I don’t think it would be good enough to simply move it to contribute/design as-is. I proposed a rough plan on ~~Bug #17063#note-6~~. Should I take care of splitting the blueprint and moving pieces to contribute/*, or would you folks like us to first discuss together what should go where, or would one of you want to take care of thinking this through by themselves, or something else?

@intrigeri I’m fine with you making this choice and implement it as you see fit.

> * Structure-wise, it’s not clear to me why “Machine translation” is in its own section instead of being part of “Setup of our translation platform”. For example, I don’t understand why this section is different in nature from the one about the staging website, which is in “Setup of our translation platform”. I would move it there, if you agree.

@intrigeri this part was moved from translate-server.git. I agree with you, it should be part of our setup. Please move it there.

#37 Updated by hefee 2019-10-04 23:42:05

> I’ve reviewed this piece of doc. First, congrats: this work is amazing! By reading it, I understood a few things that were previously not 100% clear to me.

great ;D

> As agreed beforehand, I’ve pushed a number of (mostly minor) improvements: commit:161b3436732e950ae3c84897bc7db0a7d3a3fac3. I’ll let you review them if you’d like to.

ACK

> * There’s a “XXX” left. It seems to me that this comment is correct and the script should be renamed. I’m not sure how important this is and given we’re likely running out of budget, I would be fine with leaving things as-is.

lets create a ticket for this and remove the XXX.

> * The bits about Docker are outdated, no? I propose we leave them in the blueprint when we’ll move the rest of it elsewhere.

ACK

> * I think I would move the specification (“Choosing a translation web platform”) to a dedicated sub-page. If that’s fine by you, I volunteer to do it myself.

ACK

> * More generally, this blueprint mixes rather different kinds of information together. I think it’s been fine so far, as a single place to write all this made this work simpler; but I don’t think it would be good enough to simply move it to contribute/design as-is. I proposed a rough plan on ~~Bug #17063#note-6~~. Should I take care of splitting the blueprint and moving pieces to contribute/*, or would you folks like us to first discuss together what should go where, or would one of you want to take care of thinking this through by themselves, or something else?

ACK

> * The blueprint claims that touch and changing mtime triggers a Git commit, which is incorrect AFAICT: Git does not care about such filesystem metadata. So the explanation of why we need to avoid creating useless merge commits seems buggy to me. hefee, if you agree, could you please fix it?

@intrigeri: This was my explanation, why git allows me to commit, because from cmd line I know that git stops me from commiting empty commits. I see this is a highlevel check, that is not activated within python-git.

> * I don’t understand the logical “then” relationship there “Weblate’s Git repository is not bare. Hence […]”. Even if it were a bare repo, we would have to merge its changes, no?

@intrigeri: This logical “then” is about that we need to pull and can’t push, not about the integration of commits from Weblate. But yeah I’m not happy with this paragraph.

> * Structure-wise, it’s not clear to me why “Machine translation” is in its own section instead of being part of “Setup of our translation platform”. For example, I don’t understand why this section is different in nature from the one about the staging website, which is in “Setup of our translation platform”. I would move it there, if you agree.

Move it.

#38 Updated by hefee 2019-10-07 15:26:58

related to ~~Bug #17129~~: rename puppet-tails:update_weblate_git.py → merge_canonical_changes.py added

#39 Updated by hefee 2019-10-07 15:37:31

Status changed from In Progress to Needs Validation
Assignee changed from hefee to intrigeri

hefee wrote:
> > As agreed beforehand, I’ve pushed a number of (mostly minor) improvements: commit:161b3436732e950ae3c84897bc7db0a7d3a3fac3. I’ll let you review them if you’d like to.

I reviewed this one and others by u on top of mine.

> > * There’s a “XXX” left. It seems to me that this comment is correct and the script should be renamed. I’m not sure how important this is and given we’re likely running out of budget, I would be fine with leaving things as-is.
>
> lets create a ticket for this and remove the XXX.

created ~~Bug #17129~~.

> > * The blueprint claims that touch and changing mtime triggers a Git commit, which is incorrect AFAICT: Git does not care about such filesystem metadata. So the explanation of why we need to avoid creating useless merge commits seems buggy to me. hefee, if you agree, could you please fix it?
>
> intrigeri: This was my explanation, why git allows me to commit, because from cmd line I know that git stops me from commiting empty commits. I see this is a highlevel check, that is not activated within python-git.

improved the paragraph.

> > * I don’t understand the logical “then” relationship there “Weblate’s Git repository is not bare. Hence […]”. Even if it were a bare repo, we would have to merge its changes, no?
>
> intrigeri: This logical “then” is about that we need to pull and can’t push, not about the integration of commits from Weblate. But yeah I’m not happy with this paragraph.