Bug #15819

Make sure ikiwiki generated PO files satisfy the requirements we want check_po to set

Added by Anonymous 2018-08-19 17:38:28 . Updated 2019-06-27 17:19:31 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Start date:
2018-08-19
Due date:
% Done:

100%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Translation Platform
Deliverable for:

Description

One related question: does ikiwiki itself generate PO files that satisfy the new requirements introduced by this branch? I hope so but if not, then we have a problem: every time ikiwiki on our live website updates/creates a PO file we will get test suite failures.

First iteration (Feature #16328) will disable checks that are not satisfied by the PO files generated by ikiwiki. And then we can decide which ones of these disabled new checks we really would like to enable, and then we’ll have to fix ikiwiki/po4a before we can do that.

Subtasks

Related issues

Related to Tails - Bug #15408: Consider forcing wrapping of po files at 79 chars per line Rejected 2018-03-13
Related to Tails - Feature #16103: Handle translations outside wiki/src WRT check_po: document or opt-out Resolved 2018-11-05
Related to Tails - Feature #16328: Merge a stricter version of check_po whose expectations are realistic into our master branch Resolved 2019-01-08
Related to Tails - Bug #15403: Unify po headers Resolved 2018-08-19

History

#1 Updated by Anonymous 2018-08-19 17:38:45

  • Deliverable for set to Sponsor_L

#2 Updated by intrigeri 2018-10-24 17:03:34

  • Target version changed from Tails_3.10.1 to Tails_3.11

#3 Updated by Anonymous 2018-10-29 16:13:20

  • Assignee set to hefee

@hefee: in order to move this forward, would you be able to look into that?

#4 Updated by Anonymous 2018-10-29 16:13:46

  • related to Bug #15408: Consider forcing wrapping of po files at 79 chars per line added

#5 Updated by hefee 2018-11-05 16:46:39

  • related to Feature #16102: List of potential checks we might want to do on PO files added

#6 Updated by hefee 2018-11-05 16:48:39

  • Assignee deleted (hefee)
  • QA Check set to Info Needed

To test this I need to understand, what script(s) is/are executed to update the translations.

#7 Updated by Anonymous 2018-11-09 12:18:27

  • Assignee set to hefee

I think you should ask intrigeri to get all the information, because I would probably do exactly the same thing :)

#8 Updated by hefee 2018-11-12 11:07:53

  • Assignee changed from hefee to intrigeri

I need some intput how the server build the website and summit git commits, so I can test this locally if the automatical build is breaking the proposed git precommit script.

#9 Updated by Anonymous 2018-11-12 13:54:40

All I know is that we have ikiwiki.setup and ikiwiki-cgi.setup at the root of tails.git.
And a script to build the website called “build-website”. I don’t know if the server uses only this script or if it uses also other things.

#10 Updated by intrigeri 2018-11-12 15:12:06

  • Assignee changed from intrigeri to hefee
  • QA Check deleted (Info Needed)

It’s funny, I’ve documented the production website config on https://tails.boum.org/contribute/how/website/production/ yesterday :) It does not use build-website (that would be arbitrary code exec on the web server by anyone who can push changes to tails.git’s master branch). Instead, the website is refreshed on Git push using the pingee ikiwiki plugin (called in a post-update hook.

But anyway, all this info is probably useless to you because there’s nothing relevant that’s specific to the production website: all you need to check is what PO files are produced by ikiwiki when building the website locally. To force a refresh of those PO files, I think touch’ing the corresponding “master” (sic) pages is enough, e.g.: touch wiki/src/**/*.{html,mdwn} then run ./build-website. Note that you need po4a from Stretch: more recent versions behave differently and we’re not quite ready for that.

#11 Updated by intrigeri 2018-11-12 16:04:37

Also, as part of our release process check_po is used against PO files that come from Transifex, e.g. for tails.git:po/*.po and our custom packages (tails-greeter, tails-installer, etc.). So if check_po suddenly starts rejecting all those files, we’ll have a problem. I suggest creating a sibling ticket to check that :)

#12 Updated by hefee 2018-11-12 16:23:13

intrigeri wrote:
> Also, as part of our release process check_po is used against PO files that come from Transifex, e.g. for tails.git:po/*.po and our custom packages (tails-greeter, tails-installer, etc.). So if check_po suddenly starts rejecting all those files, we’ll have a problem. I suggest creating a sibling ticket to check that :)

We have two modes for the new check_po script one is like before, and than we have a extended check mode that checks additional stuff. So it should be fine. Plese send updates for this issue to Feature #16103.

#13 Updated by hefee 2018-11-12 16:23:30

  • related to Feature #16103: Handle translations outside wiki/src WRT check_po: document or opt-out added

#14 Updated by hefee 2018-11-13 11:45:01

  • Assignee changed from hefee to intrigeri
  • QA Check set to Info Needed

intrigeri wrote:
> It’s funny, I’ve documented the production website config on https://tails.boum.org/contribute/how/website/production/ yesterday :) It does not use build-website (that would be arbitrary code exec on the web server by anyone who can push changes to tails.git’s master branch). Instead, the website is refreshed on Git push using the pingee ikiwiki plugin (called in a post-update hook.

okay thanks.

> But anyway, all this info is probably useless to you because there’s nothing relevant that’s specific to the production website: all you need to check is what PO files are produced by ikiwiki when building the website locally. To force a refresh of those PO files, I think touch’ing the corresponding “master” (sic) pages is enough, e.g.: touch wiki/src/**/*.{html,mdwn} then run ./build-website. Note that you need po4a from Stretch: more recent versions behave differently and we’re not quite ready for that.

the touch wiki/src/**/*.{html,mdwn} then run ./build-website was successfull - no change was requested from ikiwiki. I think this is mostly successful, because Ikiwiki/Plugins/po.pm using msgmerge to merge pot file with po file. And if the pot file is the same, the po file is not touched (msgmerge —previous -q -U —backup=none pofile potfile). So a simple rebuild won’t break anything.

But what you haven’t described:

  • If the rebuild of the website updates the po files, how these are commited? E.g. activating a new langauge and the rebuild will create the new po files and wants to push them back to main git. Can we somehow execute unify headers in that case?
    One thing could be done if, we modify the msgmerge call and add -w 79 for the line length. But we also want control the po headers and I don’t see where ikiwiki sets those headers for new created files.

#15 Updated by intrigeri 2018-11-15 18:11:31

  • Assignee changed from intrigeri to hefee
  • QA Check changed from Info Needed to Dev Needed

> * If the rebuild of the website updates the po files, how these are commited? E.g. activating a new langauge and the rebuild will create the new po files and wants to push them back to main git. Can we somehow execute unify headers in that case?

One could surely extend the po plugin to do that.

> One thing could be done if, we modify the msgmerge call and add -w 79 for the line length. But we also want control the po headers and I don’t see where ikiwiki sets those headers for new created files.

Then I guess that’s done by po4a.

#16 Updated by hefee 2018-11-16 13:21:41

  • Assignee changed from hefee to intrigeri
  • QA Check changed from Dev Needed to Info Needed

intrigeri wrote:
> > * If the rebuild of the website updates the po files, how these are commited? E.g. activating a new langauge and the rebuild will create the new po files and wants to push them back to main git. Can we somehow execute unify headers in that case?
>
> One could surely extend the po plugin to do that.

Can you point me to the correct place to modify, or do this on your own? As I’m not very familiar with perl it would take a while to implement it.

> > One thing could be done if, we modify the msgmerge call and add -w 79 for the line length. But we also want control the po headers and I don’t see where ikiwiki sets those headers for new created files.
>
> Then I guess that’s done by po4a.

I think the most easiest way it to just allow to run our unify script before the push, that is at least also a good solution for other teams, that have other ideas how they what to format their po files. IMO we should concentrate on running the unifiy script instead of fixing Ikiwi’s creation of PO files.

#17 Updated by intrigeri 2018-11-16 13:43:37

  • Assignee changed from intrigeri to hefee

> intrigeri wrote:
>> > * If the rebuild of the website updates the po files, how these are commited? E.g. activating a new langauge and the rebuild will create the new po files and wants to push them back to main git. Can we somehow execute unify headers in that case?
>>
>> One could surely extend the po plugin to do that.

> Can you point me to the correct place to modify, or do this on your own? As I’m not very familiar with perl it would take a while to implement it.

I’m happy to do whatever work we need for this project on the PO plugin, as long as it’s not urgent. But I feel the need to formalize things: I’ve been spending quite time on this project with no formally defined role/budget; it’s been a while since the last time I was told by u “please do task X, clock and report your time”; this makes it hard for me to organize my work (I have no clue whether I should book some time for this project, how much and when; every request comes as a surprise and at the moment and don’t particularly need surprise bonus work :)

I would act as an external consultant so we would need to agree on a timeline and budget (so it probably needs to go through your team lead for budgeting matters).

>> > One thing could be done if, we modify the msgmerge call and add -w 79 for the line length. But we also want control the po headers and I don’t see where ikiwiki sets those headers for new created files.
>>
>> Then I guess that’s done by po4a.

> I think the most easiest way it to just allow to run our unify script before the push, that is at least also a good solution for other teams, that have other ideas how they what to format their po files. IMO we should concentrate on running the unifiy script instead of fixing Ikiwi’s creation of PO files.

So if I understand/guess correctly, we would need to add a new setting to the PO plugin (or ikiwiki as a whole), whose value is an arbitrary command line, and then between refreshing stuff and pushing (or between committing and pushing), ikiwiki would run that command, commit resulting changes, and refresh the website again. Is that what you have in mind?

Happy to discuss this further once my role here has been clarified :)

#18 Updated by hefee 2018-11-16 14:52:28

intrigeri wrote:
> every request comes as a surprise and at the moment and don’t particularly need surprise bonus work :)

well for me it is like stepping deeper and deeper into the dark and beeing surprised, how things are actually done, because I’m not familiar with the Tails infrastructure. I thoght that the ikiwiki pushes changes to PO files is a script that can be easily modified…

> >> > One thing could be done if, we modify the msgmerge call and add -w 79 for the line length. But we also want control the po headers and I don’t see where ikiwiki sets those headers for new created files.
> >>
> >> Then I guess that’s done by po4a.
>
> > I think the most easiest way it to just allow to run our unify script before the push, that is at least also a good solution for other teams, that have other ideas how they what to format their po files. IMO we should concentrate on running the unifiy script instead of fixing Ikiwi’s creation of PO files.
>
> So if I understand/guess correctly, we would need to add a new setting to the PO plugin (or ikiwiki as a whole), whose value is an arbitrary command line, and then between refreshing stuff and pushing (or between committing and pushing), ikiwiki would run that command, commit resulting changes, and refresh the website again. Is that what you have in mind?

The other solution would be to replace the pingee plugin with a script, with that we can control the git commit by ourselves.

#19 Updated by hefee 2018-11-16 14:53:32

  • Assignee deleted (hefee)

intrigeri wrote:
> I’m happy to do whatever work we need for this project on the PO plugin, as long as it’s not urgent. But I feel the need to formalize things: I’ve been spending quite time on this project with no formally defined role/budget; it’s been a while since the last time I was told by u “please do task X, clock and report your time”; this makes it hard for me to organize my work (I have no clue whether I should book some time for this project, how much and when; every request comes as a surprise and at the moment and don’t particularly need surprise bonus work :)

@u: can your clearify this?

#20 Updated by Anonymous 2018-11-18 14:25:07

  • Assignee set to hefee

See my email :)

#21 Updated by CyrilBrulebois 2018-12-16 14:04:31

  • Target version changed from Tails_3.11 to Tails_3.12

#22 Updated by intrigeri 2019-01-08 17:14:35

  • related to Feature #16328: Merge a stricter version of check_po whose expectations are realistic into our master branch added

#23 Updated by intrigeri 2019-01-08 17:17:37

  • Subject changed from Make sure ikiwiki generated PO files satisfy check_po's requirements to Make sure ikiwiki generated PO files satisfy the requirements we want check_po to set
  • Description updated
  • Assignee deleted (hefee)
  • Target version deleted (Tails_3.12)
  • QA Check deleted (Info Needed)
  • Type of work changed from Test to Code

#26 Updated by intrigeri 2019-03-12 17:08:38

  • Deliverable for deleted (Sponsor_L)

#27 Updated by intrigeri 2019-03-12 17:16:39

#28 Updated by intrigeri 2019-03-12 17:16:45

#29 Updated by intrigeri 2019-06-26 10:58:14

  • related to deleted (Feature #16102: List of potential checks we might want to do on PO files)

#30 Updated by intrigeri 2019-06-26 10:58:20

  • blocks Feature #16102: List of potential checks we might want to do on PO files added

#31 Updated by intrigeri 2019-06-26 10:58:24

  • blocked by deleted (Feature #16102: List of potential checks we might want to do on PO files)

#32 Updated by intrigeri 2019-06-26 10:58:29

  • blocked by Feature #16102: List of potential checks we might want to do on PO files added

#33 Updated by hefee 2019-06-27 17:16:53

  • blocks deleted (Feature #16102: List of potential checks we might want to do on PO files)

#34 Updated by hefee 2019-06-27 17:19:31

  • Status changed from Confirmed to Resolved
  • % Done changed from 0 to 100
  • Affected tool set to Translation Platform

The scripts are got updated life now and the files from ikiwki are satisfying our needs.