Bug #17005

Upgrade to po4a 0.55 in Tails itself, in the Vagrant build box, on {translate,www}.lizard, and on RM's systems

Added by intrigeri 2019-08-29 17:53:53 . Updated 2020-04-29 13:33:44 .

Status:
Resolved
Priority:
Elevated
Assignee:
intrigeri
Category:
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
tails.git:feature/17005-po4a-0.55 and doc/17005-unfuzzy, puppet-tails.git:feature/17005-po4a-0.55
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

po4a changed behavior between Stretch and Buster, e.g. wrt. linebreaks at the end of strings, and better extraction of bullet points. This is an improvement.

But this means that when updating with Buster’s po4a a PO file that was generated with Stretch’s po4a, tons of strings will be marked as fuzzy; and reciprocally. We’ve been bitten by this a couple times already, e.g. when a technical writer was using a recent version of po4a from Ubuntu, or when another used the one that’s currently installed in Tails by our devel branch.

So we need to go through this upgrade in a coordinated way, taking the drawbacks into account.

Before we release 4.0, we need to upgrade to po4a 0.55 all systems that are responsible for generating or updating PO files:

  • Tails itself: some translators and technical writers work in there
  • Vagrant build box
  • www.lizard: ikiwiki refreshes PO files
  • translate.lizard: for the staging website
  • every RM’s system: we update PO files as part of the release process

Tentative migration plan:

  1. Coordinate with stakeholders
  2. On a topic branch forked off master (feature/17005-po4a-0.55):
    1. Update our doc to require po4a 0.55 instead of the Stretch version in wiki/src/contribute/build/website.mdwn and wiki/src/contribute/release_process.mdwn.
    2. Upgrade po4a to the version from Buster in Tails itself (config/chroot_apt/preferences) and in our Vagrant box (vagrant/definitions/tails-builder/postinstall.sh).
    3. Refresh all website PO files with po4a 0.55. It’ll make lots of strings fuzzy, which will have an immediate cost for translators, but on the long term it’s worth it.
    4. Commit and push.
    5. To validate this branch and ease review, prepare and push a feature/17005-po4a-0.55-stable that will actually build on Jenkins, forked off feature/17005-po4a-0.55, with current stable merged into it.
    6. Review.
  3. On a topic branch for our Puppet code (feature/17005-po4a-0.55 in puppet-tails.git):
    1. Upgrade po4a on {translate,www}.lizard to the version from stretch-backports.
    2. Review
  4. Once everything is ready and reviewed:
    1. Soft-freeze our website
      1. Ask tech writers to avoid pushing until we’re done here.
      2. Disable Weblate’s automated push.
    2. Merge the Puppet branch, deploy, ensure po4a is upgraded.
    3. Merge the tails.git topic branch into master, refresh PO files again with po4a 0.55, commit.
    4. Merge master into stable, then stable into devel.
    5. Push stable and devel.
    6. Push master.
    7. Tell tech writers that they can push stuff again.
  5. Announce the change to people who maintain other systems themselves to build the website:
    1. people who translate in Git: canceled as nobody does that on a regular basis anymore
    2. release managers
    3. technical writers
  6. Weblate (blocked by Bug #17619)
    1. Re-enable Weblate automatic push
    2. Ensure Weblate picks up the updates
    3. Announce the change to Weblate translators

Ideally, to maximize the chances as many strings as possible get unfuzzied in the static version of our doc included in the first release that follows this change, we would merge all this early in a release cycle.


Subtasks


Related issues

Related to Tails - Bug #17127: Downgrade po4a to Stretch's 0.47-2 Resolved
Related to Tails - Bug #16868: Upgrade Vagrant box to Buster Resolved
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by intrigeri 2019-08-29 17:54:16

#2 Updated by intrigeri 2019-08-31 15:04:45

I wrote “the earlier, the better”, but let’s not do this before 3.16 and 4.0~beta2 are out: if anything goes wrong, we don’t want to disrupt the release process.

#3 Updated by intrigeri 2019-09-04 10:29:31

  • Description updated

#4 Updated by intrigeri 2019-09-12 12:55:19

If we don’t manage to get this done in time for 4.0, we can downgrade po4a to the Stretch version in 4.0 and postpone.

#5 Updated by intrigeri 2019-10-06 05:43:36

  • related to Bug #17127: Downgrade po4a to Stretch's 0.47-2 added

#6 Updated by intrigeri 2019-10-06 05:45:30

  • Target version changed from Tails_4.0 to Tails_4.1

intrigeri wrote:
> If we don’t manage to get this done in time for 4.0, we can downgrade po4a to the Stretch version in 4.0 and postpone.

We still have quite some non-trivial FT work to do in time for 4.0~rc1/4.0, that cannot be postponed, so to be on the safe side, I think we should postpone this one and apply the aforementioned workaround → Bug #17127.

#7 Updated by intrigeri 2019-10-06 05:46:08

  • Subject changed from Use the same version of po4a in Tails 4.0, on www.lizard, and on RM's systems to Upgrade to po4a 0.55 in Tails itself, on www.lizard, and on RM's systems

#8 Updated by intrigeri 2019-10-19 12:56:41

  • related to Bug #16868: Upgrade Vagrant box to Buster added

#9 Updated by intrigeri 2019-10-19 12:57:43

  • Subject changed from Upgrade to po4a 0.55 in Tails itself, on www.lizard, and on RM's systems to Upgrade to po4a 0.55 in Tails itself, in the Vagrant build box, on www.lizard, and on RM's systems

(Once Bug #16868 is merged, we’ll have another place where we forcefully install po4a 0.47.)

#10 Updated by intrigeri 2019-11-08 18:38:06

  • Target version deleted (Tails_4.1)

It would be nice if we could give translators the benefits of po4a 0.55 but have plenty of other important things to do.

#11 Updated by emmapeel 2020-01-14 10:06:51

At the moment the instructions on the website ( https://tails.boum.org/contribute/build/website/ ) tell you to install po4a/stretch but when trying to do it from Tails 4.2.x, the server says something like: could not find po4a in stretch distribution.

Installing po4a works well, though.

#12 Updated by intrigeri 2020-01-15 10:32:18

Hi @emmapeel,

> At the moment the instructions on the website
> ( https://tails.boum.org/contribute/build/website/ ) tell you to
> install po4a/stretch but when trying to do it from Tails 4.2.x, the
> server says something like: could not find po4a in
> stretch distribution.

AFAICT, the instructions in the “Build the website in Tails” section don’t have this problem.
So I’ve updated that doc page a bit:

  • increase the chances Tails users follow the relevant instructions
  • improve a bit the non-Tails instructions (not relevant here, but it was cheap while I was at it)

Better?

#13 Updated by intrigeri 2020-01-28 12:46:20

  • Subject changed from Upgrade to po4a 0.55 in Tails itself, in the Vagrant build box, on www.lizard, and on RM's systems to Upgrade to po4a 0.55 in Tails itself, in the Vagrant build box, on {translate,www}.lizard, and on RM's systems
  • Description updated

I think we should give up on developers unfuzzying the strings that this mass-change to PO files will make fuzzy:

  • This requirement is the reason why we’ve been delaying this upgrade so far and I suspect it would delay it for at least 5 more months.
  • The upgrade will make our translators’ life easier because the new po4a handles newlines at the end of strings in a way that’s more friendly (basically, it hides them from translators). Given the frequency at which we see inconsistent-trailing-newlines errors, i.e. at which translators get these newlines wrong, it seems clear that this will give us recurring benefits ⇒ IMO it’s worth paying the migration cost up-front ASAP, even with the drawback that translators have to (trivially) unfuzzy a large number of newly-fuzzy strings.

I’ve updated the migration plan in the description accordingly.

#14 Updated by intrigeri 2020-01-28 12:49:05

  • Description updated
  • Assignee set to intrigeri
  • Target version set to Tails_4.3

I’ll try to get all the bits ready in time to merge them immediately after 4.3 is out.

#15 Updated by intrigeri 2020-02-09 08:32:17

  • Target version changed from Tails_4.3 to Tails_4.4

#16 Updated by CyrilBrulebois 2020-03-01 16:49:33

Hey, I was just wading through some tickets targetting 4.4, and I was wondering whether this might be merged in time for it, so that I can prepare my host environment accordingly?

#17 Updated by intrigeri 2020-03-01 17:08:26

> Hey, I was just wading through some tickets targetting 4.4, and I was wondering whether this might be merged in time for it, so that I can prepare my host environment accordingly?

Thank you for caring. The plan is to merge this early in a release cycle, so that translators have as much time as we can give them to unfuzzy tons of strings.
So no, it won’t get merged in 4.4. Best case, during this dev cycle, I’ll prepare branches that can be merged soon after 4.4 is released.

#18 Updated by intrigeri 2020-03-08 13:44:02

  • Description updated

#19 Updated by intrigeri 2020-03-08 14:32:39

  • Status changed from Confirmed to In Progress

Applied in changeset commit:tails|74286e277285c1e2e1a7bd0065981b00affec152.

#20 Updated by intrigeri 2020-03-08 14:40:45

  • Description updated
  • Feature Branch set to tails.git:feature/17005-po4a-0.55, puppet-tails.git:feature/17005-po4a-0.55

#21 Updated by CyrilBrulebois 2020-03-12 09:55:57

  • Target version changed from Tails_4.4 to Tails_4.5

#22 Updated by intrigeri 2020-03-14 11:16:49

  • Target version changed from Tails_4.5 to Tails_4.6

That’s postponed to early-post-4.5 days, to avoid fiddling with the stable branch at a time when our RMs need to see it change as little as possible.

#23 Updated by hans 2020-03-24 15:56:08

FYI we are currently in the process of porting f-droid.org to build entirely using Debian/buster packages, including po4a. There were some Markdown bugs fixed recently, post 0.55, that are very much worth having. I recommend using at least 0.58 from git. We have one last bug open, after that I’m going to push for a 0.58 release which can then go to buster-backports. This is the bug in question: https://github.com/mquinson/po4a/issues/196

These are the bugs that were fixed:

#24 Updated by intrigeri 2020-03-25 10:23:09

Hi @hans,

> There were some Markdown bugs fixed recently, post 0.55, that are very much worth having. I recommend using at least 0.58 from git. We have one last bug open, after that I’m going to push for a 0.58 release which can then go to buster-backports. This is the bug in question: https://github.com/mquinson/po4a/issues/196
>
> These are the bugs that were fixed: […]

Great, thanks for the pointers!
I’m glad we’re on the same boat here, even if we recently were not able to jump on other boats that would have been nice to share :)

These bug fixes indeed look potentially useful. On the Tails side, the improvements we care about most are in 0.55 so I think I’ll first go through this upgrade (which is basically ready to go), and cross the 0.58+ bridge in another iteration, once 0.58 is in buster-backports and all relevant Tails persons run Buster or newer.

#25 Updated by hans 2020-03-25 11:23:01

From what I’ve seen, adding the step to 0.55 in between will create new issues that you will have to migrate from, while jumping straight to 0.58 looks like the same amount of work as 0.55.

#26 Updated by intrigeri 2020-03-26 08:12:49

Hi @hans,

> From what I’ve seen, adding the step to 0.55 in between will create new issues that you will have to migrate from,

Thank you. This makes me very curious!
Are you referring to regressions in 0.55 compared to what we’re currently using (0.47)?
Or anything else?

#27 Updated by hans 2020-03-26 08:48:26

I wouldn’t call them regressions, but https://github.com/mquinson/po4a/issues/194 is a good example. Basically, using 0.55 will give your translators lots of

```bash

and

```

strings to translate, while 0.58 will not. Going from 0.55 to 0.58 would then remove all those as strings.

#28 Updated by intrigeri 2020-03-27 09:12:59

Hi @hans,

hans wrote:
> I wouldn’t call them regressions, but https://github.com/mquinson/po4a/issues/194 is a good example. Basically, using 0.55 will give your translators lots of

```bash

and

```

strings to translate, while 0.58 will not.

Thank you. Luckily (for the matter at hand), we don’t use code fences.

I’ll come back to it in a few weeks and we’ll see what’s the 0.58 status and whether we still need to support Stretch at that time.

#29 Updated by hans 2020-04-01 20:27:31

FYI, there has been a bunch of other work on smoothing out the process of using the `po4a` command to do all the syncing. That new work means f-droid.org no longer needs a custom script to run `po4a`. The maintainer is soliciting feedback on improving the po4a.conf format, in case you are interested: https://lists.po4a.org/archives/list/devel@lists.po4a.org/thread/NUVSGUGKTYDTA66RSMHGE6ZZH3Q2CMNY/

#30 Updated by intrigeri 2020-04-02 07:44:56

> FYI, there has been a bunch of other work on smoothing out the process of using the `po4a` command to do all the syncing. That new work means f-droid.org no longer needs a custom script to run `po4a`. The maintainer is soliciting feedback on improving the po4a.conf format, […]

Thanks. Actually, we don’t use the po4a(1p) program. We use the ikiwiki po plugin, which itself uses the Locale::Po4a::* Perl libraries.

#31 Updated by intrigeri 2020-04-12 13:55:01

  • Description updated

#32 Updated by intrigeri 2020-04-14 14:04:23

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to anonym

@anonym, can you please review but not merge these branches?

I’d like to do the deployment (which has to be coordinated with various other things & people) tomorrow.

#33 Updated by anonym 2020-04-14 14:25:11

  • Assignee changed from anonym to intrigeri

Code and doc changes seem correct (not tested, though), and I cannot find any places referring to po4a that you missed, so LGTM!

As for the changes in the PO files, oops, lots of new fuzzy lines with trivial fixes (almost 1500 occurrences AFAICT) so it sure does look like this will add lots of busy work for our translators. :/ I wonder, are these ones “trivial” enough to fix automatically with some sed-fu or similar? Or perhaps I’m exaggerating how annoying this will be for translators?

#34 Updated by intrigeri 2020-04-14 14:36:16

Hi @anonym,

> As for the changes in the PO files, oops, lots of new fuzzy lines with trivial fixes (almost 1500 occurrences AFAICT) so it sure does look like this will add lots of busy work for our translators. :/

Right.

The end-goal here is to make translators’ life better on the long term: the new po4a extracts strings in a way that’s easier to translate, and harder to make mistakes about (e.g. the infamous inconsistent newline at end of string error we get spammed with regularly).

Initially, the plan here was to fix the problem you’re mentioning, to avoid making translators’ life worse on the short term.
But when fixing this problem was part of this issue, for months nobody on the FT volunteered to tackle it.
So in the end I gave up and decided that the long-term savings (for translators) were worth the short-term annoyance (for translators too), and that the perfect was the enemy of the good. It’s a trade-off.

> I wonder, are these ones “trivial” enough to fix automatically with some sed-fu or similar?

I considered this. I believe the difficult part is not to remove the now-extraneous \n. It’s to decide whether after doing so, it’s OK to drop the fuzzy marker: to do so, one has to check if the string became fuzzy solely because of the po4a upgrade. So one needs to essentially interface with Git-diffs-of-PO-files. That does not sound trivial to me.

#35 Updated by anonym 2020-04-14 18:38:41

intrigeri wrote:
> So in the end I gave up and decided that the long-term savings (for translators) were worth the short-term annoyance (for translators too), and that the perfect was the enemy of the good. It’s a trade-off.

Makes perfect sense! Still, do you think it would be worth me spending an hour on it tomorrow to look for a non-perfect solution that cheaply/safely kills a large amount of them?

BTW, why is po4a (0.47-2) in in our (base branches’) APT suites? In Bug #17572 I noticed this, and speculated it was because we wanted to make this desired version easily available for our translators that do their work inside Tails, otherwise they would have to add stretch APT sources themselves. Any clue?

Any way, it seems to me that we definitely should remove po4a 0.47-2 from our (base branches’) APT suites as part of moving to po4a 0.55-1 (i.e. resolving this ticket), right?

#36 Updated by intrigeri 2020-04-15 07:17:16

  • Description updated

#37 Updated by intrigeri 2020-04-15 07:39:39

  • Description updated

#38 Updated by intrigeri 2020-04-15 07:59:44

  • Description updated

#39 Updated by intrigeri 2020-04-15 08:04:01

  • blocked by Bug #17619: update_weblate_components.py broken due to unclean local Git state added

#40 Updated by intrigeri 2020-04-15 08:17:24

Hi @anonym,

> Makes perfect sense! Still, do you think it would be worth me spending an hour on it tomorrow to look for a non-perfect solution that cheaply/safely kills a large amount of them?

I think so, yes, as long as we don’t unduly remove fuzzy markers that were there before the post-po4a-upgrade PO files refresh.
I’ll happily review such a branch.
Please base this work on our current master branch (that had the po4a upgrade merged in already).

> BTW, why is po4a (0.47-2) in in our (base branches’) APT suites? In Bug #17572 I noticed this, and speculated it was because we wanted to make this desired version easily available for our translators that do their work inside Tails, otherwise they would have to add stretch APT sources themselves. Any clue?

It was mostly for technical writers (AFAIK, no regular+active translator works in Git anymore).

> Any way, it seems to me that we definitely should remove po4a 0.47-2 from our (base branches’) APT suites as part of moving to po4a 0.55-1 (i.e. resolving this ticket), right?

This would be good. Can you please handle this on Bug #17572?

FWIW, I don’t understand the “definitely” part: it sounds like there will be problems if we don’t remove it. I don’t know what these problems are, besides having unused cruft in our APT suites, which AFAICT, is closer to the normal/usual situation than to an exceptional situation that definitely needs solving.

#41 Updated by intrigeri 2020-04-15 08:17:59

  • Description updated

#42 Updated by intrigeri 2020-04-15 08:19:10

  • Status changed from Needs Validation to In Progress

Status update tl;dr (details in the issue description): deployed! Only remaining bits are Weblate related; I’ll do them once Bug #17619 is done.

#43 Updated by anonym 2020-04-15 08:47:42

intrigeri wrote:
> Hi @anonym,
>
> > Makes perfect sense! Still, do you think it would be worth me spending an hour on it tomorrow to look for a non-perfect solution that cheaply/safely kills a large amount of them?
>
> I think so, yes, as long as we don’t unduly remove fuzzy markers that were there before the post-po4a-upgrade PO files refresh.
> I’ll happily review such a branch.
> Please base this work on our current master branch (that had the po4a upgrade merged in already).

Ok, will do a very careful attempt! Stay tuned!

> > Any way, it seems to me that we definitely should remove po4a 0.47-2 from our (base branches’) APT suites as part of moving to po4a 0.55-1 (i.e. resolving this ticket), right?
>
> This would be good. Can you please handle this on Bug #17572?

“Definitely!” >:)

> FWIW, I don’t understand the “definitely” part: it sounds like there will be problems if we don’t remove it. I don’t know what these problems are, besides having unused cruft in our APT suites, which AFAICT, is closer to the normal/usual situation than to an exceptional situation that definitely needs solving.

Indeed, not sure why I felt the need to make such a strong claim. I had no adverse consequences in mind, just “let’s properly clean up”.

#44 Updated by intrigeri 2020-04-15 10:12:46

> Indeed, not sure why I felt the need to make such a strong claim. I had no adverse consequences in mind, just “let’s properly clean up”.

Thank you, @anonym, for the clarification. We’re on the same page :)

#45 Updated by anonym 2020-04-15 11:07:18

  • Feature Branch changed from tails.git:feature/17005-po4a-0.55, puppet-tails.git:feature/17005-po4a-0.55 to tails.git:feature/17005-po4a-0.55 and doc/17005-unfuzzy, puppet-tails.git:feature/17005-po4a-0.55

I managed to unfuzzy 1386 strings! Please have a look at the branch: doc/17005-unfuzzy

#46 Updated by intrigeri 2020-04-15 14:09:25

> I managed to unfuzzy 1386 strings! Please have a look at the branch: doc/17005-unfuzzy

Merged, thanks!

#47 Updated by intrigeri 2020-04-29 13:31:53

  • blocks deleted (Bug #17619: update_weblate_components.py broken due to unclean local Git state)

#48 Updated by intrigeri 2020-04-29 13:33:44

  • Status changed from In Progress to Resolved

2 weeks later, I’ll stop waiting for the Weblate blocker.