Tails

#5 Updated by intrigeri 2019-03-18 12:35:27

> We currently have many languages or languages variants that are poorly or not translated at all (either in Debian and GNOME in general or for our internal tools). It’s the case for example of “Chinese, mandarin” and probably many others.

I’m fine with hiding options that would result in most of the desktop being in English, at least when there’s a close enough alternative that’s better translated (e.g. a close variant of the same language). When there’s no such alternate option, I’m worried that removing the option implies the user will have no choice but using English, and I’m not sure if it’s better or worse than, say, 90% English and 10% in their preferred language; I guess it’s a matter of “at least some bits are translated and thus easier to understand i.e. better than nothing” vs. “context/languages switching has a cost and is evil”.

> If we base ourselves on the PO files for our internal tools, we might be able to automatically generate a list of languages during the build. Making sure that our internal tools are well translated in a given language before listing it sounds like a good criteria too.

I’d be curious to see what this heuristics yields and whether the resulting list indeed would be likely to help users pick an option that will work better for them.

#7 Updated by sajolida 2019-03-21 13:29:09

Ok, so what’s the next step here?

Is it to build a table with:

• Language name, all the ones we have in Tails Greeter
• Percentage of translation of: core GNOME? all packages included in Tails?
• Percentage of translation of Tails software

So we can have concrete examples?

I would know how to compute the percentage of translations of Tails software but not the rest.

#14 Updated by sajolida 2019-08-11 18:48:55

Here is the list. I used as PO files:

• greeter.git:po/
• liveusb-creator.git:po/
• tails.git:po/
• persistence-setup.git:po/
• whisperback.git:po/

The script is in ux.git:greeter/percentage-translated.rb.

The result is:

de           98.6
he           98.4
es           98.4
ca           98.4
tr           98.4
ka           98.4
ru           98.4
it           98.4
da           98.4
ro           98.4
zh_CN        98.4
is           98.3
pt_PT        98.3
fr           98.3
es_AR        98.2
en_GB        98.0
pl           98.0
ga           98.0
sv           97.8
el           97.4
zh_TW        93.4
fa           85.4
pt_BR        85.4
bn           85.2
bn_BD        84.1
pt           75.4
hr_HR        75.0
id           71.5
cs           71.1
nb           70.8
ko           70.5
bg           70.4
fr_CA        70.2
lv           70.2
lt           69.4
vi           68.3
uk           64.3
nl           61.6
hu           61.3
fi           56.5
ar           54.1
sq           49.8
zh_HK        49.0
sk_SK        49.0
ms_MY        45.2
sk           44.0
sl_SI        43.7
az           42.9
km           42.4
hr           37.5
ja           36.6
nl_BE        31.7
eu           31.5
nn           30.7
si_LK        30.2
gl           28.1
th           24.1
et           23.9
sr           21.8
lb           16.7
en           11.4
pl_PL         9.0
ta            8.1
hy            8.1
cy            8.0
ru@petr1708   7.2
nds           4.3
sr@latin      4.1
bs            3.3
ast           3.3
pa            3.3
bn_IN         2.1
ms            0.1
zh            0.0

I’d be interested in seeing a list of all current locale listed in Tails Greeter and study a bit the difference. I briefly tried to understand how the list was built in the code of Tails Greeter but failed.

Is it listing all locales in /usr/share/locale/?

#16 Updated by sajolida 2019-08-11 21:47:17

We currently have 74 languages with at least some translation of our custom programs.

The full list of locales in /usr/share/locale/, which I suspect is the base for the listing in Tails Greeter, has 268 locales.

By only listing in Tails Greeter the locales for which we have at least 1 string of custom software translated, we would reduce the list of options by a factor of 4. Pretty good already!

Once our custom software are all in the main git repo (Feature #7036), I guess that it will become even easier to compute this list at build time. For the time being, we could limit ourselves to the list of languages that are already in tails.git:po/, if that’s easier.

I compared this list with the list of languages that have at least 1 string of custom software translated. This time ignoring liveusb-creator.git because some translations are probably inherited from upstream and might not even be relevant in the context of Tails. The missing languages are the following (ie. the languages that are in either greeter.git, persistence-setup.git, or whisperback.git):

ar
en
eu
gl
hr
hu
hy
ja
nl
nl_BE
pl_PL
ru@petr1708
si_LK
sk
ta
th

From these:

• hr, pl_PL, and sk have another locale from the same language but another territory in tails.git.
• Only ja, hu, and ar are translated at more than 25% across all repositories.

So, if my simplification is worth it in terms of integration code, I propose to only list in Tails Greeter, the languages that have a PO file in tails.git:po/. To add a new language in Tails Greeter, people should start translating for this language in tails.git.

@intrigeri: What do you think?

#17 Updated by intrigeri 2019-08-12 13:28:17

> I’d be interested in seeing a list of all current locale listed in Tails Greeter and study a bit the difference. I briefly tried to understand how the list was built in the code of Tails Greeter but failed. Is it listing all locales in /usr/share/locale/?

It’s generated from /usr/share/i18n/SUPPORTED, which I guess is the list of locales supported by Debian.

#18 Updated by intrigeri 2019-08-14 07:59:37

> So, if my simplification is worth it in terms of integration code,

It would be easier to code, indeed. In any case, I would not want us to spend time now on writing code that will be obsoleted by Feature #7036.

> I propose to only list in Tails Greeter, the languages that have a PO file in tails.git:po/. To add a new language in Tails Greeter, people should start translating for this language in tails.git.

Currently tails.git:po/ only has languages for which 100% of these strings are translated (but emma peel proposed to change this on Bug #16774). Also, the RM often has to remove invalid PO files from there during the release process, e.g. in May I removed: ar, fa, hu, ja, kk, nl, sk. The risk of such removal happening and its problematic impact will only increase with Feature #7036. That’s surely another problem, which we should probably think about somewhere else (and possibly as part of Feature #7036), but if we’re talking about “for the time being” we must consider this problem as very much alive.

So in the current state of things, your proposal is equivalent to: “in order to offer a language in Tails Greeter, people must translate 100% of the tails-misc resource on Transifex without doing any mistake that would make the resulting PO file invalid”. This seems a very high bar to me. For example, with this heuristic, we would have removed the Japanese language option from the Greeter since 3.13.2, even though the GNOME desktop/apps and Tor Browser are almost entirely translated, which feels like a bad move to me.

Sorry for being a killjoy here, but I prefer giving you this info ASAP instead of waiting until I make time to reply with more constructive ideas.

#19 Updated by sajolida 2019-08-16 17:24:29

> It’s generated from /usr/share/i18n/SUPPORTED, which I guess is the list of locales supported by Debian.

Thanks!

So that would be a list of 284 locales (instead of 268 as I thought
initially):

sed -r 's/^([^\.@ ]+).+/\1/' /usr/share/i18n/SUPPORTED | sort -u | wc -l

#20 Updated by sajolida 2019-08-16 17:49:04

> Currently tails.git:po/ only has languages for which 100% of these strings are translated (but emma peel proposed to change this on Bug #16774).

I see something different in tails.git. For example:

• po/zh.po is translated at 0% and after a quick scan I don’t think
  that it has ever seen a translation.
• po/cy.po is translated at 22, pt at 38, es_AR at 99%, etc.

But it sounds like a good idea anyway to block this ticket on Bug #16774 and
there seems to be a solution for Bug #16744 already :)

> Also, the RM often has to remove invalid PO files from there during the release process, e.g. in May I removed: ar, fa, hu, ja, kk, nl, sk. The risk of such removal happening and its problematic impact will only increase with Feature #7036. That’s surely another problem, which we should probably think about somewhere else (and possibly as part of Feature #7036), but if we’re talking about “for the time being” we must consider this problem as very much alive.

Do you mean that if a glitch is introduced in fr.po between 3.15 and
3.16, we won’t have any French translation of everything in po/tails.pot
in 3.16 and later until the glitch is fixed?

If so, could we consider asking RMs to instead either:

• Remove the msgstr with the glitch but keep the rest of fr.po
• Roll back fr.po to its version prior to the glitch

#22 Updated by intrigeri 2019-08-17 16:08:21

>> Currently tails.git:po/ only has languages for which 100% of these strings are translated (but emma peel proposed to change this on Bug #16774).

> I see something different in tails.git. For example:

> * po/zh.po is translated at 0% and after a quick scan I don’t think that it has ever seen a translation.

That one does not come from Transifex. It’s a leftover from some old stuff of ours ⇒ deleted!

> * po/cy.po is translated at 22, pt at 38, es_AR at 99%, etc.

I suspect these ones are all caused by a bug in the script Tor runs to import translations from Transifex into Git. On top of that, our own import-translations script is not deleting PO files that were removed from the Tor Git repo we copy them from. So there are two bugs that make the current behaviour not match what the intent is/was.

>> Also, the RM often has to remove invalid PO files from there during the release process, e.g. in May I removed: ar, fa, hu, ja, kk, nl, sk. The risk of such removal happening and its problematic impact will only increase with Feature #7036. That’s surely another problem, which we should probably think about somewhere else (and possibly as part of Feature #7036), but if we’re talking about “for the time being” we must consider this problem as very much alive.

> Do you mean that if a glitch is introduced in fr.po between 3.15 and 3.16, we won’t have any French translation of everything in po/tails.pot in 3.16 and later until the glitch is fixed?

Exactly.

> If so, could we consider asking RMs to instead either:

> * Remove the msgstr with the glitch but keep the rest of fr.po

I’d rather not to.

> * Roll back fr.po to its version prior to the glitch

Sure, that’s cheap ⇒ done with commit:256a340a6627c9d1d3009bd96300bc88aa7c129b.

#23 Updated by sajolida 2019-08-18 08:40:28

>> * Roll back fr.po to its version prior to the glitch
>
> Sure, that’s cheap ⇒ done with commit:256a340a6627c9d1d3009bd96300bc88aa7c129b.

Cool!

So the next steps are:

• Work on Bug #16774, which includes agreeing on a translation threshold
  level for inclusion of languages.
• Finish agreeing on the mechanism I’m proposing here: include in Tails
  Greeter only the locale that have a translation file in tails.git:po.

#24 Updated by intrigeri 2019-08-19 11:25:03

> So the next steps are:

> * Work on Bug #16774, which includes agreeing on a translation threshold level for inclusion of languages.

Done, left to be implemented (emmapeel for the Tor bits + myself for the Tails ones).

> * Finish agreeing on the mechanism I’m proposing here: include in Tails Greeter only the locale that have a translation file in tails.git:po.

I agree with the general idea but I think some refining is needed.

As data points, with the threshold we picked, in the current state of the tails-misc resource:

• we would have: ca, cs, de, el, es_AR, es, fi, fr, ga, he, hu, it, km, lt, pt_BR, pt_PT, ro, sv, tr, zh_CN
• we would lose: az, bg, bn_BD, bn, cy, da, en_GB, fa, fr_CA, hr_HR, id, is, ka, ko, lv, ms_MY, nb, nn, pl, pt, ru, sk_SK, sl_SI, sq, sr, uk, vi, zh_HK, zh_TW

This will change over time, as we merge custom software into tails.git.

This is related to the discussion we had on Feature #15807, where we agreed about this list of tier-1 supported languages:

• Arabic - AR
• German - DE
• English - EN
• Spanish - ES
• Farsi - FA
• French - FR
• Hindi
• Indonesian
• Italian - IT
• Portuguese - PT-BR
• Russian - RU
• Turkish - TR
• Simplified Chinese - zh-CN

It would not make sense to me to hide, from the Greeter, languages that are on this list, and for which we ship language packs etc. So even if we don’t have (enough reviewed) translations for these languages, IMO we should always propose them in the Greeter. In the current state of our translations, it makes a difference for Arabic, Hindi, Indonesian, and Russian.

Finally, I’m worried about the UX churn aspect of “include in Tails Greeter only the locale that have a translation file in tails.git:po”. The extreme (admittedly theoretical) example is: if my language is translated around 25% on Transifex, it could be proposed in Tails 4.0, not proposed in 4.1, proposed again in 4.2, etc. More generally, training users to choose their preferred language in the Greeter, and then dropping it, may feel like a UX regression to them: even if our custom software is not translated well enough anymore for us to ship the corresponding translations, it may be that GNOME and Tor Browser are well translated into that language. So what about this: we use the mechanism you propose (+ tier-1 supported languages) to add languages to the Greeter, but not to remove them immediately as soon as they drop below the threshold. Instead, we could remove non-tier-1 languages from the Greeter only once we’ve not had PO files in tails.git:po for them since 6-12 months. The mechanism we have to update these PO files could update whatever file will be used to generate the list of languages visible in the Greeter, and in there, maintain a “last seen” timestamp; then when we generate the list of languages visible in the Greeter, we ignore those for which the “last seen” timestamp is too old, unless it’s a tier-1 supported language.

What do you think?

#25 Updated by sajolida 2019-08-22 17:50:28

> IMO we should always propose [tier-1 languages] in the Greeter.

+1

> we use the mechanism you propose (+ tier-1 supported languages) to add languages to the Greeter

+1

> Instead, we could remove non-tier-1 languages from the Greeter only once we’ve not had PO files in tails.git:po for them since 6-12 months.

Why not. I believe that my proposal to use a 0% threshold on
Bug #16774#note-16 would achieve the same in practice.

#26 Updated by intrigeri 2019-08-23 08:21:21

>> Instead, we could remove non-tier-1 languages from the Greeter only once we’ve not had PO files in tails.git:po for them since 6-12 months.

> Why not. I believe that my proposal to use a 0% threshold on Bug #16774#note-16 would achieve the same in practice.

I understand this reasoning relies on the implicit assumption that with:

• X = the probability that a given language’s percentage of reviewed strings varies between zero and a strictly positive number
• Y = the probability that a given language’s percentage of reviewed strings varies around 25% (sometimes strictly less, sometimes more)

… X << Y, and X is sufficiently small that we can live with the risk of problematic UX consequences.

Intuitively, this sounds reasonable to me: I guess there’s little chances that translators translate & review precisely the strings that are going to be obsoleted soon after, and only these ones, hence dropping to 0%, getting removed, then they translate & review a couple strings again and their language comes back.

To sum up, wrt. removing languages from the Greeter, we have three options:

• A. Only propose languages that have a PO file in tails.git + tier-1 languages, period. This embeds a removal mechanism, by definition.
• B. Propose languages that have a PO file in tails.git + tier-1 languages + any language that was proposed in our last release. And then, manually clean up “any language that was proposed in our last release” from time to time.
• C. Same as B except we automate the cleaning process as I’ve suggested.

I understand you’re in favour of option A, and skip the added implementation complexity of options B and C, on the grounds that we can assume option A won’t cause too much flapping (languages added/removed/added/etc.) that would be detrimental to users. I’m fine with it. If we notice it causes trouble later, we can come back to the drawing board. Deal?

#27 Updated by sajolida 2019-08-27 15:21:45

> I understand you’re in favour of option A

I’m in favor of any of A, B, or C and will let me beloved coders choose
the one they prefer implementing, especially since “if we notice it
causes trouble later, we can come back to the drawing board” :)

#36 Updated by segfault 2019-09-23 20:59:04

> I would also like to curate the list of keyboard layouts, which is huge and, since based on Buster, contains things like “Czech Slovak and German” - which is the first result when you filter the layouts for “german” (there is also “Czech” and “German”, so I really don’t see the point of this layout).

See Feature #17089.

#51 Updated by segfault 2019-09-30 22:55:10

intrigeri wrote:
> > I curated config/chroot_local-includes/usr/share/tails-greeter/default_langcodes according to Option A in the description.
>
> Awesome!
>
> I realize I’ve been unclear: the idea was to propose tier-1 languages + languages that have a PO file in tails.git at the time the Tails image is built. Hard-coding this list based on today’s state of translations has two problems:
>
> * We’re in a time of flux: we just changed the criteria for inclusion of PO files in tails.git and I expect this will affect the list of languages that make it there quite a bit in the next few months, as translators realize what the updated expectations are (tl;dr: reviewed translations).
> * Sooner or later, some of the languages you’ve hard-coded won’t meet the criteria that lead us to choose “PO file in tails.git” as a classifier anymore. And vice versa. So we would need a process to regularly update the list of languages. I’d rather automate it :)
>
> Does this make sense to you?
> Do you want to implement the automation I’m proposing?

Sure, makes sense and I would like to implement it. But the translations in po/ are mostly only for a language, not a full locale - and I don’t know how to automatically get a locale from a language code without implementing a manual mapping from language code to country. I searched quite a lot but couldn’t find anything. Neither the Python locale module nor GNOME seems to support that.

If we can’t do this automatically and have to maintain a mapping from language to locale, I don’t think that would be much better than the manually maintained list of locales I added.

#52 Updated by intrigeri 2019-10-02 07:30:52

Hi!

> Sure, makes sense and I would like to implement it.

:)))

> But the translations in po/ are mostly only for a language, not a full locale - and I don’t know how to automatically get a locale from a language code without implementing a manual mapping from language code to country.

The options I can think of are:

• If we have a LL.po file, then we display all LL_* locales in the Greeter.
  • Pros: probably straightforward to implement with basic regexp skills, no ongoing maintenance cost.
  • Cons: we don’t curate the list of languages as much as we would like to.
• Decide ourselves manually which is the primary country for language “LL”, that is, which region should be picked when one chooses “LL”. This is the “manual mapping from language code to country”.
  • Pros:
    • We FTBFS when we have no mapping for a given newly introduced language, which avoids the risk that one forgets to add it to the list of locales.
    • Given this is a political decision, it’s good that we have control over it (for example, currently for Portuguese we default to Portugal, while there are way more people speaking Portuguese in Brazil than in Portugal; defaulting to Portugal reinforces colonialist heritage, which IMO is not ideal ethically speaking).
  • Cons: whenever a new LL.po appears, the RM has to make this decision and add a line to a config file. This happens in the critical path to a release, so it’s stressful, which is not the ideal context to make good decisions.
• Have some code decide which is the primary country for language “LL”, that is, which region should be picked when one chooses “LL”. I understand this is what you researched and did not find a solution for. But the current Greeter apparently is able to make this decision already: for example, if I choose “Deutsch (German)” without specifying a region, the Greeter somehow decides that I want “Deutsch - Deutschland (German - Germany)”. Is there any reason why we can’t reuse the same code here?
  • Pros: we already have most of the needed code; no ongoing maintenance cost
  • Cons: we don’t get (for free) manual control over such political decisions

> If we can’t do this automatically and have to maintain a mapping from language to locale, I don’t see how that would be any better than the manually maintained list of locales I added.

That’s the 2nd option I’ve described above. IMO it’s still better than a manually maintained list of locales, because:

• If a new LL.po appears, it’s obvious that an action is needed: we FTBFS. While with the manually maintained list of locales, we rely on the RM to not skip a manual step of the release process. Experience shows that RMs are prone to skipping random steps, be it because of a mistaken judgement call (“probably not needed this time”) or of a mere oversight.
• If a LL.po is removed, then the corresponding language gets automatically removed from the Greeter, which is the idea here. Of course, we could ask the RM to do it, but again, that’s unreliable.

#53 Updated by segfault 2019-10-02 18:06:20

intrigeri wrote:
> > But the translations in po/ are mostly only for a language, not a full locale - and I don’t know how to automatically get a locale from a language code without implementing a manual mapping from language code to country.
>
> The options I can think of are:
>
> * If we have a LL.po file, then we display all LL_* locales in the Greeter.
> Pros: probably straightforward to implement with basic regexp skills, no ongoing maintenance cost.
> Cons: we don’t curate the list of languages as much as we would like to.

This could actually be a good solution. According to the ticket’s description, the motivation for curating the list is to remove languages which are not well translated. This proposal wouldn’t add languages which are not well translated, because all the LL_* locales should be translated by the LL.po file.

> * Decide ourselves manually which is the primary country for language “LL”, that is, which region should be picked when one chooses “LL”. This is the “manual mapping from language code to country”.
> Pros:
> * We FTBFS when we have no mapping for a given newly introduced language, which avoids the risk that one forgets to add it to the list of locales.
> * Given this is a political decision, it’s good that we have control over it (for example, currently for Portuguese we default to Portugal, while there are way more people speaking Portuguese in Brazil than in Portugal; defaulting to Portugal reinforces colonialist heritage, which IMO is not ideal ethically speaking).
> Cons: whenever a new LL.po appears, the RM has to make this decision and add a line to a config file. This happens in the critical path to a release, so it’s stressful, which is not the ideal context to make good decisions.
> * Have some code decide which is the primary country for language “LL”, that is, which region should be picked when one chooses “LL”. I understand this is what you researched and did not find a solution for. But the current Greeter apparently is able to make this decision already: for example, if I choose “Deutsch (German)” without specifying a region, the Greeter somehow decides that I want “Deutsch - Deutschland (German - Germany)”. Is there any reason why we can’t reuse the same code here?
> Pros: we already have most of the needed code; no ongoing maintenance cost
> Cons: we don’t get (for free) manual control over such political decisions

The greeter’s algorithm for this is not very sophisticated. It just tests whether there is locale with the same country code as the language code, for example pt_PT, and then uses that. Else it uses the first pt_* locale in its list of locales - and this list begins with a hardcoded list of default locales (config/chroot_local-includes/usr/share/tails-greeter/default_langcodes).

So this algorithm has the issue that (1.) it doesn’t allow good manual control over political decisions, and (2.) it also requires a manually maintained list of locales to give useful results (without this list of defaults, for example the locale chosen for en would be en_AG).

If you want to take a look, the function which implements this is get_default_locale in config/chroot_local-includes/usr/lib/python3/dist-packages/tailsgreeter/language.py.

> > If we can’t do this automatically and have to maintain a mapping from language to locale, I don’t see how that would be any better than the manually maintained list of locales I added.
>
> That’s the 2nd option I’ve described above. IMO it’s still better than a manually maintained list of locales, because:
>
> * If a new LL.po appears, it’s obvious that an action is needed: we FTBFS. While with the manually maintained list of locales, we rely on the RM to not skip a manual step of the release process. Experience shows that RMs are prone to skipping random steps, be it because of a mistaken judgement call (“probably not needed this time”) or of a mere oversight.
> * If a LL.po is removed, then the corresponding language gets automatically removed from the Greeter, which is the idea here. Of course, we could ask the RM to do it, but again, that’s unreliable.

I agree. That’s why I edited the quoted sentence after posting the comment :)

My favorite is your 1st option. But if we decide against that, I propose that we do the following:

• Manually maintain a mapping from language code to country, and use the already existing list in config/chroot_local-includes/usr/share/tails-greeter/default_langcodes as the basis.
• During build time, generate a list from tier-1 languages and languages in po/
• In the greeter, compile the list of locales from the list above. For each entry in that list which does not already contain a country code, use the mapping to obtain the country code.
• We can still decide whether we want to FTBFS if the language list contains a language which has no mapping, or whether the Greeter should choose a country instead.

#54 Updated by intrigeri 2019-10-02 19:01:41

> My favorite is your 1st option.

I say go for it: it should be cheap and is a reasonable step in the spirit of the goals set on this ticket. Also, it has a good chance to land into 4.0.
Then we can ask sajolida to try it out and tell us if he would like us to curate the list further, possibly in some future release, and then we can consider the marginal cost/benefit of this next iteration. Unless, of course, he states his opinion here before you code the 1st option.

#55 Updated by segfault 2019-10-03 00:05:49

I implemented it, but IMO we still have the need for specifying a default country per language:

In the Greeter the languages are grouped into one entry. If the user clicks on that entry instead of expanding it, the Greeter uses the algorithm I described above to choose the default language. But since we don’t add the config/chroot_local-includes/usr/share/tails-greeter/default_langcodes to the beginning of the supported locales list, it will only try to use the country with the same code as the language, and if there is none it will choose the first country alphabetically.

The effect is that, when clicking English, en_AB is used (Antigua and Barbuda), when clicking Arabic it’s ar_AE (United Arab Emirates), which both seem like bad choices.

To fix this, we could use the old default_langcodes file as the basis for our new locales list. I will implement something.

#56 Updated by intrigeri 2019-10-03 04:40:57

> To fix this, we could use the old default_langcodes file as the basis for our new locales list (locales). I will implement something.

Yeah, in doubt, I think it’s OK to lean towards preserving the current behavior, that does not seem problematic in itself.

#62 Updated by intrigeri 2019-10-04 06:43:31

> Status changed from Needs Validation to In Progress

Was this intentional?

#67 Updated by intrigeri 2019-10-05 07:46:12

I’ll run the full test suite locally, as some of our automated tests that exercise the Greeter are tagged fragile, and thus not run for this branch on Jenkins.

#69 Updated by sajolida 2019-10-05 15:33:10

Yay! You rock!