Bug #17554

lint_po bails out when fed locales with an "@" e.g. ru@petr1708

Added by CyrilBrulebois 2020-03-26 00:13:55 . Updated 2020-04-12 12:48:20 .

Status:
Resolved
Priority:
Normal
Assignee:
intrigeri
Category:
Internationalization
Target version:
Start date:
Due date:
% Done:

100%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Syncing PO/POT files from Transifex for Whisperback leads to simply two updates:

kibi@armor:~/work/clients/tails/whisperback.git$ git diff --stat
 po/fr.po |  4 ++--
 po/hu.po | 72 ++++++++++++++++++++++++++++++++++++++++++------------------------------
 2 files changed, 44 insertions(+), 32 deletions(-)

but lint_po is quite unhappy about the language suffix:

kibi@armor:~/work/clients/tails/whisperback.git$ "${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/lint_po
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/kibi/work/clients/tails/release/release-checkout/submodules/jenkins-tools/slaves/lint_po", line 231, in check_po_file
    issues = poFile.i18nspector(my_env)
  File "/home/kibi/work/clients/tails/release/release-checkout/submodules/jenkins-tools/slaves/lint_po", line 186, in i18nspector
    self.lang_without_script(), self.fname]
  File "/home/kibi/work/clients/tails/release/release-checkout/submodules/jenkins-tools/slaves/lint_po", line 118, in lang_without_script
    lang = self.lang()
  File "/home/kibi/work/clients/tails/release/release-checkout/submodules/jenkins-tools/slaves/lint_po", line 113, in lang
    raise NoLanguageError(self.fname)
NoLanguageError: Can't detect expect file suffix .XX.po for 'po/ru@petr1708.po'.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/kibi/work/clients/tails/release/release-checkout/submodules/jenkins-tools/slaves/lint_po", line 351, in <module>
    main(logging.getLogger())
  File "/home/kibi/work/clients/tails/release/release-checkout/submodules/jenkins-tools/slaves/lint_po", line 334, in main
    files, 10):
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 314, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 695, in next
    raise value
__main__.NoLanguageError: Can't detect expect file suffix .XX.po for 'po/ru@petr1708.po'.

That doesn’t seem to be quite new, the only recent fixes (going back to June 2019) seem to be about Python versions instead.

(BTW: s/expect/expected/)

So I’ve decided to proceed with the rest of the release process for that component, but I suppose we should fix the jenkins-tools submodule at some point to avoid further loss of time in the future?


Subtasks


Related issues

Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by intrigeri 2020-03-26 08:39:41

#2 Updated by intrigeri 2020-03-26 08:40:39

  • Subject changed from lint_po vs. whisperback vs. ru@petr1708 to lint_po bails out when fed locales with an "@" e.g. ru@petr1708
  • Category set to Internationalization

#3 Updated by intrigeri 2020-03-26 08:41:00

  • Assignee set to hefee

Hi @hefee, could you please take a look?

For the curious, “petr1708” refers to:

This locale is unlikely to ever be made available via our Welcome Screen to Tails users, but I see a bunch of other locales that have an at-sign in their name, so perhaps it’s worth supporting this case?
Alternatively, we could filter those locales out when we import files from Transifex (import-translations script).

#4 Updated by hefee 2020-03-26 17:05:19

  • Status changed from Confirmed to Needs Validation
  • Assignee changed from hefee to intrigeri

intrigeri wrote:
> This locale is unlikely to ever be made available via our Welcome Screen to Tails users, but I see a bunch of other locales that have an at-sign in their name, so perhaps it’s worth supporting this case?

As the “normal” po files in Tails had no `@` sign, I ignored this case, when writing the script.

The manpage of i18nspector tells us, that is only likes only `language_territory`. I scanned through the code of `lint_po` and there are only two places to add the support this. `lang()` - should return the language with all modifiers. `lang_without_script()` - shoul return the language code like `i18nspector` likes it.

I post a diff, as creating a branch etc. for a submodule takes more time and needs interaction of you anyways.

<code class="diff">
diff --git a/slaves/lint_po b/slaves/lint_po
index 9cc6432..c7db193 100755
--- a/slaves/lint_po
+++ b/slaves/lint_po
@@ -108,7 +108,7 @@ class PoFile:
         """@returns: language of filename, possibly ending with an underscore
         followed by the territory or script"""
         name = os.path.basename(self.fname)
-        m = re.match(r"^(?:[^.].*\.)?(?P<lang>[A-Za-z_]+)\.po$", name)
+        m = re.match(r"^(?:[^.].*\.)?(?P<lang>[A-Za-z0-9_@]+)\.po$", name)
         if not m:
             raise NoLanguageError(self.fname)
         return m.group("lang")
@@ -119,7 +119,7 @@ class PoFile:
         # a script suffix (e.g. "_Latn") starts with an underscore,
         # followed by the script name in title case, which we
         # approximate as an upper case letter followed by a lower case one
-        m = re.match(r"(?P<lang_without_script>.*)_[A-Z][a-z][A-Za-z]*$", lang)
+        m = re.match(r"^(?P<lang_without_script>.*?)(_[A-Z][a-z][A-Za-z]*)?(@[A-Za-z0-9]+)?$", lang)
         if m:
             return m.group("lang_without_script")
         else:
diff --git a/slaves/test/testLintPo.py b/slaves/test/testLintPo.py
index 3ed3ead..bba5455 100644
--- a/slaves/test/testLintPo.py
+++ b/slaves/test/testLintPo.py
@@ -97,6 +97,7 @@ class TestCheckPo(unittest.TestCase):
         self.assertEqual(lint_po.PoFile("index.de.po").lang(), "de")
         self.assertEqual(lint_po.PoFile("x/a/a.fb.xx.po").lang(), "xx")
         self.assertEqual(lint_po.PoFile("a.po").lang(), "a")
+        self.assertEqual(lint_po.PoFile("ru@petr1708.po").lang(), "ru@petr1708")

         _p = lint_po.PoFile(".de.po")
         with self.assertRaises(lint_po.NoLanguageError, msg=_p.fname) as e:
@@ -117,6 +118,8 @@ class TestCheckPo(unittest.TestCase):
         self.assertEqual(lint_po.PoFile("index.de_DE.po").lang_without_script(), "de_DE")
         self.assertEqual(lint_po.PoFile("index.sr_Latn.po").lang_without_script(), "sr")
         self.assertEqual(lint_po.PoFile("index.sr_LAtn.po").lang_without_script(), "sr_LAtn")
+        self.assertEqual(lint_po.PoFile("ru@petr1708.po").lang_without_script(), "ru")
+        self.assertEqual(lint_po.PoFile("ru_RU@petr1708.po").lang_without_script(), "ru_RU")

     def test_needs_rewrap(self):
         with lint_po.pofile_readonly(os.path.join(DIRNAME, "checkPo/length")) as poFile:
</code>

#5 Updated by intrigeri 2020-03-27 10:25:37

  • Target version set to Tails_4.6

#6 Updated by intrigeri 2020-04-12 12:48:20

  • Status changed from Needs Validation to Resolved
  • % Done changed from 0 to 100

Applied in changeset commit:tails|20ab48eecaa45bdb36757ce078c40f2633c455be.