Bug #15481

Weblate bash UTF8 broken

Added by Anonymous 2018-03-30 12:58:38 . Updated 2019-06-27 17:16:34 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Start date:
2018-03-30
Due date:
% Done:

0%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:
309

Description

When doing a git log -p on the weblate VM as user weblate I get broken UTF-8 output.
When doing the same on my computer, things work out correctly, thus I suspect a problem on the VM.
I think it would be useful to fix this because we will be using scripts to verify that po files are indeed in UTF-8 and we want to be able to let a human confirm things if this is not the case. Hence my request to please make it work?

 #~ msgstr ""
-#~ "<a href=\"https://www.lush.com/\" title=\"Lush Digital Fund - 11 000<E2><82><AC>\">"
-#~ "[[!img lib/partners/lush.png link=\"no\"]]</a>"
+#~ "<a href=\"https://www.lush.com/\" title=\"Lush Digital Fund - 11 000<E2><82><AC>"
+#~ "\">[[!img lib/partners/lush.png link=\"no\"]]</a>"

Subtasks

History

#1 Updated by groente 2018-04-13 11:29:19

  • Assignee deleted (groente)
  • QA Check set to Info Needed

Can you provide exact steps to reproduce? UTF-8 seems to render fine here..

#2 Updated by Anonymous 2018-04-24 15:18:56

  • Assignee set to groente
  • QA Check deleted (Info Needed)

This happens in Weblate’s VCS folder. I can correctly not connect so I cannot really tell you where it is. From the top of my head it’s in in /var/lib/weblate/vcs/tails or something like that.

There, as user weblate when doing a git fetch tails master and then “git diff master —” I get this broken UTF8. Note that on other machines I connect to this works fine - so I think it’s something on the VM that is not quite right.

#3 Updated by Anonymous 2018-04-24 15:19:54

u wrote:
> This happens in Weblate’s VCS folder. I can correctly not connect so I cannot really tell you where it is. From the top of my head it’s in in /var/lib/weblate/vcs/tails or something like that.
>
> There, as user weblate when doing a git fetch tails master and then “git diff master —” I get this broken UTF8. Note that on other machines I connect to this works fine - so I think it’s something on the VM that is not quite right.

As written previously you can see this when doing git log -p as user weblate too - easier than the previous approach.

#4 Updated by groente 2018-04-26 12:37:54

  • Assignee deleted (groente)
  • QA Check set to Info Needed

That’s odd, I see UTF8 rendered just fine in the git log (running as user weblate), also the euro symbol after Lush Digital Fund that you mentioned renders fine. Are there any particularities concerning your terminal? Running screen perhaps or so?

#5 Updated by emmapeel 2018-04-27 07:38:20

I can reproduce this without screen, using sudo -u weblate git log -p from /var/lib/weblate/repositories/vcs/tails/index

#6 Updated by Anonymous 2018-05-29 09:21:50

  • Assignee set to groente
  • QA Check deleted (Info Needed)

groente wrote:
> That’s odd, I see UTF8 rendered just fine in the git log (running as user weblate), also the euro symbol after Lush Digital Fund that you mentioned renders fine. Are there any particularities concerning your terminal? Running screen perhaps or so?

No. I use gnome-terminal on a more a less default Debian Stretch machine.
Can you try from a Tails instance for example?
It looks like emmapeel can also reproduce this problem.

#7 Updated by intrigeri 2018-05-29 10:28:04

I’d be curious to see the output of these commands in the shell that exposes this problem:

locale
echo $TERM

#8 Updated by Anonymous 2018-05-29 10:51:38

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = “de_DE.UTF-8”
are supported and installed on your system.
perl: warning: Falling back to the standard locale (“C”).

#9 Updated by Anonymous 2018-05-29 10:51:49

intrigeri wrote:
> I’d be curious to see the output of these commands in the shell that exposes this problem:
>
> […]

locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE=“de_DE.UTF-8”
LC_NUMERIC=“de_DE.UTF-8”
LC_TIME=“de_DE.UTF-8”
LC_COLLATE=“de_DE.UTF-8”
LC_MONETARY=“de_DE.UTF-8”
LC_MESSAGES=“de_DE.UTF-8”
LC_PAPER=“de_DE.UTF-8”
LC_NAME=“de_DE.UTF-8”
LC_ADDRESS=“de_DE.UTF-8”
LC_TELEPHONE=“de_DE.UTF-8”
LC_MEASUREMENT=“de_DE.UTF-8”
LC_IDENTIFICATION=“de_DE.UTF-8”
LC_ALL=

#10 Updated by intrigeri 2018-05-29 11:03:00

So you’re using a locale that the system does not support.
I think it’s because your SSH client forwards these environment variables (SendEnv LANG LC_* or similar).
I guess it would be fixed by tweaking SendEnv for this host to avoid forwarding these settings :)

#11 Updated by intrigeri 2018-05-29 11:03:15

  • Assignee deleted (groente)

#12 Updated by Anonymous 2018-05-29 13:26:55

  • Status changed from Confirmed to Resolved

SendEnv cannot be overridden. The server could instead AcceptEnv to not accept client variables.

I will tweak my bashrc instead to forward LANG=C.UTF-8 - just in case anybody else needs to fix this, they can maybe find my comment :)

#13 Updated by Anonymous 2019-02-07 15:31:24

#14 Updated by intrigeri 2019-06-27 17:16:34

  • Assignee deleted ()