Bug #17343
[Weblate Database] switch from utf8 to utf8mb4 charset.
0%
Description
We are facing utf-8 encoing issues with the mariadb database.
those issues point to
https://github.com/WeblateOrg/weblate/issues/2218
https://github.com/WeblateOrg/weblate/issues/1054
and those pointing to the documentation:
https://docs.weblate.org/en/weblate-3.5.1/admin/install.html#mysql-or-mariadb
the proposed solution is to switch to `utf8mb4`.
The database already using `utf8mb4` but not all tables. At least `trans_component` is using `utf8` instead of `utf8mb4`.
2019-12-12 15:11:40,251 - UWC - root(ERROR): Adding new component triggered by 'wiki/src/news/celebrating_10_years/mafe.de.po' failed:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.5/dist-packages/django/db/backends/mysql/base.py", line 101, in execute
return self.cursor.execute(query, args)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 226, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorvalue
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 217, in execute
res = self._query(query)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 378, in _query
rowcount = self._do_query(q)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 341, in _do_query
db.query(q)
File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 280, in query
_mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x8D\\xA0 (...' for column 'source' at row 1")
Subtasks
History
#1 Updated by hefee 2019-12-13 21:30:05
After we updated the database, we need to update the components:
2019-12-12 15:10:52,418 - UWC - root(INFO): Updated remote ffd5682881192702cd01e9986053b41a149eda82..54e2b58a7bb8276cb84de83e6b5eed938ee37cd1#2 Updated by zen 2019-12-13 22:22:43
Here’s MariaDB’s doc on Setting Character Sets and Collations.
#3 Updated by hefee 2019-12-13 22:32:31
- Assignee set to zen
- Priority changed from Normal to Elevated
- Target version set to Tails_4.2
#4 Updated by CyrilBrulebois 2020-01-07 18:00:44
- Target version changed from Tails_4.2 to Tails_4.3
#5 Updated by anonym 2020-02-11 15:26:37
- Target version changed from Tails_4.3 to Tails_4.4
#6 Updated by zen 2020-02-21 18:53:09
If we do Bug #16994 first, we might not need to do this one.
#7 Updated by CyrilBrulebois 2020-03-12 09:56:02
- Target version changed from Tails_4.4 to Tails_4.5
#8 Updated by zen 2020-03-16 20:38:18
- Due date set to 2020-03-18
- Start date set to 2020-03-18
I plan to do this together with the change from MariaDB to PostgreSQL on 03/18/2020.
#9 Updated by zen 2020-03-26 17:25:01
- Due date changed from 2020-03-18 to 2020-04-01
- Start date changed from 2020-03-18 to 2020-04-01
Pandemic context made me had to do something else on the day I had organized to do this, so I’m rescheduling for April 1st 2020.
#10 Updated by CyrilBrulebois 2020-04-07 17:05:20
- Target version changed from Tails_4.5 to Tails_4.6
#11 Updated by zen 2020-04-22 12:28:02
- Status changed from Confirmed to Rejected
This is not needed now that we are using PostgreSQL (see Bug #16994).
#12 Updated by hefee 2020-04-22 14:30:50
I also double checked, that the file that triggered the problem initially, is now added to Weblate and that the string is displayed correctly: