Bug #16436

Make the setup production-ready, adjust resource allocation and optimize stuff if needed

Added by Anonymous 2019-02-07 15:13:10 . Updated 2019-09-13 09:14:39 .

Status:
Resolved
Priority:
Normal
Assignee:
groente
Category:
Target version:
Start date:
2017-02-11
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Translation Platform
Deliverable for:
309

Description

Sysadmin work

Budgeted time: 16h

Suggestions:

  • Give the VM one more vCPU? Sometimes the webapp (run within the apache process) and mysqld are competing for resources and there are timeouts in the web UI.
  • Optimize MariaDB config according to the size and usage of our database? Sometimes MariaDB is eating a full CPU core for a long time, while there’s quite some free RAM. Presumably bigger caches etc. might help.

Subtasks

Feature #12220: Set up monitoring for weblate Resolved

100

Feature #15359: List parts of code/packages/configs to be puppetized for translation platform & its clone Resolved

0

Feature #16135: Consider filtering abusive requests to Weblate upstream Resolved

100

Feature #16450: Use puppet logic to not copy language list Resolved

100

Bug #16525: translation-server: logrotate logs of weblate script. Resolved

100


History

#1 Updated by intrigeri 2019-03-12 17:28:21

  • Estimated time changed from 160 h to 16 h

#2 Updated by intrigeri 2019-06-29 08:14:11

  • Description updated

#3 Updated by hefee 2019-07-05 15:37:52

#4 Updated by Anonymous 2019-07-19 11:27:41

  • Status changed from Confirmed to Needs Validation

@groente: 10 days ago you said you’d like to add Apache mod_security before we send out the call for testing. As said over email (July 9th 2019), I am preparing to send out this call today. Is there a chance you’ll still implement this, or should I ignore this for now?

#5 Updated by groente 2019-07-19 12:37:18

u wrote:
> @groente: 10 days ago you said you’d like to add Apache mod_security before we send out the call for testing. As said over email (July 9th 2019), I am preparing to send out this call today. Is there a chance you’ll still implement this, or should I ignore this for now?

working on this right now!

#6 Updated by Anonymous 2019-07-19 16:44:00

#7 Updated by groente 2019-08-01 09:27:23

u, hefee: Sofar, I’ve:

- deployed mod_sec

- given the host extra diskspace for the audit log

- given the host an extra vcpu

- adjusted the mariadb configuration according to mysqltuner recommendations
- given the host considerable extra memory to deal with the new mariadb config

Please let me know if you encounter any problems due to lack of system resources and/or 403 errors due to mod_sec false positives.

#8 Updated by intrigeri 2019-08-01 14:25:52

Hi @groente!

> Please let me know if you encounter any problems due to lack of system resources and/or 403 errors due to mod_sec false positives.

I’m seeing 403 for every Weblate page I’m trying now (using Tor Browser, FWIW). It did work just fine until a dozen minutes ago or so.
Is there any additional info I shall provide so you can debug & fix this?

#9 Updated by groente 2019-08-01 14:29:37

intrigeri wrote:
> Hi @groente!
>
> I’m seeing 403 for every Weblate page I’m trying now (using Tor Browser, FWIW). It did work just fine until a dozen minutes ago or so.
> Is there any additional info I shall provide so you can debug & fix this?

timestamps and url’s please! thank you!

#10 Updated by intrigeri 2019-08-01 14:48:23

> timestamps and url’s please! thank you!

https://translate.tails.boum.org/translate/tails/wikisrcsandboxpo/fr/?type=suggestions at 14:47 and 14:48 UTC today.

#11 Updated by zen 2019-08-01 19:16:17

groente wrote:
> u, hefee: Sofar, I’ve:
>
> - deployed mod_sec
> - given the host extra diskspace for the audit log

Maybe you are aware, but I will note here that we currently have about 8G of mod_security logs and that is giving a disk warning in the monitoring system. Logs are being rotated, though.

#12 Updated by groente 2019-08-01 19:24:00

zen wrote:
> groente wrote:
> > u, hefee: Sofar, I’ve:
> >
> > - deployed mod_sec
> > - given the host extra diskspace for the audit log
>
> Maybe you are aware, but I will note here that we currently have about 8G of mod_security logs and that is giving a disk warning in the monitoring system. Logs are being rotated, though.

I am aware, yes, lateron I’ll readjust the rather excessive logging, but currently in the testing phase, it’s spitting out useful info to adjust our ruleset more effectively.

#13 Updated by groente 2019-08-15 11:45:20

  • Assignee changed from groente to hefee

Hey Hefee,

If you don’t mind, I’d like your thoughts on the following:

I’ve been seeing these kind of error messages pop up on a few occassions in the logfiles:

Unable to connect to WSGI daemon process ‘translate.tails.boum.org’ on ‘/var/run/apache2/wsgi.21667.5.1.sock’ after multiple attempts as listener backlog limit was exceeded.

Do you think this is related to https://github.com/GrahamDumpleton/mod_wsgi/issues/181 ? To get the mod_wsgi version which supports the new settings, we would have to upgrade to buster…

#14 Updated by hefee 2019-08-15 22:45:28

  • Assignee changed from hefee to groente

groente wrote:
> Hey Hefee,
>
> If you don’t mind, I’d like your thoughts on the following:
>
> I’ve been seeing these kind of error messages pop up on a few occasions in the logfiles:
>
> Unable to connect to WSGI daemon process ‘translate.tails.boum.org’ on ‘/var/run/apache2/wsgi.21667.5.1.sock’ after multiple attempts as listener backlog limit was exceeded.
>
> Do you think this is related to https://github.com/GrahamDumpleton/mod_wsgi/issues/181 ?

I don’t think it is related.

What may be worth looking into is using uwsgi to ship Weblate and not use mod_wsgi from Apache. I switched several years ago from mod_wsgi to uwsgi and I never regret it, when it comes to ship python apps. As uswgi is performing very well and it is not bundled with Apache, so it is easy to use Nginx etc as Webserver. We need https://httpd.apache.org/docs/2.4/en/mod/mod_proxy_uwsgi.html as the uwsgi protocol is faster than the http, or use an http socket from uwsgi.

an initial ini file for uwsgi:


# cat /etc/uwsgi/apps-enabled/translate.tails.boum.org.ini 
[uwsgi]
master = true
touch-reload = %p  # this is a nice feature, if you touch this file, the app will be restarted automatically with the new settings

chown-socket = www-data:www-data

uid = weblate
gid = weblate

plugin = python3
chdir = <%= @code_git_checkout %>
file = <%= @code_git_checkout %>/weblate/wsgi.py

vacuum=True
max-requests=5000

and that creates the uwsgi-socket /run/uwsgi/app/translate.tails.boum.org/socket where Apache can connect to.

#15 Updated by intrigeri 2019-08-16 07:40:58

> What may be worth looking into is using uwsgi to ship Weblate and not use mod_wsgi from Apache. I switched several years ago from mod_wsgi to uwsgi and I never regret it, when it comes to ship python apps. As uswgi is performing very well and it is not bundled with Apache, so it is easy to use Nginx etc as Webserver.

Our Weblate is already sitting behind a nginx reverse proxy so if we’re going the uwsgi road, it would be tempting to drop the additional hop via Apache. But we rely on Apache mod_security. ModSecurity supports nginx but AFAICT that’s not in Debian, which is a show-stopper. So my understanding is that the only workable option is replacing mod_wsgi with uwsgi, i.e. we would do nginx → Apache → uwsgi, instead of the current nginx → Apache → mod_wsgi.

#16 Updated by groente 2019-08-16 19:59:04

  • Status changed from Needs Validation to In Progress

thanks for the feedback! i haven’t actually seen the wsgi error again since the last parameter adjustments i made a few days ago. if it keeps behaving, i’ll just stick with the setup as is, else i’ll look into mod_proxy_uwsgi.

#17 Updated by CyrilBrulebois 2019-09-05 00:05:29

  • Target version changed from Tails_3.16 to Tails_3.17

#18 Updated by intrigeri 2019-09-12 09:04:25

I think we’re good here and could close this ticket. Objections, anyone?

#19 Updated by intrigeri 2019-09-12 14:25:09

  • Target version changed from Tails_3.17 to Tails_4.0

#20 Updated by Anonymous 2019-09-13 09:14:39

  • Status changed from In Progress to Resolved

Yay!