Feature #15360

Refresh the Weblate staging website more often

Added by Anonymous 2018-03-02 11:12:53 . Updated 2019-07-02 08:57:44 .

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
Start date:
2018-03-02
Due date:
% Done:

0%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Translation Platform
Deliverable for:

Description

- find out how we could do it
- if it would be possible to do it only for one file in one language

currently, the build with all languages enabled takes ~25 minutes, so it’s much too long to ask for a rebuild of the entire wiki.

This is not part of the deliverable, but we would want that.


Subtasks


Related issues

Related to Tails - Feature #16872: Make it obvious to visitors of the staging website that it is not trustworthy Confirmed
Blocked by Tails - Feature #15080: Integrate the staging website with the interface Confirmed 2018-06-11

History

#1 Updated by Anonymous 2018-05-29 13:02:04

  • Parent task deleted (Feature #15077)
  • Deliverable for deleted (Sponsor_L)

I think this is out of scope for the grant and we can do that as a later improvement to the staging website.

#2 Updated by Anonymous 2018-05-29 13:02:12

#3 Updated by Anonymous 2018-05-29 13:06:41

#4 Updated by Anonymous 2018-05-29 13:06:47

#5 Updated by Anonymous 2018-06-12 14:29:01

Our current idea is to build the staging wiki once a day and I think this is sufficient for now.

#6 Updated by Anonymous 2018-11-02 19:57:33

  • Assignee set to emmapeel

hi emma, if you would like to look deeper into this, we could collect ideas on this ticket. But building once a day currently seems sufficient to me.

#7 Updated by hefee 2018-11-13 12:28:44

The the main website using the ikiwiki pingee plugin to trigger a rebuild:
https://git.tails.boum.org/puppet-tails/tree/files/gitolite/hooks/www_website_ping-post-update.hook

#8 Updated by intrigeri 2019-06-27 17:13:01

  • Affected tool set to Translation Platform

#9 Updated by intrigeri 2019-06-27 17:13:53

#10 Updated by intrigeri 2019-06-27 17:49:50

  • related to Feature #15080: Integrate the staging website with the interface added

#11 Updated by intrigeri 2019-06-27 17:55:20

  • Subject changed from Check for mechanisms that (safely) trigger staging builds accessible to all translators to Refresh the Weblate staging website more often
  • Category set to Infrastructure
  • Assignee deleted (emmapeel)
  • Type of work changed from Research to Sysadmin

We discussed this with @hefee and our current take on it:

  • Measure how long it takes to refresh (not rebuild) the staging website in common (not worst case) situations.
  • Assuming this refresh time is expressed in minutes rather than hours, run it as often as we can (without monopolizing a vCPU permanently), using a systemd timer (which gives us not only cron-like scheduling but also avoids running 2 instances of the refresh job at the same time).

In any case, this goes hand in hand with Feature #15080 (either we do both or we do none of them):

  • If the staging website for the page one is reviewing is not readily available from the Weblate interface, then chances are translators don’t use this staging website much (if at all).
  • If the staging website is lagging most of the time behind the changes one is reviewing, chances are that translators get used to ignore it because it’s not useful.

#12 Updated by intrigeri 2019-06-29 08:17:50

We currently don’t refresh, we always rebuild.

I’ve tried --refresh and that fails because save-suggestions.py does git clean -fdx, which deletes the files ikiwiki needs to know what to refresh, namely config/chroot_local-includes/usr/share/doc/tails/website/.ikiwiki/. Would git clean -fd suffice?

#13 Updated by hefee 2019-07-01 07:41:25

  • Assignee set to intrigeri

intrigeri wrote:
> I’ve tried --refresh and that fails because save-suggestions.py does git clean -fdx, which deletes the files ikiwiki needs to know what to refresh, namely config/chroot_local-includes/usr/share/doc/tails/website/.ikiwiki/. Would git clean -fd suffice?

I think that should be suffice. Go for it.

#14 Updated by intrigeri 2019-07-01 07:58:51

  • Status changed from Confirmed to In Progress
  • Target version set to Tails_3.16

> I think that should be suffice. Go for it.

Done in https://git.tails.boum.org/puppet-tails/commit/?id=1bfb6238b172ac33cdb6f6592dc4b72ce934a127. We’ll see how it goes tomorrow morning.

@hefee, please read the commit message, especially the part about POT files, and let me know if you can already predict trouble :)

#15 Updated by intrigeri 2019-07-02 08:57:44

  • Assignee deleted (intrigeri)
  • Target version deleted (Tails_3.16)

intrigeri wrote:
> * Measure how long it takes to refresh (not rebuild) the staging website in common (not worst case) situations.

It takes about 1 hour:

weblate@translate:~$ time /var/lib/weblate/scripts/suggestions-cronjob.sh

real    60m18.435s
user    52m40.796s
sys     0m15.492s

> * Assuming this refresh time is expressed in minutes rather than hours, run it as often as we can (without monopolizing a vCPU permanently), using a systemd timer (which gives us not only cron-like scheduling but also avoids running 2 instances of the refresh job at the same time).

I think it would be reasonable to run this every 4 hours.

I’m done with this initial analysis and will come back to it once we’ve discussed with our team how we prioritize this + Feature #15080.

#16 Updated by intrigeri 2019-09-13 09:46:24

  • related to Feature #16872: Make it obvious to visitors of the staging website that it is not trustworthy added

#17 Updated by intrigeri 2019-09-13 09:54:31

  • related to deleted (Feature #15080: Integrate the staging website with the interface)

#18 Updated by intrigeri 2019-09-13 09:54:41

  • blocked by Feature #15080: Integrate the staging website with the interface added