Bug #15679

update puppet-tails to have scripts in weblate vm

Added by hefee 2018-06-20 15:13:46 . Updated 2019-06-27 17:16:36 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Start date:
2018-06-20
Due date:
% Done:

0%

Feature Branch:
https://salsa.debian.org/hefee/puppet-tails/tree/feature/integrate-weblate
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:
309

Description

This is a first bunch of changes of the weblate vm to have the script we wrot to be available in weblate vm.

  • cron.sh is needed because otherwise the git repo can be blocked by other cron jobs, thats why we should streamline the functions into one script
  • save-suggests.py is for creating a copy with all suggestions in weblate for the stating wiki
  • update-weblate-components.py should replace the cron job, but currently it is not tested, that’s why I only added it and do not update cron.sh

On production weblate the different cronjobs block each other and the .git/lock is created. When we tried to update the cronjobs via crontab -e some service added the removed cronjobs again (is this the puppet client?) Can we disable this feature for some time to be able to test our new update-weblate-components.py script?

Subtasks

Related issues

Related to Tails - Bug #15568: Script create staging wiki with suggestions Resolved 2018-05-03
Related to Tails - Bug #15185: Automatically update Weblate components on mdwn file add/rename/delete in tails.bo git Resolved 2019-02-27
Related to Tails - Bug #15356: Automate adding new components to Weblate Duplicate 2018-03-01
Related to Tails - Feature #15190: Cronjobs for translate.lizard are not running Resolved 2018-01-17

History

#1 Updated by hefee 2018-06-20 15:14:53

  • related to Bug #15568: Script create staging wiki with suggestions added

#2 Updated by hefee 2018-06-20 15:15:26

  • related to Bug #15185: Automatically update Weblate components on mdwn file add/rename/delete in tails.bo git added

#3 Updated by hefee 2018-06-20 15:15:38

  • related to Bug #15356: Automate adding new components to Weblate added

#4 Updated by emmapeel 2018-06-21 07:41:24

  • related to Feature #15190: Cronjobs for translate.lizard are not running added

#5 Updated by Anonymous 2018-07-04 12:35:15

  • Assignee set to groente

Can you review this please?

#6 Updated by mercedes508 2018-07-09 07:21:43

  • Status changed from New to Confirmed

#7 Updated by hefee 2018-07-17 18:28:15

Updated the scripts to use a logging config file so that we can adjust what we want to log and what is printed to commandline/triggers a failure for cron.

#8 Updated by Anonymous 2018-08-19 16:16:58

@groente. may you please review this? Thanks!

#9 Updated by groente 2018-09-26 11:31:13

  • Assignee changed from groente to hefee
  • QA Check changed from Ready for QA to Pass

I’ve incorporated your scripts in puppet-tails and deployed them on translate. I did make a few minor changes (e.g., writing to ${mutable_data_dir}/repositories/home instead of ${mutable_data_dir/home) and I’m not sure what the purpose of the empty config files in the files/scripts directory is? Can you clarify that last bit and let me know whether everything works according to plan on translate now? Thank you!

#10 Updated by hefee 2018-09-26 21:05:59

  • Assignee changed from hefee to groente

Thanks a lot for finding time to work on this. Where can I see what you’ll want to push?

groente wrote:
> I’ve incorporated your scripts in puppet-tails and deployed them on translate. I did make a few minor changes (e.g., writing to ${mutable_data_dir}/repositories/home instead of ${mutable_data_dir/home)

It was intended to move the scripts from ${mutable_data_dir}/repositories/home -> ${mutable_data_dir/home} as it has nothing to do with repositories (IMO).

> and I’m not sure what the purpose of the empty config files in the files/scripts directory is? Can you clarify that last bit and let me know whether everything works according to plan on translate now? Thank you!

Well the file saveSuggestions.conf shouldn’t be empty. Just copy the updateWeblateComponents.conf to saveSuggestions.conf and change the filepath:
/var/lib/weblate/repositories/home/update.log -> /var/lib/weblate/repositories/home/saveSuggestions.log

(argh another repositories/home)

As I now have access to the translate I can fix the confs afterwards to have the correct logging, when you run puppet with my stuff. But I’m blocked from the whole git stuff, so having the files in place would help a lot.

#11 Updated by groente 2018-09-27 11:39:55

  • Assignee changed from groente to hefee
  • QA Check changed from Pass to Info Needed

hefee wrote:
> Thanks a lot for finding time to work on this. Where can I see what you’ll want to push?

You can see the changes I made in https://git.tails.boum.org/puppet-tails/

> It was intended to move the scripts from ${mutable_data_dir}/repositories/home -> ${mutable_data_dir/home} as it has nothing to do with repositories (IMO).

Ah, that kindof makes sense. I thought it prudent to stick to the existing directory structure, but indeed there seems to be little logic in placing ‘home’ inside ‘repositories’. Or is there perphaps any reason these scripts should reside within weblate’s data dir?

> > and I’m not sure what the purpose of the empty config files in the files/scripts directory is? Can you clarify that last bit and let me know whether everything works according to plan on translate now? Thank you!
>
> Well the file saveSuggestions.conf shouldn’t be empty. Just copy the updateWeblateComponents.conf to saveSuggestions.conf and change the filepath:
> /var/lib/weblate/repositories/home/update.log -> /var/lib/weblate/repositories/home/saveSuggestions.log

Ah, okay, done!

> As I now have access to the translate I can fix the confs afterwards to have the correct logging, when you run puppet with my stuff. But I’m blocked from the whole git stuff, so having the files in place would help a lot.

Okay, the files are now in placed, but mind you, they’re managed by puppet now, so any changes you make manually will be overwritten. If you want permanent changes to these files, please submit a patch and I’ll incorporate it in puppet-tails.

#12 Updated by hefee 2018-09-27 12:10:53

  • Assignee changed from hefee to groente
  • % Done changed from 0 to 80
  • QA Check changed from Info Needed to Pass

groente wrote:
> hefee wrote:
> > Thanks a lot for finding time to work on this. Where can I see what you’ll want to push?
>
> You can see the changes I made in https://git.tails.boum.org/puppet-tails/
>
>
> > It was intended to move the scripts from ${mutable_data_dir}/repositories/home -> ${mutable_data_dir/home} as it has nothing to do with repositories (IMO).
>
> Ah, that kindof makes sense. I thought it prudent to stick to the existing directory structure, but indeed there seems to be little logic in placing ‘home’ inside ‘repositories’. Or is there perphaps any reason these scripts should reside within weblate’s data dir?

well it was the only directory for weblate_admins to write to that’s why things where moved there. But any better place is welcomed.

> > As I now have access to the translate I can fix the confs afterwards to have the correct logging, when you run puppet with my stuff. But I’m blocked from the whole git stuff, so having the files in place would help a lot.
>
> Okay, the files are now in placed, but mind you, they’re managed by puppet now, so any changes you make manually will be overwritten. If you want permanent changes to these files, please submit a patch and I’ll incorporate it in puppet-tails.

How often puppet will overwrite stuff? Just to know how I could do testing…

Otherwise, we can mark this as resolved from my side, if everything works on the server.

#13 Updated by groente 2018-09-27 12:25:55

  • Assignee changed from groente to hefee
  • % Done changed from 80 to 0
  • QA Check changed from Pass to Info Needed

> > Ah, that kindof makes sense. I thought it prudent to stick to the existing directory structure, but indeed there seems to be little logic in placing ‘home’ inside ‘repositories’. Or is there perphaps any reason these scripts should reside within weblate’s data dir?
>
> well it was the only directory for weblate_admins to write to that’s why things where moved there. But any better place is welcomed.

Okay, shall i move everything from /var/lib/weblate/repositories/home to /var/lib/weblate/scripts ?

> How often puppet will overwrite stuff? Just to know how I could do testing…

Twice an hour, around 24 and 54.

#14 Updated by hefee 2018-10-03 13:03:35

  • Assignee changed from hefee to groente
  • QA Check changed from Info Needed to Dev Needed

groente wrote:
> > > Ah, that kindof makes sense. I thought it prudent to stick to the existing directory structure, but indeed there seems to be little logic in placing ‘home’ inside ‘repositories’. Or is there perphaps any reason these scripts should reside within weblate’s data dir?
> >
> > well it was the only directory for weblate_admins to write to that’s why things where moved there. But any better place is welcomed.
>
> Okay, shall i move everything from /var/lib/weblate/repositories/home to /var/lib/weblate/scripts ?

well the scripts, that are managed by puppet - yes. Don’t delete/move other stuff, I’ll take care of this.

> > How often puppet will overwrite stuff? Just to know how I could do testing…
>
> Twice an hour, around 24 and 54.

can you change this to just once a day? Than trying out things is easier, as I than have round 12h instead of just several minutes.

#15 Updated by groente 2018-10-08 10:38:39

  • Assignee changed from groente to hefee
  • QA Check changed from Dev Needed to Ready for QA

> > Okay, shall i move everything from /var/lib/weblate/repositories/home to /var/lib/weblate/scripts ?
>
> well the scripts, that are managed by puppet - yes. Don’t delete/move other stuff, I’ll take care of this.

Okay, the scripts have now been moved. I’ve also added a /var/lib/weblate/logs directory and set the config file to write its logs there.

> > > How often puppet will overwrite stuff? Just to know how I could do testing…
> >
> > Twice an hour, around 24 and 54.
>
> can you change this to just once a day? Than trying out things is easier, as I than have round 12h instead of just several minutes.

I’m afraid not, no. Feel free to send patches / merge requests whenever you want to include something in puppet, though. If you give me a shout on xmpp, we can coordinate the timing for patches to be rolled out, so testing goes a bit more smooth.

That said, I think it’s time to close this ticket, no?

#16 Updated by intrigeri 2018-10-08 11:37:47

>> > > How often puppet will overwrite stuff? Just to know how I could do testing…
>> >
>> > Twice an hour, around 24 and 54.
>>
>> can you change this to just once a day? Than trying out things is easier, as I than have round 12h instead of just several minutes.

> I’m afraid not, no. Feel free to send patches / merge requests whenever you want to include something in puppet, though. If you give me a shout on xmpp, we can coordinate the timing for patches to be rolled out, so testing goes a bit more smooth.

Or, ideally: test candidate changes on a local clone of the Weblate setup and not on the production setup :)

#17 Updated by hefee 2018-10-11 19:31:27

groente wrote:
> I’m afraid not, no. Feel free to send patches / merge requests whenever you want to include something in puppet, though. If you give me a shout on xmpp, we can coordinate the timing for patches to be rolled out, so testing goes a bit more smooth.

okay, we’ll coordinate via xmpp if i see things to test on the live setup

> That said, I think it’s time to close this ticket, no?

yes.

#18 Updated by hefee 2018-10-11 19:35:29

intrigeri wrote:
> > I’m afraid not, no. Feel free to send patches / merge requests whenever you want to include something in puppet, though. If you give me a shout on xmpp, we can coordinate the timing for patches to be rolled out, so testing goes a bit more smooth.
>
> Or, ideally: test candidate changes on a local clone of the Weblate setup and not on the production setup :)

Well the issues that needs interactions are broken merges between the two git repositories or other things, that are realy hard to simulate. Additionally I’m very unhappy that the translation server is called as “production” because from my point of view it is not ready and we have to test and make sure that things are working like expected. And this different views about the state of the translation server made it very hard to improve the situation. Hopefully we are now on the right track and can tackle the last bits and make it production ready.

#19 Updated by hefee 2018-10-11 19:36:06

  • Assignee changed from hefee to groente
  • QA Check changed from Ready for QA to Pass

#20 Updated by groente 2018-10-11 20:53:22

  • Status changed from Confirmed to Resolved

#21 Updated by intrigeri 2018-10-12 08:08:29

> Additionally I’m very unhappy that the translation server is called as “production” because from my point of view it is not ready and we have to test and make sure that things are working like expected. And this different views about the state of the translation server made it very hard to improve the situation.

Understood.

  • I’m not saying the currently running translation service is production-ready: I’m aware it’s not and I don’t want us to claim it is.
  • What I’m saying is that the VM this translation service is running on is a production one, not a development one, hence the way it’s managed. That’s the agreement that was reached with the sysadmin team before this project started. Hence, for example, our reluctance to make Puppet run less often and my repeated suggestions to test stuff elsewhere first (see the proposal drebs and I came up with in May). This agreement appears to be obsolete in practice but AFAIK it was never updated so it’s hard for me to know on which grounds decisions shall be made at the moment and clearly, I’m stuck to the previous formal agreement and have a hard time acknowledging it does not work anymore (sorry about that!). One alternate strategy could be to acknowledge that you folks need a dev VM (managed in a more relaxed / by hand way), find some suitable place for that VM (i.e. decide whether the current one is an OK place for it or find another location), make the setup production ready, Puppetize it, and finally deploy the production-ready service on translate.lizard. But I’m just guessing what your team’s needs might be and I’m obviously not the best placed to do that.

> Hopefully we are now on the right track and can tackle the last bits and make it production ready.

Yeah! \o/

#22 Updated by Anonymous 2019-02-07 15:36:02

#23 Updated by intrigeri 2019-06-27 17:16:36

  • Assignee deleted (groente)