Tails

Today on misc.lizard there were 3 instances of that cronjob running: 1 current + 2 that started respectively on April 22 and 23. The 2 oldest ones were all stuck waiting for a wget command to complete.

This might explain some of the leftover temporary directories we’ve seen recently (Feature #17679): one hypothesis is that these directories were not leftovers of “old instances of check-mirrors.rb was killed and did not clean up after itself”, but rather a consequence of “weeks-old check-mirrors.rb instances were still running”.

Either way, I think we need 2 things:

• sysadmin: a lock on the cronjob (like we have for most similar cronjobs already) and possibly a timeout (see e.g. timeout(1) which seems to fit the bill), in the cronjob. In case the timeout expires or the lock can’t be taken by a new cronjob instance, I think the mirrors team should be made aware.
• check-mirrors.rb itself: IMO the script should not wait indefinitely for a wget command to complete; it should run it under some sort of timeout.

I’m filing a single issue for both suggestions to limit overhead. I think they can be implemented in parallel.