Feature #16545

Make contributors aware of stalled work regularly

Added by Anonymous 2019-03-07 14:10:52 . Updated 2019-11-17 05:27:39 .

Status:
Resolved
Priority:
Normal
Assignee:
intrigeri
Category:
Target version:
Start date:
2019-03-07
Due date:
% Done:

0%

Feature Branch:
tails.git:16455+triagingdoc
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Create automated emails every 45 days to individuals

We’ve created two Redmine views:

“Stalled: Unblock other people’s work”
https://redmine.tails.boum.org/code/projects/tails/issues?query_id=317
This view shows tickets assigned to a contributor, while the ticket status shows that someone else is waiting for their input.

“Stalled: WIP & not updated for a while”
https://redmine.tails.boum.org/code/projects/tails/issues?query_id=316
This view shows tickets assigned to a contributor, that are work in progress but have not seen activity since a while. Mostly tickets on which 80% of the work is done and that can either be finished or could be assigned to a relevant team to take over.

Using the YAML list (Feature #16544), send everyone on that list a link to this Redmine view + filtered for them,
asking them if they think they can realistically come back to it and finish the work in the next 6 months;
expected valid answers / possible actions are:
- yes ⇒ add suitable target version and when relevant, blocks the “Core work: $team in $quarter” ticket
- no ⇒ deassign yourself, then: if relevant for a core team, bring it to their attention; else, forget it, make some tea and have a bath :)


Files


Subtasks


Related issues

Blocked by Tails - Feature #16544: Create YAML of assignee names from Redmine view for ticket triaging Resolved 2019-03-07

History

#1 Updated by Anonymous 2019-03-07 14:11:12

  • blocked by Feature #16544: Create YAML of assignee names from Redmine view for ticket triaging added

#2 Updated by intrigeri 2019-03-10 16:36:49

  • Status changed from New to Confirmed

#3 Updated by intrigeri 2019-03-12 18:12:11

  • Description updated

#4 Updated by intrigeri 2019-03-13 13:27:22

  • Target version changed from Tails_3.15 to Tails_3.14

#5 Updated by intrigeri 2019-05-02 17:26:22

  • Target version changed from Tails_3.14 to Tails_3.15

I first need to implement the workflow change (dropping the “QA Check” field) I’ve proposed, then to update Enrico’s script accordingly.

#6 Updated by intrigeri 2019-07-06 17:32:23

  • Status changed from Confirmed to Needs Validation
  • Assignee deleted (intrigeri)

Hi @u!

I have something that works locally. All pieces are deployed in production already except the cronjob, because I’d like your feedback about the content of the email it’s going to send. My draft lives there: https://git.tails.boum.org/puppet-tails/tree/files/redmine/reminder/email_body

A few things that we did not consider initially IIRC, though, and for which I need your input (or consent, in some cases):

  • I’m not sure what the From field should be. Possibly you, as our ticket triager? Except this might stop working if your email server ever does DKIM/SPF/whatever. For now, I’ve set From: root$FQDN, which is not ideal. * Similarly, we need a To@ field makes it too likely that the message is classified as spam. Here as well, I’d be inclined to set your address there, and set all other recipients as Bcc. Shall I do that?
  • cron does not easily allow doing something every 45 days; the closest cheap options to that are: either every month or every 2 months; I think every 2 months is sufficient; what do you think?
  • Be it wrt. “stalled WIP” or “reviews waiting for a long time”, we have at least as many problematic tickets that this code will not identify as those it will point people to. These problematic tickets, that our setup will miss, are those which have a target version that’s been postponed after the N last releases by the Release Manager: every such Target version bump counts as an update so there’s no chance the ticket will ever be identitied as stalled. That of course does not imply that what we did is useless nor that we should block on this. But I’d like us to think this through and address it if we can do so easily. For example, we could do one pass every year or so on the tickets whose target version is the next Tails release, and whenever the target version was bumped by the RM since 6+ months after each release, with no update by the assignee, we drop the target version and paste a nice comment from a template. I’d be fine with doing the first pass in the next few months myself. Then our queries about stalled WIP and reviews will be able to notice those tickets, starting 6 months later. What do you think?

#7 Updated by Anonymous 2019-07-08 10:02:27

@intrigeri,

> I have something that works locally. All pieces are deployed in production already except the cronjob, because I’d like your feedback about the content of the email it’s going to send. My draft lives there: https://git.tails.boum.org/puppet-tails/tree/files/redmine/reminder/email_body

Nice!

> A few things that we did not consider initially IIRC, though, and for which I need your input (or consent, in some cases):
>
> * I’m not sure what the From field should be. Possibly you, as our ticket triager? Except this might stop working if your email server ever does DKIM/SPF/whatever. For now, I’ve set From: root$FQDN@, which is not ideal.

My address as from will not work, my mail server migrated last week to a new one and SPF/DKIM will be in place if it’s not already.

Either, we keep using root@FQDN, or we set up some sort of alias (that can forward emails to me). I’m not sure how to request such an alias.

> * Similarly, we need a To field makes it too likely that the message is classified as spam. Here as well, I’d be inclined to set your address there, and set all other recipients as Bcc. Shall I do that?

If I get it right, we do send out one email per person, is that correct? So why do addresses need a Bcc:, when they can have a clear To: field? Not sure why the one or the other would make spam detectors cough more or less.

> * cron does not easily allow doing something every 45 days; the closest cheap options to that are: either every month or every 2 months; I think every 2 months is sufficient; what do you think?

Agreed.

> * Be it wrt. “stalled WIP” or “reviews waiting for a long time”, we have at least as many problematic tickets that this code will not identify as those it will point people to. These problematic tickets, that our setup will miss, are those which have a target version that’s been postponed after the N last releases by the Release Manager: every such Target version bump counts as an update so there’s no chance the ticket will ever be identitied as stalled. That of course does not imply that what we did is useless nor that we should block on this. But I’d like us to think this through and address it if we can do so easily. For example, we could do one pass every year or so on the tickets whose target version is the next Tails release, and whenever the target version was bumped by the RM since 6+ months after each release, with no update by the assignee, we drop the target version and paste a nice comment from a template. I’d be fine with doing the first pass in the next few months myself. Then our queries about stalled WIP and reviews will be able to notice those tickets, starting 6 months later. What do you think?

That sounds like a very good plan. Should I ask enrico if he thinks this could be automated? It would avoid us a lot of boring and recurring work.

#8 Updated by Anonymous 2019-07-08 10:05:12

@intrigeri

> I have something that works locally. All pieces are deployed in production already except the cronjob, because I’d like your feedback about the content of the email it’s going to send. My draft lives there: https://git.tails.boum.org/puppet-tails/tree/files/redmine/reminder/email_body

I like the text and the signature :)

So we don’t include a list of ticket numbers in that email?

#9 Updated by Anonymous 2019-07-08 10:05:29

  • Status changed from Needs Validation to In Progress
  • Assignee set to intrigeri

#10 Updated by intrigeri 2019-07-09 10:22:31

> So we don’t include a list of ticket numbers in that email?

No, the plan was to do the cheapest possible thing, i.e. point folks to the place where they will find their list (after logging in).

#11 Updated by intrigeri 2019-07-09 10:53:54

Dear @u,

>> * I’m not sure what the From field should be. Possibly you, as our ticket triager? Except this might stop working if your email server ever does DKIM/SPF/whatever. For now, I’ve set From: root$FQDN@, which is not ideal.

> My address as from will not work, my mail server migrated last week to a new one and SPF/DKIM will be in place if it’s not already.

> Either, we keep using root@FQDN, or we set up some sort of alias (that can forward emails to me). I’m not sure how to request such an alias.

Thanks, that’s the info I needed. I’ll try to find & implement a cheap solution, which should be easier now that we host a MX.

>> * Similarly, we need a To field makes it too likely that the message is classified as spam. Here as well, I’d be inclined to set your address there, and set all other recipients as Bcc. Shall I do that?

> If I get it right, we do send out one email per person, is that correct?

Well, I did the cheapest thing i.e. send one single email to all affected folks. Indeed, sending one email per person would fix this problem. I’ll do that. Thanks!

>> * Be it wrt. “stalled WIP” or “reviews waiting for a long time”, we have at least as many problematic tickets that this code will not identify as those it will point people to. These problematic tickets, that our setup will miss, are those which have a target version that’s been postponed after the N last releases by the Release Manager: every such Target version bump counts as an update so there’s no chance the ticket will ever be identitied as stalled. That of course does not imply that what we did is useless nor that we should block on this. But I’d like us to think this through and address it if we can do so easily. For example, we could do one pass every year or so on the tickets whose target version is the next Tails release, and whenever the target version was bumped by the RM since 6+ months after each release, with no update by the assignee, we drop the target version and paste a nice comment from a template. I’d be fine with doing the first pass in the next few months myself. Then our queries about stalled WIP and reviews will be able to notice those tickets, starting 6 months later. What do you think?

> That sounds like a very good plan. Should I ask enrico if he thinks this could be automated? It would avoid us a lot of boring and recurring work.

I’d love to be surprised but I expect that:

  • It’s not cheap to specify the exact algorithm the automation should implement in all corner cases (e.g. what to do if the assignee was the RM back then?)
  • It’s not cheap to implement said specification (who was the RM back then?)
  • In quite a few cases, for human/social/accounting/management-ish reasons, doing the automated thing would cause all sorts of trouble, such as workers confusion (“so, what’s my deadline, in the end?”). So I think that someone will need to review all the resulting Redmine changes and occasionally revert some of them with personalized explanations/apologies. I’m not sure that all in all it’s better (for us and contributors) than doing it by hand in the first place.

Right now that’s 87 tickets we would need to go through for the initial pass. It’s indeed quite some work but the number should drop quickly, once we’ve dealt with the initial backlog and folks get used to a new, stricter meaning of “target version”. Given this and the fact I’m proposing to do it only once a year, I feel more optimistic than “lot of boring and recurring work”.

In any case, if we go for the automated way, I suspect we’ll need to go through a bunch of these tickets by hand first, in order to tell what the algorithm should be and provide examples of what we expect it to do.

Oh, but actually, what about this: the automation does not do any change on Redmine, but instead computes a list of tickets that are candidates for dropping target version? This relaxes quite a bit the need for the algorithm to be excellent and it addresses my concern about automatic changes being undesirable in some cases. It still saves quite some of the boring work and allows humans to focus on what we’re better at than computers. There’s some boring/repetitive work remaining but perhaps not much more than reviewing automatic changes?

#12 Updated by intrigeri 2019-07-09 11:48:38

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (intrigeri)
  • Target version changed from Tails_3.15 to Tails_3.16

I’ve implemented everything discussed above and enabled the cronjob, which should run every 2 months, on the 17th. Please let me know if something goes wrong (or if it does not run at all).

#13 Updated by zen 2019-07-17 13:21:52

  • related to #16887 added

#14 Updated by intrigeri 2019-07-18 14:02:55

A first batch of email was sent!

#15 Updated by Anonymous 2019-07-19 12:08:37

intrigeri wrote:
> A first batch of email was sent!

It worked :)

#16 Updated by Anonymous 2019-07-19 12:11:38

@intrigeri

> Right now that’s 87 tickets we would need to go through for the initial pass. It’s indeed quite some work but the number should drop quickly, once we’ve dealt with the initial backlog and folks get used to a new, stricter meaning of “target version”. Given this and the fact I’m proposing to do it only once a year, I feel more optimistic than “lot of boring and recurring work”.

Agreed.

> In any case, if we go for the automated way, I suspect we’ll need to go through a bunch of these tickets by hand first, in order to tell what the algorithm should be and provide examples of what we expect it to do.
>
> Oh, but actually, what about this: the automation does not do any change on Redmine, but instead computes a list of tickets that are candidates for dropping target version? This relaxes quite a bit the need for the algorithm to be excellent and it addresses my concern about automatic changes being undesirable in some cases. It still saves quite some of the boring work and allows humans to focus on what we’re better at than computers. There’s some boring/repetitive work remaining but perhaps not much more than reviewing automatic changes?

Yes, I don’t expect the algorithm to do anything besides computing the list.
If you think it would be doable and have fun doing it, I would say go ahead and clock as ticket triaging work.

#17 Updated by intrigeri 2019-08-30 16:42:07

I’ve written a template (attached) for tickets postponed many times. I’ll now go through all open 3.16 tickets, and:

  • If a ticket has been postponed more than N times (I’m not sure yet about N, I’ll adjust as I go through them), consider dropping the target version and pasting this template as a comment. As discussed above, this is not adequate in all cases, so I’ll have to draw the line on a case-by-case basis.
  • Take notes about whatever criteria I use to make the decision, to inform future discussions about whether we can automate this, and what the algorithm should be.

#18 Updated by intrigeri 2019-08-30 18:03:23

I’ve gone through 86 tickets that all have target version = 3.16. Most of them won’t be solved in time for 3.16, which will be released in 4 days. I did this:

As a result, there’s only 62 tickets left with a 3.16 target version. It’s not ideal, but that’s about 20 more tickets that now have a chance to show up in the stalled work views and trigger email notifications.

The criteria I’ve consciously used are (some of those I realized too late, and did not apply initially):

  • When the last action on the ticket by the assignee was performed. For example, if they did not update the ticket since 3 months or more, it suggests the target version should be dropped.
  • When the ticket was (re)assigned to the assignee. If the ticket landed on their plate recently, we should probably do nothing.
  • Whether there’s been progress but it’s not documented on the ticket. For example, sajolida did steady progress on Feature #16539 (see discussion on -project@) but the ticket does not reflect it. It’s not ideal but in this case, dropping the target version would be inadequate.
  • Who postponed the ticket recently. If the assignee did, well, it suggests they have it in mind and are updating their plans, which is good (note, however, that Some People® — sajolida and myself — are usually postponing their tickets themselves before the release, which skews things a bit ;). If a RM did, it suggests the opposite.
  • What the ticket state is. We’ve been focused primarily on “In Progress” and “Needs Validation” so far, so they are the best candidates for dropping target version. But when someone is holding a “Confirmed” ticket for 6+ months and it’s postponed every 6 weeks, possibly dropping the target version will help them take a step back and reconsider their commitment to it.
  • Whether it’s part of a sponsor deliverable.
    • If the official deadline is in the future, most of the time we don’t want to touch the Target version.
    • Else, if we’ve already delivered the work to the sponsor but the ticket is about a follow-up task: it depends™ :P No, seriously, this sort of situations is too tricky for me to try to formalize criteria by myself right now, so I’ll just ignore these tickets. In the past we’ve generally discussed them on a case-by-case basis, between the PM, accounting team, and other relevant stakeholders.
  • Whether there’s an assignee. If there’s none but the ticket was postponed a few times, then:
    • Either it’s important and it should be put on the plate of a core team.
    • Or it’s unimportant and the target version should be dropped: keeping it has little value but forces the RMs to do extra work. If someone wants to track those tickets, there surely are better ways.
  • Whether the ticket is blocked by anything (other tickets or what not) that’s out of the hands of the assignee.
  • How safe/privileged is the assignee’s position in Tails. I expect some of us to feel comfortable reverting my changes, for example if they’re used to using their own, personal definition of the “Target version” field, which does not match how we’ve documented we collectively use it. So I’ve been a bit bolder with dropping target versions for contributors who hold more power than average.

I propose that we:

  1. Do the same thing manually once or twice a year. It took me less than 2 hours, including the time to write this entire comment, including documenting the criteria I’ve used. I think that’s time well invested but I’m curious to see what feedback I get. It may be that doing it twice a year does not take much more time than once a year in total, and it would be more beneficial for Tails. I’m fine with doing the next round or two myself.
  2. Document these criteria somewhere under wiki/src/contribute/working_together/roles/ticket_gardener/.
  3. Don’t spent time automating this now: many of these criteria are pretty hard to automate, doing it by hand does not require that much time, I did not feel it was painful, and it’s not a good time for us to invest more into Redmine-specific code (upcoming migration to GitLab). We can reconsider next time we do this by hand.

What do you think?

#19 Updated by CyrilBrulebois 2019-09-05 00:05:39

  • Target version changed from Tails_3.16 to Tails_3.17

#20 Updated by intrigeri 2019-09-12 14:25:31

  • Target version changed from Tails_3.17 to Tails_4.0

#21 Updated by Anonymous 2019-09-18 09:50:30

  • Assignee set to intrigeri
  • Feature Branch set to tails.git:16455+triagingdoc

I agree with this reasoning and have not much to add. I’ve added a branch with these criteria, please refine as needed and merge when ready.

#22 Updated by intrigeri 2019-09-18 10:25:22

  • Status changed from Needs Validation to In Progress

Applied in changeset commit:tails|71b96d5ec23fd51af60caf1e628c3d14aaff04fe.

#23 Updated by intrigeri 2019-09-18 10:31:19

  • Status changed from In Progress to Resolved

Thank you! I’ve:

  1. merged your branch
  2. pushed one important fix + a bunch of minor improvements on top
  3. added the template comment so we don’t have to rewrite it from scratch every time and can improve it collectively

So I think we’re done here \o/

#24 Updated by intrigeri 2019-11-17 05:27:39

intrigeri wrote:
> A first batch of email was sent!

FTR, today’s run notified 12 folks and tried to notify 3 other people (including the “Sysadmins” group) but failed because the automation bits don’t know their email address yet — which I’ll fix right away.