Tails

#2 Updated by Anonymous 2019-03-07 14:11:12

• blocks Feature #16545: Make contributors aware of stalled work regularly added

#3 Updated by enrico 2019-03-11 20:56:35

• File redmine-remind added
• Assignee changed from enrico to intrigeri

I made an initial version of the script:

$ ./redmine-remind  --api-key @access-key  --verbose
INFO Querying https://redmine.tails.boum.org/code/issues.json?query_id=316&project_id=tails
INFO Querying https://redmine.tails.boum.org/code/issues.json?query_id=317&project_id=tails
{}

--access-key takes the Redmine access key. Use @filename to make it read it from a file, so it does not get leaked in the environment.

Currently the two custom queries you indicated give empty results. To test it, try different queries:

$ ./redmine-remind  --api-key @access-key -v --query-ids=268 
INFO Querying https://redmine.tails.boum.org/code/issues.json?query_id=268&project_id=tails
INFO Querying https://redmine.tails.boum.org/code/users/675.json
INFO Querying https://redmine.tails.boum.org/code/users/516.json
INFO Querying https://redmine.tails.boum.org/code/users/519.json
INFO Querying https://redmine.tails.boum.org/code/users/1158.json
INFO Querying https://redmine.tails.boum.org/code/users/776.json
INFO Querying https://redmine.tails.boum.org/code/users/71.json
alant:
  issues:
  - created_on: '2017-03-19T17:32:53Z'
    id: 12384
    subject: Forward Greeter accessibility settings to the GNOME session
    updated_on: '2017-09-09T10:33:10Z'
  - created_on: '2016-08-19T05:30:23Z'
    id: 11664
    subject: Improve usability of the Greeter with a screenreader
    updated_on: '2019-01-22T17:58:26Z'
  - created_on: '2014-12-16T10:37:56Z'
    id: 8444
    subject: Monitor development of screen keyboard in GNOME
    updated_on: '2017-09-15T17:51:32Z'
intrigeri:
  issues:
  - created_on: '2017-08-30T09:49:39Z'
    id: 14522
    subject: Make Tails usable for blind users
    updated_on: '2019-01-22T17:58:26Z'
…

This is the command line help with a list of all the options:

$ ./redmine-remind --help
usage: redmine-remind [-h] [--verbose] [--debug] --api-key API_KEY
                      [--server SERVER] [--query-ids QUERY_IDS]

Fetch reminder information from open redmine issyes

optional arguments:
  -h, --help            show this help message and exit
  --verbose, -v         verbose output
  --debug               debug output
  --api-key API_KEY     Redmine API key; use @file to read it from a file
                        (required)
  --server SERVER       Redmine URL. Default:
                        https://redmine.tails.boum.org/code/
  --query-ids QUERY_IDS
                        IDs of queries to run. Default: 316,317

I’m attaching the script.

@intrigeri let me know how you’d like the YAML structure to look like

#4 Updated by Anonymous 2019-03-12 14:05:32

• QA Check set to Info Needed

#5 Updated by Anonymous 2019-03-12 14:06:03

• Description updated

#6 Updated by intrigeri 2019-03-12 18:01:11

• Description updated

#7 Updated by intrigeri 2019-03-12 18:23:09

• Assignee changed from intrigeri to enrico
• QA Check changed from Info Needed to Dev Needed

Hi @enrico!

> I made an initial version of the script:

Wow, that was fast. The CLI looks great to me. In particular, I’m very happy you added support for reading the API key from a file. I did not look at the code yet because I know you have way more experience than me at Python so I doubt I’ll have much to say except nitpicks, which I expect won’t help you at this point.

> Currently the two custom queries you indicated give empty results.

Indeed, in the meantime we repurposed these custom queries (to be about the currently logged in user only) but did not give you new ones that would allow you to generate the list of usernames we need here. Thanks for coping with this and managing to produce useful code anyway!

So I’ve created new queries that reflect what we had in mind when we decided we would create this ticket for you: https://redmine.tails.boum.org/code/projects/tails/issues?query_id=318 and https://redmine.tails.boum.org/code/projects/tails/issues?query_id=319.

> @intrigeri let me know how you’d like the YAML structure to look like

A list of Redmine usernames would be great. The content of the list would be: the set of users who have at least one ticket assigned to them on the view.

Regarding the format, I doubt we need YAML: if whitespace is forbidden in Redmine usernames, a space-separated list would allow me to do Feature #16545 just fine.

(The YAML I mentioned during the conversation that lead to creating this ticket was: a static mapping from Redmine usernames to email addresses, which would be given as its other input to Feature #16545. That’s because I thought we could not query the Redmine API for this info without giving high privileges to the owner of the API key used by your script. But I did not check yet and if you know the answer by heart, I’d be happy to learn :)

#8 Updated by enrico 2019-03-13 09:03:29

• File redmine-remind added
• Assignee changed from enrico to intrigeri

intrigeri wrote:

> I doubt I’ll have much to say except nitpicks, which I expect won’t help you at this point.

Feel free to have a cursory look for obvious stupid mistakes I might have done: the code is really quite short.

> So I’ve created new queries that reflect what we had in mind when we decided we would create this ticket for you: https://redmine.tails.boum.org/code/projects/tails/issues?query_id=318 and https://redmine.tails.boum.org/code/projects/tails/issues?query_id=319.

Ok, I put the new query IDs in DEFAULT_QUERY_IDS at the top of the script, feel free to change there if you want to use different queries in the future. The script also takes a comma separated list of IDs as the --query-ids argument, if you want to use it in other contexts.

> A list of Redmine usernames would be great. The content of the list would be: the set of users who have at least one ticket assigned to them on the view.
>
> Regarding the format, I doubt we need YAML: if whitespace is forbidden in Redmine usernames, a space-separated list would allow me to do Feature #16545 just fine.

I wouldn’t trust redmine not to have spaces in usernames, see: https://www.redmine.org/issues/811

I made the script now output one username per line. I added a --one-line option to make it print names space-separated, and a --print0 option, like in find(1), to print with null characters instead of newlines. You can play around and see what works for you.

To help with idempotence, output names are always sorted before printing.

> (The YAML I mentioned during the conversation that lead to creating this ticket was: a static mapping from Redmine usernames to email addresses, which would be given as its other input to Feature #16545. That’s because I thought we could not query the Redmine API for this info without giving high privileges to the owner of the API key used by your script. But I did not check yet and if you know the answer by heart, I’d be happy to learn :)

I’m not sure i follow here. In the first version of the script, I made it look up user records to find email addresses, but email data doesn’t seem to be accessible for all users, only for some, at least to me. I know next to nothing about Redmine access control, unfortunately.

I’m attaching the new iteration of the script.

#9 Updated by intrigeri 2019-03-13 09:34:51

• Status changed from Confirmed to In Progress
• QA Check changed from Dev Needed to Ready for QA

#10 Updated by intrigeri 2019-03-13 10:15:48

• Assignee changed from intrigeri to enrico

Hi @enrico!

I’ve imported your script into our puppet-tails Git repository (https://git.tails.boum.org/puppet-tails/tree/files/redmine/reminder/redmine-remind) and pushed a few additional commits there. Please base future work on the (slightly modified) version that’s in there. Sorry, no “Fork” button but I’m happy to create a Git repo on our infra, to host your personal fork, if it helps.

> Feel free to have a cursory look for obvious stupid mistakes I might have done: the code is really quite short.

Done, fixed a couple typos and that’s all :)

> I wouldn’t trust redmine not to have spaces in usernames, see: https://www.redmine.org/issues/811

Thanks for the pointer. In this light, I’ve removed the --one-line option, which we won’t use (thanks to your research) and thus seems risky to keep around.

> I made the script now output one username per line. I added a --one-line option to make it print names space-separated, and a --print0 option, like in find(1), to print with null characters instead of newlines. You can play around and see what works for you.

Great! I’ll use either the default one-per-line output or the null-separated one, we’ll see. Each is 3 lines of code so I see no harm in keeping both around :)

> To help with idempotence, output names are always sorted before printing.

Neat.

I’ve tested the script and it works fine for me, except the generated list of usernames seems to be incomplete. The script outputs:

1sstomars
CyrilBrulebois
Dr_Whax
alant
anonym
bertagaz
emmapeel
geb
grey-helm
huertanix
infinity0

… while on both queries (318, 319) I see e.g. tickets assigned to muri and segfault.

I’ve applied https://git.tails.boum.org/puppet-tails/commit/files/redmine/reminder/redmine-remind?id=3e94048f229ee9f8748e3698fc5213740f9f4144 which fixes the problem for me (until any of these queries returns more than 200 issues, that is). Does it look OK to you? Is there a better way to fix this problem?

#12 Updated by enrico 2019-03-13 12:18:35

• Assignee changed from enrico to intrigeri

#13 Updated by intrigeri 2019-03-13 13:26:59

• Status changed from In Progress to Resolved
• Target version changed from Tails_3.14 to Tails_3.13
• QA Check changed from Ready for QA to Pass

hi again @enrico!

> I propose this tweak, to let urlencode do proper encoding of query strings

Oh yeah, that’s much nicer :) Applied.

> I had forgotten the unpleasantness of Redmine doing mandatory pagination on APIs, which makes it impossible to extract large datasets without race conditions, but they seem to like it that way :/

Gah :/ IIRC the Python Redmine library deals with pagination issues behind the hood but indeed it has to be racy.

> If it’s not unreasonable to expect more than 100 issues per query, let me know and I’ll code a cycle that repeats the query with an increasing offset until they come out empty.

Don’t bother. One goal of the work we’re doing here is to lower the number of issues returned by these queries at the end of the day, which is mostly a social & organizational matter. For now, let assume it won’t utterly fail. If it does fail, most likely we’ll need to take a step back and think about why the process does not work as well as we hoped, rather than increasing that limit.

Next step is Feature #16545. I would love to see you review the code I’ll come up with there (mostly copied’n’pasted from bits we already have): I bet you could help me get better at Python along the way.

#14 Updated by intrigeri 2019-03-13 13:27:40

• Assignee deleted (intrigeri)

#17 Updated by zen 2019-07-17 13:21:36

• related to #16887 added