Feature #10507
Document how to get the statistics we need out of Jenkins
100%
Description
For SponsorS reports we need:
- average daily number of topic branches we’ve built at least one ISO image from (note that it’s not exactly what I asked for
Feature #9939, sorry); - number of test suite runs per release cycle.
For Feature #9939, bertagaz compiled close enough stats “According to the GolbalBuildStat Jenkins plugin”, which is great, but it is not documented how someone else would do it themselves. A very rough doc draft in the sysadmin team Git repo would be good enough IMO, as long as at least I am able to follow it to successfully compile such stats.
This process will also allow us to identify any manual steps in this process that should be automated, if any (but automating them is not part of this ticket, of course).
Subtasks
History
#1 Updated by intrigeri 2015-11-07 01:41:22
- blocks #8668 added
#2 Updated by bertagaz 2015-11-08 04:38:02
- Assignee changed from bertagaz to intrigeri
- % Done changed from 0 to 80
- QA Check set to Ready for QA
Added a page in the sysadmins repo where I explain the method I used to get this statistics.
I think there’s all that’s needed to automate for the build jobs, but as explained for the test jobs it’s a bit different for the moment.
#3 Updated by intrigeri 2015-12-01 10:29:18
- Status changed from Confirmed to In Progress
- Assignee changed from intrigeri to bertagaz
- % Done changed from 80 to 90
For successful ISO builds: thanks for the draft! For many reasons that you can guess, I disliked using grep to “parse” XML so I took a couple dozen minutes to rewrite it using an actual XML parser: https://git-tails.immerda.ch/puppet-tails/tree/files/jenkins/master/successful-ISO-builds. For October it gives me the same result (935) as what you reported, so it seems that it works fine, but still: please review it.
For test suite runs: thanks for the explanation. You wrote you “modified the URL of this page” but I don’t know what page you’re referring to. Anyway, the details are mostly specific to the October special case, so I’ve ignored them and instead focussed on the general case, starting with November. I really have a hard time making any sense of the Global Build Stats interface, even with your explanations; I guess I would need step-by-step instructions; but wait! To avoid the web UI I instead wrote a script that counts runs of the test suite that were not aborted: https://git-tails.immerda.ch/puppet-tails/tree/files/jenkins/master/ISO-test-suite-runs. Please review it too, and if that’s not exactly what we should could, please clarify and I can adjust the code.
Now I want to add timer units (systemd.timer(5)) to run these scripts at the beginning of each month, against the results file of past month, and publish their results. It would allow non-sysadmins to check metrics and write reports. This is why I put the script there instead of in the sysadmins private repo. We’ll see :)
(Off-topic starts here. In November I see an amount of aborted test suite runs (209) that seems pretty large compared to the number of non-aborted runs (314). I guess I am missing some info to make sense of this, but from my standpoint it seems concerning. If this problem is tracked by a ticket, please point me to it. Otherwise, please ensure it’s tracked, or explain me why it’s not a problem. Thanks! :)
#4 Updated by bertagaz 2015-12-08 04:37:14
- Status changed from In Progress to Fix committed
- Assignee deleted (
bertagaz) - QA Check changed from Ready for QA to Pass
intrigeri wrote:
> For successful ISO builds: thanks for the draft! For many reasons that you can guess, I disliked using grep to “parse” XML so I took a couple dozen minutes to rewrite it using an actual XML parser: https://git-tails.immerda.ch/puppet-tails/tree/files/jenkins/master/successful-ISO-builds. For October it gives me the same result (935) as what you reported, so it seems that it works fine, but still: please review it.
That was a draft, nice script!
> For test suite runs: thanks for the explanation. You wrote you “modified the URL of this page” but I don’t know what page you’re referring to. Anyway, the details are mostly specific to the October special case, so I’ve ignored them and instead focussed on the general case, starting with November. I really have a hard time making any sense of the Global Build Stats interface, even with your explanations; I guess I would need step-by-step instructions; but wait! To avoid the web UI I instead wrote a script that counts runs of the test suite that were not aborted: https://git-tails.immerda.ch/puppet-tails/tree/files/jenkins/master/ISO-test-suite-runs. Please review it too, and if that’s not exactly what we should could, please clarify and I can adjust the code.
Can’t say much about the perl aspect, but considering the previous script, it makes sense.
> Now I want to add timer units (systemd.timer(5)) to run these scripts at the beginning of each month, against the results file of past month, and publish their results. It would allow non-sysadmins to check metrics and write reports. This is why I put the script there instead of in the sysadmins private repo. We’ll see :)
I’ll let you manage the way you’ll encode this publication step in Redmine ticket/your todo list.
> (Off-topic starts here. In November I see an amount of aborted test suite runs (209) that seems pretty large compared to the number of non-aborted runs (314). I guess I am missing some info to make sense of this, but from my standpoint it seems concerning. If this problem is tracked by a ticket, please point me to it. Otherwise, please ensure it’s tracked, or explain me why it’s not a problem.
Thanks for ticket Bug #10717. I was aware of this high umber of aborted job, my priority was to fix the test suite robustness first, then have a look at it.
#5 Updated by intrigeri 2015-12-08 05:41:34
- Status changed from Fix committed to Resolved
(We’re not going to wait for a release.)
#6 Updated by bertagaz 2016-02-09 12:33:11
- % Done changed from 90 to 100