Bug #9660: Jenkins notifications doesn't cut our needs

Bug #9660

Jenkins notifications doesn't cut our needs

Added by intrigeri 2015-06-30 00:32:35 . Updated 2015-08-26 06:07:42 .

Status:

Resolved

Priority:

Elevated

Assignee:

Category:

Continuous Integration

Target version:

Tails_1.5

Start date:

2015-06-30

Due date:

% Done:

100%

Feature Branch:

Type of work:

Sysadmin

Blueprint:

Starter:

Affected tool:

Deliverable for:

266

Description

We discovered two issues with the Mailer plugin:

https://jenkins.tails.boum.org/job/build_Tails_ISO_bugfix-8007-AppArmor-hardening/48/console says An attempt to send an e-mail to empty list of recipients, ignored. and indeed, no emails are sent.

=> The send-to-individuals setting do not notify on scheduled builds as no scm changes happened.

https://jenkins.tails.boum.org/job/build_Tails_ISO_bugfix-8007-AppArmor-hardening/49/console sent an email to all the last committers since the last build.

Subtasks

Related issues

Related to Tails - ~~Feature #6216~~: Email notification on failed Jenkins build

Resolved

2013-08-07

History

#1 Updated by intrigeri 2015-06-30 00:33:15

blocks #8668 added

#2 Updated by intrigeri 2015-06-30 00:33:29

related to ~~Feature #6216~~: Email notification on failed Jenkins build added

#3 Updated by bertagaz 2015-06-30 06:07:25

This job was triggered by a timer according to the build page, might be the reason: no changes detected in the git repo, so the mail plugin doesn’t send to the last author.

#4 Updated by intrigeri 2015-06-30 11:07:25

> This job was triggered by a timer according to the build page, might be the reason: no changes detected in the git repo, so the mail plugin doesn’t send to the last author.

OK, sounds like a reasonable explanation. How do we fix that?

#5 Updated by bertagaz 2015-07-03 04:45:40

Subject changed from Jenkins sometimes does not notify on build failure to Jenkins notifications doesn't cut our needs
Description updated

#6 Updated by bertagaz 2015-07-06 03:19:33

So the Mailer plugin is clearly too simple for our specified use cases.

The most obvious path to fix the situation is probably to use the email-ext plugin, but it does not seem to have better default behaviors than the Mailer plugin.

It has a send-to: - culprits option that has the same behavior than the Mailer plugin send-to-individuals setting (notify all committers since last build), and it’s probably not working with scheduled builds as well.

We’ll probably have to use the email-ext plugin ability to dynamically generate the recipient list at build time (for the development branches, as the base one are always notifying tails-rm).

One path I’ll give a try is to use it with the EnvInject plugin, to set the last commit author in an environment variable after the git checkout, and then use it in a script set as the recipient field of the jobs email-ext publisher.

#7 Updated by intrigeri 2015-07-07 06:57:20

Status changed from Confirmed to In Progress

> So the Mailer plugin is clearly too simple for our specified use cases.

Agreed.

> We’ll probably have to use the email-ext plugin ability to dynamically generate the recipient list at build time […]

I had a look, and indeed this seems to be the way to go. And our specs is quite simple (simpler than the “culprits” behaviour of these two plugins, actually), which will ease things a bit.

#8 Updated by bertagaz 2015-07-08 07:02:30

Assignee changed from bertagaz to intrigeri
% Done changed from 0 to 20
QA Check set to Ready for QA

I’ve experimented with the smarter_notifications job in our jenkins-jobs repo, and found what I believe is a simple workaround.

It adds a new builder, which uses a shell step after the git checkout to get the last git commit author email and export it in a file. This file is then used in a EnvInject plugin step to set a variable in the build environment.

As the Mailer plugin can use this kind of variable in its recipients setting, it just even works with it.

Two builds are interesting in this job regarding this ticket:

build 3 was triggered manually, and I received a build failure email as being the last author on this branch by that time.
build 4 was triggered with a gigantic git changelog (thanks to a merge of stable in the job branch), and I was the only recipient of the build failure email, as being the one who did this merge.

So I’ve deployed this new build step in every jobs that used the send-to-indivuals setting. We’ll see when the next build triggered by a timer if the email gets send to the right recipient. Meanwhile, if you want to have a look… :)

#9 Updated by bertagaz 2015-07-08 10:26:52

% Done changed from 20 to 50

bertagaz wrote:
>
> So I’ve deployed this new build step in every jobs that used the send-to-indivuals setting. We’ll see when the next build triggered by a timer if the email gets send to the right recipient. Meanwhile, if you want to have a look… :)

Build 56 of the bugfix-8007-AppArmor-hardening branch is the first to be triggered by a timer without any changes since then.
It says


09:42:28  > git checkout -f 5d333a2d1f368dee808cf098e4fba149ea855e87
09:42:29  > git rev-list 5d333a2d1f368dee808cf098e4fba149ea855e87 # timeout=10
09:42:29 [build_Tails_ISO_bugfix-8007-AppArmor-hardening] $ /bin/sh -xe /tmp/hudson7931451108922639500.sh
09:42:29 + mkdir /var/lib/jenkins/workspace/build_Tails_ISO_bugfix-8007-AppArmor-hardening/tmp
09:42:29 + git show -s --pretty=format:%ae 5d333a2d1f368dee808cf098e4fba149ea855e87
09:42:29 + echo NOTIFY_TO=intrigeri@boum.org
09:42:29 [EnvInject] - Injecting environment variables from a build step.
09:42:29 [EnvInject] - Injecting as environment variables the properties file path '/var/lib/jenkins/workspace/build_Tails_ISO_bugfix-8007-AppArmor-hardening/tmp/recipient'
09:42:29 [EnvInject] - Variables injected successfully.
[...]
10:13:09 Build step 'Execute shell' marked build as failure
10:13:09 Sending e-mails to: intrigeri@boum.org

which is the last commit author.

Similary, build 24 of the feature-wiperam-deb branch has been triggered by a timer with many changes.

It says:

09:16:04  > git checkout -f 42826e4178bf09f10faa80b0fdc22ece10773c76
09:16:04  > git rev-list 5ed0efe2d1431b9a067d8422960336df1c22c584 # timeout=10
09:16:05 [build_Tails_ISO_feature-wiperam-deb] $ /bin/sh -xe /tmp/hudson5710137626733789472.sh
09:16:05 + mkdir /var/lib/jenkins/workspace/build_Tails_ISO_feature-wiperam-deb/tmp
09:16:05 + git show -s --pretty=format:%ae 42826e4178bf09f10faa80b0fdc22ece10773c76
09:16:05 + echo NOTIFY_TO=intrigeri@boum.org
09:16:05 [EnvInject] - Injecting environment variables from a build step.
09:16:05 [EnvInject] - Injecting as environment variables the properties file path '/var/lib/jenkins/workspace/build_Tails_ISO_feature-wiperam-deb/tmp/recipient'
09:16:05 [EnvInject] - Variables injected successfully.
[...]
09:56:05 Sending e-mails to: intrigeri@boum.org

Which is also the last committer of this branch.

So to me, it sounds like the behavior we specified is met: email last commit author when a development branch build fails.

#10 Updated by intrigeri 2015-07-09 09:34:27

> I’ve experimented with the smarter_notifications job in our jenkins-jobs repo, and found what I believe is a simple workaround.

> It adds a new builder, which uses a shell step after the git checkout to get the last git commit author email and export it in a file. This file is then used in a EnvInject plugin step to set a variable in the build environment.

Sounds like a very good idea. Pushed a minor robustness improvement to the jenkins-jobs repo.

> As the Mailer plugin can use this kind of variable in its recipients setting, it just even works with it.

So we can de-install the email-ext plugin and its dependencies, right? If you agree, please proceed or file a ticket for that.

#11 Updated by intrigeri 2015-07-09 09:36:52

Assignee changed from intrigeri to bertagaz
% Done changed from 50 to 80
QA Check changed from Ready for QA to Pass

> So to me, it sounds like the behavior we specified is met: email last commit author when a development branch build fails.

Agreed, excellent! Please apply to all relevant jobs, and then feel free to close this ticket :)

#12 Updated by bertagaz 2015-07-09 14:22:51

intrigeri wrote:
> Sounds like a very good idea.

In fact it seems to be the usual way people workaround this in Jenkins, according to most of the online sources.

> Pushed a minor robustness improvement to the jenkins-jobs repo.

Nice, that’s why I like reviews! :)

> So we can de-install the email-ext plugin and its dependencies, right? If you agree, please proceed or file a ticket for that.

Yeah, I think I’ll leave the token-macro plugin installed, because it is of the kind we might need for other tasks, and other plugins use it. Later maybe drop it if no use raise. No need for a ticket, I’ll take care of this while deploying in the coming days.

#13 Updated by bertagaz 2015-07-09 15:37:24

Status changed from In Progress to Resolved
Assignee deleted (~~bertagaz~~)
% Done changed from 80 to 100
Starter set to No

intrigeri wrote:
> Agreed, excellent! Please apply to all relevant jobs, and then feel free to close this ticket :)

Already did applied it everywhere btw!

#14 Updated by intrigeri 2015-07-09 22:38:21

> Already did applied it everywhere btw!

… except on three topic branch build jobs. Fixed.

#15 Updated by intrigeri 2015-08-26 06:07:42

Deliverable for set to 266