Bug #8692

Fix double-triggered build post Git push

Added by intrigeri 2015-01-13 19:25:12 . Updated 2015-08-26 06:09:49 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Tails_1.4
Start date:
2015-01-13
Due date:
% Done:

100%

Feature Branch:
Type of work:
Research
Blueprint:

Starter:
0
Affected tool:
Deliverable for:
266

Description

It seems to me that sometimes, when pushing to Git, the very same commit is built from twice in a row by Jenkins. E.g.

… were “Started by an SCM change”, and both built from commit:038ead81fa7c261b6600e60ccd8144b1b0fc1b00.

This of course will be a serious performance issue whenever we start building more branches, and will make the feedback loop longer than it should be.

Looks like upstream bug 25639, that’s supposedly fixed in Jenking Git plugin 2.2.11.

It might be that this problem happens only when pushing several branches at a time, or when pushing several branches in a row. To be investigated.

Currently, we’re using the curl $JENKINS_URL/git/notifyCommit?url=$GIT_REPO mechanism in the Git post-receive hook, in tails.git on puppet-git.lizard. That repo is fed via git push --mirror from our primary Git repo. Maybe we should first check that the post-receive hook is run only once per updated branch? Also, this URL supports a branches= parameter, which would be worth trying when debugging this problem.

Might be a bug in the pollscm plugin: it might be that the polling+triggering code only checks what commit a build was last completed from, instead of the commit a build was last started from. But we wouldn’t be the only ones experiencing it.

Subtasks

History

#1 Updated by intrigeri 2015-01-13 19:31:28

  • Description updated

#2 Updated by intrigeri 2015-01-13 19:45:03

  • Status changed from Confirmed to In Progress
  • Assignee changed from bertagaz to intrigeri
  • % Done changed from 0 to 10

Testing a fix.

#3 Updated by intrigeri 2015-01-13 19:49:49

  • Assignee changed from intrigeri to bertagaz

Upgraded the git plugin (and its dependencies) that’s supposed to fix http://issues.jenkins-ci.org/browse/JENKINS-2563. I’ll let it to bertagaz to check if the bug is still here.

#4 Updated by intrigeri 2015-01-13 19:58:29

Apparently not fixed: I’ve pushed (in one go) changes to devel and experimental, which has rightfully triggered respectively builds devel 790 and experimental 1575. Then, I’ve pushed feature/jessie, which has rightfully triggered Jessie build 283, but has also triggered builds devel 791 and experimental 1576.

I notice that my second push (feature/jessie only) says:

remote: To tails@git.tails.boum.org:tails
remote:    038ead8..6e41183  feature/jessie -> feature/jessie
remote:    93b93e7..c6b76ff  immerda/devel -> immerda/devel
remote:    dc525a2..647babe  immerda/experimental -> immerda/experimental
remote:    038ead8..6e41183  labs/feature/jessie -> labs/feature/jessie
remote:    038ead8..6e41183  lizard/feature/jessie -> lizard/feature/jessie

… so I wonder if pushing the other mirrors’ branch references (e.g. lab/feature/jessie) might be responsible for the double-builds. Would be worth trying with a non-mirror push.

#5 Updated by bertagaz 2015-01-14 10:59:48

I also wonder if this trouble is related to the number of remotes our tails.git repo on puppet-git knows about. It seems to have 3 of them now configured (labs,lizard and immerda), and I’m not sure it’s necessary for this repo to know them all. Maybe we should remove some?

#6 Updated by intrigeri 2015-01-14 11:32:59

> I also wonder if this trouble is related to the number of remotes our tails.git repo on puppet-git knows about. It seems to have 3 of them now configured (labs,lizard and immerda), and I’m not sure it’s necessary for this repo to know them all. Maybe we should remove some?

These are not remotes per-se: they’re not configured in tails.git/config. They are remote refs pushed via git push --mirror, as explained above. But yeah, looks like we agree, this might be the root cause, hence the debugging steps I’m suggesting above.

#7 Updated by intrigeri 2015-02-24 18:04:27

FTR, this still happens all the time.

#8 Updated by intrigeri 2015-03-09 11:29:01

  • % Done changed from 10 to 20
  • Starter set to No

The Git hook was adjusted to not use --mirror a couple weeks ago, and I’ve not noticed this bug anymore. Let’s wait a bit more before calling it fixed.

#9 Updated by bertagaz 2015-03-13 13:21:51

  • Assignee changed from bertagaz to intrigeri

intrigeri wrote:
> The Git hook was adjusted to not use --mirror a couple weeks ago, and I’ve not noticed this bug anymore. Let’s wait a bit more before calling it fixed.

After having monitored a bit the build history, I don’t see either any duplicated builds since at least February 24th, which IIRC correlates to the time where you changed the hook. So it seems to me this change fixed the issue. I’ll keep on watching from time to time, but feel free to close this ticket whenever you feel it’s ok.

#10 Updated by intrigeri 2015-03-13 19:10:15

  • Status changed from In Progress to Resolved
  • % Done changed from 20 to 100

#11 Updated by intrigeri 2015-05-29 11:37:47

  • blocks #8668 added

#12 Updated by intrigeri 2015-06-19 04:25:36

  • Assignee deleted (intrigeri)

#13 Updated by intrigeri 2015-08-26 06:09:49

  • Deliverable for set to 266