I am troubleshooting a scenario where jobs with a wildcard tag refspec (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and rebuild a successfully built tag referencing the same commit.
When a rerun occurs, prior job runs for the tag are no longer present in the Git Build History for the job. I am seeking help to further investigate and trace polling activity to try to determine what changes are being detected that trigger the job to run again, or learn if i should submit this as a bug report for assistance instead. Thanks, everyone! --------------- Existing Bugs Found: This was the most relative bug I could find, yet my situation is different. https://issues.jenkins-ci.org/browse/JENKINS-17614?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true --------------- *My Environment:* Jenkins 2.176.3 on traditional VMs (not containers) GitHub Plugin 1.29.5 Git Plugin 4.0.0 *Steps to reproduce:* 1. Add a webhook to a GitHub repo which sends the following events to Jenkins: Pushes, Branch or tag creation, Releases. 2. Create two “Pipeline” jobs in Jenkins, both referencing the same GitHub repo. The jobs execute a Declarative Pipeline Jenkinsfile script. *NOTE: JJB is used to create the jobs (jobs are not copied).* Settings for job 1 (master branch job): Do not allow concurrent builds GitHub hook trigger for GITScm polling Pipeline script from SCM: Git Repo ID: origin Refspec: +refs/heads/*:refs/remotes/origin/* Branch Specifier: refs/heads/master Additional Behaviors: Wipe out repository & force clone Lightweight checkout: false Settings for job 2 (tag job): Do not allow concurrent builds GitHub hook trigger for GITScm polling Pipeline script from SCM: Git Repo ID: origin Refspec: +refs/tags/*:refs/remotes/origin/tags/* Branch Specifier: */tags/* Additional Behaviors: Wipe out repository & force clone Lightweight checkout: false 3. When commits are merged to the master branch of the repo, the webhook fires and Job 1 executes. 4. When new releases (tags) are published, the webhook fires and Job 2 executes for the most recent (new) tag. (Things have been running smoothly for me with this type of setup for a couple years.) 5. Occasionally there is a situation with jobs configured like job 2, specifically: - New commits are pushed to master which triggers the webhook to Jenkins. - Job 1 polls and detects the new commits to master and runs. - Job 2 polls and detects changes even though no tags have been created and previously built tags still reference the same commit. - Job 2 then runs and rebuilds the last tag it had already built successfully. - The polling log shows last built revision is the same commit and tag, no diff is performed, but changes are found. - The changes page for the job shows no changes. Observations when this occurs: - Let's say the tag has been built 4 times. We will have a polling log and change logs for the first job run that initially built the tag, then jobs 2 and 3 will have no polling log on the filesystem but will have an empty changelog0.xml, then the last build of the tag has a polling log and change log files. - When a tag is being re-built, when you look at Git Build Data for the job rebuilding that tag, all prior job runs that built that tag successfully are missing from the history. - The problem does not always happen in my environment. It can happen for a repo one day then not happen the next. Perhaps I am simply overlooking some activity in the repo that results in polling detecting changes with my wildcard tag specifier. - Workspaces appear to be intact as required for the git plugin to perform a wildcard tag poll (post-clone). There are no indications of workspaces needing to be created as job runs start. - The same build node (same workspace) can be used for all job runs and the issue can still occur. *Example polling log and Git Build Data for a re-run:* Started on Dec 30, 2019 7:54:51 AM Started by event from 10.25.59.190 ⇒ https://xxxx/github-webhook/ on Mon Dec 30 07:54:51 EST 2019 Using strategy: Default [poll] Last Built Revision: Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0) using credential 28d104ae-ad72-401a-8dc2-d72cf4b8e913 > /data/git-client/bin/git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repositories > /data/git-client/bin/git config remote.origin.url https://github.com/repo.git # timeout=10 Fetching upstream changes from https://github.com/repo.git > /data/git-client/bin/git --version # timeout=10 using GIT_ASKPASS to set credentials xxxx Service Account > /data/git-client/bin/git fetch --tags --force --progress -- https://github.com/repo.git +refs/tags/*:refs/remotes/origin/tags/* # timeout=10 Polling for changes in Seen branch in repository origin/2.2.0 Seen branch in repository origin/2.3.0 Seen branch in repository origin/2.4.0 Seen branch in repository origin/gh-pages Seen branch in repository origin/master Seen branch in repository origin/openshift-migration Seen branch in repository origin/rhel-build Seen branch in repository origin/tags/v0.10.0 Seen branch in repository origin/tags/v0.3 Seen branch in repository origin/tags/v0.4 Seen branch in repository origin/tags/v0.7.0 Seen branch in repository origin/tags/v0.7.1 Seen branch in repository origin/tags/v0.8.0 Seen branch in repository origin/tags/v0.9.0 Seen branch in repository origin/tags/v0.9.1 Seen branch in repository origin/tags/v1.0.0 Seen branch in repository origin/tags/v1.0.0-secure Seen branch in repository origin/tags/v1.1.0 Seen branch in repository origin/tags/v1.2.0 Seen branch in repository origin/tags/v1.3.0 Seen branch in repository origin/tags/v1.3.1 Seen branch in repository origin/tags/v1.4.0 Seen branch in repository origin/tags/v1.5.0 Seen branch in repository origin/tags/v1.6.0 Seen branch in repository origin/tags/v1.6.1 Seen branch in repository origin/tags/v1.7.0 Seen branch in repository origin/tags/v1.7.1 Seen branch in repository origin/tags/v1.7.2 Seen branch in repository origin/tags/v1.8.0 Seen branch in repository origin/tags/v1.9.0 Seen branch in repository origin/tags/v2.0.0 Seen branch in repository origin/tags/v2.1.0 Seen branch in repository origin/tags/v2.2.0 Seen branch in repository origin/tags/v2.3.0 Seen 34 remote branches > /data/git-client/bin/git show-ref --tags -d # timeout=10 Using strategy: Default [poll] Last Built Revision: Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0) Done. Took 0.53 sec Changes found --------------------------------- Git Build Data (for Build #9) Revision: 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc origin/tags/v2.4.0 refs/tags/v2.4.0 Built Branches refs/tags/v2.1.0: Build #4 of Revision 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, refs/tags/v2.1.0) origin/tags/v2.2.0: Build #5 of Revision 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, refs/tags/v2.2.0) refs/tags/v2.2.0: Build #5 of Revision 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, refs/tags/v2.2.0) origin/tags/v2.1.0: Build #4 of Revision 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, refs/tags/v2.1.0) origin/tags/v2.0.0: Build #2 of Revision 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, refs/tags/v2.0.0) refs/tags/v2.4.0: Build #9 of Revision 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, refs/tags/v2.4.0) refs/tags/v2.3.0: Build #8 of Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0) origin/tags/v1.0.0-secure: Build #1 of Revision 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, refs/tags/v1.0.0-secure) origin/tags/v2.4.0: Build #9 of Revision 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, refs/tags/v2.4.0) refs/tags/v1.0.0-secure: Build #1 of Revision 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, refs/tags/v1.0.0-secure) refs/tags/v2.0.0: Build #2 of Revision 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, refs/tags/v2.0.0) origin/tags/v2.3.0: Build #8 of Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0) NOTE: Build #6 was the initial build of refs/tags/v2.3.0 Build #7 was the second rerun of refs/tags/v2.3.0 Build #8 was the last rerun of refs/tags/v2.3.0 Build #9 was for refs/tags/v2.4.0 -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b114b34b-3f60-4dc5-87cd-072bcfb81f8c%40googlegroups.com.
