Thank you for your response, Mark! I have not tried to run any scripts to prune history as I see described in JENKINS-19022. There are a lot of comments in this one and I am still re-reading them to be sure I am not missing something else relevant folks discovered along the way.
I do have "Discard old builds" enabled with Strategy "Log Rotation" and "Max # of builds to keep" = 10. Forgive me for omitting the fact that I use Discard old builds initially. Reading more about "Discard old builds": - Build count: discard the *oldest build* when a certain number of builds already exist - Note that Jenkins does not discard items immediately when this configuration is updated, or as soon as any of the configured values are exceeded; *these rules are evaluated each time a build of this project completes.* I am now focusing on the "Discard old builds" settings and build.xml content, and what may be happening to the build history when a build completes. I have one job that has only run a total of 7 times that had the issue on builds #3 and #4, and the example I initially provided has only run a total of 12 times and we see the issue on multiple builds there, too. So while there are indeed many tags in the repo, the build history is not "large" yet. I have another Jenkins environment I used for a while where I did not have this issue. It had the same settings (including "Discard old builds"), but when I was running jobs through this environment on a regular basis, Jenkins and plugins would have been at older versions. To hone in on git plugin versions just in the change they are relevant, I compared my build.xml: New Environment <hudson.plugins.git.util.BuildData plugin="git@4.0.0"> <marked plugin="git-client@3.0.0"> <revision plugin="git-client@3.0.0"> Old Environment <hudson.plugins.git.util.BuildData plugin="git@3.9.3"> <marked plugin="git-client@2.7.7"> <revision plugin="git-client@2.7.7"> --------------- Given this additional information, please let me know if you have any new thoughts or if you feel I should file a bug report (for perhaps the logRotate or BuildDiscarder area?) as I continue to research and test. Thank you again for your help and everything you do for the Jenkins community! -Eric On Fri, Jan 3, 2020 at 8:38 PM Mark Waite <mark.earl.wa...@gmail.com> wrote: > > > On Fri, Jan 3, 2020 at 2:51 PM Eric Pierson <pierson...@gmail.com> wrote: > >> I am troubleshooting a scenario where jobs with a wildcard tag refspec >> (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and >> rebuild a successfully built tag referencing the same commit. >> >> When a rerun occurs, prior job runs for the tag are no longer present in >> the Git Build History for the job. >> >> > That sounds to me as though you are relying on the JENKINS-19022 memory > bloat bug to retain history of more jobs in the BuildData than are actually > in the build history. > > If someone executed one of the scripts that is mentioned in JENKINS-19022 > to clean the excess BuildData from the history, that seems like it might > cause the SHA-1's referenced by some of the tags to be removed from the > history. > > If build history were being removed based on a specific number of builds, > then it also could be that the history entry which included that tag was > removed from the list, and thus was no longer visible. > > Mark Waite > > >> I am seeking help to further investigate and trace polling activity to >> try to determine what changes are being detected that trigger the job to >> run again, or learn if i should submit this as a bug report for assistance >> instead. >> >> Thanks, everyone! >> >> >> --------------- >> Existing Bugs Found: >> >> This was the most relative bug I could find, yet my situation is >> different. >> >> >> https://issues.jenkins-ci.org/browse/JENKINS-17614?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true >> --------------- >> >> >> *My Environment:* >> >> Jenkins 2.176.3 on traditional VMs (not containers) >> GitHub Plugin 1.29.5 >> Git Plugin 4.0.0 >> >> >> *Steps to reproduce:* >> >> 1. Add a webhook to a GitHub repo which sends the following events to >> Jenkins: Pushes, Branch or tag creation, Releases. >> >> 2. Create two “Pipeline” jobs in Jenkins, both referencing the same >> GitHub repo. The jobs execute a Declarative Pipeline Jenkinsfile script. >> >> *NOTE: JJB is used to create the jobs (jobs are not copied).* >> >> >> >> Settings for job 1 (master branch job): >> Do not allow concurrent builds >> GitHub hook trigger for GITScm polling >> Pipeline script from SCM: Git >> Repo ID: origin >> Refspec: +refs/heads/*:refs/remotes/origin/* >> Branch Specifier: refs/heads/master >> Additional Behaviors: Wipe out repository & force clone >> Lightweight checkout: false >> >> >> >> Settings for job 2 (tag job): >> Do not allow concurrent builds >> GitHub hook trigger for GITScm polling >> Pipeline script from SCM: Git >> Repo ID: origin >> Refspec: +refs/tags/*:refs/remotes/origin/tags/* >> Branch Specifier: */tags/* >> Additional Behaviors: Wipe out repository & force clone >> Lightweight checkout: false >> >> >> 3. When commits are merged to the master branch of the repo, the webhook >> fires and Job 1 executes. >> >> 4. When new releases (tags) are published, the webhook fires and Job 2 >> executes for the most recent (new) tag. >> >> (Things have been running smoothly for me with this type of setup for a >> couple years.) >> >> 5. Occasionally there is a situation with jobs configured like job 2, >> specifically: >> >> - New commits are pushed to master which triggers the webhook to >> Jenkins. >> - Job 1 polls and detects the new commits to master and runs. >> - Job 2 polls and detects changes even though no tags have been >> created and previously built tags still reference the same commit. >> - Job 2 then runs and rebuilds the last tag it had already built >> successfully. >> - The polling log shows last built revision is the same commit and >> tag, no diff is performed, but changes are found. >> - The changes page for the job shows no changes. >> >> >> Observations when this occurs: >> >> - Let's say the tag has been built 4 times. We will have a polling >> log and change logs for the first job run that initially built the tag, >> then jobs 2 and 3 will have no polling log on the filesystem but will have >> an empty changelog0.xml, then the last build of the tag has a polling log >> and change log files. >> - When a tag is being re-built, when you look at Git Build Data for >> the job rebuilding that tag, all prior job runs that built that tag >> successfully are missing from the history. >> - The problem does not always happen in my environment. It can >> happen for a repo one day then not happen the next. Perhaps I am simply >> overlooking some activity in the repo that results in polling detecting >> changes with my wildcard tag specifier. >> - Workspaces appear to be intact as required for the git plugin to >> perform a wildcard tag poll (post-clone). There are no indications of >> workspaces needing to be created as job runs start. >> - The same build node (same workspace) can be used for all job runs >> and the issue can still occur. >> >> >> >> *Example polling log and Git Build Data for a re-run:* >> >> Started on Dec 30, 2019 7:54:51 AM >> Started by event from 10.25.59.190 ⇒ https://xxxx/github-webhook/ on Mon >> Dec 30 07:54:51 EST 2019 >> Using strategy: Default >> [poll] Last Built Revision: Revision >> f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, >> refs/tags/v2.3.0) >> using credential 28d104ae-ad72-401a-8dc2-d72cf4b8e913 >> > /data/git-client/bin/git rev-parse --is-inside-work-tree # timeout=10 >> Fetching changes from the remote Git repositories >> > /data/git-client/bin/git config remote.origin.url >> https://github.com/repo.git # timeout=10 >> Fetching upstream changes from https://github.com/repo.git >> > /data/git-client/bin/git --version # timeout=10 >> using GIT_ASKPASS to set credentials xxxx Service Account >> > /data/git-client/bin/git fetch --tags --force --progress -- >> https://github.com/repo.git +refs/tags/*:refs/remotes/origin/tags/* # >> timeout=10 >> Polling for changes in >> Seen branch in repository origin/2.2.0 >> Seen branch in repository origin/2.3.0 >> Seen branch in repository origin/2.4.0 >> Seen branch in repository origin/gh-pages >> Seen branch in repository origin/master >> Seen branch in repository origin/openshift-migration >> Seen branch in repository origin/rhel-build >> Seen branch in repository origin/tags/v0.10.0 >> Seen branch in repository origin/tags/v0.3 >> Seen branch in repository origin/tags/v0.4 >> Seen branch in repository origin/tags/v0.7.0 >> Seen branch in repository origin/tags/v0.7.1 >> Seen branch in repository origin/tags/v0.8.0 >> Seen branch in repository origin/tags/v0.9.0 >> Seen branch in repository origin/tags/v0.9.1 >> Seen branch in repository origin/tags/v1.0.0 >> Seen branch in repository origin/tags/v1.0.0-secure >> Seen branch in repository origin/tags/v1.1.0 >> Seen branch in repository origin/tags/v1.2.0 >> Seen branch in repository origin/tags/v1.3.0 >> Seen branch in repository origin/tags/v1.3.1 >> Seen branch in repository origin/tags/v1.4.0 >> Seen branch in repository origin/tags/v1.5.0 >> Seen branch in repository origin/tags/v1.6.0 >> Seen branch in repository origin/tags/v1.6.1 >> Seen branch in repository origin/tags/v1.7.0 >> Seen branch in repository origin/tags/v1.7.1 >> Seen branch in repository origin/tags/v1.7.2 >> Seen branch in repository origin/tags/v1.8.0 >> Seen branch in repository origin/tags/v1.9.0 >> Seen branch in repository origin/tags/v2.0.0 >> Seen branch in repository origin/tags/v2.1.0 >> Seen branch in repository origin/tags/v2.2.0 >> Seen branch in repository origin/tags/v2.3.0 >> Seen 34 remote branches >> > /data/git-client/bin/git show-ref --tags -d # timeout=10 >> Using strategy: Default >> [poll] Last Built Revision: Revision >> f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, >> refs/tags/v2.3.0) >> Done. Took 0.53 sec >> Changes found >> >> --------------------------------- >> >> Git Build Data (for Build #9) >> Revision: 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc >> origin/tags/v2.4.0 >> refs/tags/v2.4.0 >> Built Branches >> refs/tags/v2.1.0: Build #4 of Revision >> 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, >> refs/tags/v2.1.0) >> origin/tags/v2.2.0: Build #5 of Revision >> 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, >> refs/tags/v2.2.0) >> refs/tags/v2.2.0: Build #5 of Revision >> 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, >> refs/tags/v2.2.0) >> origin/tags/v2.1.0: Build #4 of Revision >> 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, >> refs/tags/v2.1.0) >> origin/tags/v2.0.0: Build #2 of Revision >> 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, >> refs/tags/v2.0.0) >> refs/tags/v2.4.0: Build #9 of Revision >> 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, >> refs/tags/v2.4.0) >> refs/tags/v2.3.0: Build #8 of Revision >> f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, >> refs/tags/v2.3.0) >> origin/tags/v1.0.0-secure: Build #1 of Revision >> 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, >> refs/tags/v1.0.0-secure) >> origin/tags/v2.4.0: Build #9 of Revision >> 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, >> refs/tags/v2.4.0) >> refs/tags/v1.0.0-secure: Build #1 of Revision >> 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, >> refs/tags/v1.0.0-secure) >> refs/tags/v2.0.0: Build #2 of Revision >> 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, >> refs/tags/v2.0.0) >> origin/tags/v2.3.0: Build #8 of Revision >> f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, >> refs/tags/v2.3.0) >> >> NOTE: >> Build #6 was the initial build of refs/tags/v2.3.0 >> Build #7 was the second rerun of refs/tags/v2.3.0 >> Build #8 was the last rerun of refs/tags/v2.3.0 >> Build #9 was for refs/tags/v2.4.0 >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jenkinsci-users+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/b114b34b-3f60-4dc5-87cd-072bcfb81f8c%40googlegroups.com >> <https://groups.google.com/d/msgid/jenkinsci-users/b114b34b-3f60-4dc5-87cd-072bcfb81f8c%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Thanks! > Mark Waite > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Jenkins Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/jenkinsci-users/_JJM5DZoPvA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > jenkinsci-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGFkmt76_HEU4h_Aey%2Bai8UiD3hYkwNE%3Da3ZaF70SK-JA%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGFkmt76_HEU4h_Aey%2Bai8UiD3hYkwNE%3Da3ZaF70SK-JA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAKh5vxkMvHNW%2B8JpOUPFj%2BqRHgf%2Bi-6%3DR-R0Ppmam4XjDPXO-A%40mail.gmail.com.
