Re: Jobs with a wildcard tag refspec sometimes rebuild tags at same commit
Issue entered and will update this thread as things develop. https://issues.jenkins-ci.org/browse/JENKINS-60717 Thanks, Eric On Thu, Jan 9, 2020 at 7:53 AM Eric Pierson wrote: > Yes, I intend to write up the issue today for tracking. > > I have been troubleshooting more but unfortunately don't have anything > valuable to add just yet. I will reply to this thread once the issue has > been opened with more details. > > Eric > > On Thu, Jan 9, 2020, 05:49 Beushausen, Christian < > christian.beushau...@continental-corporation.com> wrote: > >> Hi Eric, Mark, >> >> >> >> Could you keep the list updated? Or have you created a new issue on Jira >> for this? >> We do see this (or a very similar) issue also for at least one job in our >> environment >> >> >> >> Thank you and cheers. >> >> >> >> Mit freundlichen Gruessen/Best regards, >> >> *Christian Beushausen* >> I S PD SW SWF >> Interior Systems & Technology >> >> *E-Mail:* christian.beushau...@continental-corporation.com >> >> >> *From:* jenkinsci-users@googlegroups.com < >> jenkinsci-users@googlegroups.com> *On Behalf Of *Mark Waite >> *Sent:* Dienstag, 7. Januar 2020 01:21 >> *To:* Jenkins Users >> *Subject:* Re: Jobs with a wildcard tag refspec sometimes rebuild tags >> at same commit >> >> >> >> Discard old builds is very likely the problem. If a tag build is removed >> from the history by "Discard old builds" then future searches for the that >> tag (or the SHA-1 of the commit associated with the tag) will fail and it >> will build again. >> >> >> >> On Mon, Jan 6, 2020 at 5:18 PM Eric Pierson wrote: >> >> Thank you for your response, Mark! >> >> >> >> I have not tried to run any scripts to prune history as I see described >> in JENKINS-19022. There are a lot of comments in this one and I am still >> re-reading them to be sure I am not missing something else relevant folks >> discovered along the way. >> >> >> >> I do have "Discard old builds" enabled with Strategy "Log Rotation" and >> "Max # of builds to keep" = 10. Forgive me for omitting the fact that I >> use Discard old builds initially. >> >> >> >> Reading more about "Discard old builds": >> >>- Build count: discard the *oldest build* when a certain number of >>builds already exist >>- Note that Jenkins does not discard items immediately when this >>configuration is updated, or as soon as any of the configured values are >>exceeded; *these rules are evaluated each time a build of this >>project completes.* >> >> I am now focusing on the "Discard old builds" settings and build.xml >> content, and what may be happening to the build history when a build >> completes. I have one job that has only run a total of 7 times that had >> the issue on builds #3 and #4, and the example I initially provided has >> only run a total of 12 times and we see the issue on multiple builds there, >> too. So while there are indeed many tags in the repo, the build history is >> not "large" yet. >> >> >> >> I have another Jenkins environment I used for a while where I did not >> have this issue. It had the same settings (including "Discard old >> builds"), but when I was running jobs through this environment on a regular >> basis, Jenkins and plugins would have been at older versions. >> >> >> >> To hone in on git plugin versions just in the change they are relevant, I >> compared my build.xml: >> >> >> >> New Environment >> >> >> >> >> >> >> >> >> >> Old Environment >> >> >> >> >> >> >> >> >> >> --- >> >> >> >> Given this additional information, please let me know if you have any new >> thoughts or if you feel I should file a bug report (for perhaps the >> logRotate or BuildDiscarder area?) as I continue to research and test. >> >> >> >> Thank you again for your help and everything you do for the Jenkins >> community! >> >> >> >> -Eric >> >> >> >> >> >> >> >> On Fri, Jan 3, 2020 at 8:38 PM Mark Waite >> wrote: >> >> >> >> >> >> On Fri, Jan 3, 2020 at 2:51 PM Eric Pierson wrote: >> >> I am tr
Re: Jobs with a wildcard tag refspec sometimes rebuild tags at same commit
Yes, I intend to write up the issue today for tracking. I have been troubleshooting more but unfortunately don't have anything valuable to add just yet. I will reply to this thread once the issue has been opened with more details. Eric On Thu, Jan 9, 2020, 05:49 Beushausen, Christian < christian.beushau...@continental-corporation.com> wrote: > Hi Eric, Mark, > > > > Could you keep the list updated? Or have you created a new issue on Jira > for this? > We do see this (or a very similar) issue also for at least one job in our > environment > > > > Thank you and cheers. > > > > Mit freundlichen Gruessen/Best regards, > > *Christian Beushausen* > I S PD SW SWF > Interior Systems & Technology > > *E-Mail:* christian.beushau...@continental-corporation.com > > > *From:* jenkinsci-users@googlegroups.com > *On Behalf Of *Mark Waite > *Sent:* Dienstag, 7. Januar 2020 01:21 > *To:* Jenkins Users > *Subject:* Re: Jobs with a wildcard tag refspec sometimes rebuild tags at > same commit > > > > Discard old builds is very likely the problem. If a tag build is removed > from the history by "Discard old builds" then future searches for the that > tag (or the SHA-1 of the commit associated with the tag) will fail and it > will build again. > > > > On Mon, Jan 6, 2020 at 5:18 PM Eric Pierson wrote: > > Thank you for your response, Mark! > > > > I have not tried to run any scripts to prune history as I see described in > JENKINS-19022. There are a lot of comments in this one and I am still > re-reading them to be sure I am not missing something else relevant folks > discovered along the way. > > > > I do have "Discard old builds" enabled with Strategy "Log Rotation" and > "Max # of builds to keep" = 10. Forgive me for omitting the fact that I > use Discard old builds initially. > > > > Reading more about "Discard old builds": > >- Build count: discard the *oldest build* when a certain number of >builds already exist >- Note that Jenkins does not discard items immediately when this >configuration is updated, or as soon as any of the configured values are >exceeded; *these rules are evaluated each time a build of this project >completes.* > > I am now focusing on the "Discard old builds" settings and build.xml > content, and what may be happening to the build history when a build > completes. I have one job that has only run a total of 7 times that had > the issue on builds #3 and #4, and the example I initially provided has > only run a total of 12 times and we see the issue on multiple builds there, > too. So while there are indeed many tags in the repo, the build history is > not "large" yet. > > > > I have another Jenkins environment I used for a while where I did not have > this issue. It had the same settings (including "Discard old builds"), but > when I was running jobs through this environment on a regular basis, > Jenkins and plugins would have been at older versions. > > > > To hone in on git plugin versions just in the change they are relevant, I > compared my build.xml: > > > > New Environment > > > > > > > > > > Old Environment > > > > > > > > > > --- > > > > Given this additional information, please let me know if you have any new > thoughts or if you feel I should file a bug report (for perhaps the > logRotate or BuildDiscarder area?) as I continue to research and test. > > > > Thank you again for your help and everything you do for the Jenkins > community! > > > > -Eric > > > > > > > > On Fri, Jan 3, 2020 at 8:38 PM Mark Waite > wrote: > > > > > > On Fri, Jan 3, 2020 at 2:51 PM Eric Pierson wrote: > > I am troubleshooting a scenario where jobs with a wildcard tag refspec > (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and > rebuild a successfully built tag referencing the same commit. > > > > When a rerun occurs, prior job runs for the tag are no longer present in > the Git Build History for the job. > > > > > > That sounds to me as though you are relying on the JENKINS-19022 memory > bloat bug to retain history of more jobs in the BuildData than are actually > in the build history. > > > > If someone executed one of the scripts that is mentioned in JENKINS-19022 > to clean the excess BuildData from the history, that seems like it might > cause the SHA-1's referenced by some of the tags to be removed from the > history. > > > > If build history w
RE: Jobs with a wildcard tag refspec sometimes rebuild tags at same commit
Hi Eric, Mark, Could you keep the list updated? Or have you created a new issue on Jira for this? We do see this (or a very similar) issue also for at least one job in our environment Thank you and cheers. Mit freundlichen Gruessen/Best regards, Christian Beushausen I S PD SW SWF Interior Systems & Technology E-Mail: christian.beushau...@continental-corporation.com<mailto:christian.beushau...@continental-corporation.com> From: jenkinsci-users@googlegroups.com On Behalf Of Mark Waite Sent: Dienstag, 7. Januar 2020 01:21 To: Jenkins Users Subject: Re: Jobs with a wildcard tag refspec sometimes rebuild tags at same commit Discard old builds is very likely the problem. If a tag build is removed from the history by "Discard old builds" then future searches for the that tag (or the SHA-1 of the commit associated with the tag) will fail and it will build again. On Mon, Jan 6, 2020 at 5:18 PM Eric Pierson mailto:pierson...@gmail.com>> wrote: Thank you for your response, Mark! I have not tried to run any scripts to prune history as I see described in JENKINS-19022. There are a lot of comments in this one and I am still re-reading them to be sure I am not missing something else relevant folks discovered along the way. I do have "Discard old builds" enabled with Strategy "Log Rotation" and "Max # of builds to keep" = 10. Forgive me for omitting the fact that I use Discard old builds initially. Reading more about "Discard old builds": * Build count: discard the oldest build when a certain number of builds already exist * Note that Jenkins does not discard items immediately when this configuration is updated, or as soon as any of the configured values are exceeded; these rules are evaluated each time a build of this project completes. I am now focusing on the "Discard old builds" settings and build.xml content, and what may be happening to the build history when a build completes. I have one job that has only run a total of 7 times that had the issue on builds #3 and #4, and the example I initially provided has only run a total of 12 times and we see the issue on multiple builds there, too. So while there are indeed many tags in the repo, the build history is not "large" yet. I have another Jenkins environment I used for a while where I did not have this issue. It had the same settings (including "Discard old builds"), but when I was running jobs through this environment on a regular basis, Jenkins and plugins would have been at older versions. To hone in on git plugin versions just in the change they are relevant, I compared my build.xml: New Environment mailto:git@4.0.0>"> mailto:git-client@3.0.0>"> mailto:git-client@3.0.0>"> Old Environment mailto:git@3.9.3>"> mailto:git-client@2.7.7>"> mailto:git-client@2.7.7>"> --- Given this additional information, please let me know if you have any new thoughts or if you feel I should file a bug report (for perhaps the logRotate or BuildDiscarder area?) as I continue to research and test. Thank you again for your help and everything you do for the Jenkins community! -Eric On Fri, Jan 3, 2020 at 8:38 PM Mark Waite mailto:mark.earl.wa...@gmail.com>> wrote: On Fri, Jan 3, 2020 at 2:51 PM Eric Pierson mailto:pierson...@gmail.com>> wrote: I am troubleshooting a scenario where jobs with a wildcard tag refspec (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and rebuild a successfully built tag referencing the same commit. When a rerun occurs, prior job runs for the tag are no longer present in the Git Build History for the job. That sounds to me as though you are relying on the JENKINS-19022 memory bloat bug to retain history of more jobs in the BuildData than are actually in the build history. If someone executed one of the scripts that is mentioned in JENKINS-19022 to clean the excess BuildData from the history, that seems like it might cause the SHA-1's referenced by some of the tags to be removed from the history. If build history were being removed based on a specific number of builds, then it also could be that the history entry which included that tag was removed from the list, and thus was no longer visible. Mark Waite I am seeking help to further investigate and trace polling activity to try to determine what changes are being detected that trigger the job to run again, or learn if i should submit this as a bug report for assistance instead. Thanks, everyone! --- Existing Bugs Found: This was the most relative bug I could find, yet my situation is different. https://issues.jenkins-ci.org/browse/JENKINS-17614?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=true --- My Environment: Jenkins 2.176.3 on traditional VMs (not containers
Re: Jobs with a wildcard tag refspec sometimes rebuild tags at same commit
Discard old builds is very likely the problem. If a tag build is removed from the history by "Discard old builds" then future searches for the that tag (or the SHA-1 of the commit associated with the tag) will fail and it will build again. On Mon, Jan 6, 2020 at 5:18 PM Eric Pierson wrote: > Thank you for your response, Mark! > > I have not tried to run any scripts to prune history as I see described in > JENKINS-19022. There are a lot of comments in this one and I am still > re-reading them to be sure I am not missing something else relevant folks > discovered along the way. > > I do have "Discard old builds" enabled with Strategy "Log Rotation" and > "Max # of builds to keep" = 10. Forgive me for omitting the fact that I > use Discard old builds initially. > > Reading more about "Discard old builds": > >- Build count: discard the *oldest build* when a certain number of >builds already exist >- Note that Jenkins does not discard items immediately when this >configuration is updated, or as soon as any of the configured values are >exceeded; *these rules are evaluated each time a build of this project >completes.* > > I am now focusing on the "Discard old builds" settings and build.xml > content, and what may be happening to the build history when a build > completes. I have one job that has only run a total of 7 times that had > the issue on builds #3 and #4, and the example I initially provided has > only run a total of 12 times and we see the issue on multiple builds there, > too. So while there are indeed many tags in the repo, the build history is > not "large" yet. > > I have another Jenkins environment I used for a while where I did not have > this issue. It had the same settings (including "Discard old builds"), but > when I was running jobs through this environment on a regular basis, > Jenkins and plugins would have been at older versions. > > To hone in on git plugin versions just in the change they are relevant, I > compared my build.xml: > > > New Environment > > > > > Old Environment > > > > > > --- > > Given this additional information, please let me know if you have any new > thoughts or if you feel I should file a bug report (for perhaps the > logRotate or BuildDiscarder area?) as I continue to research and test. > > Thank you again for your help and everything you do for the Jenkins > community! > > -Eric > > > > On Fri, Jan 3, 2020 at 8:38 PM Mark Waite > wrote: > >> >> >> On Fri, Jan 3, 2020 at 2:51 PM Eric Pierson wrote: >> >>> I am troubleshooting a scenario where jobs with a wildcard tag refspec >>> (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and >>> rebuild a successfully built tag referencing the same commit. >>> >>> When a rerun occurs, prior job runs for the tag are no longer present in >>> the Git Build History for the job. >>> >>> >> That sounds to me as though you are relying on the JENKINS-19022 memory >> bloat bug to retain history of more jobs in the BuildData than are actually >> in the build history. >> >> If someone executed one of the scripts that is mentioned in JENKINS-19022 >> to clean the excess BuildData from the history, that seems like it might >> cause the SHA-1's referenced by some of the tags to be removed from the >> history. >> >> If build history were being removed based on a specific number of builds, >> then it also could be that the history entry which included that tag was >> removed from the list, and thus was no longer visible. >> >> Mark Waite >> >> >>> I am seeking help to further investigate and trace polling activity to >>> try to determine what changes are being detected that trigger the job to >>> run again, or learn if i should submit this as a bug report for assistance >>> instead. >>> >>> Thanks, everyone! >>> >>> >>> --- >>> Existing Bugs Found: >>> >>> This was the most relative bug I could find, yet my situation is >>> different. >>> >>> >>> https://issues.jenkins-ci.org/browse/JENKINS-17614?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=true >>> --- >>> >>> >>> *My Environment:* >>> >>> Jenkins 2.176.3 on traditional VMs (not containers) >>> GitHub Plugin 1.29.5 >>> Git Plugin 4.0.0 >>> >>> >>> *Steps to reproduce:* >>> >>> 1. Add a webhook to a GitHub repo which sends the following events to >>> Jenkins: Pushes, Branch or tag creation, Releases. >>> >>> 2. Create two “Pipeline” jobs in Jenkins, both referencing the same >>> GitHub repo. The jobs execute a Declarative Pipeline Jenkinsfile script. >>> >>> *NOTE: JJB is used to create the jobs (jobs are not copied).* >>> >>> >>> >>> Settings for job 1 (master branch job): >>> Do not allow concurrent builds >>> GitHub hook trigger for GITScm polling >>> Pipeline script from SCM: Git >>> Repo ID: origin >>> Refspec: +refs/heads/*:refs/remotes/origin/* >>> Branch Specifier: refs/heads/master >>> Additional Behaviors: Wipe out repository
Re: Jobs with a wildcard tag refspec sometimes rebuild tags at same commit
Thank you for your response, Mark! I have not tried to run any scripts to prune history as I see described in JENKINS-19022. There are a lot of comments in this one and I am still re-reading them to be sure I am not missing something else relevant folks discovered along the way. I do have "Discard old builds" enabled with Strategy "Log Rotation" and "Max # of builds to keep" = 10. Forgive me for omitting the fact that I use Discard old builds initially. Reading more about "Discard old builds": - Build count: discard the *oldest build* when a certain number of builds already exist - Note that Jenkins does not discard items immediately when this configuration is updated, or as soon as any of the configured values are exceeded; *these rules are evaluated each time a build of this project completes.* I am now focusing on the "Discard old builds" settings and build.xml content, and what may be happening to the build history when a build completes. I have one job that has only run a total of 7 times that had the issue on builds #3 and #4, and the example I initially provided has only run a total of 12 times and we see the issue on multiple builds there, too. So while there are indeed many tags in the repo, the build history is not "large" yet. I have another Jenkins environment I used for a while where I did not have this issue. It had the same settings (including "Discard old builds"), but when I was running jobs through this environment on a regular basis, Jenkins and plugins would have been at older versions. To hone in on git plugin versions just in the change they are relevant, I compared my build.xml: New Environment Old Environment --- Given this additional information, please let me know if you have any new thoughts or if you feel I should file a bug report (for perhaps the logRotate or BuildDiscarder area?) as I continue to research and test. Thank you again for your help and everything you do for the Jenkins community! -Eric On Fri, Jan 3, 2020 at 8:38 PM Mark Waite wrote: > > > On Fri, Jan 3, 2020 at 2:51 PM Eric Pierson wrote: > >> I am troubleshooting a scenario where jobs with a wildcard tag refspec >> (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and >> rebuild a successfully built tag referencing the same commit. >> >> When a rerun occurs, prior job runs for the tag are no longer present in >> the Git Build History for the job. >> >> > That sounds to me as though you are relying on the JENKINS-19022 memory > bloat bug to retain history of more jobs in the BuildData than are actually > in the build history. > > If someone executed one of the scripts that is mentioned in JENKINS-19022 > to clean the excess BuildData from the history, that seems like it might > cause the SHA-1's referenced by some of the tags to be removed from the > history. > > If build history were being removed based on a specific number of builds, > then it also could be that the history entry which included that tag was > removed from the list, and thus was no longer visible. > > Mark Waite > > >> I am seeking help to further investigate and trace polling activity to >> try to determine what changes are being detected that trigger the job to >> run again, or learn if i should submit this as a bug report for assistance >> instead. >> >> Thanks, everyone! >> >> >> --- >> Existing Bugs Found: >> >> This was the most relative bug I could find, yet my situation is >> different. >> >> >> https://issues.jenkins-ci.org/browse/JENKINS-17614?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=true >> --- >> >> >> *My Environment:* >> >> Jenkins 2.176.3 on traditional VMs (not containers) >> GitHub Plugin 1.29.5 >> Git Plugin 4.0.0 >> >> >> *Steps to reproduce:* >> >> 1. Add a webhook to a GitHub repo which sends the following events to >> Jenkins: Pushes, Branch or tag creation, Releases. >> >> 2. Create two “Pipeline” jobs in Jenkins, both referencing the same >> GitHub repo. The jobs execute a Declarative Pipeline Jenkinsfile script. >> >> *NOTE: JJB is used to create the jobs (jobs are not copied).* >> >> >> >> Settings for job 1 (master branch job): >> Do not allow concurrent builds >> GitHub hook trigger for GITScm polling >> Pipeline script from SCM: Git >> Repo ID: origin >> Refspec: +refs/heads/*:refs/remotes/origin/* >> Branch Specifier: refs/heads/master >> Additional Behaviors: Wipe out repository & force clone >> Lightweight checkout: false >> >> >> >> Settings for job 2 (tag job): >> Do not allow concurrent builds >> GitHub hook trigger for GITScm polling >> Pipeline script from SCM: Git >> Repo ID: origin >> Refspec: +refs/tags/*:refs/remotes/origin/tags/* >> Branch Specifier: */tags/* >> Additional Behaviors: Wipe out repository & force clone >> Lightweight checkout: false >> >> >> 3. When commits are merged to the
Re: Jobs with a wildcard tag refspec sometimes rebuild tags at same commit
On Fri, Jan 3, 2020 at 2:51 PM Eric Pierson wrote: > I am troubleshooting a scenario where jobs with a wildcard tag refspec > (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and > rebuild a successfully built tag referencing the same commit. > > When a rerun occurs, prior job runs for the tag are no longer present in > the Git Build History for the job. > > That sounds to me as though you are relying on the JENKINS-19022 memory bloat bug to retain history of more jobs in the BuildData than are actually in the build history. If someone executed one of the scripts that is mentioned in JENKINS-19022 to clean the excess BuildData from the history, that seems like it might cause the SHA-1's referenced by some of the tags to be removed from the history. If build history were being removed based on a specific number of builds, then it also could be that the history entry which included that tag was removed from the list, and thus was no longer visible. Mark Waite > I am seeking help to further investigate and trace polling activity to try > to determine what changes are being detected that trigger the job to run > again, or learn if i should submit this as a bug report for assistance > instead. > > Thanks, everyone! > > > --- > Existing Bugs Found: > > This was the most relative bug I could find, yet my situation is different. > > > https://issues.jenkins-ci.org/browse/JENKINS-17614?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=true > --- > > > *My Environment:* > > Jenkins 2.176.3 on traditional VMs (not containers) > GitHub Plugin 1.29.5 > Git Plugin 4.0.0 > > > *Steps to reproduce:* > > 1. Add a webhook to a GitHub repo which sends the following events to > Jenkins: Pushes, Branch or tag creation, Releases. > > 2. Create two “Pipeline” jobs in Jenkins, both referencing the same > GitHub repo. The jobs execute a Declarative Pipeline Jenkinsfile script. > > *NOTE: JJB is used to create the jobs (jobs are not copied).* > > > > Settings for job 1 (master branch job): > Do not allow concurrent builds > GitHub hook trigger for GITScm polling > Pipeline script from SCM: Git > Repo ID: origin > Refspec: +refs/heads/*:refs/remotes/origin/* > Branch Specifier: refs/heads/master > Additional Behaviors: Wipe out repository & force clone > Lightweight checkout: false > > > > Settings for job 2 (tag job): > Do not allow concurrent builds > GitHub hook trigger for GITScm polling > Pipeline script from SCM: Git > Repo ID: origin > Refspec: +refs/tags/*:refs/remotes/origin/tags/* > Branch Specifier: */tags/* > Additional Behaviors: Wipe out repository & force clone > Lightweight checkout: false > > > 3. When commits are merged to the master branch of the repo, the webhook > fires and Job 1 executes. > > 4. When new releases (tags) are published, the webhook fires and Job 2 > executes for the most recent (new) tag. > > (Things have been running smoothly for me with this type of setup for a > couple years.) > > 5. Occasionally there is a situation with jobs configured like job 2, > specifically: > >- New commits are pushed to master which triggers the webhook to >Jenkins. >- Job 1 polls and detects the new commits to master and runs. >- Job 2 polls and detects changes even though no tags have been >created and previously built tags still reference the same commit. >- Job 2 then runs and rebuilds the last tag it had already built >successfully. >- The polling log shows last built revision is the same commit and >tag, no diff is performed, but changes are found. >- The changes page for the job shows no changes. > > > Observations when this occurs: > >- Let's say the tag has been built 4 times. We will have a polling >log and change logs for the first job run that initially built the tag, >then jobs 2 and 3 will have no polling log on the filesystem but will have >an empty changelog0.xml, then the last build of the tag has a polling log >and change log files. >- When a tag is being re-built, when you look at Git Build Data for >the job rebuilding that tag, all prior job runs that built that tag >successfully are missing from the history. >- The problem does not always happen in my environment. It can happen >for a repo one day then not happen the next. Perhaps I am simply >overlooking some activity in the repo that results in polling detecting >changes with my wildcard tag specifier. >- Workspaces appear to be intact as required for the git plugin to >perform a wildcard tag poll (post-clone). There are no indications of >workspaces needing to be created as job runs start. >- The same build node (same workspace) can be used for all job runs >and the issue can still occur. > > > > *Example polling log and Git Build Data for a re-run:* > > Started on Dec 30, 2019 7:54:51
