Thanks for the responses and the issue link Mark, I've increased the Git timeout value as described, and confirmed in the log that the 30 minute timeout is in effect, but I'm still seeing the same failure (i.e. fatal error in Scan Multibranch Pipeline Now). I was pretty confident that there's not a timeout issue as the failure occurs in a matter of seconds, so I'm going to try the reference repo approach which seems like it should work, but it would appear to be an issue within the git client parsing to me. I'll confirm my outcome.
Regards, Kevin On Tuesday, 2 July 2019 13:37:41 UTC+1, Mark Waite wrote: > > Sorry, I didn't read your description thoroughly enough. You said: > > I'm getting a fatal error reported when doing a repository scan (Scan >> Multibranch Pipeline Now) which was previously working. >> > > The command line git log output indicates that your repository is large > enough that it may be reaching the default 10 minute clone timeout. Refer > to https://issues.jenkins-ci.org/browse/JENKINS-38973 for a discussion of > possible workarounds. > > On Tue, Jul 2, 2019 at 6:31 AM Mark Waite <mark.e...@gmail.com > <javascript:>> wrote: > >> The repository in the workspace on the agent running that build is >> probably damaged. Wipe the workspace on the agent and run the job again. >> >> If wiping the workspace and running the job again does not resolve it, >> you may also be encountering a timeout while cloning the repository. >> Increase the timeout for the repository clone from its default of 10 >> minutes to something large enough that it will allow you to clone the >> repository. >> >> The amount of output you're showing hints that the repository is large or >> the network connection between the agent and the upstream cloned repository >> is slow. In either of those cases, you probably want to reduce the clone >> time and the disc space use by applying one or more of the techniques >> described in "Git in the Large <https://youtu.be/jBGFjFc6Jf8?t=6434>". >> Those techniques include (1) reference repositories on the agent, (2) >> narrow refspecs, (3) shallow clones, and (4) sparse checkouts. >> >> If this is a Pipeline repository and the failure is during the initial >> clone of the repository, then you may also need to use lightweight checkout >> to only checkout the Jenkinsfile rather than the entire repository. >> Alternately for Pipelines, if your git provider is GitHub, Bitbucket, or >> Gitea, you can significantly improve performance by using those branch >> source plugins to manage the Pipeline instead of relying on low-level >> command line git calls. >> >> Reference repositories are usually the most effective technique to reduce >> data transfer time and disc space use for large git repositories. Allan >> Burdajewicz of CloudBees wrote a great article on reference repositories at >> https://support.cloudbees.com/hc/en-us/articles/115001728812-Using-a-Git-reference-repository >> . >> >> On Tue, Jul 2, 2019 at 5:51 AM Kevin Stevens <kevs...@gmail.com >> <javascript:>> wrote: >> >>> I'm getting a fatal error reported when doing a repository scan (Scan >>> Multibranch Pipeline Now) which was previously working. >>> I've run the git commands manually from the command line (copied and >>> pasted) and they appear to work correctly, so it seems like a problem with >>> the Jenkins git client plugin. >>> >>> The scan log is below (repository name and account information modified >>> for security reasons). >>> I'm not clear if this is a GIT or a Jenkins git client problem. I've >>> cleared the Jenkins workspace and updated to the latest Jenkins plugins >>> (running on Ubuntu 18.04). >>> Is anyone able to offer help in debugging this issue please? >>> >>> >>> [Tue Jul 02 09:48:00 BST 2019] Starting branch indexing... >>> > git --version # timeout=10 >>> using GIT_ASKPASS to set credentials Jenkins Bitbucket User >>> > git ls-remote --symref >>> g...@bitbucket.org:my-company-name/my-repository-name.git # timeout=10 >>> > git rev-parse --is-inside-work-tree # timeout=10 >>> Setting origin to >>> g...@bitbucket.org:my-company-name/my-repository-name.git >>> > git config remote.origin.url >>> g...@bitbucket.org:my-company-name/my-repository-name.git # timeout=10 >>> Fetching & pruning origin... >>> Listing remote references... >>> > git config --get remote.origin.url # timeout=10 >>> > git --version # timeout=10 >>> using GIT_ASKPASS to set credentials Jenkins Bitbucket User >>> > git ls-remote -h >>> g...@bitbucket.org:my-company-name/my-repository-name.git # timeout=10 >>> Fetching upstream changes from origin >>> > git config --get remote.origin.url # timeout=10 >>> using GIT_ASKPASS to set credentials Jenkins Bitbucket User >>> > git fetch --tags --progress origin >>> +refs/heads/*:refs/remotes/origin/* --prune >>> ERROR: [Tue Jul 02 09:48:10 BST 2019] Could not fetch branches from >>> source 498872a6-3888-419f-9d2f-a2baa4520968 >>> hudson.plugins.git.GitException: Command "git fetch --tags --progress >>> origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code >>> 128: >>> stdout: >>> stderr: remote: Counting objects: 114, done. >>> remote: Compressing objects: 0% (1/114) >>> remote: Compressing objects: 1% (2/114) >>> remote: Compressing objects: 2% (3/114) >>> remote: Compressing objects: 3% (4/114) >>> remote: Compressing objects: 4% (5/114) >>> remote: Compressing objects: 5% (6/114) >>> remote: Compressing objects: 6% (7/114) >>> remote: Compressing objects: 7% (8/114) >>> remote: Compressing objects: 8% (10/114) >>> remote: Compressing objects: 9% (11/114) >>> remote: Compressing objects: 10% (12/114) >>> remote: Compressing objects: 11% (13/114) >>> remote: Compressing objects: 12% (14/114) >>> remote: Compressing objects: 13% (15/114) >>> remote: Compressing objects: 14% (16/114) >>> remote: Compressing objects: 15% (18/114) >>> remote: Compressing objects: 16% (19/114) >>> remote: Compressing objects: 17% (20/114) >>> remote: Compressing objects: 18% (21/114) >>> remote: Compressing objects: 19% (22/114) >>> remote: Compressing objects: 20% (23/114) >>> remote: Compressing objects: 21% (24/114) >>> remote: Compressing objects: 22% (26/114) >>> remote: Compressing objects: 23% (27/114) >>> remote: Compressing objects: 24% (28/114) >>> remote: Compressing objects: 25% (29/114) >>> remote: Compressing objects: 26% (30/114) >>> remote: Compressing objects: 27% (31/114) >>> remote: Compressing objects: 28% (32/114) >>> remote: Compressing objects: 29% (34/114) >>> remote: Compressing objects: 30% (35/114) >>> remote: Compressing objects: 31% (36/114) >>> remote: Compressing objects: 32% (37/114) >>> remote: Compressing objects: 33% (38/114) >>> remote: Compressing objects: 34% (39/114) >>> remote: Compressing objects: 35% (40/114) >>> remote: Compressing objects: 36% (42/114) >>> remote: Compressing objects: 37% (43/114) >>> remote: Compressing objects: 38% (44/114) >>> remote: Compressing objects: 39% (45/114) >>> remote: Compressing objects: 40% (46/114) >>> remote: Compressing objects: 41% (47/114) >>> remote: Compressing objects: 42% (48/114) >>> remote: Compressing objects: 43% (50/114) >>> remote: Compressing objects: 44% (51/114) >>> remote: Compressing objects: 45% (52/114) >>> remote: Compressing objects: 46% (53/114) >>> remote: Compressing objects: 47% (54/114) >>> remote: Compressing objects: 48% (55/114) >>> remote: Compressing objects: 49% (56/114) >>> remote: Compressing objects: 50% (57/114) >>> remote: Compressing objects: 51% (59/114) >>> remote: Compressing objects: 52% (60/114) >>> remote: Compressing objects: 53% (61/114) >>> remote: Compressing objects: 54% (62/114) >>> remote: Compressing objects: 55% (63/114) >>> remote: Compressing objects: 56% (64/114) >>> remote: Compressing objects: 57% (65/114) >>> remote: Compressing objects: 58% (67/114) >>> remote: Compressing objects: 59% (68/114) >>> remote: Compressing objects: 60% (69/114) >>> remote: Compressing objects: 61% (70/114) >>> remote: Compressing objects: 62% (71/114) >>> remote: Compressing objects: 63% (72/114) >>> remote: Compressing objects: 64% (73/114) >>> remote: Compressing objects: 65% (75/114) >>> remote: Compressing objects: 66% (76/114) >>> remote: Compressing objects: 67% (77/114) >>> remote: Compressing objects: 68% (78/114) >>> remote: Compressing objects: 69% (79/114) >>> remote: Compressing objects: 70% (80/114) >>> remote: Compressing objects: 71% (81/114) >>> remote: Compressing objects: 72% (83/114) >>> remote: Compressing objects: 73% (84/114) >>> remote: Compressing objects: 74% (85/114) >>> remote: Compressing objects: 75% (86/114) >>> remote: Compressing objects: 76% (87/114) >>> remote: Compressing objects: 77% (88/114) >>> remote: Compressing objects: 78% (89/114) >>> remote: Compressing objects: 79% (91/114) >>> remote: Compressing objects: 80% (92/114) >>> remote: Compressing objects: 81% (93/114) >>> remote: Compressing objects: 82% (94/114) >>> remote: Compressing objects: 83% (95/114) >>> remote: Compressing objects: 84% (96/114) >>> remote: Compressing objects: 85% (97/114) >>> remote: Compressing objects: 86% (99/114) >>> remote: Compressing objects: 87% (100/114) >>> remote: Compressing objects: 88% (101/114) >>> remote: Compressing objects: 89% (102/114) >>> remote: Compressing objects: 90% (103/114) >>> remote: Compressing objects: 91% (104/114) >>> remote: Compressing objects: 92% (105/114) >>> remote: Compressing objects: 93% (107/114) >>> remote: Compressing objects: 94% (108/114) >>> remote: Compressing objects: 95% (109/114) >>> remote: Compressing objects: 96% (110/114) >>> remote: Compressing objects: 97% (111/114) >>> remote: Compressing objects: 98% (112/114) >>> remote: Compressing objects: 99% (113/114) >>> remote: Compressing objects: 100% (114/114) >>> remote: Compressing objects: 100% (114/114), done. >>> Receiving objects: 0% (1/114) >>> Receiving objects: 1% (2/114) >>> Receiving objects: 2% (3/114) >>> Receiving objects: 3% (4/114) >>> Receiving objects: 4% (5/114) >>> Receiving objects: 5% (6/114) >>> Receiving objects: 6% (7/114) >>> Receiving objects: 7% (8/114) >>> Receiving objects: 8% (10/114) >>> Receiving objects: 9% (11/114) >>> Receiving objects: 10% (12/114) >>> Receiving objects: 11% (13/114) >>> Receiving objects: 12% (14/114) >>> Receiving objects: 13% (15/114) >>> Receiving objects: 14% (16/114) >>> error: object file >>> .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty >>> fatal: cannot read existing object info >>> 21a28795e9c6fa55f026860e0c2f0b08d1b31611 >>> fatal: index-pack failed >>> >>> at >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042) >>> at >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761) >>> at >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72) >>> at >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442) >>> at >>> jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575) >>> at >>> jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556) >>> at >>> jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367) >>> at >>> jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556) >>> at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373) >>> at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283) >>> at >>> jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634) >>> at >>> com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277) >>> at >>> com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164) >>> at >>> jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025) >>> at hudson.model.ResourceController.execute(ResourceController.java:97) >>> at hudson.model.Executor.run(Executor.java:429) >>> [Tue Jul 02 09:48:10 BST 2019] Finished branch indexing. Indexing took >>> 10 sec >>> FATAL: Failed to recompute children of my-repository-name >>> hudson.plugins.git.GitException: Command "git fetch --tags --progress >>> origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code >>> 128: >>> stdout: >>> stderr: remote: Counting objects: 114, done. >>> remote: Compressing objects: 0% (1/114) >>> remote: Compressing objects: 1% (2/114) >>> remote: Compressing objects: 2% (3/114) >>> remote: Compressing objects: 3% (4/114) >>> remote: Compressing objects: 4% (5/114) >>> remote: Compressing objects: 5% (6/114) >>> remote: Compressing objects: 6% (7/114) >>> remote: Compressing objects: 7% (8/114) >>> remote: Compressing objects: 8% (10/114) >>> remote: Compressing objects: 9% (11/114) >>> remote: Compressing objects: 10% (12/114) >>> remote: Compressing objects: 11% (13/114) >>> remote: Compressing objects: 12% (14/114) >>> remote: Compressing objects: 13% (15/114) >>> remote: Compressing objects: 14% (16/114) >>> remote: Compressing objects: 15% (18/114) >>> remote: Compressing objects: 16% (19/114) >>> remote: Compressing objects: 17% (20/114) >>> remote: Compressing objects: 18% (21/114) >>> remote: Compressing objects: 19% (22/114) >>> remote: Compressing objects: 20% (23/114) >>> remote: Compressing objects: 21% (24/114) >>> remote: Compressing objects: 22% (26/114) >>> remote: Compressing objects: 23% (27/114) >>> remote: Compressing objects: 24% (28/114) >>> remote: Compressing objects: 25% (29/114) >>> remote: Compressing objects: 26% (30/114) >>> remote: Compressing objects: 27% (31/114) >>> remote: Compressing objects: 28% (32/114) >>> remote: Compressing objects: 29% (34/114) >>> remote: Compressing objects: 30% (35/114) >>> remote: Compressing objects: 31% (36/114) >>> remote: Compressing objects: 32% (37/114) >>> remote: Compressing objects: 33% (38/114) >>> remote: Compressing objects: 34% (39/114) >>> remote: Compressing objects: 35% (40/114) >>> remote: Compressing objects: 36% (42/114) >>> remote: Compressing objects: 37% (43/114) >>> remote: Compressing objects: 38% (44/114) >>> remote: Compressing objects: 39% (45/114) >>> remote: Compressing objects: 40% (46/114) >>> remote: Compressing objects: 41% (47/114) >>> remote: Compressing objects: 42% (48/114) >>> remote: Compressing objects: 43% (50/114) >>> remote: Compressing objects: 44% (51/114) >>> remote: Compressing objects: 45% (52/114) >>> remote: Compressing objects: 46% (53/114) >>> remote: Compressing objects: 47% (54/114) >>> remote: Compressing objects: 48% (55/114) >>> remote: Compressing objects: 49% (56/114) >>> remote: Compressing objects: 50% (57/114) >>> remote: Compressing objects: 51% (59/114) >>> remote: Compressing objects: 52% (60/114) >>> remote: Compressing objects: 53% (61/114) >>> remote: Compressing objects: 54% (62/114) >>> remote: Compressing objects: 55% (63/114) >>> remote: Compressing objects: 56% (64/114) >>> remote: Compressing objects: 57% (65/114) >>> remote: Compressing objects: 58% (67/114) >>> remote: Compressing objects: 59% (68/114) >>> remote: Compressing objects: 60% (69/114) >>> remote: Compressing objects: 61% (70/114) >>> remote: Compressing objects: 62% (71/114) >>> remote: Compressing objects: 63% (72/114) >>> remote: Compressing objects: 64% (73/114) >>> remote: Compressing objects: 65% (75/114) >>> remote: Compressing objects: 66% (76/114) >>> remote: Compressing objects: 67% (77/114) >>> remote: Compressing objects: 68% (78/114) >>> remote: Compressing objects: 69% (79/114) >>> remote: Compressing objects: 70% (80/114) >>> remote: Compressing objects: 71% (81/114) >>> remote: Compressing objects: 72% (83/114) >>> remote: Compressing objects: 73% (84/114) >>> remote: Compressing objects: 74% (85/114) >>> remote: Compressing objects: 75% (86/114) >>> remote: Compressing objects: 76% (87/114) >>> remote: Compressing objects: 77% (88/114) >>> remote: Compressing objects: 78% (89/114) >>> remote: Compressing objects: 79% (91/114) >>> remote: Compressing objects: 80% (92/114) >>> remote: Compressing objects: 81% (93/114) >>> remote: Compressing objects: 82% (94/114) >>> remote: Compressing objects: 83% (95/114) >>> remote: Compressing objects: 84% (96/114) >>> remote: Compressing objects: 85% (97/114) >>> remote: Compressing objects: 86% (99/114) >>> remote: Compressing objects: 87% (100/114) >>> remote: Compressing objects: 88% (101/114) >>> remote: Compressing objects: 89% (102/114) >>> remote: Compressing objects: 90% (103/114) >>> remote: Compressing objects: 91% (104/114) >>> remote: Compressing objects: 92% (105/114) >>> remote: Compressing objects: 93% (107/114) >>> remote: Compressing objects: 94% (108/114) >>> remote: Compressing objects: 95% (109/114) >>> remote: Compressing objects: 96% (110/114) >>> remote: Compressing objects: 97% (111/114) >>> remote: Compressing objects: 98% (112/114) >>> remote: Compressing objects: 99% (113/114) >>> remote: Compressing objects: 100% (114/114) >>> remote: Compressing objects: 100% (114/114), done. >>> Receiving objects: 0% (1/114) >>> Receiving objects: 1% (2/114) >>> Receiving objects: 2% (3/114) >>> Receiving objects: 3% (4/114) >>> Receiving objects: 4% (5/114) >>> Receiving objects: 5% (6/114) >>> Receiving objects: 6% (7/114) >>> Receiving objects: 7% (8/114) >>> Receiving objects: 8% (10/114) >>> Receiving objects: 9% (11/114) >>> Receiving objects: 10% (12/114) >>> Receiving objects: 11% (13/114) >>> Receiving objects: 12% (14/114) >>> Receiving objects: 13% (15/114) >>> Receiving objects: 14% (16/114) >>> error: object file >>> .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty >>> fatal: cannot read existing object info >>> 21a28795e9c6fa55f026860e0c2f0b08d1b31611 >>> fatal: index-pack failed >>> >>> at >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042) >>> at >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761) >>> at >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72) >>> at >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442) >>> at >>> jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575) >>> at >>> jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556) >>> at >>> jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367) >>> at >>> jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556) >>> at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373) >>> at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283) >>> at >>> jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634) >>> at >>> com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277) >>> at >>> com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164) >>> at >>> jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025) >>> at hudson.model.ResourceController.execute(ResourceController.java:97) >>> at hudson.model.Executor.run(Executor.java:429) >>> Finished: FAILURE >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to jenkins...@googlegroups.com <javascript:>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> Thanks! >> Mark Waite >> > > > -- > Thanks! > Mark Waite > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/183327b0-b243-4c31-bbca-8a843651492a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
