Is it always "kex_exchange_identification: Connection closed by remote host" as the failure reason?
If so, it usually means the remote host is closing it before exchanging SSH keys (maybe because of connection limits, or authentication problems), or perhaps there is a firewall/network proxy or something else in between your agent and the git repository host that is closing the connection from both the git client and the git server's perspective. I can't think of anything obvious that would have changed due just to the MacOS upgrade, unless something has changed with firewall settings, or other such software you have installed on the agents. Are you sure nothing changed in the network environment your agents are deployed, or on the Gitlab instance at the same time? It looks like you are using SSH Git materials, so once you can replicate with git commands directly on the agent you might need to use environment variable GIT_SSH_COMMAND="ssh -vvv" to see all of the raw SSH debug information (GIT_CURL_VERBOSE is for HTTPS connections). If you discover something here you can probably google to debug it further. *But the issue still exists. on the GoCD, we are still using pre-installed git, do I need install git with brew on the GoCD as well.* If the problem is only on the agents, then no need to change things on the servers, especially if using homebrew Git didn't change anything. -Chad On Sun, Mar 17, 2024 at 12:31 AM SATYA R <satya96...@gmail.com> wrote: > Hi Chad, > Thank you for responding to my issue. > Just wanted to give some basic info on the GoCD version. > We are using GoCD 23.1.0 aarch version, all of our servers are Mac OS M1 > servers(for both GoCD and agents) > > - What are the specific clone errors you are getting? > > > > > > > > > > > *we get the below error when the job tried to clone more the repositories, > at first we thorught the git server is closing the connection, the gitlab > team took the logs from ec2 instance where it says the source ip which is > go-agent is requesting the disconnection.pipelines logs:STDERR: Cloning > into > '/Users/****/go-agent-23.1.0/pipelines/Git_Clone/daf-mock-server'...STDERR: > kex_exchange_identification: Connection closed by remote hostSTDERR: > Connection closed by 10.112.205.90 port 22STDERR: fatal: Could not read > from remote repository.STDERR: STDERR: Please make sure you have the > correct access rightsSTDERR: and the repository exists.Failed to run git > clone command* > > > *Logs from Git server: the Ip in the logs is go-agent ip.Mar 5 23:30:23 > ip-10-112-205-125 sshd[3315655]: pam_unix(sshd:session): session closed for > user git Mar 5 23:30:23 ip-10-112-205-125 sshd[3315640]: Received > disconnect from 10.254.56.146 port 63239:11: disconnected by user* > > - Have you tried running clones manually to see if you can replicate > the problem? That way you can enable git debugging flags to debug the > issue. (e.g GIT_CURL_VERBOSE=1) > > *I have not done this part, I can try this on the agent.* > > - Which exact git version are you using? A Mac pre-installed one, or > something from Brew or MacPorts? > > *Before this issue, we used pre-installed git version on all go-agents. In > order to resolve this issue, I have installed git with brew on the > go-agents and started using it. the version is 2.44.0. But the issue still > exists. on the GoCD, we are still using pre-installed git, do I need > install git with brew on the GoCD as well.* > > - Are the errors always on agents or always on the server, or both? > > *This error is occurring on the agents when the job starts cloning the > repos mentioned in the materials.* > > - What's the filesystem path that GoCD is cloning into when there are > failures? > > *The failures are happening on the agents. the agent path is > /Users/****/go-agent-23.1.0/* > *We installed GoCD in /Applications/.* > > *The artifacts path is /Volumes/Artifacts.* > Please let me know if you need more information. > Thank you. > > On Saturday, March 16, 2024 at 9:37:21 AM UTC-4 Chad Wilson wrote: > >> GoCD doesn't do anything special with Git - it just uses whatever git >> version you have available to the agent/server on the path. If something >> has changed after the Sonoma upgrade, it is probably affecting all Git >> usage on the machine. >> >> - What are the specific clone errors you are getting? >> - Have you tried running clones manually to see if you can replicate >> the problem? That way you can enable git debugging flags to debug the >> issue. (e.g GIT_CURL_VERBOSE=1) >> - Which exact git version are you using? A Mac pre-installed one, or >> something from Brew or MacPorts? >> - Are the errors always on agents or always on the server, or both? >> - What's the filesystem path that GoCD is cloning into when there are >> failures? >> >> -Chad >> >> On Sat, Mar 16, 2024 at 2:51 AM SATYA R <satya...@gmail.com> wrote: >> >>> Hi Team, >>> >>> Recently, we upgraded all of our go-agents to macos Sonoma from ventura. >>> GoCD also runs on Sonoma version. >>> After the upgrade, we are seeing intermittent git clone failures in the >>> pipelines. >>> Each pipeline has 3 or more repos to clone, when multiple pipelines >>> around 20 pipelines try to clone the 3 or more repositories from git, we >>> are facing this git clone issue. >>> sometimes the pipeline are able to clone all the repos but sometimes >>> they fail. >>> >>> is there anything that we need to particularly look into in gocd and >>> go-agents configuration due to sonoma upgrade. do we need to make any >>> changes in go-agent service in order to support the sonoma version? >>> >>> Kindly help us. >>> >>> Thank you. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to go-cd+un...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/63efcffd-c537-4855-8039-9de9eafb885en%40googlegroups.com >>> <https://groups.google.com/d/msgid/go-cd/63efcffd-c537-4855-8039-9de9eafb885en%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to go-cd+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/f725ced6-703d-480d-a801-049fc13f474fn%40googlegroups.com > <https://groups.google.com/d/msgid/go-cd/f725ced6-703d-480d-a801-049fc13f474fn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH-8qE0jL4J0%3DRdKkOBT_F5mBm7S5Q3vQaGoOn9%3DNZ72bQ%40mail.gmail.com.
