On Fri, May 24, 2019 at 7:37 PM Clark Boylan <[email protected]> wrote:
> On Fri, May 24, 2019, at 10:23 AM, Alfredo Moralejo Alonso wrote: > > Hi, > > > > Since last hours puppet-openstack-integrating jobs in RDO are failing > > to deploy puppet modules from > > https://opendev.org/openstack/puppet-<modules> in CentOS7. > > > > The sympthom is jobs get stuck randomly when installing modules using > > r10k. Investigating it, I've been able to reproduce the issue running: > > > > git clone --mirror https://github.com/openstack/puppet-<module> > > > > About 50% of the times, commands get stuck for any project in > > opendev.org/openstack. I've checked that it works fine if i don't use > > "--mirror". > > > > As it seems to work fine with fedora, I've compiled and installed > > latest version of git from upstream and in a CentOS7 and then, git > > clone --mirror works fine, but in this case r10k also gets stuck when > > running: > > > > git --git-dir > > /home/centos/.r10k/git/https---opendev.org-openstack-puppet-keystone > > fetch origin --prune > > > > In this case, it only happens with puppet-keystone. > > > > Note that it always works fine using github.com/openstack mirrors > > instead of opendev.org so the problem seems to be some combination of > > git repos from opendev with CentOS7 git. Has there been any change that > > may be affecting git repos behavior somehow? > > Yesterday we replicated all of refs/changes and refs/notes from Gerrit to > Gitea. We had these replicated on the old Cgit farm, but gitea didn't > support this replication until wednesday when we upgraded Gitea. The > manpage for git clone --mirror says it will "map all refs (including > remote-tracking branches, notes etc.)" so this is possibly related. > > When you say it gets stuck randomly does that mean the git process never > completes the clone? or it stops with failure? In either case stracing the > clone process might help to figure out if it is these new refs that it is > having problems with. > > When fails, it never completes, example output with GIT_TRACE enabled: $ git clone --mirror https://opendev.org/openstack/puppet-neutron trace: built-in: git 'clone' '--mirror' ' https://opendev.org/openstack/puppet-neutron' Cloning into bare repository 'puppet-neutron.git'... trace: run_command: 'git-remote-https' 'origin' ' https://opendev.org/openstack/puppet-neutron' trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' 'https://opendev.org/openstack/puppet-neutron/' trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' 'https://opendev.org/openstack/puppet-neutron/' trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' 'https://opendev.org/openstack/puppet-neutron/' It always get stuck in: trace: built-in: git 'fetch-pack' ... I have a 2MB strace output, but it's probably more useful for you to just replicate it in a centos:latest container?, simply: " docker run -it docker.io/centos:latest /bin/bash yum install -y git strace git clone --mirror https://opendev.org/openstack/puppet-nova " you can run the git clone several times if it finishes fine. > Clark > > _______________________________________________ > OpenStack-Infra mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
