On Tue, May 28, 2019 at 11:01 PM Clark Boylan <cboy...@sapwetik.org> wrote:
> On Fri, May 24, 2019, at 12:17 PM, Alfredo Moralejo Alonso wrote: > > > > > > On Fri, May 24, 2019 at 9:00 PM Jeremy Stanley <fu...@yuggoth.org> > wrote: > > > On 2019-05-24 20:41:58 +0200 (+0200), Alfredo Moralejo Alonso wrote: > > > > On Fri, May 24, 2019 at 8:33 PM Alfredo Moralejo Alonso < > amora...@redhat.com> > > > > wrote: > > > [...] > > > > > For periodic jobs we run every 4 hours i see: > > > > > - Last successful jobs started at 23-May-2019 21:06:42 UTC > > > > > - First failed jobs with this issue started at 24-May-2019 > 03:06:37 UTC > > > > > > > > We just got one that failed and sarted at 24-May-2019 02:25:00 UTC > > > [...] > > > > > > Thanks, between the timing and symptoms, we're starting to suspect > > > this could be related to how older Git clients' requests are being > > > distributed between our load-balanced backends with inconsistent > > > packing. We're weighing some ideas for how to improve things there, > > > so stay tuned. The good news is that within the next 24 hours the > > > backends will be mostly in sync most of the time, so the symptoms > > > you're observing may subside for the most part once that's the case > > > (but to be entirely honest, we're still not quite sure just yet). > > > > ok, thanks for the update. I'll switch some jobs to use github mirrors > > as workaround until we find it more stable. > > Can you test opendev.org again? We've updated the load balancer method > from least connections to a source IP hash. We think this will address the > state mismatch problems that you noticed previously. I have since been able > to run a git clone --mirror and git fetch origin --prune myself (when it > didn't work reliably in the past). > > Please let us know if it works now or if it doesn't. That data will be > useful either way. > > I've tested it both locally and in CI jobs and it's working fine in all cases. Thanks for your support, Alfredo > Clark > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra