On Fri, May 24, 2019, at 12:17 PM, Alfredo Moralejo Alonso wrote: > > > On Fri, May 24, 2019 at 9:00 PM Jeremy Stanley <[email protected]> wrote: > > On 2019-05-24 20:41:58 +0200 (+0200), Alfredo Moralejo Alonso wrote: > > > On Fri, May 24, 2019 at 8:33 PM Alfredo Moralejo Alonso > > <[email protected]> > > > wrote: > > [...] > > > > For periodic jobs we run every 4 hours i see: > > > > - Last successful jobs started at 23-May-2019 21:06:42 UTC > > > > - First failed jobs with this issue started at 24-May-2019 03:06:37 UTC > > > > > > We just got one that failed and sarted at 24-May-2019 02:25:00 UTC > > [...] > > > > Thanks, between the timing and symptoms, we're starting to suspect > > this could be related to how older Git clients' requests are being > > distributed between our load-balanced backends with inconsistent > > packing. We're weighing some ideas for how to improve things there, > > so stay tuned. The good news is that within the next 24 hours the > > backends will be mostly in sync most of the time, so the symptoms > > you're observing may subside for the most part once that's the case > > (but to be entirely honest, we're still not quite sure just yet). > > ok, thanks for the update. I'll switch some jobs to use github mirrors > as workaround until we find it more stable.
Can you test opendev.org again? We've updated the load balancer method from least connections to a source IP hash. We think this will address the state mismatch problems that you noticed previously. I have since been able to run a git clone --mirror and git fetch origin --prune myself (when it didn't work reliably in the past). Please let us know if it works now or if it doesn't. That data will be useful either way. Clark _______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
