AlinsRan commented on issue #2749: URL: https://github.com/apache/apisix-ingress-controller/issues/2749#issuecomment-5472426423
Still relevant, and there are really two different things in this thread. @hanxinimm's original report is the mixed-management case: an Admin API write pushed the version past what the controller had, and after that every sync was rejected. On 2.2.0 with ADC 0.27.1 that part should behave better. #2785 makes the controller re-derive its baseline from the data plane when a push comes back rejected over a `conf_version`, and then retry, so it should recover on its own instead of needing the delete-pod-and-restart-controller dance. @hanxinimm, if you're still running this, could you try 2.2.0 + ADC 0.27.1 and see whether the errors clear up by themselves? @arjunbhut's case looks different, and I don't think the multi-writer checklist I posted in May covers it. Before anything else it's still worth confirming there's only one `GatewayProxy` pointing at that cluster and only one controller actively writing, since that rules out the easy explanation. But assuming both are already true, here's the mechanism I suspect you're hitting, because it fits the rotation you described. In standalone mode the controller hands all data plane addresses to ADC as a single task with one cache key derived from the `GatewayProxy`. ADC keeps one cached snapshot per cache key: on the first dump it HEADs every server, picks the one with the most recent `X-Last-Modified`, and adopts that server's config as the baseline, including its per-type `*_conf_version` values. From then on the cache is updated from what ADC sends rather than re-read from the pods, and each sync is pushed to every server. That model assumes the pods hold identical config. Once a push lands on some pods and not others, they drift, and a baseline taken from a single pod can be behind a different pod on a different resource type. Which is exactly the shape of your samples: one pod rejecting on `ssls_conf_version`, another on `upstreams_conf_version`, and the set rotating between cycles. Picking the most recently updated pod doesn't help here, because "most recent overall" isn't the same as "highest for every type". If that's what's happening, upgrading alone won't make it go away, though it may hide it: the safety net in #2785 re-derives the baseline and retries whenever a push is rejected, so cycles that used to fail may now recover. If you do upgrade, the `conf_version_conflict` metric is a good thing to watch, since it counts those retries and will show the drift even when the sync eventually succeeds. So the ask is the same either way: could you retest on 2.2.0 + ADC 0.27.1 with a single `GatewayProxy` and a single writer? If the rotating rejections are gone, this can be closed. If they're still there, I'd rather split them into their own issue, since the fix would be in how the initial baseline is built across multiple data plane replicas, which is a different change from anything discussed here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
