moonming commented on PR #13894: URL: https://github.com/apache/apisix/pull/13894#issuecomment-6786859521
Thanks @BetterAndBetterII for the analysis and the reproduction. You found the right consequence: routing reads the worker-local copy and keys the picker on `checker.status_ver`, and nothing ever reconciles that copy with shm. After tracing the cause, I'd like to fix this in lua-resty-healthcheck rather than on APISIX's request path, so I won't merge this PR as it is: - The divergence starts in the library. `healthcheck.new()` reads the initial state from shm, calls `poll()`, and only registers the event callback later in `start()`, so events arriving in between are dropped for that checker (membphis reproduced this in #13888). Events can also be lost in transport, and the library never reconciles afterwards. - Reading every node's shm state on every selection makes selection O(N) (see membphis's numbers above), depends on the library's private key layout, and leaves `ai-proxy-multi` keyed on `status_ver`. The library fix is in api7/lua-resty-healthcheck#61: resync from shm right after the callback is registered, keep a version counter in shm that every worker checks every 0.1 s (resyncing targets and states and bumping `status_ver` when it changes), take the state from shm when a target is re-added, and relax the lua-resty-events version check. With that, the balancer and `ai-proxy-multi` stay unchanged and APISIX only bumps the dependency. Your TEST 2 scenario is exactly the regression test APISIX needs. Would you like to turn this PR into that? Run it with 2 workers, inject the missed event through the library API (unregister this worker's callback, then call `set_target_status`) instead of writing shm directly, and assert that real requests fail over to the backup within ~0.2 s and return after recovery. It would land together with the dependency bump once the library is released. If you'd rather not, no problem, I'll make sure it gets written and credit this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
