nic-6443 opened a new issue, #13777: URL: https://github.com/apache/apisix/issues/13777
PR #13721 restores correctness by no longer moving `watch_ctx.rev` to a revision sampled on a separate connection after a watch timeout. It also reuses unchanged objects during compact recovery. One cost remains: recovery still issues a Range request for each config type in every worker, transfers all values, and has `lua-resty-etcd` deserialize them before `load_full_data()` can compare `modifiedIndex`. Once etcd has returned `compacted`, skipping reconciliation is not safe. If the watcher last confirmed revision `R` and etcd has compacted through `C`, these two histories are indistinguishable after `(R, C]` has been removed: - no key under the watched prefix changed; - the watch stream silently missed a change or deletion. A revision sampled by a separate Range request does not prove delivery on the watch stream. The revision in a `created` response is not a delivery barrier either: etcd sends the create response before flushing buffered historical events for that watcher. `WatchProgressRequest` provides the barrier we need. etcd only emits a progress response after the watchers on that same stream are synchronized, so its header revision certifies that all earlier events have been delivered. An idle watcher can then advance safely without waiting until its old revision is compacted. The reference implementation I think we should pursue is: - Change `lua-resty-etcd` to represent `/v3/watch` as a full-duplex session with operations such as `recv()`, `request_progress()`, and `close()`. The etcd HTTP gateway already accepts multiple `WatchRequest` messages from one streaming request body, so this should not require restoring APISIX's native gRPC etcd mode. - When the APISIX watch read times out, send `WatchProgressRequest` on the same stream and wait for a bounded response. If a progress response reports revision `P`, set the next revision to `P + 1` and continue or reopen the watch. - If sending the request fails or no progress response arrives, close the stream and reconnect from the unchanged revision. Never advance based on another connection. - If etcd actually returns `compacted` before a progress barrier is received, keep the snapshot reload as the correctness fallback; at that point it cannot be skipped safely. Periodic `progress_notify` alone is not enough as a default solution. etcd's default progress interval is 10 minutes, while APISIX normally restarts an idle watch after about 50 seconds, so the stream is usually closed before a notification can arrive. An explicit request avoids requiring an etcd server-side interval change. The tests should cover at least: - an idle APISIX prefix while another prefix advances the global revision through aggressive compaction: progress responses should advance the watcher without a recovery Range request; - a put or delete immediately before a progress request: the event must be delivered before the progress revision is accepted; - a silently blackholed watch stream: APISIX must not advance, and must replay from the old revision or reload if that revision was compacted. Relevant protocol references: - [`WatchProgressRequest` is available since etcd 3.4](https://github.com/etcd-io/etcd/blob/v3.6.4/api/etcdserverpb/rpc.proto#L757-L826) - [etcd sends progress only when the watchers are synchronized](https://github.com/etcd-io/etcd/blob/v3.6.4/server/storage/mvcc/watchable_store.go#L536-L560) - [the HTTP gateway continuously decodes requests from the watch request body](https://github.com/etcd-io/etcd/blob/v3.6.4/api/etcdserverpb/gw/rpc.pb.gw.go#L161-L194) This should remove the normal idle-prefix compact recovery path without reintroducing the event-loss window fixed by #13721. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
