On Mon, Jul 27, 2026 at 7:03 PM Xuneng Zhou <[email protected]> wrote: > > Hi Alexander, > > On Thu, Jul 9, 2026 at 8:24 PM Xuneng Zhou <[email protected]> wrote: > > > > Hi! > > > > As Noah pointed out in [1], auxiliary processes are supported in the > > facility, but proper cleanup is absent when they exit. I have attached > > a patch trying to address that as suggested before getting sidetracked > > for too long with the issue raised in [2]. > > I'd like to propose a series of patches to do some further clean-ups & > enhancements. The first few are mostly mechanical, straightforward, > and the last two are more subtle. I appreciate your inputs/thoughts on > them. > > 1) Patch 1 to address the issue raised by Noah. > > 2) Patch 2 to fix two small docs mismatches. > > 3) Patch 3 to optimize the WaitLSNLock acquisition in deleteLSNWaiter. > > 4) Patch 4 to clarify lsn waiter cleanup after wakeup. > > 5) Patch 5 to address a rare issue of deregistration of waiters whose > target lsn has reached but then flashed back. > > Currently, the waiters are removed from the heap by the wakers if > their target lsn has reached. We offer no re-registration service in > the reception desk for them assuming they won't need to extend their > stay. This assumption holds if the lsn is monotonically increasing, > which is true for most cases. However, there seems to be a catch for > the wal write/flush lsn from the wal receiver side. In the cases of > restarting wal receiver, the write/flush lsn could be reset to > positions lagging behind(the attached restarting cases file summarized > by Sol with inputs from me shows a matrix of this). > > One problematic interleaving, for reset cases 2–4 in the attached > file, is the following: > > Let W be the old published write position, X the waiter’s target, S > the new streaming segment start, and R the replay position, with: > max(N, R) < X ≤ W > > [T0] A backend registers a standby_write waiter for target X and sleeps. > > [T1] The walreceiver publishes writtenUpto = W and calls > WaitLSNWakeup(W). Because X ≤ W, the waker removes the waiter. from > the heap, sets inHeap = false, and sets its latch. > > [T2] Before the waiter runs, streaming is restarted. > RequestXLogStreaming() resets shared writtenUpto from W to S. The > effective standby_write position is now: max(writtenUpto, replay) = > max(S, R) = R. Therefore the effective position has regressed below X. > > [T3] The waiter consumes the old latch notification and rechecks its > condition. It observes R < X. In the pre-fix code, it goes back to > sleep without re-registering, even though it is no longer in the > waiters heap.
I took a second look at the wording. I had asked Sol to polish this part for clarity, but the revised version is inaccurate. It should be: [T2] Before the waiter runs, streaming is restarted. RequestXLogStreaming() resets writtenUpto from W to S. The effective standby_write position becomes max(S, R), which is below X. [T3] The waiter consumes the earlier latch notification and rechecks its condition. It observes that max(S, R) < X. In the pre-fix code, it waits again without re-registering, even though the waker has already removed it from the heap. Sorry for not double-checking it. > [T4] The walreceiver later receives and publishes WAL through X and > calls WaitLSNWakeup() again. The waiter is absent from the heap, so it > receives no notification and can remain asleep even though its target > has now been reached. > > One possible fix for the issue that I have considered is to make > write/flush lsn monotonic so it won't flash back once advanced. This > seems to work well at first glance. However, it has two shortcomings: > one is that it can blur the definition of write lsn and another is > that it won't fix the cross-timeline lsn regression. Currently, > writtenUpto is the current walreceiver stream’s write frontier. Giving > that value monotonicity would turn it into the greatest numeric lsn > ever written by any receiver incarnation. Cross-timeline lsn flashback > is expected given the fork point is smaller than the pre-restart lsns. > Extending it across timelines would incorrectly associate progress on > an old timeline with the new timeline. The patch places the fix in the > waiter side -- let the backend do a recheck and re-registration if the > published lsn is regressed. > > Another aspect is the test, it's relatively simple to construct the > failure case, but seems hard to do so deterministically. I spent a lot > of time(maybe too much for an unconfirmed rare issue) trying to tame > it in a simple way; it seems hard to orchestrate the startup process, > wal receiver, waiting backend into above-like interleavings end-to-end > without using complex synchronization like three injection points. The > tricky part is to ensure max(N, R) < X ≤ W. We cannot stop the startup > then the wal receiver to make sure that because we need the former to > manipulate the later one. The current workaround is to add a synthetic > helper for removing the heap node even if its target lsn has not > actually reached, then trigger the real wake-up call by advancing the > published lsn to check whether the waiter is notified and the wait > completes. The rationale behind this is that the premature removal > from the heap without re-registration is the underlying issue > regardless of the specific lsn types and failure scenarios. That said, > I am unsure whether this is a proper way or better alternatives exist. > > 6) Patch 6 to cover a missing wake-up point for primary-flush waiters [WIP] -- Regards, Xuneng Zhou HighGo Software Co., Ltd.
