Hi, On Wed, Sep 2, 2026 at 5:52 PM Paul Kim <[email protected]> wrote: > > On Tue, Mar 19, 2024 at 04:58:57AM +0000, Jeff Davis wrote: > > I'm not clear on why the callers of WaitXLogInsertionsToFinish() are > > handling errors the way they are. XLogWrite PANICs, XLogFlush ERRORs > > (which is likely to be escalated to a PANIC anyway), and the other > > callers ignore the return value and leave it up to XLogWrite() to > > PANIC. > > I hit a production incident on PostgreSQL 15.13 with physical > streaming replication: the primary logged "request to flush past end > of generated WAL" for a position just past a segment boundary, and the > standby then got stuck retrying "record with incorrect prev-link" at > that same position. That led me to the ignored return value in > XLogBackgroundFlush(). In a non-assert build, XLogWrite() does not > necessarily PANIC for this caller. > > I reproduced the following sequence on PostgreSQL 15.13 with 1MB WAL > segments: > > 1. Inject an asyncXactLSN at a segment boundary plus the 40-byte long > page header. > 2. WaitXLogInsertionsToFinish() logs "request to flush past end of > generated WAL" and clamps the request to the reserved position. > 3. XLogBackgroundFlush() discards that return value. XLogWrite() writes > the initialized WAL buffer page and advertises the original partial > position, so a physical walsender sends only the new page header. > 4. If that header overwrites a recycled segment on the standby, the > remaining bytes are stale. Recovery can interpret them as a record > and report an incorrect prev-link. I reproduced the subsequent > five-second retry loop as well.
Nice! Do you mind adding the reproducer as a TAP test for HEAD? Also, I suggest adding an entry for this bug in the current CF: https://commitfest.postgresql.org/. I will try to find some time to review this. -- Bharath Rupireddy Amazon Web Services: https://aws.amazon.com
