Hi Andres, Hi all, We found an event in the cluster I somehow missed. The 6th of May, a sysadmin operation moved the VM's storage to another datastore using svmotion. The storages were moved without shutting down the VMs nor PostgreSQL themselves. In this three node Patroni cluster, instances were all moved as standbys. Which means a swichover occured. Let's call the nodes: n21, n22, n23.
Here is a sum up of the events: 0. Node n23 is the current primary (since March) 1. 2026-05-06 11h30: n21 storage moved using svmotion without shutting it down 2. 2026-05-06 22h30: switchover. n21 becomes primary, n23 a standby 3. 2026-05-07 11h30: n22 storage moved using svmotion without shutting it down 4. 2026-05-11 11h30: n23 storage moved using svmotion without shutting it down 5. 2026-05-13 19h00: a daily batch issues a lot of INSERT+INIT continuous blocks, including the 2547 affected ones. Each block is filled with about 27 xacts inserting only one row and being **all** immediately aborted. There's at least 500 blocks before and after the affected zone having the same exact workload. Only inserted data differ. Xact id increase monotonically. 6. 2026-05-14 02h00: daily vacuum starts on the affected table. Among usual other tasks, all blocks filled with only dead rows are "CLEAN remxid 0" and set "VISIBLE cutoff xid 0". All? no. Not the 2547 affected ones. For some reason, VACUUM ignored them. 7. between 2026-06-02 21h00 and 2026-06-03 21h00: an old row has been cleaned somewhere, moving the xid horizon from 3,498,165,236 to 3,596,946,949 and removing all pg_xact segments from 0D08 to 0D65 8. 2026-06-04 02h20: daily vacuum analyze raise the first error about missing file "pg_xact/0D44". The xact status of the 2547 affected blocks were all in this file. 9. 2026-06-12: the customer contact us about this issue 10. 2026-06-15: pg_check_frozen() exposed all the affected blocks Note that step 8 was repeating every day until we fixed the production (see first email of the thread). The error was always raised during the ANALYZE phase. So it _seems_ (but I can't prove it) VACUUM never cleaned these pages at all between May 13th and June 12th. I can accept the n21 datastore hot move hadn't been really atomic, seeding some incoherence here and there. We already discover a corruption after a vmotion (not svmotion) with another customer. In short, standby n21 might been promoted with a silent logical corruption. However, I can not be 100% sure it would be the root cause of the problem here. Especially since I fail to find what would be the corruption. Indeed, the affected data have been written **after** this switchover. What kind of corruption would have trigger this behavior? The Visibility Map should have been updated during the writes… shouldn't they? But according to pg_check_frozen(), it wasn't! What could forbid the vacuum to clean this specific region for such a long time if it's not the visibility map? Looking at the affected data we collected in june, they all have flags: HEAP_HASNULL, HEAP_HASVARWIDTH, HEAP_XMAX_INVALID Some have an additional HEAP_XMIN_INVALID that could have been set when some readings or a vacuum analyze land here. But nothing would keep vacuum away from them... Does this story ring any bell to someone? I build some graphical map of the relation to visualize various actions on each blocks, but "decent" png are about 5MB and I don't want to spam everyone on this list with them. If you're interested, I can share them offlist. Regards,
