Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2025-06-09 Thread Vitaly Davydov
Hi Amit, > I think we can use this approach for HEAD and probably keep the > previous idea for backbranches. Keeping some value in shared_memory > per slot sounds risky to me in terms of introducing new bugs. Not sure, what kind of problems may occur. I propose to allocate in shmem an array of la

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2025-06-05 Thread Vitaly Davydov
Dear Alexander, Amit Alexander Korotkov wrote: > Also, I've changed ReplicationSlotsComputeRequiredLSN() call to > CheckPointReplicationSlots() to update required LSN after > SaveSlotToPath() updated last_saved_restart_lsn. This helps to pass > checks in 001_stream_rep.pl without additional hacks

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2025-05-26 Thread Vitaly Davydov
Dear Amit, > OTOH, if we don't want to adjust physical > slot machinery, it seems saving the logical slots to disk immediately > when its restart_lsn is updated is a waste of effort after your patch, > no? If so, why are we okay with that? I agree, that saving logical slots at advance is a possib

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2025-05-26 Thread Vitaly Davydov
segments cleanup; * logical-replication-slot-advance-segment - triggered in LogicalConfirmReceivedLocation() when restart_lsn was changed enough to point to the next WAL segment. Discussion: https://postgr.es/m/flat/1d12d2-67235980-35-19a406a0%4063439497 Author: Vitaly Davydov Author:

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2025-05-20 Thread Vitaly Davydov
[PATCH 2/5] Add TAP test to check logical repl slot advance during checkpoint The test verifies that logical replication slot is still valid after immediate restart on checkpoint completion in case when the slot was advanced during checkpoint. Original patch by: Tomas Vondra Modified by: Vitaly Dav

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2025-05-02 Thread Vitaly Davydov
l-repl-slot-advance-dur.v2.patch [4] 0004-Keep-WAL-segments-by-slot-s-flushed-restart-LSN.v2.patch [5] 0005-Remove-redundant-ReplicationSlotsComputeRequiredLSN-.v2.patch With best regards, Vitaly From a8693c3003df7f9850af0be5284bb6f0e7a82fa6 Mon Sep 17 00:00:00 2001 From: Vitaly Davydov Date: Wed,

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2025-04-24 Thread Vitaly Davydov
prepare a patch where restart_lsn_flushed stored outside of ReplicationSlot structure and doesn't affect the existing API. With best regards, Vitaly On Friday, April 04, 2025 06:22 MSK, Alexander Korotkov wrote: > Hi, Vitaly! > > On Mon, Mar 3, 2025 at 5:12 PM Vitaly Davydov > wrote

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2025-03-03 Thread Vitaly Davydov
which is used to calculate the oldest LSN for WAL segments removal. This state is updated every time with the current restart_lsn at the moment, when the slot is saving to disk. With best regards, Vitaly From 480ab108499d95c8befd95911524c4d77cec6e2e Mon Sep 17 00:00:00 2001 From: Vitaly Davydov Date

Re: Issues with 2PC at recovery: CLOG lookups and GlobalTransactionData

2025-01-31 Thread Vitaly Davydov
On Friday, January 31, 2025 03:21 MSK, Michael Paquier wrote: > > Thoughts and comments are welcome. I'm looking at the v13 patch. I see, there is the only file for v13: v2-0002-Fix-issues-with-2PC-file-handling-at-recovery-13.txt There are two points I would like to highlight: #1. In Recover

Re: An improvement of ProcessTwoPhaseBuffer logic

2025-01-29 Thread Vitaly Davydov
It seems, there are much deeper problems with twophase transactions as I thought. I'm interested in fixing twophase transactions, because I support a solution which actively uses twophase transactions. I'm interested to get more deeply into the twophase functionality. Below, I just want to clari

Re: An improvement of ProcessTwoPhaseBuffer logic

2025-01-22 Thread Vitaly Davydov
tside of the current range. Based on these assumptions (AdjustToFullTransactionId or use full xids) some other functions were modified as shown in the patch. With best regards, Vitaly From 6fa76418466a511a8af32fc3032b4f2ac9daae77 Mon Sep 17 00:00:00 2001 From: Vitaly Davydov Date: Wed, 22 Jan 20

Re: An improvement of ProcessTwoPhaseBuffer logic

2024-12-27 Thread Vitaly Davydov
nsactionId in global transaction state instead of TransactionId. I'm not sure, it will consume significant additional memory, but it make the code more clear and probably result into less number of locks. With best regards, Vitaly From 1e74e01a2485e2b3a59f880a5eaf86af1af81cc5 Mon Sep 17 00:00:00

Re: An improvement of ProcessTwoPhaseBuffer logic

2024-12-25 Thread Vitaly Davydov
Hi Michael, Thank you for the explanation and the patch! I'm happy, that I seem to be on the right way. On Wednesday, December 25, 2024 08:04 MSK, Michael Paquier wrote: > Vitaly, have you seen that in the wild as an effect of future 2PC files? I haven't heard about this problem in production

An improvement of ProcessTwoPhaseBuffer logic

2024-12-24 Thread Vitaly Davydov
Dear Hackers, I would like to discuss ProcessTwoPhaseBuffer function. It reads two-phase transaction states from disk or the WAL. It takes xid as well as some other input parameters and executes the following steps: Step #1: Check if xid is committed or aborted in clog (TransactionIdDidCommit,

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2024-12-13 Thread Vitaly Davydov
nt I have no other ideas how to deal with invalid restart_lsn_flushed. With best regards, Vitaly From a6fb33969213e5f5dd994853ac052df64372b85f Mon Sep 17 00:00:00 2001 From: Vitaly Davydov Date: Thu, 31 Oct 2024 12:29:12 +0300 Subject: [PATCH 1/2] Keep WAL segments by slot's flushed resta

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2024-11-21 Thread Vitaly Davydov
Hi Tomas,   Thank you for the reply and your interest to the investigation. On Wednesday, November 20, 2024 20:24 MSK, Tomas Vondra wrote:   > If an existing physical slot is advanced in the middle of checkpoint > execution, WAL segments, which are related to saved on disk restart LSN > may be r

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2024-11-20 Thread Vitaly Davydov
. The disadvantage is that physical slots do not guarantee WAL keeping starting from its' restart lsns in general.   I would be happy to get some advice, whether I am on the right or wrong way.   Thank you in advance.   With best regards, Vitaly On Thursday, November 07, 2024 16:30 MSK, "

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2024-11-07 Thread Vitaly Davydov
gards, Vitaly Davydov On Thursday, October 31, 2024 13:32 MSK, "Vitaly Davydov" wrote:   Sorry, attached the missed patch. On Thursday, October 31, 2024 13:18 MSK, "Vitaly Davydov" wrote: Dear Hackers,   I'd like to discuss a problem with replication slots's restar

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2024-10-31 Thread Vitaly Davydov
Sorry, attached the missed patch. On Thursday, October 31, 2024 13:18 MSK, "Vitaly Davydov" wrote: Dear Hackers,   I'd like to discuss a problem with replication slots's restart LSN. Physical slots are saved to disk at the beginning of checkpoint. At the end of checkpoi

Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2024-10-31 Thread Vitaly Davydov
Dear Hackers,   I'd like to discuss a problem with replication slots's restart LSN. Physical slots are saved to disk at the beginning of checkpoint. At the end of checkpoint, old WAL segments are recycled or removed from disk, if they are not kept by slot's restart_lsn values.   If an existing

Re: Fsync (flush) all inserted WAL records

2024-08-20 Thread Vitaly Davydov
per. I cannot be absolutely without seeing a patch, but adding SQL functions in this area is usually very useful for monitoring purposes of external solutions. -- Michael   From ba82d6c6f8570fbbff14b4b52fa7720122bfb8ad Mon Sep 17 00:00:00 2001 From: Vitaly Davydov Date: Tue, 20 Aug 2024 18:03

Re: Fsync (flush) all inserted WAL records

2024-08-07 Thread Vitaly Davydov
On Wednesday, August 07, 2024 16:55 MSK, Aleksander Alekseev wrote:   Perhaps you could give more context on the use cases for this function? The value of it is not quite clear. What people typically need is making sure if a given LSN was fsync'ed and/or replicated and/or applied on a replica. Y

Re: Fsync (flush) all inserted WAL records

2024-08-07 Thread Vitaly Davydov
Hi Aleksander, On Wednesday, August 07, 2024 12:19 MSK, Aleksander Alekseev wrote:  > Does pg_current_wal_flush_lsn() [1] return what you need? > > [1]: > https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-RECOVERY-CONTROL If not, take a look at its implementation and func

Fsync (flush) all inserted WAL records

2024-08-05 Thread Vitaly Davydov
Hi Hackers, I use async commits. At some moment, I would like to make sure that all inserted WAL records are fsync-ed. I can use XLogFlush function but I have some doubts which LSN to specify. There is a number of functions which return write or insert LSNs but they are not applicable. I can'

RE: Slow catchup of 2PC (twophase) transactions on replica in LR

2024-07-05 Thread Vitaly Davydov
Hi Kuroda-san, Thank you very much for the patch. In general, it seem to work well for me, but there seems to be a memory access problem in libpqrcv_alter_slot -> quote_identifier in case of NULL slot_name. It happens, if the two_phase option is altered on a subscription without slot. I think,

RE: Slow catchup of 2PC (twophase) transactions on replica in LR

2024-04-22 Thread Vitaly Davydov
or your help! I will definitely try your patch. I need some time to test and incorporate it. I also plan to port my stuff to the master branch to simplify testing of patches. With best regards, ​Vitaly Davydov  

Re: How to accurately determine when a relation should use local buffers?

2023-11-21 Thread Vitaly Davydov
an in backends or there is some work-in-progress to allow some other usage contexts? If so, the check of rd_rel->relpersistence is enough. Not sure why we use SmgrIsTemp instead of RelationUsesLocalBuffers in ReadBuffer_common. With best regards, Vitaly Davydov вт, 21 нояб. 2023 г. в 11:52, Al