Re: Impact of checkpointer during pg_upgrade

2023-09-10 Thread Dilip Kumar
On Fri, Sep 8, 2023 at 11:59 AM Amit Kapila wrote: > > On Fri, Sep 8, 2023 at 10:10 AM Michael Paquier wrote: > > > > On Fri, Sep 08, 2023 at 09:14:59AM +0530, Amit Kapila wrote: > > > On Fri, Sep 8, 2023 at 9:00 AM Zhijie Hou (Fujitsu) > > > wrote: > > >>> I > > >>> mean that doing the latter

Re: Impact of checkpointer during pg_upgrade

2023-09-08 Thread Amit Kapila
On Fri, Sep 8, 2023 at 10:10 AM Michael Paquier wrote: > > On Fri, Sep 08, 2023 at 09:14:59AM +0530, Amit Kapila wrote: > > On Fri, Sep 8, 2023 at 9:00 AM Zhijie Hou (Fujitsu) > > wrote: > >>> I > >>> mean that doing the latter is benefitial for the sake of any patch > >>> committed and > >>>

Re: Impact of checkpointer during pg_upgrade

2023-09-08 Thread Michael Paquier
On Fri, Sep 08, 2023 at 11:59:41AM +0530, Amit Kapila wrote: > I agree with doing something simple. So, to conclude, we agree on two > things in this thread (a) Use max_slot_wal_keep_size to -1 to start > postmaster for the old cluster during the upgrade; (b) Have an > elog(ERROR) to avoid

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Michael Paquier
On Fri, Sep 08, 2023 at 09:14:59AM +0530, Amit Kapila wrote: > On Fri, Sep 8, 2023 at 9:00 AM Zhijie Hou (Fujitsu) > wrote: >>> I >>> mean that doing the latter is benefitial for the sake of any patch >>> committed and >>> as a long-term method to rely on. > > What is your worry here? Are you

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Amit Kapila
On Fri, Sep 8, 2023 at 9:00 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, September 8, 2023 10:58 AM Michael Paquier > wrote: > > > > On Fri, Sep 08, 2023 at 08:18:14AM +0530, Amit Kapila wrote: > > > This validation tries to ensure that we don't have any bugs/issues in > > > our patch. It may

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Michael Paquier
On Fri, Sep 08, 2023 at 03:30:23AM +, Zhijie Hou (Fujitsu) wrote: > I feel adding a check at pg_upgrade may not 100% detect the slot invalidation > if we check by querying the old cluster to get the slot info, because the > invalidation can happen before the first time we fetch the info or

RE: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Zhijie Hou (Fujitsu)
On Friday, September 8, 2023 10:58 AM Michael Paquier wrote: > > On Fri, Sep 08, 2023 at 08:18:14AM +0530, Amit Kapila wrote: > > This validation tries to ensure that we don't have any bugs/issues in > > our patch. It may be a candidate for assert but I feel even if we > > encounter any bug it

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Michael Paquier
On Fri, Sep 08, 2023 at 08:18:14AM +0530, Amit Kapila wrote: > This validation tries to ensure that we don't have any bugs/issues in > our patch. It may be a candidate for assert but I feel even if we > encounter any bug it is better to fix the bug. My guess is that an elog-like error is more

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Amit Kapila
On Fri, Sep 8, 2023 at 5:37 AM Michael Paquier wrote: > > On Thu, Sep 07, 2023 at 03:33:52PM +0530, Amit Kapila wrote: > > I think if we just make max_slot_wal_keep_size to -1 that should be > > sufficient to not let any slots get invalidated during upgrade. Do you > > have anything else in mind?

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Michael Paquier
On Thu, Sep 07, 2023 at 03:33:52PM +0530, Amit Kapila wrote: > I think if we just make max_slot_wal_keep_size to -1 that should be > sufficient to not let any slots get invalidated during upgrade. Do you > have anything else in mind? Forcing wal_keep_size while on it may be a good thing. > If we

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Dilip Kumar
On Thu, Sep 7, 2023 at 3:34 PM Amit Kapila wrote: > > On Thu, Sep 7, 2023 at 12:55 PM Michael Paquier wrote: > > > > (Just dumping what I have in mind while reading the thread.) > > > > On Sat, Sep 02, 2023 at 10:08:51AM +0530, Amit Kapila wrote: > > > During pg_upgrade, we start the server for

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Amit Kapila
On Thu, Sep 7, 2023 at 12:55 PM Michael Paquier wrote: > > (Just dumping what I have in mind while reading the thread.) > > On Sat, Sep 02, 2023 at 10:08:51AM +0530, Amit Kapila wrote: > > During pg_upgrade, we start the server for the old cluster which can > > allow the checkpointer to remove

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Michael Paquier
(Just dumping what I have in mind while reading the thread.) On Sat, Sep 02, 2023 at 10:08:51AM +0530, Amit Kapila wrote: > During pg_upgrade, we start the server for the old cluster which can > allow the checkpointer to remove the WAL files. It has been noticed > that we do generate certain

Re: Impact of checkpointer during pg_upgrade

2023-09-05 Thread Dilip Kumar
On Tue, Sep 5, 2023 at 10:55 AM Amit Kapila wrote: > > > Earlier I was thinking that ERRORing out is better so that the user > > can take necessary action for the invalidated slots and then retry > > upgrade. But thinking again I could not find what are the advantages > > of this because if we

Re: Impact of checkpointer during pg_upgrade

2023-09-04 Thread Amit Kapila
On Tue, Sep 5, 2023 at 10:09 AM Dilip Kumar wrote: > > On Tue, Sep 5, 2023 at 9:38 AM Amit Kapila wrote: > > > > On Mon, Sep 4, 2023 at 4:19 PM Dilip Kumar wrote: > > > > > > Said that there is a possibility that some of the slots which got > > > invalidated even on the previous checkpoint

Re: Impact of checkpointer during pg_upgrade

2023-09-04 Thread Dilip Kumar
On Tue, Sep 5, 2023 at 9:38 AM Amit Kapila wrote: > > On Mon, Sep 4, 2023 at 4:19 PM Dilip Kumar wrote: > > > > Said that there is a possibility that some of the slots which got > > invalidated even on the previous checkpoint might get the same LSN as > > the slot which got invalidated later if

Re: Impact of checkpointer during pg_upgrade

2023-09-04 Thread Amit Kapila
On Mon, Sep 4, 2023 at 4:19 PM Dilip Kumar wrote: > > Said that there is a possibility that some of the slots which got > invalidated even on the previous checkpoint might get the same LSN as > the slot which got invalidated later if there is no activity between > these two checkpoints. So if we

Re: Impact of checkpointer during pg_upgrade

2023-09-04 Thread Dilip Kumar
On Mon, Sep 4, 2023 at 1:41 PM Dilip Kumar wrote: > > > > I think we can do better, like we can just read the latest > > > checkpoint's LSN before starting the old cluster. And now while > > > checking the slot can't we check if the the slot is invalidated then > > > their confirmed_flush_lsn >=

Re: Impact of checkpointer during pg_upgrade

2023-09-04 Thread Dilip Kumar
On Mon, Sep 4, 2023 at 11:18 AM Amit Kapila wrote: > > On Mon, Sep 4, 2023 at 10:33 AM Dilip Kumar wrote: > > > > On Mon, Sep 4, 2023 at 8:41 AM Amit Kapila wrote: > > > > > > On Sat, Sep 2, 2023 at 6:12 PM Dilip Kumar wrote: > > > > > > > > On Sat, Sep 2, 2023 at 10:09 AM Amit Kapila > > >

Re: Impact of checkpointer during pg_upgrade

2023-09-03 Thread Amit Kapila
On Mon, Sep 4, 2023 at 10:33 AM Dilip Kumar wrote: > > On Mon, Sep 4, 2023 at 8:41 AM Amit Kapila wrote: > > > > On Sat, Sep 2, 2023 at 6:12 PM Dilip Kumar wrote: > > > > > > On Sat, Sep 2, 2023 at 10:09 AM Amit Kapila > > > wrote: > > > > > > > The other possibilities apart from not allowing

Re: Impact of checkpointer during pg_upgrade

2023-09-03 Thread Dilip Kumar
On Mon, Sep 4, 2023 at 8:41 AM Amit Kapila wrote: > > On Sat, Sep 2, 2023 at 6:12 PM Dilip Kumar wrote: > > > > On Sat, Sep 2, 2023 at 10:09 AM Amit Kapila wrote: > > > > > The other possibilities apart from not allowing an upgrade in such a > > > case could be (a) Before starting the old

Re: Impact of checkpointer during pg_upgrade

2023-09-03 Thread Amit Kapila
On Sat, Sep 2, 2023 at 6:12 PM Dilip Kumar wrote: > > On Sat, Sep 2, 2023 at 10:09 AM Amit Kapila wrote: > > > The other possibilities apart from not allowing an upgrade in such a > > case could be (a) Before starting the old cluster, we fetch the slots > > directly from the disk using some tool

Re: Impact of checkpointer during pg_upgrade

2023-09-02 Thread Dilip Kumar
On Sat, Sep 2, 2023 at 10:09 AM Amit Kapila wrote: > > During pg_upgrade, we start the server for the old cluster which can > allow the checkpointer to remove the WAL files. It has been noticed > that we do generate certain types of WAL records (e.g > XLOG_RUNNING_XACTS, XLOG_CHECKPOINT_ONLINE,

Impact of checkpointer during pg_upgrade

2023-09-01 Thread Amit Kapila
During pg_upgrade, we start the server for the old cluster which can allow the checkpointer to remove the WAL files. It has been noticed that we do generate certain types of WAL records (e.g XLOG_RUNNING_XACTS, XLOG_CHECKPOINT_ONLINE, and XLOG_FPI_FOR_HINT) even during pg_upgrade for old cluster,