Re: [HACKERS] Restricting maximum keep segments by repslots

2020-06-20 Thread Justin Pryzby
Minor language tweak: diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 7050ce6e2e..08142d64cb 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -3800,8 +3800,8 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows slots are

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-05-28 Thread Kyotaro Horiguchi
Thank you for looking this and trouble rebasing! At Mon, 30 Mar 2020 20:03:27 -0300, Alvaro Herrera wrote in > I rebased this patch; it's failing to apply due to minor concurrent > changes in PostgresNode.pm. I squashed the patches in a series that > made the most sense to me. > > I have a

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-05-18 Thread Alvaro Herrera
On 2020-May-19, Michael Paquier wrote: > On Mon, May 18, 2020 at 07:44:59PM -0400, Alvaro Herrera wrote: > > BTW while you're messing with checkpointer, I propose this patch to > > simplify things. > > It seems to me that this would have a benefit if we begin to have a > code path in

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-05-18 Thread Michael Paquier
On Mon, May 18, 2020 at 07:44:59PM -0400, Alvaro Herrera wrote: > BTW while you're messing with checkpointer, I propose this patch to > simplify things. It seems to me that this would have a benefit if we begin to have a code path in CreateCheckpoint() where where it makes sense to let the

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-05-18 Thread Alvaro Herrera
BTW while you're messing with checkpointer, I propose this patch to simplify things. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 9148a6defa2e8b3fd81b982de53f73584a8b3d10 Mon Sep 17 00:00:00 2001 From:

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-05-17 Thread Alvaro Herrera
On 2020-May-16, Andres Freund wrote: > Hi, > > On 2020-05-16 22:51:50 -0400, Alvaro Herrera wrote: > > On 2020-May-16, Andres Freund wrote: > > > > > I, independent of this patch, added a few additional paths in which > > > checkpointer's latch is reset, and I found a few shutdowns in

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-05-16 Thread Andres Freund
Hi, On 2020-05-16 22:51:50 -0400, Alvaro Herrera wrote: > On 2020-May-16, Andres Freund wrote: > > > I, independent of this patch, added a few additional paths in which > > checkpointer's latch is reset, and I found a few shutdowns in regression > > tests to be extremely slow / timing out. The

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-05-16 Thread Alvaro Herrera
On 2020-May-16, Andres Freund wrote: > I, independent of this patch, added a few additional paths in which > checkpointer's latch is reset, and I found a few shutdowns in regression > tests to be extremely slow / timing out. The reason for that is that > the only check for interrupts is at the

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-05-16 Thread Andres Freund
Hi, On 2020-04-29 18:58:16 -0400, Alvaro Herrera wrote: > On 2020-Apr-28, Alvaro Herrera wrote: > > > On 2020-Apr-28, Kyotaro Horiguchi wrote: > > > > > > Anyway I think this patch should fix it also -- instead of adding a new > > > > flag, we just rely on the existing flags (since

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-29 Thread Kyotaro Horiguchi
At Wed, 29 Apr 2020 18:58:16 -0400, Alvaro Herrera wrote in > On 2020-Apr-28, Alvaro Herrera wrote: > > > On 2020-Apr-28, Kyotaro Horiguchi wrote: > > > > > > Anyway I think this patch should fix it also -- instead of adding a new > > > > flag, we just rely on the existing flags (since

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-29 Thread Alvaro Herrera
On 2020-Apr-30, Kyotaro Horiguchi wrote: > At Tue, 28 Apr 2020 20:47:10 -0400, Alvaro Herrera > wrote in > > /* > > * After the sanity checks in CreateDecodingContext, make sure > > the > > * restart_lsn is valid. Avoid "cannot get changes" wording in

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-29 Thread Kyotaro Horiguchi
Thank you for polishing and committing this. At Tue, 28 Apr 2020 20:47:10 -0400, Alvaro Herrera wrote in > I pushed this one. Some closing remarks: > > On 2020-Apr-28, Alvaro Herrera wrote: > > > On 2020-Apr-28, Kyotaro Horiguchi wrote: > > > > Agreed to describe what is failed rather than

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-29 Thread Alvaro Herrera
On 2020-Apr-28, Alvaro Herrera wrote: > On 2020-Apr-28, Kyotaro Horiguchi wrote: > > > > Anyway I think this patch should fix it also -- instead of adding a new > > > flag, we just rely on the existing flags (since do_checkpoint must have > > > been set correctly from the flags earlier in that

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-28 Thread Alvaro Herrera
I pushed this one. Some closing remarks: On 2020-Apr-28, Alvaro Herrera wrote: > On 2020-Apr-28, Kyotaro Horiguchi wrote: > > Agreed to describe what is failed rather than the cause. However, > > logical replications slots are always "previously reserved" at > > creation. > > Bah, of course.

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-28 Thread Alvaro Herrera
On 2020-Apr-28, Kyotaro Horiguchi wrote: > > Anyway I think this patch should fix it also -- instead of adding a new > > flag, we just rely on the existing flags (since do_checkpoint must have > > been set correctly from the flags earlier in that block.) > > Since the added (!do_checkpoint)

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-28 Thread Alvaro Herrera
On 2020-Apr-28, Kyotaro Horiguchi wrote: > At Mon, 27 Apr 2020 18:33:42 -0400, Alvaro Herrera > wrote in > > On 2020-Apr-08, Kyotaro Horiguchi wrote: > > > > > At Wed, 08 Apr 2020 09:37:10 +0900 (JST), Kyotaro Horiguchi > > > wrote in > > Thanks for the fix! I propose two changes: > > >

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-28 Thread Kyotaro Horiguchi
At Mon, 27 Apr 2020 19:40:07 -0400, Alvaro Herrera wrote in > On 2020-Apr-08, Kyotaro Horiguchi wrote: > > > I understand how it happens. > > > > The latch triggered by checkpoint request by CHECKPOINT command has > > been absorbed by ConditionVariableSleep() in > >

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-27 Thread Kyotaro Horiguchi
At Mon, 27 Apr 2020 18:33:42 -0400, Alvaro Herrera wrote in > On 2020-Apr-08, Kyotaro Horiguchi wrote: > > > At Wed, 08 Apr 2020 09:37:10 +0900 (JST), Kyotaro Horiguchi > > wrote in > > > > Just avoiding starting replication when restart_lsn is invalid is > > sufficient (the attached,

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-27 Thread Alvaro Herrera
On 2020-Apr-08, Kyotaro Horiguchi wrote: > I understand how it happens. > > The latch triggered by checkpoint request by CHECKPOINT command has > been absorbed by ConditionVariableSleep() in > InvalidateObsoleteReplicationSlots. The attached allows checkpointer > use MyLatch for other than

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-27 Thread Alvaro Herrera
On 2020-Apr-08, Kyotaro Horiguchi wrote: > At Wed, 08 Apr 2020 09:37:10 +0900 (JST), Kyotaro Horiguchi > wrote in > > Just avoiding starting replication when restart_lsn is invalid is > sufficient (the attached, which is equivalent to a part of what the > invalidated flag did). I thing that

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-09 Thread Kyotaro Horiguchi
At Wed, 08 Apr 2020 14:19:56 +0900 (JST), Kyotaro Horiguchi wrote in me> Just avoiding starting replication when restart_lsn is invalid is me> sufficient (the attached, which is equivalent to a part of what the me> invalidated flag did). I thing that the error message needs a Hint but me> it

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-08 Thread Kyotaro Horiguchi
At Wed, 08 Apr 2020 16:46:05 +0900 (JST), Kyotaro Horiguchi wrote in > At Wed, 08 Apr 2020 14:19:56 +0900 (JST), Kyotaro Horiguchi > wrote in > The latch triggered by checkpoint request by CHECKPOINT command has > been absorbed by ConditionVariableSleep() in >

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-08 Thread Kyotaro Horiguchi
At Wed, 08 Apr 2020 14:19:56 +0900 (JST), Kyotaro Horiguchi wrote in > I saw another issue, the following sequence on the primary freezes > when invalidation happens. > > =# create table tt(); drop table tt; select pg_switch_wal();create table > tt(); drop table tt; select

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-07 Thread Kyotaro Horiguchi
At Wed, 08 Apr 2020 09:37:10 +0900 (JST), Kyotaro Horiguchi wrote in > > I pushed version 26, with a few further adjustments. > > > > I think what we have now is sufficient, but if you want to attempt this > > "invalidated" flag on top of what I pushed, be my guest. > > I don't think the

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-07 Thread Kyotaro Horiguchi
Thank you for committing this. At Tue, 7 Apr 2020 18:45:22 -0400, Alvaro Herrera wrote in > On 2020-Apr-07, Kyotaro Horiguchi wrote: > > > > Mmm. Couldn't we have a new member 'invalidated' in ReplicationSlot? > > > > I did that in the attached. The invalidated is shared-but-not-saved > >

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-07 Thread Alvaro Herrera
On 2020-Apr-07, Kyotaro Horiguchi wrote: > > Mmm. Couldn't we have a new member 'invalidated' in ReplicationSlot? > > I did that in the attached. The invalidated is shared-but-not-saved > member of a slot and initialized to false then irreversibly changed to > true when the slot loses required

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-07 Thread Kyotaro Horiguchi
At Tue, 07 Apr 2020 12:09:05 +0900 (JST), Kyotaro Horiguchi wrote in > > it seems to me that it suffices to check restart_lsn for being invalid > > in the couple of places where the slot's owner advances (which is the > > two auxiliary functions for ProcessStandbyReplyMessage). I have done so

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Kyotaro Horiguchi
At Mon, 6 Apr 2020 14:58:39 -0400, Alvaro Herrera wrote in > > LOG: slot rep1 is invalidated at 0/1C0 due to exceeding > > max_slot_wal_keep_size > > Sounds good. Here's a couple of further adjustments to your v24. This > passes the existing tests (pg_basebackup exception noted below),

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Alvaro Herrera
On 2020-Apr-07, Kyotaro Horiguchi wrote: > At Mon, 6 Apr 2020 12:54:56 -0400, Alvaro Herrera > wrote in > > Thanks for the revised version. Please note that you forgot to "git > > add" the test file, to it's not in the patch. > > Oops! I forgot that I was working after just doing patch -p1

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Kyotaro Horiguchi
At Mon, 6 Apr 2020 12:54:56 -0400, Alvaro Herrera wrote in > Thanks for the revised version. Please note that you forgot to "git > add" the test file, to it's not in the patch. Oops! I forgot that I was working after just doing patch -p1 on my working directory. This is the version that

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Alvaro Herrera
On 2020-Apr-06, Alvaro Herrera wrote: > I think there's a race condition in this: if we kill a walsender and it > restarts immediately before we (checkpoint) can acquire the slot, we > will wait for it to terminate on its own. Fixing this requires changing > the ReplicationSlotAcquire API so

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Alvaro Herrera
On 2020-Apr-06, Alvaro Herrera wrote: > Lastly, I noticed that we're now changing the slot's restart_lsn to > Invalid without being the slot's owner, which goes counter to what is > said in slot.h: > > * - Individual fields are protected by mutex where only the backend owning > * the slot is

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Alvaro Herrera
On 2020-Apr-06, Kyotaro Horiguchi wrote: > At Fri, 3 Apr 2020 20:14:03 -0300, Alvaro Herrera > wrote in > Agreed and thanks for the code. The patch is enough to express the > intention. I fixed some compilation errors and made a clean up of > KeepLogSeg. InvalidateObsoleteReplicationSlots

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Alvaro Herrera
On 2020-Apr-06, Kyotaro Horiguchi wrote: > > * Andres complained that the "distance" column was not a great value to > > expose (20171106132050.6apzynxrqrzgh...@alap3.anarazel.de). That's > > right: it changes both by the insertion LSN as well as the slot's > > consumption. Maybe we can

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Kyotaro Horiguchi
At Fri, 3 Apr 2020 20:14:03 -0300, Alvaro Herrera wrote in > So, the more I look at this patch, the less I like the way the slots are > handled. > > * I think it's a mistake to try to do anything in KeepLogSeg itself; > that function is merely in charge of some arithmetic. I propose to >

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-03 Thread Alvaro Herrera
So, the more I look at this patch, the less I like the way the slots are handled. * I think it's a mistake to try to do anything in KeepLogSeg itself; that function is merely in charge of some arithmetic. I propose to make that function aware of the new size limitation (so that it doesn't

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-31 Thread Kyotaro Horiguchi
At Tue, 31 Mar 2020 18:01:36 -0300, Alvaro Herrera wrote in > I noticed some other things: > > 1. KeepLogSeg sends a warning message when slots fall behind. To do > this, it searches for "the most affected slot", that is, the slot that > lost the most data. But it seems to me that that's a

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-31 Thread Kyotaro Horiguchi
At Tue, 31 Mar 2020 16:59:05 -0300, Alvaro Herrera wrote in > On 2020-Mar-31, Alvaro Herrera wrote: > > > On 2020-Mar-31, Alvaro Herrera wrote: > > > > > I'm not sure if I explained my proposal clearly. What if > > > XLogGetLastRemovedSegno returning zero means that every segment is > > >

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-31 Thread Kyotaro Horiguchi
At Tue, 31 Mar 2020 14:18:36 -0300, Alvaro Herrera wrote in > On 2020-Mar-31, Alvaro Herrera wrote: > > > I'm not sure if I explained my proposal clearly. What if > > XLogGetLastRemovedSegno returning zero means that every segment is > > valid? We don't need to scan pg_xlog at all. > > I

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-31 Thread Alvaro Herrera
On 2020-Mar-31, Alvaro Herrera wrote: > I think we should kill(SIGTERM) the walsender using the slot > (slot->active_pid), > then acquire the slot and set it to some state indicating that it is now > useless, no longer reserving WAL; so when the walsender is restarted, it > will find the slot

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-31 Thread Alvaro Herrera
On 2020-Mar-31, Alvaro Herrera wrote: > /* release lock before syscalls */ > foreach(l, pids_to_kill) > { > kill(SIGTERM, lfirst_int(l)); > } > > I sense some attempt to salvage

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-31 Thread Alvaro Herrera
I noticed some other things: 1. KeepLogSeg sends a warning message when slots fall behind. To do this, it searches for "the most affected slot", that is, the slot that lost the most data. But it seems to me that that's a bit pointless; if a slot data, it's now useless and anything that was

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-31 Thread Alvaro Herrera
On 2020-Mar-31, Alvaro Herrera wrote: > On 2020-Mar-31, Alvaro Herrera wrote: > > > I'm not sure if I explained my proposal clearly. What if > > XLogGetLastRemovedSegno returning zero means that every segment is > > valid? We don't need to scan pg_xlog at all. > > I mean this: [v21 does it

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-31 Thread Alvaro Herrera
On 2020-Mar-31, Alvaro Herrera wrote: > I'm not sure if I explained my proposal clearly. What if > XLogGetLastRemovedSegno returning zero means that every segment is > valid? We don't need to scan pg_xlog at all. I mean this: XLogSegNo FindOldestXLogFileSegNo(void) { XLogSegNo segno =

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-31 Thread Alvaro Herrera
On 2020-Mar-31, Kyotaro Horiguchi wrote: > Thank you for looking this and trouble rebasing! > > At Mon, 30 Mar 2020 20:03:27 -0300, Alvaro Herrera > wrote in > > I rebased this patch; it's failing to apply due to minor concurrent > > changes in PostgresNode.pm. I squashed the patches in a

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-30 Thread Kyotaro Horiguchi
Thank you for looking this and trouble rebasing! At Mon, 30 Mar 2020 20:03:27 -0300, Alvaro Herrera wrote in > I rebased this patch; it's failing to apply due to minor concurrent > changes in PostgresNode.pm. I squashed the patches in a series that > made the most sense to me. > > I have a

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-30 Thread Alvaro Herrera
I rebased this patch; it's failing to apply due to minor concurrent changes in PostgresNode.pm. I squashed the patches in a series that made the most sense to me. I have a question about static variable lastFoundOldestSeg in FindOldestXLogFileSegNo. It may be set the first time the function

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-01-23 Thread Kyotaro Horiguchi
At Thu, 23 Jan 2020 21:28:54 +0900 (JST), Kyotaro Horiguchi wrote in > > In the same function, I think that setting restBytes to -1 when > > "useless" is bad style. I would just leave that variable alone when the > > returned status is not one that receives the number of bytes. So the > >

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-01-23 Thread Kyotaro Horiguchi
Hello, Jehan. At Wed, 22 Jan 2020 17:47:23 +0100, Jehan-Guillaume de Rorthais wrote in > Hi, > > First, it seems you did not reply to Alvaro's concerns in your new set of > patch. See: > > https://www.postgresql.org/message-id/20190917195800.GA16694%40alvherre.pgsql Mmmm. Thank you very

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-01-22 Thread Jehan-Guillaume de Rorthais
Hi, First, it seems you did not reply to Alvaro's concerns in your new set of patch. See: https://www.postgresql.org/message-id/20190917195800.GA16694%40alvherre.pgsql On Tue, 24 Dec 2019 21:26:14 +0900 (JST) Kyotaro Horiguchi wrote: [...] > > Indeed, "loosing" is a better match for this

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-12-26 Thread Kyotaro Horiguchi
At Tue, 24 Dec 2019 21:26:14 +0900 (JST), Kyotaro Horiguchi wrote in > The attached v17 patch is changed in the follwing points. > > - Rebased to the current master. > > - Change KeepLogSeg not to emit the message "Slot %s lost %ld > segment(s)" if the slot list is not changed. > > -

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-12-24 Thread Kyotaro Horiguchi
I'm very sorry for being late to reply. At Wed, 2 Oct 2019 17:08:07 +0200, Jehan-Guillaume de Rorthais wrote in > On Tue, 30 Jul 2019 21:30:45 +0900 (Tokyo Standard Time) > Kyotaro Horiguchi wrote: > > > In "pg_replication_slots" view, the new "wal_status" field is misleading. > > > Consider

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-11-27 Thread Michael Paquier
On Wed, Oct 02, 2019 at 05:08:07PM +0200, Jehan-Guillaume de Rorthais wrote: > I wonder if this is useful to show these messages for slots that were already > dead before this checkpoint? This thread has been waiting for input from the patch author, Horiguchi-san, for a couple of months now, so I

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-10-02 Thread Jehan-Guillaume de Rorthais
On Tue, 30 Jul 2019 21:30:45 +0900 (Tokyo Standard Time) Kyotaro Horiguchi wrote: > Thanks for reviewing! > > At Thu, 27 Jun 2019 16:22:56 +0200, Jehan-Guillaume de Rorthais > wrote in <20190627162256.4f4872b8@firost> > > Hi all, > > > > Being interested by this feature, I did a patch review.

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-09-17 Thread Alvaro Herrera
Hello I have a couple of API-level reservation about this patch series. Firstly, "behind" when used as a noun refers to buttocks. Therefore, the ReplicationSlotsEnumerateBehinds function name seems funny (I think when used as a preposition you wouldn't put it in plural). I don't suggest a

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-07-31 Thread Kyotaro Horiguchi
At Tue, 30 Jul 2019 21:30:45 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190730.213045.221405075.horikyota@gmail.com> > I attach the revised patch. I'll repost the polished version > sooner. This is the revised patch. - Status criteria has been changed. "streaming" :

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-07-30 Thread Kyotaro Horiguchi
At Tue, 30 Jul 2019 21:30:45 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190730.213045.221405075.horikyota@gmail.com> > I attach the revised patch. I'll repost the polished version > sooner. (Mainly TAP test and documentation, code comments will be revised) regards. --

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-07-30 Thread Kyotaro Horiguchi
Thanks for reviewing! At Thu, 27 Jun 2019 16:22:56 +0200, Jehan-Guillaume de Rorthais wrote in <20190627162256.4f4872b8@firost> > Hi all, > > Being interested by this feature, I did a patch review. > > This features adds the GUC "max_slot_wal_keep_size". This is the maximum > amount > of WAL

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-06-27 Thread Jehan-Guillaume de Rorthais
Hi all, Being interested by this feature, I did a patch review. This features adds the GUC "max_slot_wal_keep_size". This is the maximum amount of WAL that can be kept in "pg_wal" by active slots. If the amount of WAL is superior to this limit, the slot is deactivated and its status (new filed

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-02-21 Thread Kyotaro HORIGUCHI
At Fri, 22 Feb 2019 10:12:51 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20190222.101251.0542.horiguchi.kyot...@lab.ntt.co.jp> horiguchi.kyotaro> At Fri, 15 Feb 2019 19:13:23 -0800, Andres Freund wrote in <20190216031323.t7tfrae4l6zqt...@alap3.anarazel.de> > > Maybe I'm

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-02-21 Thread Kyotaro HORIGUCHI
At Fri, 15 Feb 2019 19:13:23 -0800, Andres Freund wrote in <20190216031323.t7tfrae4l6zqt...@alap3.anarazel.de> > Hi, > > On 2019-01-30 10:42:04 +0900, Kyotaro HORIGUCHI wrote: > > From 270aff9b08ced425b4c4e23b53193285eb2359a6 Mon Sep 17 00:00:00 2001 > > From: Kyotaro Horiguchi > > Date: Thu,

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-02-15 Thread Andres Freund
Hi, On 2019-01-30 10:42:04 +0900, Kyotaro HORIGUCHI wrote: > From 270aff9b08ced425b4c4e23b53193285eb2359a6 Mon Sep 17 00:00:00 2001 > From: Kyotaro Horiguchi > Date: Thu, 21 Dec 2017 21:20:20 +0900 > Subject: [PATCH 1/6] Add WAL relief vent for replication slots > > Adds a capability to limit

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-01-29 Thread Kyotaro HORIGUCHI
At Thu, 20 Dec 2018 16:24:38 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20181220.162438.121484007.horiguchi.kyot...@lab.ntt.co.jp> > Thank you for piking this and sorry being late. > > At Mon, 19 Nov 2018 13:39:58 +0900, Michael Paquier > wrote in

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-12-19 Thread Kyotaro HORIGUCHI
Thank you for piking this and sorry being late. At Mon, 19 Nov 2018 13:39:58 +0900, Michael Paquier wrote in <20181119043958.ge4...@paquier.xyz> > ereport should not be called within xlogreader.c as a base rule: Ouch! I forgot that. Fixed to use report_invalid_record slightly changing the

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-11-19 Thread Michael Paquier
On Mon, Nov 19, 2018 at 01:39:58PM +0900, Michael Paquier wrote: > I was just coming by to look at bit at the patch series, and bumped > into that: So I have been looking at the last patch series 0001-0004 posted on this thread, and coming from here:

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-11-18 Thread Michael Paquier
On Fri, Oct 26, 2018 at 11:26:36AM +0900, Kyotaro HORIGUCHI wrote: > The reason for doing that in the fucntion is it can happen also > for physical replication when walsender is active but far > behind. The removed(renamed)-but-still-open segment may be > recycled and can be overwritten while

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-10-29 Thread Masahiko Sawada
On Thu, Oct 25, 2018 at 9:56 PM Kyotaro HORIGUCHI wrote: > > Hello. > > At Mon, 22 Oct 2018 19:35:04 +0900, Masahiko Sawada > wrote in > > > On Thu, Sep 13, 2018 at 6:30 PM Kyotaro HORIGUCHI > > wrote: > > Sorry for the late response. The patch still can be applied to the > > It's alright.

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-10-25 Thread Kyotaro HORIGUCHI
At Thu, 25 Oct 2018 21:55:18 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20181025.215518.189844649.horiguchi.kyot...@lab.ntt.co.jp> > > =# alter system set max_slot_wal_keep_size to '64MB'; -- while > > wal_keep_segments is 0 > > =# select pg_reload_conf(); > > =# select slot_name,

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-10-25 Thread Kyotaro HORIGUCHI
Hello. At Mon, 22 Oct 2018 19:35:04 +0900, Masahiko Sawada wrote in > On Thu, Sep 13, 2018 at 6:30 PM Kyotaro HORIGUCHI > wrote: > Sorry for the late response. The patch still can be applied to the It's alright. Thanks. > curent HEAD so I reviewed the latest patch. > The value of 'remain'

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-10-22 Thread Masahiko Sawada
On Thu, Sep 13, 2018 at 6:30 PM Kyotaro HORIGUCHI wrote: > > Hello. > > Thank you for the comments, Sawada-san, Peter. > > At Mon, 10 Sep 2018 19:52:24 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20180910.195224.22629595.horiguchi.kyot...@lab.ntt.co.jp> > > At Thu, 6 Sep 2018

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-09-13 Thread Kyotaro HORIGUCHI
Hello. Thank you for the comments, Sawada-san, Peter. At Mon, 10 Sep 2018 19:52:24 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180910.195224.22629595.horiguchi.kyot...@lab.ntt.co.jp> > At Thu, 6 Sep 2018 22:32:21 +0200, Peter Eisentraut > wrote in >

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-09-11 Thread Masahiko Sawada
On Mon, Sep 10, 2018 at 7:19 PM, Kyotaro HORIGUCHI wrote: > Hello. > > At Thu, 6 Sep 2018 19:55:39 +0900, Masahiko Sawada > wrote in >> On Thu, Sep 6, 2018 at 4:10 PM, Kyotaro HORIGUCHI >> wrote: >> > Thank you for the comment. >> > >> > At Wed, 5 Sep 2018 14:31:10 +0900, Masahiko Sawada >>

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-09-10 Thread Kyotaro HORIGUCHI
Hello. At Thu, 6 Sep 2018 22:32:21 +0200, Peter Eisentraut wrote in <29bbd79d-696b-509e-578a-0fc38a3b9...@2ndquadrant.com> > This documentation > > + > +Specify the maximum size of WAL files > +that replication > +slots are allowed to retain in the > pg_wal > +

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-09-10 Thread Kyotaro HORIGUCHI
Hello. At Thu, 6 Sep 2018 19:55:39 +0900, Masahiko Sawada wrote in > On Thu, Sep 6, 2018 at 4:10 PM, Kyotaro HORIGUCHI > wrote: > > Thank you for the comment. > > > > At Wed, 5 Sep 2018 14:31:10 +0900, Masahiko Sawada > > wrote in > > > >> On Tue, Sep 4, 2018 at 7:52 PM, Kyotaro HORIGUCHI

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-09-06 Thread Peter Eisentraut
This documentation + +Specify the maximum size of WAL files +that replication +slots are allowed to retain in the pg_wal +directory at checkpoint time. +If max_slot_wal_keep_size is zero (the default), +replication slots retain unlimited size

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-09-06 Thread Masahiko Sawada
On Thu, Sep 6, 2018 at 4:10 PM, Kyotaro HORIGUCHI wrote: > Thank you for the comment. > > At Wed, 5 Sep 2018 14:31:10 +0900, Masahiko Sawada > wrote in >> On Tue, Sep 4, 2018 at 7:52 PM, Kyotaro HORIGUCHI >> Thank you for updating! Here is the review comment for v8 patch. >> >> +/*

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-09-06 Thread Kyotaro HORIGUCHI
Thank you for the comment. At Wed, 5 Sep 2018 14:31:10 +0900, Masahiko Sawada wrote in > On Tue, Sep 4, 2018 at 7:52 PM, Kyotaro HORIGUCHI > Thank you for updating! Here is the review comment for v8 patch. > > +/* > + * This slot still has all required segments.

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-09-04 Thread Masahiko Sawada
On Tue, Sep 4, 2018 at 7:52 PM, Kyotaro HORIGUCHI wrote: > At Mon, 3 Sep 2018 18:14:22 +0900, Masahiko Sawada > wrote in >> Thank you for updating the patch! >> >> On Tue, Jul 31, 2018 at 6:11 PM, Kyotaro HORIGUCHI >> wrote: >> > Hello. >> > >> > At Tue, 24 Jul 2018 16:47:41 +0900, Masahiko

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-09-04 Thread Kyotaro HORIGUCHI
At Mon, 3 Sep 2018 18:14:22 +0900, Masahiko Sawada wrote in > Thank you for updating the patch! > > On Tue, Jul 31, 2018 at 6:11 PM, Kyotaro HORIGUCHI > wrote: > > Hello. > > > > At Tue, 24 Jul 2018 16:47:41 +0900, Masahiko Sawada > > wrote in > > > >> On Mon, Jul 23, 2018 at 4:16 PM,

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-09-03 Thread Masahiko Sawada
Thank you for updating the patch. On Tue, Jul 31, 2018 at 6:11 PM, Kyotaro HORIGUCHI wrote: > Hello. > > At Tue, 24 Jul 2018 16:47:41 +0900, Masahiko Sawada > wrote in >> On Mon, Jul 23, 2018 at 4:16 PM, Kyotaro HORIGUCHI >> wrote: >> > Hello. >> > >> > At Fri, 20 Jul 2018 10:13:58 +0900,

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-08-02 Thread Kyotaro HORIGUCHI
At Thu, 2 Aug 2018 09:05:33 -0400, Robert Haas wrote in > On Tue, Jul 31, 2018 at 9:52 PM, Kyotaro HORIGUCHI > wrote: > > I thought it's to be deprecated for some reason so I'm leaving > > wal_keep_segments in '# of segments' even though the new GUC is > > in MB. I'm a bit uneasy that the two

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-08-02 Thread Robert Haas
On Tue, Jul 31, 2018 at 9:52 PM, Kyotaro HORIGUCHI wrote: > I thought it's to be deprecated for some reason so I'm leaving > wal_keep_segments in '# of segments' even though the new GUC is > in MB. I'm a bit uneasy that the two similar settings are in > different units. Couldn't we turn it into

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-31 Thread Kyotaro HORIGUCHI
At Tue, 31 Jul 2018 12:24:13 -0700, Andres Freund wrote in <20180731192413.7lr4qbc4qbyoi...@alap3.anarazel.de> > On 2018-07-31 15:21:27 -0400, Stephen Frost wrote: > > Greetings, > > > > * Andres Freund (and...@anarazel.de) wrote: > > > On 2018-07-31 15:11:52 -0400, Bruce Momjian wrote: > > > >

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-31 Thread Andres Freund
On 2018-07-31 15:21:27 -0400, Stephen Frost wrote: > Greetings, > > * Andres Freund (and...@anarazel.de) wrote: > > On 2018-07-31 15:11:52 -0400, Bruce Momjian wrote: > > > On Tue, Jun 26, 2018 at 04:26:59PM +0900, Kyotaro HORIGUCHI wrote: > > > > Hello. This is the reabased version of slot-limit

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-31 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2018-07-31 15:11:52 -0400, Bruce Momjian wrote: > > On Tue, Jun 26, 2018 at 04:26:59PM +0900, Kyotaro HORIGUCHI wrote: > > > Hello. This is the reabased version of slot-limit feature. > > > > > > This patch limits maximum WAL segments

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-31 Thread Andres Freund
On 2018-07-31 15:11:52 -0400, Bruce Momjian wrote: > On Tue, Jun 26, 2018 at 04:26:59PM +0900, Kyotaro HORIGUCHI wrote: > > Hello. This is the reabased version of slot-limit feature. > > > > This patch limits maximum WAL segments to be kept by replication > > slots. Replication slot is useful to

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-31 Thread Bruce Momjian
On Tue, Jun 26, 2018 at 04:26:59PM +0900, Kyotaro HORIGUCHI wrote: > Hello. This is the reabased version of slot-limit feature. > > This patch limits maximum WAL segments to be kept by replication > slots. Replication slot is useful to avoid desync with replicas > after temporary disconnection

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-31 Thread Kyotaro HORIGUCHI
Hello. At Tue, 24 Jul 2018 16:47:41 +0900, Masahiko Sawada wrote in > On Mon, Jul 23, 2018 at 4:16 PM, Kyotaro HORIGUCHI > wrote: > > Hello. > > > > At Fri, 20 Jul 2018 10:13:58 +0900, Masahiko Sawada > > wrote in > > .. > > Instead, I made the field be shown in flat "bytes" using bigint,

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-24 Thread Masahiko Sawada
On Mon, Jul 23, 2018 at 4:16 PM, Kyotaro HORIGUCHI wrote: > Hello. > > At Fri, 20 Jul 2018 10:13:58 +0900, Masahiko Sawada > wrote in >> > As I reconsidered this, I noticed that "lsn - lsn" doesn't make >> > sense here. The correct formula for the value is >> > "max_slot_wal_keep_size * 1024 *

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-23 Thread Kyotaro HORIGUCHI
Hello. At Fri, 20 Jul 2018 10:13:58 +0900, Masahiko Sawada wrote in > > As I reconsidered this, I noticed that "lsn - lsn" doesn't make > > sense here. The correct formula for the value is > > "max_slot_wal_keep_size * 1024 * 1024 - ((oldest LSN to keep) - > > restart_lsn). It is not a simple

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-19 Thread Michael Paquier
On Fri, Jul 20, 2018 at 10:13:58AM +0900, Masahiko Sawada wrote: > Also, I'm not sure it's a good way to show the distance as LSN. LSN is > a monotone increasing value but in your patch, a value of the "remain" > column can get decreased. If that can happen, I think that this is a very, very bad

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-19 Thread Masahiko Sawada
On Fri, Jul 13, 2018 at 5:40 PM, Kyotaro HORIGUCHI wrote: > Hello. > > At Wed, 11 Jul 2018 15:09:23 +0900, Masahiko Sawada > wrote in >> On Mon, Jul 9, 2018 at 2:47 PM, Kyotaro HORIGUCHI >> >> min_keep_lsn in pg_replication_slots currently shows the same value in >> every slots but I felt

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-17 Thread Kyotaro HORIGUCHI
Hello. At Tue, 17 Jul 2018 13:37:59 +0900, Masahiko Sawada wrote in > >> The usage of XLogSegNoOffsetToRecPtr is wrong. Since we specify the > >> destination at 4th argument the wal_segment_size will be changed in > >> the above expression. The regression tests by PostgreSQL Patch Tester > > >

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-16 Thread Masahiko Sawada
On Fri, Jul 13, 2018 at 5:40 PM, Kyotaro HORIGUCHI wrote: > Hello. > > At Wed, 11 Jul 2018 15:09:23 +0900, Masahiko Sawada > wrote in >> On Mon, Jul 9, 2018 at 2:47 PM, Kyotaro HORIGUCHI >> wrote: > .. >> Here is review comments of v4 patches. >> >> + if (minKeepLSN) >> + { >> +

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-13 Thread Kyotaro HORIGUCHI
Hello. At Wed, 11 Jul 2018 15:09:23 +0900, Masahiko Sawada wrote in > On Mon, Jul 9, 2018 at 2:47 PM, Kyotaro HORIGUCHI > wrote: .. > Here is review comments of v4 patches. > > + if (minKeepLSN) > + { > + XLogRecPtr slotPtr = XLogGetReplicationSlotMinimumLSN(); > +

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-11 Thread Masahiko Sawada
On Mon, Jul 9, 2018 at 2:47 PM, Kyotaro HORIGUCHI wrote: > Hello. Sawada-san. > > Thank you for the comments. > Thank you for updating the patch! > At Thu, 5 Jul 2018 15:43:56 +0900, Masahiko Sawada > wrote in >> On Wed, Jul 4, 2018 at 5:28 PM, Kyotaro HORIGUCHI >> --- >> +

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-08 Thread Kyotaro HORIGUCHI
Hello. Sawada-san. Thank you for the comments. At Thu, 5 Jul 2018 15:43:56 +0900, Masahiko Sawada wrote in > On Wed, Jul 4, 2018 at 5:28 PM, Kyotaro HORIGUCHI > wrote: > > Hello. > > > > At Tue, 26 Jun 2018 16:26:59 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > > wrote in > >

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-05 Thread Masahiko Sawada
On Wed, Jul 4, 2018 at 5:28 PM, Kyotaro HORIGUCHI wrote: > Hello. > > At Tue, 26 Jun 2018 16:26:59 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20180626.162659.223208514.horiguchi.kyot...@lab.ntt.co.jp> >> The previous patche files doesn't have version number so I let >> the

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-07-04 Thread Kyotaro HORIGUCHI
Hello. At Tue, 26 Jun 2018 16:26:59 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180626.162659.223208514.horiguchi.kyot...@lab.ntt.co.jp> > The previous patche files doesn't have version number so I let > the attached latest version be v2. > > >

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-06-26 Thread Kyotaro HORIGUCHI
Hello. This is the reabased version of slot-limit feature. This patch limits maximum WAL segments to be kept by replication slots. Replication slot is useful to avoid desync with replicas after temporary disconnection but it is dangerous when some of replicas are lost. The WAL space can be

  1   2   >