Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-04-20 Thread Michael Paquier
On Mon, Apr 4, 2016 at 3:22 PM, Michael Paquier wrote: > The approach introducing the concept of WAL progress addresses the > problem of unnecessary checkpoints and of unnecessary standby > snapshots. If we take the problem only from the angle of RUNNING_XACTS > the current logic used to check if

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-04-03 Thread Michael Paquier
On Mon, Apr 4, 2016 at 6:41 AM, Simon Riggs wrote: > I'm not very excited about this patch. Too much code for so little benefit > and fragile too. > > I'm not even sure what definition of "meaningful progress" is useful. If we > commit this, a similar bug could be filed for many similar WAL record

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-04-03 Thread Simon Riggs
On 3 April 2016 at 21:32, Simon Riggs wrote: > On 14 March 2016 at 17:46, David Steele wrote: > >> On 2/24/16 12:40 AM, Michael Paquier wrote: >> >> This has the merit to be clear, thanks for the input. Whatever the >>> approach taken at the end we have two candidates: >>> - Extend XLogInsert()

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-04-03 Thread Simon Riggs
On 14 March 2016 at 17:46, David Steele wrote: > On 2/24/16 12:40 AM, Michael Paquier wrote: > > This has the merit to be clear, thanks for the input. Whatever the >> approach taken at the end we have two candidates: >> - Extend XLogInsert() with an extra argument for flags (Andres) >> - Introduc

[HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-03-15 Thread Michael Paquier
On Mon, Mar 14, 2016 at 6:46 PM, David Steele wrote: > On 2/24/16 12:40 AM, Michael Paquier wrote: > >> This has the merit to be clear, thanks for the input. Whatever the >> approach taken at the end we have two candidates: >> - Extend XLogInsert() with an extra argument for flags (Andres) >> - In

[HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-03-14 Thread David Steele
On 2/24/16 12:40 AM, Michael Paquier wrote: This has the merit to be clear, thanks for the input. Whatever the approach taken at the end we have two candidates: - Extend XLogInsert() with an extra argument for flags (Andres) - Introduce XLogInsertExtended with this extra argument and let XLogIns

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-28 Thread Michael Paquier
On Wed, Feb 24, 2016 at 2:40 PM, Michael Paquier wrote: > This has the merit to be clear, thanks for the input. Whatever the > approach taken at the end we have two candidates: > - Extend XLogInsert() with an extra argument for flags (Andres) > - Introduce XLogInsertExtended with this extra argume

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-23 Thread Robert Haas
On Fri, Feb 19, 2016 at 6:32 AM, Michael Paquier wrote: >> To be honest, my heart still balances for the Extended() interface. >> This reduces the risk of conflicts with back-patching with 9.5. > > Andres, others, what else can I do to make this thread move on? I can > produce any version of this

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-21 Thread Michael Paquier
On Fri, Feb 19, 2016 at 10:33 PM, Amit Kapila wrote: > On Fri, Feb 12, 2016 at 6:57 PM, Michael Paquier wrote: > You doesn't seem to have taken care of below typo in your patch as > pointed out by me earlier. > > + * to not rely on taking an exclusive lock an all the WAL insertion locks, > > /an al

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-19 Thread Amit Kapila
On Fri, Feb 12, 2016 at 6:57 PM, Michael Paquier wrote: > > > OK, here is attached a new version that I hope addresses all the > points raised until now. The following things are changed: > - Extend XLogInsert with a new uint8 argument to have flags. As of now > there is one flag: XLOG_INSERT_NO_P

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-18 Thread Michael Paquier
On Sat, Feb 13, 2016 at 2:26 PM, Michael Paquier wrote: > On Sat, Feb 13, 2016 at 1:01 PM, Robert Haas wrote: >> On Fri, Feb 12, 2016 at 12:56 PM, Andres Freund wrote: >>> On 2016-02-12 12:37:35 -0500, Robert Haas wrote: On Mon, Feb 8, 2016 at 4:18 AM, Andres Freund wrote: > I'm not r

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-12 Thread Michael Paquier
On Sat, Feb 13, 2016 at 1:01 PM, Robert Haas wrote: > On Fri, Feb 12, 2016 at 12:56 PM, Andres Freund wrote: >> On 2016-02-12 12:37:35 -0500, Robert Haas wrote: >>> On Mon, Feb 8, 2016 at 4:18 AM, Andres Freund wrote: >>> > I'm not really a fan. I'd rather change existing callers to add a >>> >

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-12 Thread Robert Haas
On Fri, Feb 12, 2016 at 12:56 PM, Andres Freund wrote: > On 2016-02-12 12:37:35 -0500, Robert Haas wrote: >> On Mon, Feb 8, 2016 at 4:18 AM, Andres Freund wrote: >> > I'm not really a fan. I'd rather change existing callers to add a >> > 'flags' bitmask argument. Right now there can't really be X

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-12 Thread Andres Freund
On 2016-02-12 12:37:35 -0500, Robert Haas wrote: > On Mon, Feb 8, 2016 at 4:18 AM, Andres Freund wrote: > > I'm not really a fan. I'd rather change existing callers to add a > > 'flags' bitmask argument. Right now there can't really be XLogInserts() > > in extension code, so that's pretty ok to ch

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-12 Thread Robert Haas
On Mon, Feb 8, 2016 at 4:18 AM, Andres Freund wrote: > I'm not really a fan. I'd rather change existing callers to add a > 'flags' bitmask argument. Right now there can't really be XLogInserts() > in extension code, so that's pretty ok to change. Yeah, but to what benefit? You're just turning a

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-12 Thread Michael Paquier
On Thu, Feb 11, 2016 at 5:46 PM, Michael Paquier wrote: > On Thu, Feb 11, 2016 at 5:20 PM, Andres Freund wrote: >> On 2016-02-11 09:25:30 +0530, Amit Kapila wrote: >>> On Wed, Feb 10, 2016 at 1:42 PM, Michael Paquier >>> wrote: >>> > >>> > On Wed, Feb 10, 2016 at 5:00 PM, Amit Kapila >>> wrote:

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-11 Thread Michael Paquier
On Thu, Feb 11, 2016 at 5:20 PM, Andres Freund wrote: > On 2016-02-11 09:25:30 +0530, Amit Kapila wrote: >> On Wed, Feb 10, 2016 at 1:42 PM, Michael Paquier >> wrote: >> > >> > On Wed, Feb 10, 2016 at 5:00 PM, Amit Kapila >> wrote: >> > > Okay, but isn't it better that we remove the snapshot tak

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-11 Thread Andres Freund
On 2016-02-11 09:25:30 +0530, Amit Kapila wrote: > On Wed, Feb 10, 2016 at 1:42 PM, Michael Paquier > wrote: > > > > On Wed, Feb 10, 2016 at 5:00 PM, Amit Kapila > wrote: > > > Okay, but isn't it better that we remove the snapshot taken > > > at checkpoint time in the main branch or till where th

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-10 Thread Amit Kapila
On Wed, Feb 10, 2016 at 1:42 PM, Michael Paquier wrote: > > On Wed, Feb 10, 2016 at 5:00 PM, Amit Kapila wrote: > > Okay, but isn't it better that we remove the snapshot taken > > at checkpoint time in the main branch or till where this code is > > getting back-patched. Do you see any need of s

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-10 Thread Michael Paquier
On Wed, Feb 10, 2016 at 5:00 PM, Amit Kapila wrote: > Okay, but isn't it better that we remove the snapshot taken > at checkpoint time in the main branch or till where this code is > getting back-patched. Do you see any need of same after > having the logging of snapshot in bgwriter? But this o

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-10 Thread Amit Kapila
On Wed, Feb 10, 2016 at 1:01 PM, Michael Paquier wrote: > > > On Wed, Feb 10, 2016 at 4:11 PM, Amit Kapila > wrote: > >> >> >> >>> >>> >> > - last_snapshot_lsn != GetXLogInsertRecPtr()) >>> >> > + >>> >> > GetLastCheckpointRecPtr() < GetProgressRecPtr()) >>> >> > >>> >> > How the above check in

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-09 Thread Michael Paquier
On Wed, Feb 10, 2016 at 4:11 PM, Amit Kapila wrote: > On Wed, Feb 10, 2016 at 12:16 PM, Michael Paquier < > michael.paqu...@gmail.com> wrote: > >> >> >> On Wed, Feb 10, 2016 at 12:41 PM, Amit Kapila >> wrote: >> > On Wed, Feb 10, 2016 at 7:17 AM, Michael Paquier < >> michael.paqu...@gmail.com> >

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-09 Thread Amit Kapila
On Wed, Feb 10, 2016 at 12:16 PM, Michael Paquier wrote: > > > On Wed, Feb 10, 2016 at 12:41 PM, Amit Kapila > wrote: > > On Wed, Feb 10, 2016 at 7:17 AM, Michael Paquier < > michael.paqu...@gmail.com> > > wrote: > >> > > > > Consider below code: > > > > /* > > + * Get progress before acquiring

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-09 Thread Michael Paquier
On Wed, Feb 10, 2016 at 12:41 PM, Amit Kapila wrote: > On Wed, Feb 10, 2016 at 7:17 AM, Michael Paquier < michael.paqu...@gmail.com> > wrote: >> >> On Tue, Feb 9, 2016 at 10:42 PM, Amit Kapila wrote: >> > On Tue, Feb 9, 2016 at 6:08 PM, Michael Paquier wrote: >> >> Well, the idea is to improve the

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-09 Thread Amit Kapila
On Wed, Feb 10, 2016 at 7:17 AM, Michael Paquier wrote: > > On Tue, Feb 9, 2016 at 10:42 PM, Amit Kapila wrote: > > On Tue, Feb 9, 2016 at 6:08 PM, Michael Paquier wrote: > >> Well, the idea is to improve the system responsiveness. Imagine that > >> the call to GetProgressRecPtr() is done within t

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-09 Thread Michael Paquier
On Tue, Feb 9, 2016 at 10:42 PM, Amit Kapila wrote: > On Tue, Feb 9, 2016 at 6:08 PM, Michael Paquier wrote: >> Well, the idea is to improve the system responsiveness. Imagine that >> the call to GetProgressRecPtr() is done within the exclusive lock >> portion, but that while scanning the WAL inser

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-09 Thread Amit Kapila
On Tue, Feb 9, 2016 at 6:08 PM, Michael Paquier wrote: > > On Tue, Feb 9, 2016 at 2:24 PM, Amit Kapila wrote: > > Do you see any benefit in allowing checkpoints for such cases considering > > bgwriter will anyway take care of logging standby snapshot for such > > cases? > > Well, the idea is to i

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-09 Thread Michael Paquier
On Tue, Feb 9, 2016 at 2:24 PM, Amit Kapila wrote: > Do you see any benefit in allowing checkpoints for such cases considering > bgwriter will anyway take care of logging standby snapshot for such > cases? Well, the idea is to improve the system responsiveness. Imagine that the call to GetProgres

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-08 Thread Amit Kapila
On Tue, Feb 9, 2016 at 5:37 AM, Michael Paquier wrote: > > On Mon, Feb 8, 2016 at 11:24 PM, Amit Kapila wrote: > > On Mon, Feb 8, 2016 at 12:28 PM, Michael Paquier < michael.paqu...@gmail.com> > > wrote: > >> > >> > >> >> /* > >> >> + * Fetch the progress position before taking any WAL

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-08 Thread Michael Paquier
On Mon, Feb 8, 2016 at 11:24 PM, Amit Kapila wrote: > On Mon, Feb 8, 2016 at 12:28 PM, Michael Paquier > wrote: >> >> >> >> /* >> >> + * Fetch the progress position before taking any WAL insert lock. >> >> This >> >> + * is normally an operation that does not take long, but leavin

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-08 Thread Amit Kapila
On Mon, Feb 8, 2016 at 12:28 PM, Michael Paquier wrote: > > > >> /* > >> + * Fetch the progress position before taking any WAL insert lock. This > >> + * is normally an operation that does not take long, but leaving this > >> + * lookup out of the section taken an exclusive lo

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-08 Thread Michael Paquier
On Mon, Feb 8, 2016 at 6:18 PM, Andres Freund wrote: > On 2016-02-08 15:58:49 +0900, Michael Paquier wrote: >> On Sun, Feb 7, 2016 at 2:49 AM, Andres Freund wrote: >> /* >> + * XLogInsert >> + * >> + * A shorthand for XLogInsertExtended, to update the progress of WAL >> + * activity by default.

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-08 Thread Andres Freund
On 2016-02-08 15:58:49 +0900, Michael Paquier wrote: > On Sun, Feb 7, 2016 at 2:49 AM, Andres Freund wrote: > > On 2016-02-06 22:03:15 +0900, Michael Paquier wrote: > >> The patch attached will apply on master, on 9.5 there is one minor > >> conflict. For older versions we will need another rework

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-07 Thread Michael Paquier
On Sun, Feb 7, 2016 at 2:49 AM, Andres Freund wrote: > On 2016-02-06 22:03:15 +0900, Michael Paquier wrote: >> The patch attached will apply on master, on 9.5 there is one minor >> conflict. For older versions we will need another reworked patch. > > FWIW, I don't think we should backpatch this. I

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-06 Thread Michael Paquier
On Sun, Feb 7, 2016 at 2:49 AM, Andres Freund wrote: > On 2016-02-06 22:03:15 +0900, Michael Paquier wrote: >> + /* >> + * Update the progress LSN positions. At least one WAL insertion lock >> + * is already taken appropriately before doing that, and it is just >> more >> + * s

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-06 Thread Andres Freund
On 2016-02-06 22:03:15 +0900, Michael Paquier wrote: > The patch attached will apply on master, on 9.5 there is one minor > conflict. For older versions we will need another reworked patch. FWIW, I don't think we should backpatch this. It'd look noticeably different in the back branches, and this

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-06 Thread Michael Paquier
On Thu, Feb 4, 2016 at 6:38 PM, Michael Paquier wrote: > On Thu, Feb 4, 2016 at 4:10 PM, Andres Freund wrote: >> On 2016-02-04 18:21:41 +0530, Amit Kapila wrote: >>> I think generally it is good idea, but one thing worth a thought is that >>> by doing so, we need to acquire all WAL Insertion lock

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-04 Thread Michael Paquier
On Thu, Feb 4, 2016 at 4:10 PM, Andres Freund wrote: > On 2016-02-04 18:21:41 +0530, Amit Kapila wrote: >> I think generally it is good idea, but one thing worth a thought is that >> by doing so, we need to acquire all WAL Insertion locks every >> LOG_SNAPSHOT_INTERVAL_MS to check the last_insert_

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-04 Thread Amit Kapila
On Thu, Feb 4, 2016 at 6:40 PM, Andres Freund wrote: > > On 2016-02-04 18:21:41 +0530, Amit Kapila wrote: > > I think generally it is good idea, but one thing worth a thought is that > > by doing so, we need to acquire all WAL Insertion locks every > > LOG_SNAPSHOT_INTERVAL_MS to check the last_in

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-04 Thread Andres Freund
On 2016-02-04 18:21:41 +0530, Amit Kapila wrote: > I think generally it is good idea, but one thing worth a thought is that > by doing so, we need to acquire all WAL Insertion locks every > LOG_SNAPSHOT_INTERVAL_MS to check the last_insert_pos for > every slot, do you think it is matter of concern

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-04 Thread Amit Kapila
On Wed, Feb 3, 2016 at 3:08 PM, Andres Freund wrote: > > On 2016-02-02 10:12:25 +0530, Amit Kapila wrote: > > @@ -8239,7 +8262,7 @@ CreateCheckPoint(int flags) > > if ((flags & (CHECKPOINT_IS_SHUTDOWN | > > CHECKPOINT_END_OF_RECOVERY | > >CHECKPOINT_FORCE)) == 0) > > { > > - if > > (prevPt

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-03 Thread Michael Paquier
On Tue, Feb 2, 2016 at 1:42 PM, Amit Kapila wrote: > On Sat, Jan 30, 2016 at 7:38 PM, Michael Paquier > wrote: >> >> On Fri, Jan 29, 2016 at 9:13 PM, Michael Paquier wrote: >> > Well, to put it short, I am just trying to find a way to make the >> > backend skip unnecessary checkpoints on an idle

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-03 Thread Andres Freund
On 2016-02-02 10:12:25 +0530, Amit Kapila wrote: > @@ -8239,7 +8262,7 @@ CreateCheckPoint(int flags) > if ((flags & (CHECKPOINT_IS_SHUTDOWN | > CHECKPOINT_END_OF_RECOVERY | >CHECKPOINT_FORCE)) == 0) > { > - if > (prevPtr == ControlFile->checkPointCopy.redo && > + if (GetProgressRecPtr() ==

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-01 Thread Amit Kapila
On Sat, Jan 30, 2016 at 7:38 PM, Michael Paquier wrote: > > On Fri, Jan 29, 2016 at 9:13 PM, Michael Paquier wrote: > > Well, to put it short, I am just trying to find a way to make the > > backend skip unnecessary checkpoints on an idle system, which results > > in the following WAL pattern if sy

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-31 Thread Michael Paquier
On Sat, Jan 30, 2016 at 11:08 PM, Michael Paquier wrote: > On Fri, Jan 29, 2016 at 9:13 PM, Michael Paquier wrote: >> Well, to put it short, I am just trying to find a way to make the >> backend skip unnecessary checkpoints on an idle system, which results >> in the following WAL pattern if system

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-30 Thread Michael Paquier
On Fri, Jan 29, 2016 at 9:13 PM, Michael Paquier wrote: > Well, to put it short, I am just trying to find a way to make the > backend skip unnecessary checkpoints on an idle system, which results > in the following WAL pattern if system is completely idle: > CHECKPOINT_ONLINE > RUNNING_XACTS > RUNN

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-29 Thread Michael Paquier
On Fri, Jan 29, 2016 at 5:13 PM, Andres Freund wrote: > On 2016-01-28 16:40:13 +0900, Michael Paquier wrote: >> OK, so as a first step and after thinking about the whole for a while, >> I have finished with the patch attached. This patch is aimed at >> avoiding unnecessary checkpoints on idle syst

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-29 Thread Andres Freund
On 2016-01-28 16:40:13 +0900, Michael Paquier wrote: > OK, so as a first step and after thinking about the whole for a while, > I have finished with the patch attached. This patch is aimed at > avoiding unnecessary checkpoints on idle systems when wal_level >= > hot_standby by centralizing the chec

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-27 Thread Michael Paquier
On Tue, Jan 19, 2016 at 5:10 PM, Amit Kapila wrote: > On Tue, Jan 19, 2016 at 12:41 PM, Michael Paquier > wrote: >> >> On Mon, Jan 18, 2016 at 10:19 PM, Amit Kapila >> wrote: >> > On Mon, Jan 18, 2016 at 10:54 AM, Michael Paquier >> > wrote: >> >> Yes, the thing is that, as mentioned at the beg

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-19 Thread Amit Kapila
On Tue, Jan 19, 2016 at 12:41 PM, Michael Paquier wrote: > > On Mon, Jan 18, 2016 at 10:19 PM, Amit Kapila wrote: > > On Mon, Jan 18, 2016 at 10:54 AM, Michael Paquier > > wrote: > >> Yes, the thing is that, as mentioned at the beginning of the thread, a > >> low value of archive_timeout causes

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-18 Thread Michael Paquier
On Tue, Jan 19, 2016 at 1:28 PM, Amit Kapila wrote: > On Mon, Jan 18, 2016 at 7:08 PM, Andres Freund wrote: >> I find this patch rather unsatisfactory. Yes, it kinda solves the >> problem of archive timeout, but it leaves the bigger and longer standing >> problems of unneccessary checkpoints with

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-18 Thread Michael Paquier
On Mon, Jan 18, 2016 at 10:19 PM, Amit Kapila wrote: > On Mon, Jan 18, 2016 at 10:54 AM, Michael Paquier > wrote: >> >> On Sun, Jan 17, 2016 at 1:37 PM, Amit Kapila >> wrote: >> > On Sat, Jan 16, 2016 at 6:37 PM, Michael Paquier >> > >> > wrote: >> >> >> >> >> >> >> > >> > So here if I understa

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-18 Thread Amit Kapila
On Mon, Jan 18, 2016 at 7:08 PM, Andres Freund wrote: > > On 2015-12-21 16:26:25 +0900, Michael Paquier wrote: > > On Sun, Dec 20, 2015 at 10:14 PM, Michael Paquier > > wrote: > > > Speaking of which, this patch was registered in this CF, I am moving > > > it to the next as a bug fix. > > > > I f

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-18 Thread Andres Freund
On 2015-12-21 16:26:25 +0900, Michael Paquier wrote: > On Sun, Dec 20, 2015 at 10:14 PM, Michael Paquier > wrote: > > Speaking of which, this patch was registered in this CF, I am moving > > it to the next as a bug fix. > > I found a stupid bug in my previous patch: when issuing XLOG_SWITCH it >

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-18 Thread Amit Kapila
On Mon, Jan 18, 2016 at 10:54 AM, Michael Paquier wrote: > > On Sun, Jan 17, 2016 at 1:37 PM, Amit Kapila wrote: > > On Sat, Jan 16, 2016 at 6:37 PM, Michael Paquier < michael.paqu...@gmail.com> > > wrote: > >> > >> > >> > > > > So here if I understand correctly the check related to the last segm

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-17 Thread Michael Paquier
On Sun, Jan 17, 2016 at 1:37 PM, Amit Kapila wrote: > On Sat, Jan 16, 2016 at 6:37 PM, Michael Paquier > wrote: >> >> On Sat, Jan 16, 2016 at 9:07 PM, Amit Kapila >> wrote: >> > On Sat, Jan 16, 2016 at 5:08 PM, Michael Paquier >> > >> > wrote: >> >> >> >> On Sat, Jan 16, 2016 at 7:10 PM, Amit K

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-16 Thread Amit Kapila
On Sat, Jan 16, 2016 at 6:37 PM, Michael Paquier wrote: > > On Sat, Jan 16, 2016 at 9:07 PM, Amit Kapila wrote: > > On Sat, Jan 16, 2016 at 5:08 PM, Michael Paquier < michael.paqu...@gmail.com> > > wrote: > >> > >> On Sat, Jan 16, 2016 at 7:10 PM, Amit Kapila > >> wrote: > >> > Won't this be a p

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-16 Thread Michael Paquier
On Sat, Jan 16, 2016 at 9:07 PM, Amit Kapila wrote: > On Sat, Jan 16, 2016 at 5:08 PM, Michael Paquier > wrote: >> >> On Sat, Jan 16, 2016 at 7:10 PM, Amit Kapila >> wrote: >> > On Sun, Dec 20, 2015 at 6:44 PM, Michael Paquier >> > wrote: >> >> > Looking at the code, it occurred to me that the

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-16 Thread Amit Kapila
On Sat, Jan 16, 2016 at 5:08 PM, Michael Paquier wrote: > > On Sat, Jan 16, 2016 at 7:10 PM, Amit Kapila wrote: > > On Sun, Dec 20, 2015 at 6:44 PM, Michael Paquier < michael.paqu...@gmail.com> > > wrote: > >> On Sun, Nov 8, 2015 at 9:50 PM, Michael Paquier > >> wrote: > >> > On Sat, Nov 7, 2015

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-16 Thread Michael Paquier
On Sat, Jan 16, 2016 at 7:10 PM, Amit Kapila wrote: > On Sun, Dec 20, 2015 at 6:44 PM, Michael Paquier > wrote: >> On Sun, Nov 8, 2015 at 9:50 PM, Michael Paquier >> wrote: >> > On Sat, Nov 7, 2015 at 3:54 PM, Michael Paquier wrote: >> >> I thought about something like that at some point by savi

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-16 Thread Amit Kapila
On Sun, Dec 20, 2015 at 6:44 PM, Michael Paquier wrote: > On Sun, Nov 8, 2015 at 9:50 PM, Michael Paquier > wrote: > > On Sat, Nov 7, 2015 at 3:54 PM, Michael Paquier wrote: > >> I thought about something like that at some point by saving a minimum > >> activity pointer in XLogCtl, updated each t

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-12-20 Thread Michael Paquier
On Sun, Dec 20, 2015 at 10:14 PM, Michael Paquier wrote: > Speaking of which, this patch was registered in this CF, I am moving > it to the next as a bug fix. I found a stupid bug in my previous patch: when issuing XLOG_SWITCH it is possible that the return LSN pointer is on the new segment that

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-12-20 Thread Michael Paquier
On Sun, Nov 8, 2015 at 9:50 PM, Michael Paquier wrote: > On Sat, Nov 7, 2015 at 3:54 PM, Michael Paquier wrote: >> I thought about something like that at some point by saving a minimum >> activity pointer in XLogCtl, updated each time a segment was forcibly >> switched or after inserting a checkpo

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-08 Thread Michael Paquier
On Sat, Nov 7, 2015 at 3:54 PM, Michael Paquier wrote: > I thought about something like that at some point by saving a minimum > activity pointer in XLogCtl, updated each time a segment was forcibly > switched or after inserting a checkpoint record. Then the bgwriter > looked at if the current inse

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-06 Thread Michael Paquier
On Sat, Nov 7, 2015 at 1:52 AM, Andres Freund wrote: > On 2015-11-06 11:42:56 -0500, Robert Haas wrote: >> On Fri, Nov 6, 2015 at 2:47 AM, Michael Paquier >> wrote: >> > I have as well thought a bit about adding a space-related constraint >> > on the standby snapshot generated by the bgwriter, so

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-06 Thread Robert Haas
On Fri, Nov 6, 2015 at 12:26 PM, Andres Freund wrote: > On November 6, 2015 6:21:50 PM GMT+01:00, Robert Haas > wrote: >>On Fri, Nov 6, 2015 at 11:52 AM, Andres Freund >>wrote: >>> Seems like it'd be doable in ReserveXLogInsertLocation(). >>> >>> Whether it's actually worthwhile I'm not all tha

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-06 Thread Andres Freund
On November 6, 2015 6:21:50 PM GMT+01:00, Robert Haas wrote: >On Fri, Nov 6, 2015 at 11:52 AM, Andres Freund >wrote: >> Seems like it'd be doable in ReserveXLogInsertLocation(). >> >> Whether it's actually worthwhile I'm not all that sure tho. > >Why not? Adds another instruction in one of the

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-06 Thread Robert Haas
On Fri, Nov 6, 2015 at 11:52 AM, Andres Freund wrote: > On 2015-11-06 11:42:56 -0500, Robert Haas wrote: >> On Fri, Nov 6, 2015 at 2:47 AM, Michael Paquier >> wrote: >> > I have as well thought a bit about adding a space-related constraint >> > on the standby snapshot generated by the bgwriter, s

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-06 Thread Andres Freund
On 2015-11-06 11:42:56 -0500, Robert Haas wrote: > On Fri, Nov 6, 2015 at 2:47 AM, Michael Paquier > wrote: > > I have as well thought a bit about adding a space-related constraint > > on the standby snapshot generated by the bgwriter, so as to not rely > > entirely on the interval of 15s. I finis

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-06 Thread Robert Haas
On Fri, Nov 6, 2015 at 2:47 AM, Michael Paquier wrote: > I have as well thought a bit about adding a space-related constraint > on the standby snapshot generated by the bgwriter, so as to not rely > entirely on the interval of 15s. I finished with the attached that > uses a check based on CheckPoi

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-05 Thread Michael Paquier
On Thu, Nov 5, 2015 at 3:00 PM, Michael Paquier wrote: > On Wed, Nov 4, 2015 at 7:33 PM, Andres Freund wrote: >> As soon as a single checkpoint ever happened the early-return logic in >> CreateCheckPoint() will fail to take the LogStandbySnapshot() in >> CreateCheckPoint() into account. The test i

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-04 Thread Michael Paquier
On Wed, Nov 4, 2015 at 7:33 PM, Andres Freund wrote: > On 2015-11-04 16:01:28 +0900, Michael Paquier wrote: >> On Wed, Nov 4, 2015 at 8:39 AM, Andres Freund wrote: >> > On November 4, 2015 12:37:02 AM GMT+01:00, Michael Paquier wrote: >> >>On a completely idle system, I don't think we should log

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-04 Thread Andres Freund
On 2015-11-04 16:01:28 +0900, Michael Paquier wrote: > On Wed, Nov 4, 2015 at 8:39 AM, Andres Freund wrote: > > On November 4, 2015 12:37:02 AM GMT+01:00, Michael Paquier wrote: > >>On a completely idle system, I don't think we should log any standby > >>records. This is what ~9.3 does. > > > > Ar

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-03 Thread Michael Paquier
On Wed, Nov 4, 2015 at 12:43 AM, Andres Freund wrote: > On 2015-11-03 10:23:35 -0500, Robert Haas wrote: >> On Mon, Nov 2, 2015 at 12:58 AM, Jeff Janes wrote: >> > If a transaction holding locks aborts on an otherwise idle server, perhaps >> > it will take a very long time for a log-shipping sta

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-03 Thread Michael Paquier
On Wed, Nov 4, 2015 at 8:39 AM, Andres Freund wrote: > On November 4, 2015 12:37:02 AM GMT+01:00, Michael Paquier wrote: >>On a completely idle system, I don't think we should log any standby >>records. This is what ~9.3 does. > > Are you sure? I think it'll around checkpoints, no? I thought Heikk

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-03 Thread Andres Freund
On November 4, 2015 12:37:02 AM GMT+01:00, Michael Paquier wrote: >On Wed, Nov 4, 2015 at 12:43 AM, Andres Freund >wrote: >> On 2015-11-03 10:23:35 -0500, Robert Haas wrote: >>> On Mon, Nov 2, 2015 at 12:58 AM, Jeff Janes >wrote: >>> > If a transaction holding locks aborts on an otherwise idle

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-03 Thread Michael Paquier
On Wed, Nov 4, 2015 at 12:43 AM, Andres Freund wrote: > On 2015-11-03 10:23:35 -0500, Robert Haas wrote: >> On Mon, Nov 2, 2015 at 12:58 AM, Jeff Janes wrote: >> > If a transaction holding locks aborts on an otherwise idle server, perhaps >> > it will take a very long time for a log-shipping sta

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-03 Thread Andres Freund
On 2015-11-03 10:23:35 -0500, Robert Haas wrote: > On Mon, Nov 2, 2015 at 12:58 AM, Jeff Janes wrote: > > If a transaction holding locks aborts on an otherwise idle server, perhaps > > it will take a very long time for a log-shipping standby to realize this. > > But I have hard time believing t

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-03 Thread Robert Haas
On Mon, Nov 2, 2015 at 12:58 AM, Jeff Janes wrote: > If a transaction holding locks aborts on an otherwise idle server, perhaps it > will take a very long time for a log-shipping standby to realize this. But I > have hard time believing that anyone who cares about that would be using > log-shi

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-01 Thread Jeff Janes
On Sun, Nov 1, 2015 at 11:09 PM, Michael Paquier wrote: > On Mon, Nov 2, 2015 at 2:58 PM, Jeff Janes wrote: >> Simple patch, applies and makes cleanly, does what it says and says what it >> does. >> >> If a transaction holding locks aborts on an otherwise idle server, perhaps >> it will take a

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-01 Thread Michael Paquier
On Mon, Nov 2, 2015 at 2:58 PM, Jeff Janes wrote: > Simple patch, applies and makes cleanly, does what it says and says what it > does. > > If a transaction holding locks aborts on an otherwise idle server, perhaps it > will take a very long time for a log-shipping standby to realize this. But

[HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2015-11-01 Thread Jeff Janes
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested Simple patch, applies and makes cleanly, does what it says and says w