Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-10-26 Thread Michael Paquier
On Thu, Oct 26, 2017 at 3:05 AM, Kyotaro HORIGUCHI wrote: > The largest obstacle to do that is that walreceiver is not > utterly concerned to record internals. In other words, it doesn't > know what a record is. Teaching that introduces much complexity > and the complexity slows down the walreceiv

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-10-26 Thread Kyotaro HORIGUCHI
Hello. Thank you for looking this. At Mon, 16 Oct 2017 17:58:03 +0900, Michael Paquier wrote in > On Thu, Sep 7, 2017 at 12:33 PM, Kyotaro HORIGUCHI > wrote: > > At Wed, 6 Sep 2017 12:23:53 -0700, Andres Freund wrote > > in <20170906192353.ufp2dq7wm5fd6...@alap3.anarazel.de> > >> I'm not fol

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-10-16 Thread Michael Paquier
On Thu, Sep 7, 2017 at 12:33 PM, Kyotaro HORIGUCHI wrote: > At Wed, 6 Sep 2017 12:23:53 -0700, Andres Freund wrote > in <20170906192353.ufp2dq7wm5fd6...@alap3.anarazel.de> >> I'm not following. All we need to use is the beginning of the relevant >> records, that's easy enough to keep track of. W

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-09-06 Thread Kyotaro HORIGUCHI
Hello, At Wed, 6 Sep 2017 12:23:53 -0700, Andres Freund wrote in <20170906192353.ufp2dq7wm5fd6...@alap3.anarazel.de> > On 2017-09-06 17:36:02 +0900, Kyotaro HORIGUCHI wrote: > > The problem is that the current ReadRecord needs the first one of > > a series of continuation records from the same s

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-09-06 Thread Andres Freund
On 2017-09-06 17:36:02 +0900, Kyotaro HORIGUCHI wrote: > Hi, > > At Mon, 4 Sep 2017 17:17:19 +0900, Michael Paquier > wrote in > > > On Mon, Sep 4, 2017 at 4:04 PM, Andres Freund wrote: > > > I've not read through the thread, but this seems like the wrong approach > > > to me. The receiving s

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-09-06 Thread Kyotaro HORIGUCHI
Hi, At Mon, 4 Sep 2017 17:17:19 +0900, Michael Paquier wrote in > On Mon, Sep 4, 2017 at 4:04 PM, Andres Freund wrote: > > I've not read through the thread, but this seems like the wrong approach > > to me. The receiving side should use a correct value, instead of putting > > this complexity o

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-09-04 Thread Michael Paquier
On Mon, Sep 4, 2017 at 4:04 PM, Andres Freund wrote: > I've not read through the thread, but this seems like the wrong approach > to me. The receiving side should use a correct value, instead of putting > this complexity on the sender's side. Yes I agree with that. The current patch gives me a ba

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-09-04 Thread Andres Freund
Hi, On 2017-09-04 15:51:51 +0900, Kyotaro HORIGUCHI wrote: > SpinLockAcquire(&walsnd->mutex); > + oldFlushPtr = walsnd->flush; > walsnd->write = writePtr; > walsnd->flush = flushPtr; > walsnd->apply = applyPtr; > @@ -1821,7 +1836,

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-09-03 Thread Kyotaro HORIGUCHI
Hello, Thank you for reviewing this. At Mon, 28 Aug 2017 20:14:54 +0900, Michael Paquier wrote in > On Mon, Aug 28, 2017 at 8:02 PM, Kyotaro HORIGUCHI > wrote: > > The first patch (0001-) fixes this problem, preventing the > > problematic state of WAL segments by retarding restart LSN of a >

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-08-28 Thread Michael Paquier
On Mon, Aug 28, 2017 at 8:02 PM, Kyotaro HORIGUCHI wrote: > The first patch (0001-) fixes this problem, preventing the > problematic state of WAL segments by retarding restart LSN of a > physical replication slot in a certain condition. FWIW, I have this patch marked on my list of things to look

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-08-28 Thread Kyotaro HORIGUCHI
Hello, This problem still occurs on the master. I rebased this to the current master. At Mon, 3 Apr 2017 08:38:47 +0900, Michael Paquier wrote in > On Mon, Apr 3, 2017 at 7:19 AM, Venkata B Nagothi wrote: > > As we are already past the commitfest, I am not sure, what should i change > > the p

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-04-02 Thread Michael Paquier
On Mon, Apr 3, 2017 at 7:19 AM, Venkata B Nagothi wrote: > As we are already past the commitfest, I am not sure, what should i change > the patch status to ? The commit fest finishes on the 7th of April. Even with the deadline passed, there is nothing preventing to work on bug fixes. So this item

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-04-02 Thread Venkata B Nagothi
On Fri, Mar 31, 2017 at 4:05 PM, Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > Thank you having a look on this. > > # I removed -bugs in CC:. > > At Fri, 31 Mar 2017 13:40:00 +1100, Venkata B Nagothi > wrote in gmail.com> > > On Fri, Mar 17, 2017 at 6:48 PM, Kyotaro HORIGUCHI < >

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-30 Thread Kyotaro HORIGUCHI
Thank you having a look on this. # I removed -bugs in CC:. At Fri, 31 Mar 2017 13:40:00 +1100, Venkata B Nagothi wrote in > On Fri, Mar 17, 2017 at 6:48 PM, Kyotaro HORIGUCHI < > horiguchi.kyot...@lab.ntt.co.jp> wrote: > > > Hello, > > > > At Mon, 13 Mar 2017 11:06:00 +1100, Venkata B Nagothi

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-30 Thread Venkata B Nagothi
On Fri, Mar 17, 2017 at 6:48 PM, Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > Hello, > > At Mon, 13 Mar 2017 11:06:00 +1100, Venkata B Nagothi > wrote in gmail.com> > > On Tue, Jan 17, 2017 at 9:36 PM, Kyotaro HORIGUCHI < > > horiguchi.kyot...@lab.ntt.co.jp> wrote: > > > I manag

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-30 Thread Venkata B Nagothi
On Thu, Mar 30, 2017 at 4:46 PM, Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > Hello, > > At Thu, 30 Mar 2017 15:59:14 +1100, Venkata B Nagothi > wrote in xwu0j05x_j...@mail.gmail.com> > > Yes, downloaded from the email on Windows and copied across to Linux and > > did "git apply

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-29 Thread Kyotaro HORIGUCHI
Hello, At Thu, 30 Mar 2017 15:59:14 +1100, Venkata B Nagothi wrote in > Yes, downloaded from the email on Windows and copied across to Linux and > did "git apply". The same works for me. But --keep-cr gave me the same result with you. > $ git am --keep-cr > ~/work/patches/0001-Fix-a-bug-of-p

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-29 Thread Venkata B Nagothi
On Thu, Mar 30, 2017 at 3:51 PM, Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > At Thu, 30 Mar 2017 11:12:56 +1100, Venkata B Nagothi > wrote in gmail.com> > > On Thu, Mar 30, 2017 at 10:55 AM, Michael Paquier < > michael.paqu...@gmail.com > > > wrote: > > > > > On Thu, Mar 30, 20

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-29 Thread Kyotaro HORIGUCHI
At Thu, 30 Mar 2017 11:12:56 +1100, Venkata B Nagothi wrote in > On Thu, Mar 30, 2017 at 10:55 AM, Michael Paquier > wrote: > > > On Thu, Mar 30, 2017 at 8:49 AM, Venkata B Nagothi > > wrote: > > > On Tue, Mar 28, 2017 at 5:51 PM, Kyotaro HORIGUCHI > > > wrote: > > > I tried applying this pa

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-29 Thread Tatsuo Ishii
> On Thu, Mar 30, 2017 at 9:12 AM, Tatsuo Ishii wrote: >> Committers will not apply patches which has trailing whitespace >> issues. So the patch submitter needs to fix them anyway. > > I cannot comment on that point (committers are free to pick up things > the way they want), but just using git

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-29 Thread Michael Paquier
On Thu, Mar 30, 2017 at 9:12 AM, Tatsuo Ishii wrote: > Committers will not apply patches which has trailing whitespace > issues. So the patch submitter needs to fix them anyway. I cannot comment on that point (committers are free to pick up things the way they want), but just using git commands t

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-29 Thread Venkata B Nagothi
On Thu, Mar 30, 2017 at 10:55 AM, Michael Paquier wrote: > On Thu, Mar 30, 2017 at 8:49 AM, Venkata B Nagothi > wrote: > > On Tue, Mar 28, 2017 at 5:51 PM, Kyotaro HORIGUCHI > > wrote: > > I tried applying this patch to latest master, it is not getting applied > > > > [dba@buildhost postgresql]

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-29 Thread Tatsuo Ishii
> On Thu, Mar 30, 2017 at 8:49 AM, Venkata B Nagothi wrote: >> On Tue, Mar 28, 2017 at 5:51 PM, Kyotaro HORIGUCHI >> wrote: >> I tried applying this patch to latest master, it is not getting applied >> >> [dba@buildhost postgresql]$ git apply >> /data/postgresql-patches/9.5-ReplicationSlots-Bug-P

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-29 Thread Michael Paquier
On Thu, Mar 30, 2017 at 8:49 AM, Venkata B Nagothi wrote: > On Tue, Mar 28, 2017 at 5:51 PM, Kyotaro HORIGUCHI > wrote: > I tried applying this patch to latest master, it is not getting applied > > [dba@buildhost postgresql]$ git apply > /data/postgresql-patches/9.5-ReplicationSlots-Bug-Patch/000

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-29 Thread Venkata B Nagothi
Regards, Venkata B N Database Consultant On Tue, Mar 28, 2017 at 5:51 PM, Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > This conflicts with 6912acc (replication lag tracker) so just > rebased on a6f22e8. > I tried applying this patch to latest master, it is not getting applied

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-27 Thread Kyotaro HORIGUCHI
This conflicts with 6912acc (replication lag tracker) so just rebased on a6f22e8. At Fri, 17 Mar 2017 16:48:27 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170317.164827.46663014.horiguchi.kyot...@lab.ntt.co.jp> > Hello, > > At Mon, 13 Mar 2017 11:06:00 +1100, Venkata B Nagothi

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-17 Thread Kyotaro HORIGUCHI
Hello, At Mon, 13 Mar 2017 11:06:00 +1100, Venkata B Nagothi wrote in > On Tue, Jan 17, 2017 at 9:36 PM, Kyotaro HORIGUCHI < > horiguchi.kyot...@lab.ntt.co.jp> wrote: > > I managed to reproduce this. A little tweak as the first patch > > lets the standby to suicide as soon as walreceiver sees a

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-03-12 Thread Venkata B Nagothi
On Tue, Jan 17, 2017 at 9:36 PM, Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > Hello. I added pgsql-hackers. > > This occurs also on git master and back to 9.4. > > At Fri, 13 Jan 2017 08:47:06 -0600, Jonathon Nelson > wrote in gmail.com> > > On Mon, Nov 28, 2016 at 1:39 PM, Jona

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-02-02 Thread Kyotaro HORIGUCHI
At Thu, 2 Feb 2017 15:34:33 +0900, Michael Paquier wrote in > On Thu, Feb 2, 2017 at 1:26 AM, Fujii Masao wrote: > > I'm afraid that many WAL segments would start with a continuation record > > when there are the workload of short transactions (e.g., by pgbench), and > > which would make restar

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-02-01 Thread Michael Paquier
On Thu, Feb 2, 2017 at 1:26 AM, Fujii Masao wrote: > I'm afraid that many WAL segments would start with a continuation record > when there are the workload of short transactions (e.g., by pgbench), and > which would make restart_lsn go behind very much. No? I don't quite understand this argument.

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-02-01 Thread Kyotaro HORIGUCHI
Thank you for the comment. At Thu, 2 Feb 2017 01:26:03 +0900, Fujii Masao wrote in > > The attached patch does that. Usually it reads page headers only > > on segment boundaries, but once continuation record found (or > > failed to read the next page header, that is, the first record on > > the

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-02-01 Thread Fujii Masao
On Thu, Jan 19, 2017 at 6:37 PM, Kyotaro HORIGUCHI wrote: > Hello, > > At Wed, 18 Jan 2017 12:34:51 +0900, Michael Paquier > wrote in > >> On Tue, Jan 17, 2017 at 7:36 PM, Kyotaro HORIGUCHI >> wrote: >> > I managed to reproduce this. A little tweak as the first patch >> > lets the standby to

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-01-31 Thread Kyotaro HORIGUCHI
Hello, I'll add the rebased version to the next CF. At Fri, 20 Jan 2017 11:07:29 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170120.110729.107284864.horiguchi.kyot...@lab.ntt.co.jp> > > > > - Delaying recycling a segment until the last partial record on it > > > > completes. Th

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-01-19 Thread Kyotaro HORIGUCHI
Hello, At Thu, 19 Jan 2017 18:37:31 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170119.183731.223893446.horiguchi.kyot...@lab.ntt.co.jp> > > > - Delaying recycling a segment until the last partial record on it > > > completes. This seems doable in page-wise (coarse resolution)

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-01-19 Thread Kyotaro HORIGUCHI
Hello, At Wed, 18 Jan 2017 12:34:51 +0900, Michael Paquier wrote in > On Tue, Jan 17, 2017 at 7:36 PM, Kyotaro HORIGUCHI > wrote: > > I managed to reproduce this. A little tweak as the first patch > > lets the standby to suicide as soon as walreceiver sees a > > contrecord at the beginning of

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-01-17 Thread Michael Paquier
On Tue, Jan 17, 2017 at 7:36 PM, Kyotaro HORIGUCHI wrote: > I managed to reproduce this. A little tweak as the first patch > lets the standby to suicide as soon as walreceiver sees a > contrecord at the beginning of a segment. Good idea. > I believe that a continuation record cannot be span over

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-01-17 Thread Kyotaro HORIGUCHI
Auch! It is wrong. Not decrement keep, decrement segno. 2017年1月17日(火) 19:37 Kyotaro HORIGUCHI : > Hello. I added pgsql-hackers. > > This occurs also on git master and back to 9.4. > > At Fri, 13 Jan 2017 08:47:06 -0600, Jonathon Nelson > wrote in < > cacjqam1ydczcd5docp+y5hkwto1zegw+mj8uk7avqct

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-01-17 Thread Kyotaro HORIGUCHI
Hello. I added pgsql-hackers. This occurs also on git master and back to 9.4. At Fri, 13 Jan 2017 08:47:06 -0600, Jonathon Nelson wrote in > On Mon, Nov 28, 2016 at 1:39 PM, Jonathon Nelson wrote: > > First, the postgresql configuration differs only minimally from the stock > > config: > > >