Re: [HACKERS] Timeline following for logical slots

Alvaro Herrera wrote: > Here's a proposed revert patch. Many minor changes (mostly comment > additions) that were applied as part of this series are kept intact. > The test_slot_timeline test module and corresponding recovery test > script are also reverted. Pushed. -- Álvaro Herrera

Re: [HACKERS] Timeline following for logical slots

I don't like reverting patches, but this patch is making me more and more uncomfortable. We have two open items, one of which requires writing new test code that doesn't exist yet; and we have the pg_recvlogical changes that were approved post-feature freeze, but that I now have second thoughts ab

Re: [HACKERS] Timeline following for logical slots

Craig Ringer wrote: > With this patch pg_recvlogical takes a new --endpos LSN argument, and will > exit if either: > > * it receives an XLogData message with dataStart >= endpos; or > * it receives a keepalive with walEnd >= endpos > > The latter allows it to work without needing a dummy transa

Re: [HACKERS] Timeline following for logical slots

Andres Freund wrote: > - /* oldest LSN that the client has acked receipt for */ > + /* > +* Oldest LSN that the client has acked receipt for. This is used as the > +* start_lsn point in case the client doesn't specify one, and also as a > +* safety measure to back off in case the

Re: [HACKERS] Timeline following for logical slots

On 2016-05-02 16:12:53 -0300, Alvaro Herrera wrote: > I pushed a fix to some comments, including the ones being discussed in > this subthread, which should hopefully close things here. > > I'm now going to go over Craig's pg_recvlogical changes and the proposed > for that problem. - /* oldest

Re: [HACKERS] Timeline following for logical slots

I pushed a fix to some comments, including the ones being discussed in this subthread, which should hopefully close things here. I'm now going to go over Craig's pg_recvlogical changes and the proposed for that problem. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Deve

Re: [HACKERS] Timeline following for logical slots

Robert Haas wrote: > On Tue, Apr 26, 2016 at 4:37 PM, Alvaro Herrera > wrote: > > I failed to meet this deadline, for which I apologize. This week is > > going to be hectic around here, so my new deadline is to get these two > > patches applied on Friday 29th. Ok? > > Not that you don't know t

Re: [HACKERS] Timeline following for logical slots

On Tue, Apr 26, 2016 at 4:37 PM, Alvaro Herrera wrote: > Alvaro Herrera wrote: >> Noah Misch wrote: >> >> > The above-described topic is currently a PostgreSQL 9.6 open item. Alvaro, >> > since you committed the patch believed to have created it, you own this >> > open >> > item. If that respon

Re: [HACKERS] Timeline following for logical slots

On 29 April 2016 at 15:40, Craig Ringer wrote: > I don't think pg_recvlogical can do anything about the need for that dummy > write, since the client has no way to determine the exact LSN of the commit > record of the xact of interest. It can't rely > on pg_current_xlog_insert_location() or pg_c

Re: [HACKERS] Timeline following for logical slots

On 28 April 2016 at 02:29, Andres Freund wrote: > > I don't object either, I was looking for the feature myself a number of > times (for tap tests in my case). > Exactly what I want it for. > It requires a some amount of thinking about what the limit applies to > though. "messages sent by ser

Re: [HACKERS] Timeline following for logical slots

On 2016-04-27 14:21:53 -0400, Robert Haas wrote: > > Considering that pg_recvlogical was introduced mostly as a way to test > > logical decoding features, I think this is a serious oversight and we > > should patch it. I suppose we could leave it for 9.7, thought I admit I > > would prefer it to i

Re: [HACKERS] Timeline following for logical slots

On Tue, Apr 26, 2016 at 4:35 PM, Alvaro Herrera wrote: > Craig Ringer wrote: >> The reason the new src/test/recovery/ tests don't notice this is that using >> pg_recvlogical from the TAP tests is currently pretty awkward. >> pg_recvlogical has no way to specify a stop-point or return when there's

Re: [HACKERS] Timeline following for logical slots

On Tue, Apr 26, 2016 at 05:37:40PM -0300, Alvaro Herrera wrote: > Alvaro Herrera wrote: > > Noah Misch wrote: > > > The above-described topic is currently a PostgreSQL 9.6 open item. > > > Alvaro, > > > since you committed the patch believed to have created it, you own this > > > open > > > item

Re: [HACKERS] Timeline following for logical slots

Alvaro Herrera wrote: > Noah Misch wrote: > > > The above-described topic is currently a PostgreSQL 9.6 open item. Alvaro, > > since you committed the patch believed to have created it, you own this open > > item. If that responsibility lies elsewhere, please let us know whose > > responsibility

Re: [HACKERS] Timeline following for logical slots

Craig Ringer wrote: > The reason the new src/test/recovery/ tests don't notice this is that using > pg_recvlogical from the TAP tests is currently pretty awkward. > pg_recvlogical has no way to specify a stop-point or return when there's no > immediately pending data like the SQL interface does. S

Re: [HACKERS] Timeline following for logical slots

On 2016-04-26 17:22:49 -0300, Alvaro Herrera wrote: > > - /* oldest LSN that might be required by this replication slot */ > > + /* > > +* oldest LSN that might be required by this replication slot. > > +* > > +* Points to the LSN of the most recent xl_running_xacts record where > >

Re: [HACKERS] Timeline following for logical slots

Craig Ringer wrote: > - /* oldest LSN that the client has acked receipt for */ > + /* > + * oldest LSN that the client has acked receipt for > + * > + * Decoding will start calling output plugin callbacks for commits > + * after this LSN unless a different start point i

Re: [HACKERS] Timeline following for logical slots

Noah Misch wrote: > The above-described topic is currently a PostgreSQL 9.6 open item. Alvaro, > since you committed the patch believed to have created it, you own this open > item. If that responsibility lies elsewhere, please let us know whose > responsibility it is to fix this. Since new ope

Re: [HACKERS] Timeline following for logical slots

On Sun, Apr 17, 2016 at 10:01:36PM +0800, Craig Ringer wrote: > I made an unfortunate oversight in the logical decoding timeline following > code: it doesn't work for logical decoding from the walsender, because I > left the glue code required out of the final cut of the patch. > Subject: [PATCH]

Re: [HACKERS] Timeline following for logical slots

Hi all I made an unfortunate oversight in the logical decoding timeline following code: it doesn't work for logical decoding from the walsender, because I left the glue code required out of the final cut of the patch. The reason the new src/test/recovery/ tests don't notice this is that using pg_

Re: [HACKERS] Timeline following for logical slots

On 7 April 2016 at 23:32, Robert Haas wrote: > > Yeah. I understand the reasons for that decision. Per an earlier reply I > > think we can avoid making them WAL-logged so they can be used on standbys > > and still achieve usable failover support on physical replicas. > > I think at one point we

Re: [HACKERS] Timeline following for logical slots

On 07/04/16 17:32, Robert Haas wrote: Second, I'm not sure whether it was a good design decision to make logical slots a special kind of object that sit off to the side, neither configuration (like postgresql.conf) nor WAL-protected data (like pg_clog and the data files themselves), but it was ce

Re: [HACKERS] Timeline following for logical slots

On Tue, Apr 5, 2016 at 10:34 PM, Craig Ringer wrote: >> Right, but right now you probably *aren't* do doing any kind of >> logical decoding from a master server to a standby, because there's >> squat in the core distribution that could make use of that capability. >> So you never get as far as dis

Re: [HACKERS] Timeline following for logical slots

Draft patch for comment fix.​ From 304453f67b26219dc94e6aef9dfdc1c1debf12bb Mon Sep 17 00:00:00 2001 From: Craig Ringer Date: Wed, 6 Apr 2016 14:57:26 +0800 Subject: [PATCH] Fix incorrect comments introduced in logical decoding timeline following --- src/backend/replication/logical/logicalfuncs

Re: [HACKERS] Timeline following for logical slots

I tried to address Andres's entirely valid complaints about that comment I added in this patch. I was going to add a description to restart_lsn in slot.h instead, but it's proving harder to accurately describe restart_lsn than I'd like. It's a point at which no transactions that commit after conf

Re: [HACKERS] Timeline following for logical slots

On 06/04/16 03:29, Robert Haas wrote: I don't understand why it seems to be considered OK for logical slots to just vanish on failover. The only other things I can think of where that's considered OK are unlogged tables (because that's the point and we have failover-safe ones too) and the old h

Re: [HACKERS] Timeline following for logical slots

On 6 April 2016 at 09:29, Robert Haas wrote: > > Right now if you're doing any kind of logical deocoding from a master > server > > that fails over to a standby the client just dies. The slot vanishes. > You're > > stuffed. Gee, I hope you didn't need all that nice consistent ordering, > > becau

Re: [HACKERS] Timeline following for logical slots

On Tue, Apr 5, 2016 at 3:51 AM, Craig Ringer wrote: > First, I'd like to remind everyone that logical decoding is useful for more > than replication. You can consume the change stream for audit > logging/archival, to feed into a different DBMS, etc etc. This is not just > about replicating from on

Re: [HACKERS] Timeline following for logical slots

On 5 April 2016 at 14:09, Andres Freund wrote: > On 2016-04-05 05:53:53 +0200, Petr Jelinek wrote: > > On 04/04/16 17:15, Andres Freund wrote: > > > > > >>* Robust sequence decoding and replication. If you were following the > later > > >>parts of that discussion you will've seen how fun that's g

Re: [HACKERS] Timeline following for logical slots

> On 05 Apr 2016, at 09:51, Craig Ringer wrote: > > On 5 April 2016 at 04:00, Robert Haas > wrote: > > In general, I think we'd be a lot better off if we got some kind of > logical replication into core first and then worked on lifting these > types of limitations

Re: [HACKERS] Timeline following for logical slots

On 2016-04-05 15:51:00 +0800, Craig Ringer wrote: > Review and test responses have been pretty underwhelming for pglogical, and > quite a bit seem to have boiled down to "this should live as an extension, > we don't need it in core". It often feels like we can't win: if we seek to > get it into cor

Re: [HACKERS] Timeline following for logical slots

On 5 April 2016 at 14:18, Peter Geoghegan wrote: > > I rather agree that an in-core system that solved some of the basic > problems would be a huge step forward, and would motivate people to > work on the harder problems. It's surprising to me that we don't seem > to be much closer to that then

Re: [HACKERS] Timeline following for logical slots

On 5 April 2016 at 04:00, Robert Haas wrote: > > In general, I think we'd be a lot better off if we got some kind of > logical replication into core first and then worked on lifting these > types of limitations afterwards. First, I'd like to remind everyone that logical decoding is useful for m

Re: [HACKERS] Timeline following for logical slots

On Mon, Apr 4, 2016 at 1:00 PM, Robert Haas wrote: > I don't necessarily disagree with your statement that we'd need all of > this stuff to make logical replication a substitute for physical > replication as far as failover is concerned. But I don't think that's > the most important goal, and eve

Re: [HACKERS] Timeline following for logical slots

On 2016-04-05 05:53:53 +0200, Petr Jelinek wrote: > On 04/04/16 17:15, Andres Freund wrote: > > > >>* Robust sequence decoding and replication. If you were following the later > >>parts of that discussion you will've seen how fun that's going to be, but > >>it's the simplest of all of the problems.

Re: [HACKERS] Timeline following for logical slots

On 04/04/16 17:15, Andres Freund wrote: * Robust sequence decoding and replication. If you were following the later parts of that discussion you will've seen how fun that's going to be, but it's the simplest of all of the problems. Unconvinced. People used londiste and slony for years without

Re: [HACKERS] Timeline following for logical slots

On 04/04/16 22:00, Robert Haas wrote: On Mon, Apr 4, 2016 at 10:59 AM, Craig Ringer wrote: To allow logical rep and failover to be a reasonable substitute for physical rep and failover IMO *need*: * Robust sequence decoding and replication. If you were following the later parts of that discuss

Re: [HACKERS] Timeline following for logical slots

On Mon, Apr 4, 2016 at 10:59 AM, Craig Ringer wrote: > To allow logical rep and failover to be a reasonable substitute for physical > rep and failover IMO *need*: > > * Robust sequence decoding and replication. If you were following the later > parts of that discussion you will've seen how fun tha

Re: [HACKERS] Timeline following for logical slots

On 2016-04-04 22:59:41 +0800, Craig Ringer wrote: > Assuming that here you mean separate slots on different machines > replicating via physical rep: No, I don't. > We don't currently allow the creation of a logical slot on a standby. Nor > replay from it, even to advance it without receiving the

Re: [HACKERS] Timeline following for logical slots

On 4 April 2016 at 18:01, Andres Freund wrote: > > The only way I can think of to do that really reliably right now, without > > full failover slots, is to use the newly committed pluggable WAL > mechanism > > and add a hook to SaveSlotToPath() so slot info can be captured, injected > > in WAL,

Re: [HACKERS] Timeline following for logical slots

On 2016-04-04 17:50:02 +0800, Craig Ringer wrote: > To rephrase per my understanding: The client only specifies the point it > wants to start seeing decoded commits. Decoding starts from the slot's > restart_lsn, and that's the point from which the accumulation of reorder > buffer contents begins,

Re: [HACKERS] Timeline following for logical slots

On 1 April 2016 at 14:46, Andres Freund wrote: > > However: It's safe for the slot state on the replica to be somewhat > behind > > the local replay from the master, so the slot state on the replica is > older > > than what it would've been at an equivalent time on the master... so long > > as i

Re: [HACKERS] Timeline following for logical slots

On 4 April 2016 at 14:43, Andres Freund wrote: > > OK, makes sense. And still resume decoding from restart_lsn, despite > having > > that visibility information stashed, because we also have to rebuild the > > information on invalidations for running xacts, which is not stored > > persistently a

Re: [HACKERS] Timeline following for logical slots

On 2016-04-04 14:36:29 +0800, Craig Ringer wrote: > On 4 April 2016 at 14:30, Andres Freund wrote: > > > On 2016-04-04 14:24:52 +0800, Craig Ringer wrote: > > > I don't feel like I've grasped this properly yet. I think it's referring > > to > > > the pg_logical/snapshots/ serialization, the use o

Re: [HACKERS] Timeline following for logical slots

On 2016-04-04 14:18:53 +0800, Craig Ringer wrote: > I want to go over the rest of your reply in more detail, but regarding this > and the original comment, I'm going by: > > if (start_lsn == InvalidXLogRecPtr) > { > /* continue from last position */ > start_lsn = slot->data

Re: [HACKERS] Timeline following for logical slots

On 4 April 2016 at 14:30, Andres Freund wrote: > On 2016-04-04 14:24:52 +0800, Craig Ringer wrote: > > I don't feel like I've grasped this properly yet. I think it's referring > to > > the pg_logical/snapshots/ serialization, the use of which allows us to > > avoid doing extra work in SnapBuildFi

Re: [HACKERS] Timeline following for logical slots

On 2016-04-04 14:24:52 +0800, Craig Ringer wrote: > I don't feel like I've grasped this properly yet. I think it's referring to > the pg_logical/snapshots/ serialization, the use of which allows us to > avoid doing extra work in SnapBuildFindSnapshot(...), but doesn't seem to > be crucial for corre

Re: [HACKERS] Timeline following for logical slots

On 1 April 2016 at 14:52, Andres Freund wrote: > Hi, > > On 2016-04-01 08:46:01 +0200, Andres Freund wrote: > > That's a fundamental misunderstanding on your part (perhaps created by > > imprecise docs). > > > > Speaking of which, did you see the proposed README I sent for > > > src/backend/repli

Re: [HACKERS] Timeline following for logical slots

On 1 April 2016 at 14:46, Andres Freund wrote: > > > You're thinking from the perspective of someone who knows this code > > intimately. > > Maybe. But that's not addressed by adding half-true comments to places > they only belong to peripherally. And not to all the relevant places, > since you'

Re: [HACKERS] Timeline following for logical slots

Hi, On 2016-04-01 08:46:01 +0200, Andres Freund wrote: > That's a fundamental misunderstanding on your part (perhaps created by > imprecise docs). > > Speaking of which, did you see the proposed README I sent for > > src/backend/replication/logical ? > > I skimmed it. But given we have a CF full

Re: [HACKERS] Timeline following for logical slots

Hi, On 2016-04-01 13:16:18 +0800, Craig Ringer wrote: > I think it's pretty unsafe from SQL, to be sure. > > Unless failover slots get in to 9.6 we'll need to do exactly that from > internal C stuff in pglogical to support following physical failover, I know. And this makes me scared shitless.

Re: [HACKERS] Timeline following for logical slots

On 31 March 2016 at 16:09, Andres Freund wrote: > > This gives us an option for failover of logical replication in 9.6, even > if > > it's a bit cumbersome and complex for the client, in case failover slots > > don't make the cut. And, of course, it's a pre-req for failover slots, > > which I'll

Re: [HACKERS] Timeline following for logical slots

Hi, On 2016-03-31 08:52:34 +0800, Craig Ringer wrote: > On 31 March 2016 at 07:15, Alvaro Herrera wrote: > > > > > Available attached or at > > > > > https://github.com/2ndQuadrant/postgres/tree/dev/logical-decoding-timeline-following > > > > And pushed this too. > > > > Much appreciated. Mark

Re: [HACKERS] Timeline following for logical slots

On 31 March 2016 at 07:15, Alvaro Herrera wrote: > > Available attached or at > > > https://github.com/2ndQuadrant/postgres/tree/dev/logical-decoding-timeline-following > > And pushed this too. > Much appreciated. Marked as committed at https://commitfest.postgresql.org/9/568/ . This gives us

Re: [HACKERS] Timeline following for logical slots

Craig Ringer wrote: > It removes the questionable cleanups, fixes the randAccess comment (IMO), I pushed cleanup separately, including the addition of a few comments that were documenting the original code. I actually made a mistake in extracting those, so there's one comment that's actually bog

Re: [HACKERS] Timeline following for logical slots

On 22 March 2016 at 19:48, Andres Freund wrote: > On 2016-03-15 21:04:12 +0800, Craig Ringer wrote: > > Thanks very much for the review. > > Are you planning to update the patch? > > Updated patch attached. It removes the questionable cleanups, fixes the randAccess comment (IMO), changes the "o

Re: [HACKERS] Timeline following for logical slots

On 22 March 2016 at 19:48, Andres Freund wrote: > On 2016-03-15 21:04:12 +0800, Craig Ringer wrote: > > Thanks very much for the review. > > Are you planning to update the patch? > Yes. I just spoke to Álvaro earlier. I'll pick up his modified version of my patch, remove the unrelated cleanup st

Re: [HACKERS] Timeline following for logical slots

On 2016-03-15 21:04:12 +0800, Craig Ringer wrote: > Thanks very much for the review. Are you planning to update the patch? - Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Timeline following for logical slots

Craig already replied to most points. I think we can sum up as "we've done the best we can with what we have, maybe somebody can figure how to do better in the future but for now this works." Andres Freund wrote: > On 2016-03-14 20:10:58 -0300, Alvaro Herrera wrote: > > +#ifndef FRONTEND > > +

Re: [HACKERS] Timeline following for logical slots

On 15 March 2016 at 17:12, Andres Freund wrote: > Hi > Thanks very much for the review. This patch was split out from failover slots, which its self underwent quite a few revisions, so I'm really happy to have fresh eyes on it. Especially more experienced ones. > On 2016-03-14 20:10:58 -0300,

Re: [HACKERS] Timeline following for logical slots

Andres Freund wrote: > Could you perhaps wait till tomorrow? I'd like to do a pass over it. Sure thing. I'll wait for your review. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mai

Re: [HACKERS] Timeline following for logical slots

Hi, On 2016-03-14 20:10:58 -0300, Alvaro Herrera wrote: > diff --git a/src/backend/access/transam/xlogreader.c > b/src/backend/access/transam/xlogreader.c > index fcb0872..7b60b8f 100644 > --- a/src/backend/access/transam/xlogreader.c > +++ b/src/backend/access/transam/xlogreader.c > @@ -10,9 +1

Re: [HACKERS] Timeline following for logical slots

On 15 March 2016 at 07:10, Alvaro Herrera wrote: > Petr Jelinek wrote: > > On 04/03/16 17:08, Craig Ringer wrote: > > >I'd really appreciate some review of the logic there by people who know > > >timelines well and preferably know the xlogreader. It's really just one > > >function and 2/3 comment

Re: [HACKERS] Timeline following for logical slots

On 2016-03-14 20:10:58 -0300, Alvaro Herrera wrote: > Great, thanks. I've studied this to the point where I'm confident that > it makes sense, so I'm about to push it. I didn't change any logic, > only updated comments here and there, both in the patch and in some > preexisting code. I also chan

Re: [HACKERS] Timeline following for logical slots

Petr Jelinek wrote: > On 04/03/16 17:08, Craig Ringer wrote: > >I'd really appreciate some review of the logic there by people who know > >timelines well and preferably know the xlogreader. It's really just one > >function and 2/3 comments; the code is simple but the reasoning leading > >to it is n

Re: [HACKERS] Timeline following for logical slots

On 04/03/16 17:08, Craig Ringer wrote: I'd really appreciate some review of the logic there by people who know timelines well and preferably know the xlogreader. It's really just one function and 2/3 comments; the code is simple but the reasoning leading to it is not. I think this will have to

Re: [HACKERS] Timeline following for logical slots

On 1 March 2016 at 21:00, Craig Ringer wrote: > Hi all > > Per discussion on the failover slots thread ( > https://commitfest.postgresql.org/9/488/) I'm splitting timeline > following for logical slots into its own separate patch. > > I've updated the logical decoding timeline following patch to

Re: [HACKERS] Timeline following for logical slots

Here are the tests. They depend on https://commitfest.postgresql.org/9/569/# I've also rebased the git tree for failover slots on top of the tree for TAP test improvements. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

[HACKERS] Timeline following for logical slots

Hi all Per discussion on the failover slots thread ( https://commitfest.postgresql.org/9/488/) I'm splitting timeline following for logical slots into its own separate patch. The attached patch fixes an issue I found while testing the prior revision: it would read WAL from WAL segments on the old