Re: Handle infinite recursion in logical replication setup
On Mon, 29 Aug 2022 at 16:35, Amit Kapila wrote: > > On Wed, Aug 24, 2022 at 7:27 PM vignesh C wrote: > > > > Since there was no objections to change it to throw a warning, I have > > made the changes for the same. > > > > Review comments for v42-0001* > == > 1. Can we improve the query in check_pub_table_subscribed() so that it > doesn't fetch any table that is already part of the subscription on > the subscriber? This may be better than what the patch is doing which > is to first fetch such information and then skip it. If forming this > query turns out to be too complex then we can retain your method as > well but I feel it is worth trying to optimize the query used in the > patch. I have modified the query to skip the tables as part of the query. The attached v45 patch has the changes for the same. Regards, Vignesh v45-0001-Check-and-log-a-warning-if-the-publisher-has-sub.patch Description: Binary data v45-0002-Document-the-steps-for-replication-between-prima.patch Description: Binary data
Re: Handle infinite recursion in logical replication setup
On Thu, 1 Sept 2022 at 09:19, Amit Kapila wrote:
>
> On Wed, Aug 31, 2022 at 11:35 AM Peter Smith wrote:
> >
> > Here are my review comments for v43-0001.
> >
> > ==
> >
> > 1. Commit message
> >
> > The initial copy phase has no way to know the origin of the row data,
> > so if 'copy_data = true' in the step 4 below, log a warning to notify user
> > that potentially non-local data might have been copied.
> >
> > 1a.
> > "in the step 4" -> "in step 4"
> >
> > ~
> >
> > 1b.
> > "notify user" -> "notify the user"
> >
> > ==
> >
> > 2. doc/src/sgml/ref/create_subscription.sgml
> >
> > +
> > + If the subscription is created with origin = none and
> > + copy_data = true, it will check if the publisher has
> > + subscribed to the same table from other publishers and, if so, log a
> > + warning to notify the user to check the publisher tables. Before
> > continuing
> > + with other operations the user should check that publisher tables did
> > not
> > + have data with different origins to prevent data inconsistency issues
> > on the
> > + subscriber.
> > +
> >
> > I am not sure about that wording saying "to prevent data inconsistency
> > issues" because I thought maybe is already too late because any
> > unwanted origin data is already copied during the initial sync. I
> > think the user can do it try to clean up any problems caused before
> > things become even worse (??)
> >
> > ~~~
> >
> > You asked for my thoughts (see [1] 6b):
> >
> > > There is nothing much a user can do in this case. Only option would be
> > > to take a backup before the operation and restore it and then recreate
> > > the replication setup. I was not sure if we should document these
> > > steps as similar inconsistent data could get created without the
> > > origin option . Thoughts?
> >
> > I think those cases are a bit different:
> >
> > - For a normal scenario (i.e. origin=ANY) then inconsistent data just
> > refers to problems like maybe PK violations so I think you just get
> > what you get and have to deal with the errors...
> >
> > - But when the user said origin=NONE they are specifically saying they
> > do NOT want any non-local data, yet they still might end up getting
> > data contrary to their wishes. So I think maybe there ought to be
> > something documented to warn about this potentially happening. My
> > first impression is that all this seems like a kind of terrible
> > problem because if it happens then cleanup could be difficult - if
> > that subscriber in turn was also a publisher then this bad data might
> > have propagated all over the place already! Anyway, I guess all this
> > could be mentioned in some (new ?) section of the
> > logical-replication.sgml file (i.e. patch 0002).
> >
> > IDEA: I was wondering if the documentation should recommend (or maybe
> > we can even enforce this) that when using origin=NONE the user should
> > always create the subscription with enabled=FALSE. That way if there
> > are some warnings about potential bad origin data then there it might
> > be possible to take some action or change their mind *before* any
> > unwanted data pollutes everything.
> >
> > ==
> >
> > 3. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
> >
> > + if (count == 0)
> > + tablenames = makeStringInfo();
> > + else
> > + appendStringInfoString(tablenames, ", ");
> > +
> > + appendStringInfo(tablenames, "%s.%s", nspname, relname);
> > + count++;
> >
> > I think the quotes are not correct - each separate table name should be
> > quoted.
> >
> > ~~~
> >
> > 4.
> >
> > + /*
> > + * There might be many tables present in this case, we will display a
> > + * maximum of 100 tables followed by "..." to indicate there might be
> > + * more tables.
> > + */
> > + if (count == 100)
> > + {
> > + appendStringInfoString(tablenames, ", ...");
> > + break;
> > + }
> >
> > 4a.
> > IMO this code should be part of the previous if/else
> >
> > e.g.
> > if (count == 0) ... makeStringInfo()
> > else if (count > 100) ... append ellipsis "..." and break out of the loop
> > else ... append table name to the message
> >
> > Doing it in this way means "..." definitely means there are more than
> > 100 bad tables.
> >
> > ~
> >
> > 4b.
> > Changing like above (#4a) simplifies the comment because it removes
> > doubts like "might be…" since you already know you found the 101st
> > table.
> >
> > SUGGESTION (comment)
> > The message displays a maximum of 100 tables having potentially
> > non-local data. Any more than that will be indicated by "...".
> >
> > ~
> >
>
> Are we using this style of an error message anywhere else in the code?
> If not, then I think we should avoid it here as well as it appears a
> bit adhoc to me in the sense that why restrict at 100 tables. Can we
> instead think of displaying the publications list in the message? So
> errdetail would be something like: Subscribed publications \"%s\" has
> been subscribed from some other publisher. Then we can probably giv
Re: Handle infinite recursion in logical replication setup
On Thu, 1 Sept 2022 at 08:01, shiy.f...@fujitsu.com
wrote:
>
> On Wed, Aug 31, 2022 1:06 AM vignesh C wrote:
> >
> > The attached v43 patch has the changes for the same.
> >
>
> Thanks for updating the patch.
>
> Here is a comment on the 0001 patch.
>
> + if (!isnewtable)
> + {
> + pfree(nspname);
> + pfree(relname);
> + continue;
> + }
>
> If it is a new table, in which case it would log a warning, should we also
> call
> pfree()?
Yes it should be added, I have fixed the same in the v44 patch attached at [1].
[1] -
https://www.postgresql.org/message-id/CALDaNm0NRJ1O1cYcZD%3Df7NgynozFprb7zpJSayFN5rcaS44G6Q%40mail.gmail.com
Regards,
Vignesh
Re: Handle infinite recursion in logical replication setup
On Wed, 31 Aug 2022 at 11:45, Peter Smith wrote: > > Here are some review comments for patch v43-0002: > > == > > 1. doc/src/sgml/ref/create_subscription.sgml > > @@ -403,7 +403,9 @@ CREATE SUBSCRIPTION class="parameter">subscription_name warning to notify the user to check the publisher tables. Before > continuing > with other operations the user should check that publisher tables did not > have data with different origins to prevent data inconsistency issues on > the > - subscriber. > + subscriber. Refer to for > + how copy_data and origin can be used > + to set up replication between primaries. > > > Regarding my earlier v43-0001 review (see [1] comment #2) perhaps > another pg docs section should be added in the > logical-replication.sgml (e.g. "Specifying origins during CREATE > SUBSCRIPTION"), so then this Notes text also should have more added to > it. > > SUGGESTION > Refer to for details about potential initialization > inconsistency warnings using origin=NONE. > Refer to for how copy_data and origin can be used to set up > replication between primaries. I have moved all these contents to a separate section in the logical-replication page. I have referred to this link from the documentation of origin and copy_data parameter. I have also referred to "setting up replication between primaries" in the newly added section. Since this new section is referred to from other places, I felt we need not provide a link from create_subscription notes. The changes for the same are available at [1]. [1] - https://www.postgresql.org/message-id/CALDaNm0NRJ1O1cYcZD%3Df7NgynozFprb7zpJSayFN5rcaS44G6Q%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Mon, 29 Aug 2022 at 16:35, Amit Kapila wrote:
>
> On Wed, Aug 24, 2022 at 7:27 PM vignesh C wrote:
> >
> > Since there was no objections to change it to throw a warning, I have
> > made the changes for the same.
> >
>
> Review comments for v42-0001*
> ==
> 1. Can we improve the query in check_pub_table_subscribed() so that it
> doesn't fetch any table that is already part of the subscription on
> the subscriber? This may be better than what the patch is doing which
> is to first fetch such information and then skip it. If forming this
> query turns out to be too complex then we can retain your method as
> well but I feel it is worth trying to optimize the query used in the
> patch.
I'm analysing this, I will post a reply for this soon.
> 2. I thought that it may be better to fetch all the tables that have
> the possibility to have data from more than one origin but maybe the
> list will be too long especially for FOR ALL TABLE types of cases. Is
> this the reason you have decided to give a WARNING as soon as you see
> any such table? If so, probably adding a comment for it can be good.
Yes that was the reason, now I have changed it to display the
publications based on your recent comment whose count will be
significantly very much lesser compared to the number of tables.
> 3.
> + $node_publisher->wait_for_catchup($sub_name);
> +
> + # also wait for initial table sync to finish
> + $node_subscriber->poll_query_until('postgres', $synced_query)
> + or die "Timed out while waiting for subscriber to synchronize data";
>
>
>
> +$node_B->safe_psql(
> + 'postgres', "
> +ALTER SUBSCRIPTION $subname_BA REFRESH PUBLICATION");
> +
> +$node_B->poll_query_until('postgres', $synced_query)
> + or die "Timed out while waiting for subscriber to synchronize data";
>
> You can replace this and any similar code in the patch with a method
> used in commit 0c20dd33db.
Modified
> 4.
> +###
> +# Join 3rd node (node_C) to the existing 2 nodes(node_A & node_B)
> bidirectional
> +# replication setup when the existing nodes (node_A & node_B) has
> pre-existing
> +# data and the new node (node_C) does not have any data.
> +###
> +$result = $node_A->safe_psql('postgres', "SELECT * FROM tab ORDER BY 1;");
> +is($result, qq(), 'Check existing data');
> +
> +$result = $node_B->safe_psql('postgres', "SELECT * FROM tab ORDER BY 1;");
> +is($result, qq(), 'Check existing data');
> +
> +$result = $node_C->safe_psql('postgres', "SELECT * FROM tab ORDER BY 1;");
> +is($result, qq(), 'Check existing data');
>
> The comments say that for this test, two of the nodes have some
> pre-existing data but I don't see the same in the test results. The
> test following this one will also have a similar effect. BTW, I am not
> sure all the new three node tests added by this patch are required
> because I don't see if they have additional code coverage or test
> anything which is not tested without those. This has doubled the
> amount of test timing for this test file which is okay if these tests
> make any useful contribution but otherwise, it may be better to remove
> these. Am, I missing something?
Earlier we felt with the origin patch we could support joining of
nodes to existing n-wary replication setup in various scenarios, I had
added the tests for the same scenarios. But as the patch evolved, I
could understand that this patch cannot handle the add node scenarios.
I have removed these tests.
Thanks for the comments, the v44 patch attached at [1] has the changes
for the same.
[1] -
https://www.postgresql.org/message-id/CALDaNm0NRJ1O1cYcZD%3Df7NgynozFprb7zpJSayFN5rcaS44G6Q%40mail.gmail.com
Regards,
Vignesh
Re: Handle infinite recursion in logical replication setup
On Wed, 31 Aug 2022 at 11:35, Peter Smith wrote:
>
> Here are my review comments for v43-0001.
>
> ==
>
> 1. Commit message
>
> The initial copy phase has no way to know the origin of the row data,
> so if 'copy_data = true' in the step 4 below, log a warning to notify user
> that potentially non-local data might have been copied.
>
> 1a.
> "in the step 4" -> "in step 4"
Modified
> ~
>
> 1b.
> "notify user" -> "notify the user"
Modified
> ==
>
> 2. doc/src/sgml/ref/create_subscription.sgml
>
> +
> + If the subscription is created with origin = none and
> + copy_data = true, it will check if the publisher has
> + subscribed to the same table from other publishers and, if so, log a
> + warning to notify the user to check the publisher tables. Before
> continuing
> + with other operations the user should check that publisher tables did not
> + have data with different origins to prevent data inconsistency issues on
> the
> + subscriber.
> +
>
> I am not sure about that wording saying "to prevent data inconsistency
> issues" because I thought maybe is already too late because any
> unwanted origin data is already copied during the initial sync. I
> think the user can do it try to clean up any problems caused before
> things become even worse (??)
>
> ~~~
>
> You asked for my thoughts (see [1] 6b):
>
> > There is nothing much a user can do in this case. Only option would be
> > to take a backup before the operation and restore it and then recreate
> > the replication setup. I was not sure if we should document these
> > steps as similar inconsistent data could get created without the
> > origin option . Thoughts?
>
> I think those cases are a bit different:
>
> - For a normal scenario (i.e. origin=ANY) then inconsistent data just
> refers to problems like maybe PK violations so I think you just get
> what you get and have to deal with the errors...
>
> - But when the user said origin=NONE they are specifically saying they
> do NOT want any non-local data, yet they still might end up getting
> data contrary to their wishes. So I think maybe there ought to be
> something documented to warn about this potentially happening. My
> first impression is that all this seems like a kind of terrible
> problem because if it happens then cleanup could be difficult - if
> that subscriber in turn was also a publisher then this bad data might
> have propagated all over the place already! Anyway, I guess all this
> could be mentioned in some (new ?) section of the
> logical-replication.sgml file (i.e. patch 0002).
>
> IDEA: I was wondering if the documentation should recommend (or maybe
> we can even enforce this) that when using origin=NONE the user should
> always create the subscription with enabled=FALSE. That way if there
> are some warnings about potential bad origin data then there it might
> be possible to take some action or change their mind *before* any
> unwanted data pollutes everything.
Updated document based on your suggestion.
> ==
>
> 3. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> + if (count == 0)
> + tablenames = makeStringInfo();
> + else
> + appendStringInfoString(tablenames, ", ");
> +
> + appendStringInfo(tablenames, "%s.%s", nspname, relname);
> + count++;
>
> I think the quotes are not correct - each separate table name should be
> quoted.
We will not be displaying tables anymore, the comment does not apply anymore
> ~~~
>
> 4.
>
> + /*
> + * There might be many tables present in this case, we will display a
> + * maximum of 100 tables followed by "..." to indicate there might be
> + * more tables.
> + */
> + if (count == 100)
> + {
> + appendStringInfoString(tablenames, ", ...");
> + break;
> + }
>
> 4a.
> IMO this code should be part of the previous if/else
>
> e.g.
> if (count == 0) ... makeStringInfo()
> else if (count > 100) ... append ellipsis "..." and break out of the loop
> else ... append table name to the message
>
> Doing it in this way means "..." definitely means there are more than
> 100 bad tables.
This code is removed, the comment is no more applicable.
> ~
>
> 4b.
> Changing like above (#4a) simplifies the comment because it removes
> doubts like "might be…" since you already know you found the 101st
> table.
>
> SUGGESTION (comment)
> The message displays a maximum of 100 tables having potentially
> non-local data. Any more than that will be indicated by "...".
This code is removed, the comment is no more applicable.
> ~
>
> 4c.
> It still seems a bit disconcerting to me that there can be cases where
> bad data was possibly copied but there is no record of what tables
> have been polluted. Perhaps when the maximum bad tables are exceeded
> then there can be some additional message to tell users what SQL they
> can use to discover what all the other bad tables were.
We will not be displaying the tablename anymore, we will be displaying
the publication names which has been subscribed from other
publications.
>
Re: Handle infinite recursion in logical replication setup
On Mon, 29 Aug 2022 at 12:01, Peter Smith wrote: > > Here are some review comments for patch v42-0002: > > == > > 1. doc/src/sgml/logical-replication.sgml > > copy_data = true > > There are a couple of these tags where there is a trailing space > before the . I guess it is doing no harm, but it is doing no > good either, so IMO better to get rid of the space. Modified > ~~ > > 2. doc/src/sgml/logical-replication.sgml - 31.3.1. Setting replication > between two primaries > > User need to ensure that no data changes happen on table t1 on > primary1 and primary2 until the setup is completed. > > SUGGESTION > The user must ensure that no data changes happen on table t1 of > primary1 and primary2 until the setup is completed. Modified > ~~~ > > 3. doc/src/sgml/logical-replication.sgml - 31.3.2. Adding a new > primary when there is no table data on any of the primaries > > User need to ensure that no data changes happen on table t1 on all the > primaries primary1, primary2 and primary3 until the setup is > completed. > > SUGGESTION > The user must ensure that no data changes happen on table t1 of all > the primaries primary1, primary2 and primary3 until the setup is > completed. Modified > ~~~ > > 4. doc/src/sgml/logical-replication.sgml - 31.3.3. Adding a new > primary when table data is present on the existing primaries > > User need to ensure that no operations should be performed on table t1 > on primary2 and primary3 until the setup is completed. > > SUGGESTION > The user must ensure that no operations are performed on table t1 of > primary2 and primary3 until the setup is completed. > > Here also you changed the wording - "no data changes happen" versus > "no operations should be performed". Was that a deliberate difference? > Maybe they should all be consistent wording? If they are > interchangeable IMO the "no operations are performed..." wording was > the better of the two. Modified, it was not a deliberate change. I have changed it to "no operations are performed" in other places too. > ~~~ > > 5. doc/src/sgml/logical-replication.sgml - 31.3.4. Generic steps for > adding a new primary to an existing set of primaries > > Step-2: User need to ensure that no changes happen on the required > tables of the new primary until the setup is complete. > > SUGGESTION > Step-2: The user must ensure that no changes happen on the required > tables of the new primary until the setup is complete. Modified > ~~~ > > 6. > > Step-4. User need to ensure that no changes happen on the required > tables of the existing primaries except the first primary until the > setup is complete. > > SUGGESTION > Step-4. The user must ensure that no changes happen on the required > tables of the existing primaries except the first primary until the > setup is complete. Modified > ~~~ > > 7. (the example) > > 7a. > Step-2. User need to ensure that no changes happen on table t1 on > primary4 until the setup is completed. > > SUGGESTION > Step-2. The user must ensure that no changes happen on table t1 of > primary4 until the setup is completed. Modified > ~ > > 7b. > Step-4. User need to ensure that no changes happen on table t1 on > primary2 and primary3 until the setup is completed. > > SUGGESTION > Step-4. The user must ensure that no changes happen on table t1 of > primary2 and primary3 until the setup is completed. Modified Thanks for the comments, the changes for the same are available in the v43 patch attached at [1]. [1] - https://www.postgresql.org/message-id/CALDaNm1V%2BAvzPsUcq%3DmNYG%2BJfAyovTWBe4vWKTknOgH_ko1E0Q%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Mon, 29 Aug 2022 at 11:59, Peter Smith wrote:
>
> Here are some review comments for patch v42-0001:
>
> ==
>
> 1. Commit message
>
> A later review comment below suggests some changes to the WARNING
> message so if those changes are made then the example in this commit
> message also needs to be modified.
Modified
> ==
>
> 2. doc/src/sgml/ref/alter_subscription.sgml - copy_data
>
> Refer to the Notes for the usage of true for copy_data parameter and
> its interaction with the origin parameter.
>
> I think saying "usage of true" sounded a bit strange.
>
> SUGGESTION
> Refer to the Notes about how copy_data = true can interact with the
> origin parameter.
Modified
> ==
>
> 3. doc/src/sgml/ref/create_subscription.sgml - copy data
>
> Refer to the Notes for the usage of true for copy_data parameter and
> its interaction with the origin parameter.
>
> SUGGESTION
> (same as #2)
Modified
> ~~
>
> 4. doc/src/sgml/ref/create_subscription.sgml - origin
>
> Refer to the Notes for the usage of true for copy_data parameter and
> its interaction with the origin parameter.
>
> SUGGESTION
> (same as #2)
Modified
> ~~~
>
> 5. doc/src/sgml/ref/create_subscription.sgml - Notes
>
> +
> + If the subscription is created with origin = none and
> + copy_data = true, it will check if the publisher has
> + subscribed to the same table from other publishers and, if so, throw a
> + warning to notify user to check the publisher tables. The user can ensure
>
> "throw a warning to notify user" -> "log a warning to notify the user"
Modified
> ~~
>
> 6.
>
> + warning to notify user to check the publisher tables. The user can ensure
> + that publisher tables do not have data which has an origin associated
> before
> + continuing with any other operations to prevent inconsistent data being
> + replicated.
> +
>
> 6a.
>
> I'm not so sure about this. IMO the warning is not about the
> replication – really it is about the COPY which has already happened
> anyway. So the user can't really prevent anything from going wrong;
> instead, they have to take some action to clean up if anything did go
> wrong.
>
> SUGGESTION
> Before continuing with other operations the user should check that
> publisher tables did not have data with different origins, otherwise
> inconsistent data may have been copied.
Modified based on the suggestion provided by Amit.
> ~
>
> 6b.
>
> I am also wondering what can the user do now. Assuming there was bad
> COPY then the subscriber table has already got unwanted stuff copied
> into it. Is there any advice we can give to help users fix this mess?
There is nothing much a user can do in this case. Only option would be
to take a backup before the operation and restore it and then recreate
the replication setup. I was not sure if we should document these
steps as similar inconsistent data could get created without the
origin option . Thoughts?
> ==
>
> 7. src/backend/commands/subscriptioncmds.c - AlterSubscription_refresh
>
> + /* Check whether we can allow copy of newly added relations. */
> + check_pub_table_subscribed(wrconn, sub->publications, copy_data,
> +sub->origin, subrel_local_oids,
> +subrel_count);
>
> "whether we can allow" seems not quite the right wording here anymore,
> because now there is no ERROR stopping this - so if there was unwanted
> data the COPY will proceed to copy it anyhow...
Removed the comment as the function details the necessary things.
> ~~~
>
> 8. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> @@ -1781,6 +1794,121 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
> table_close(rel, RowExclusiveLock);
> }
>
> +/*
> + * Check and throw a warning if the publisher has subscribed to the same
> table
> + * from some other publisher. This check is required only if "copy_data =
> true"
> + * and "origin = none" for CREATE SUBSCRIPTION and
> + * ALTER SUBSCRIPTION ... REFRESH statements to notify user that data having
> + * origin might have been copied.
>
> "throw a warning" -> "log a warning"
>
> "to notify user" -> "to notify the user" ?
Modified
> ~~~
>
> 9.
>
> + if (copydata == false || !origin ||
> + (pg_strcasecmp(origin, LOGICALREP_ORIGIN_NONE) != 0))
> + return;
>
> SUGGESTION
> if (!copydata || !origin ||
> (pg_strcasecmp(origin, LOGICALREP_ORIGIN_NONE) != 0))
Modified
> ~~~
>
> 10. (Question)
>
> I can't tell just by reading the code if FOR ALL TABLES case is
> handled – e.g. will this recognise the case were the publisher might
> have table data from other origins because it has a subscription on
> some other node that was publishing "FOR ALL TABLES"?
Yes it handles it.
> ~~~
>
> 11.
>
> + ereport(WARNING,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("publisher has subscribed table \"%s.%s\" from some other publisher",
> +nspname, relname),
> + errdetail("Publisher might have subscribed one or more tables from
> some other publisher."),
> + errhint("Verify
Re: Handle infinite recursion in logical replication setup
On Fri, Aug 26, 2022 at 9:52 AM Dilip Kumar wrote: > > On Mon, Aug 22, 2022 at 9:19 AM houzj.f...@fujitsu.com > wrote: > > > > > Jonathan, Sawada-San, Hou-San, and others, what do you think is the best > > > way > > > to move forward here? > > > > I think it's fine to throw a WARNING in this case given that there is a > > chance of inconsistent data. > > IMHO, since the user has specifically asked for origin=NONE but we do > not have any way to detect the origin during initial sync so I think > this could be documented and we can also issue the WARNING. So that > users notice that part and carefully set up the replication. OTOH, I > do not think that giving an error is very inconvenient because we are > already providing a new option "origin=NONE" so along with that lets > force the user to choose between copy_data=off or copy_data=force and > with that, there is no scope for mistakes. Since Jonathan also had suggested to throw a warning as in [1] and Hou-san also had suggested to throw a warning as in [2], I have made changes to throw a warning and also documented the following contents for the same: + + If the subscription is created with origin = none and + copy_data = true, it will check if the publisher has + subscribed to the same table from other publishers and, if so, throw a + warning to notify user to check the publisher tables. The user can ensure + that publisher tables do not have data which has an origin associated before + continuing with any other operations to prevent inconsistent data being + replicated. + The changes for the same are available in the v42 patch at [3]. [1] - https://www.postgresql.org/message-id/afb653ba-e2b1-33a3-a54c-849f4466e1b4%40postgresql.org [2] - https://www.postgresql.org/message-id/OS0PR01MB5716C383623ADAD64CE4841194719%40OS0PR01MB5716.jpnprd01.prod.outlook.com [3] - https://www.postgresql.org/message-id/CALDaNm2oLWsSYtOFLdOkbrpC94%3DJZzKMCYDJoiZaqAX_Hn%2BU9Q%40mail.gmail.com Regards, Vignesh
Re: Column Filtering in Logical Replication
On Tue, Aug 23, 2022 at 7:52 AM Peter Smith wrote: > > On Mon, Aug 22, 2022 at 9:25 PM vignesh C wrote: > > > ... > > > Few comments: > > 1) I felt no expressions are allowed in case of column filters. Only > > column names can be specified. The second part of the sentence > > confuses what is allowed and what is not allowed. Won't it be better > > to remove the second sentence and mention that only column names can > > be specified. > > + > > +Column list can contain only simple column references. Complex > > +expressions, function calls etc. are not allowed. > > + > > > > This wording was lifted verbatim from the commit message [1]. But I > see your point that it just seems to be overcomplicating a simple > rule. Modified as suggested. > > > 2) tablename should be table name. > > + > > +A column list is specified per table following the tablename, and > > enclosed by > > +parenthesis. See for details. > > + > > > > We have used table name in the same page in other instances like: > > a) The row filter is defined per table. Use a WHERE clause after the > > table name for each published table that requires data to be filtered > > out. The WHERE clause must be enclosed by parentheses. > > b) The tables are matched between the publisher and the subscriber > > using the fully qualified table name. > > > > Fixed as suggested. > > > 3) One small whitespace issue: > > git am v2-0001-Column-List-replica-identity-rules.patch > > Applying: Column List replica identity rules. > > .git/rebase-apply/patch:30: trailing whitespace. > >if the publication publishes only INSERT operations. > > warning: 1 line adds whitespace errors. > > > > Fixed. > > ~~~ > > PSA the v3* patch set. Thanks for the updated patch. Few comments: 1) We can shuffle the columns in publisher table and subscriber to show that the order of the column does not matter + +Create a publication p1. A column list is defined for +table t1 to reduce the number of columns that will be +replicated. + +test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, a, b, c); +CREATE PUBLICATION +test_pub=# + 2) We can try to keep the line content to less than 80 chars + +A column list is specified per table following the tablename, and enclosed by +parenthesis. See for details. + 3) tablename should be table name like it is used in other places + +A column list is specified per table following the tablename, and enclosed by +parenthesis. See for details. + 4a) In the below, you could include mentioning "Only the column list data of publication p1 are replicated." + + Insert some rows to table t1. + +test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1'); +INSERT 0 1 4b) In the above, we could mention that the insert should be done on the "publisher side" as the previous statements are executed on the subscriber side. Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Mon, Aug 22, 2022 at 9:19 AM houzj.f...@fujitsu.com wrote: > > On Thursday, August 18, 2022 11:13 AM Amit Kapila > wrote: > > > > On Wed, Aug 17, 2022 at 12:34 PM Peter Smith > > wrote: > > > > > > On Wed, Aug 17, 2022 at 4:33 PM Amit Kapila > > wrote: > > > > > > > > On Wed, Aug 17, 2022 at 8:48 AM houzj.f...@fujitsu.com > > > > wrote: > > > > > > > > > > On Tuesday, August 2, 2022 8:00 PM Amit Kapila > > wrote: > > > > > > On Tue, Jul 26, 2022 at 9:07 AM Amit Kapila > > wrote: > > > > > > > > > > Thanks for the summary. > > > > > > > > > > I think it's fine to make the user use the copy_data option more > > > > > carefully to prevent duplicate copies by reporting an ERROR. > > > > > > > > > > But I also have similar concern with Sawada-san as it's possible > > > > > for user to receive an ERROR in some unexpected cases. > > > > > > > > > > For example I want to build bi-directional setup between two nodes: > > > > > > > > > > Node A: TABLE test (has actual data) Node B: TABLE test (empty) > > > > > > > > > > Step 1: > > > > > CREATE PUBLICATION on both Node A and B. > > > > > > > > > > Step 2: > > > > > CREATE SUBSCRIPTION on Node A with (copy_data = on) > > > > > -- this is fine as there is no data on Node B > > > > > > > > > > Step 3: > > > > > CREATE SUBSCRIPTION on Node B with (copy_data = on) > > > > > -- this should be fine as user needs to copy data from Node A to > > > > > Node B, > > > > > -- but we still report an error for this case. > > > > > > > > > > It looks a bit strict to report an ERROR in this case and it seems > > > > > not easy to avoid this. So, personally, I think it might be better > > > > > to document the correct steps to build the bi-directional > > > > > replication and probably also docuemnt the steps to recover if > > > > > user accidently did duplicate initial copy if not documented yet. > > > > > > > > > > In addition, we could also LOG some additional information about > > > > > the ORIGIN and initial copy which might help user to analyze if > > > > > needed. > > > > > > > > > > > > > But why LOG instead of WARNING? I feel in this case there is a > > > > chance of inconsistent data so a WARNING like "publication "pub1" > > > > could have data from multiple origins" can be given when the user > > > > has specified > > > > options: "copy_data = on, origin = NONE" while creating a > > > > subscription. We give a WARNING during subscription creation when > > > > the corresponding publication doesn't exist, eg. > > > > > > > > postgres=# create subscription sub1 connection 'dbname = postgres' > > > > publication pub1; > > > > WARNING: publication "pub1" does not exist in the publisher > > > > > > > > Then, we can explain in docs how users can avoid data > > > > inconsistencies while setting up replication. > > > > > > > > > > I was wondering if this copy/origin case really should be a NOTICE. > > > > > > > We usually give NOTICE for some sort of additional implicit information, > > e.g., > > when we create a slot during CREATE SUBSCRIPTION > > command: "NOTICE: created replication slot "sub1" on publisher". IMO, this > > is > > likely to be a problem of data inconsistency so I think here we can choose > > between WARNING and LOG. I prefer WARNING but okay with LOG as well if > > others feel so. I think we can change this later as well if required. We do > > have an > > option to not do anything and just document it but I feel it is better to > > give user > > some indication of problem here because not everyone reads each update of > > documentation. > > > > Jonathan, Sawada-San, Hou-San, and others, what do you think is the best way > > to move forward here? > > I think it's fine to throw a WARNING in this case given that there is a > chance of inconsistent data. Since there was no objections to change it to throw a warning, I have made the changes for the same. I have made one change for the documentation patch. The current steps states that: 1. Create a publication on primary3. 2. Lock table t1 on primary2 and primary3 in EXCLUSIVE mode until the setup is completed. There is no need to lock table t1 on primary1 because any data changes made will be synchronized while creating the subscription with copy_data = true. 3. Create a subscription on primary1 to subscribe to primary3. 4. Create a subscription on primary2 to subscribe to primary3. 5. Create a subscription on primary3 to subscribe to primary1. Use copy_data = true so that the existing table data is copied during initial sync. 6. Create a subscription on primary3 to subscribe to primary2. Use copy_data = false because the initial table data would have been already copied in the previous step. 7. Now the replication setup between primaries primary1, primary2 and primary3 is complete. Incremental changes made on any primary will be replicated to the other two primaries. We found a problem with the proposed steps. Here the problem is that we lock table t1 on primary2/primary3 in step2, then we create
Re: Include the dependent extension information in describe command.
On Tue, Aug 16, 2022 at 9:04 PM Bruce Momjian wrote: > > On Mon, Aug 15, 2022 at 10:09:29PM +0530, vignesh C wrote: > > I have updated the patch to display "Objects depending on extension" > > as describe extension footer. The changes for the same are available > > in the v2 version patch attached. Thoughts? > > I wonder if we would be better off with a backslash command that showed > the dependencies of any object. Yes, If we have a backslash command which could show the dependencies of the specified object could be helpful. Can we something like below: a) Index idx1 depend on table t1 create table t1(c1 int); create index idx1 on t1(c1); postgres=# \dD idx1 Name - idx1 Depends on: table t1 b) Index idx1 depend on table t1 and extension ext1 alter index idx idx1 depends on extension ext1 postgres=# \dD idx1 Name - idx1 Depends on: table t1 extension ext1 c) materialized view mv1 depends on table t1 create materialized view mv1 as select * from t1; postgres=# \dD mv1 Name - mv1 Depends on: table t1 If you are ok with this approach, I can implement a patch on similar lines. Thoughts? Regards, Vignesh
Re: Column Filtering in Logical Replication
On Mon, Aug 22, 2022 at 1:58 PM Peter Smith wrote: > > Thanks for the view of v1-0001. > > On Wed, Aug 17, 2022 at 3:04 AM vignesh C wrote: > ... > > 1) Row filters mentions that "It has no effect on TRUNCATE commands.", > > the same is not present in case of column filters. We should keep the > > changes similarly for consistency. > > --- a/doc/src/sgml/ref/create_publication.sgml > > +++ b/doc/src/sgml/ref/create_publication.sgml > > @@ -90,8 +90,7 @@ CREATE PUBLICATION > class="parameter">name > > > >When a column list is specified, only the named columns are > > replicated. > >If no column list is specified, all columns of the table are > > replicated > > - through this publication, including any columns added later. If a > > column > > - list is specified, it must include the replica identity columns. > > + through this publication, including any columns added later. > > Modified as suggested. > > > > > 2) The document says that "if the table uses REPLICA IDENTITY FULL, > > specifying a column list is not allowed.": > > + publishes only INSERT operations. Furthermore, if the > > + table uses REPLICA IDENTITY FULL, specifying a column > > + list is not allowed. > > + > > > > Did you mean specifying a column list during create publication for > > REPLICA IDENTITY FULL table like below scenario: > > postgres=# create table t2(c1 int, c2 int, c3 int); > > CREATE TABLE > > postgres=# alter table t2 replica identity full ; > > ALTER TABLE > > postgres=# create publication pub1 for table t2(c1,c2); > > CREATE PUBLICATION > > > > If so, the document says specifying column list is not allowed, but > > creating a publication with column list on replica identity full was > > successful. > > That patch v1-0001 was using the same wording from the github commit > message [1]. I agree it was a bit vague. > > In fact the replica identity validation is done at DML execution time > so your example will fail as expected when you attempt to do a UPDATE > operation. > > e.g. > test_pub=# update t2 set c2=23 where c1=1; > ERROR: cannot update table "t2" > DETAIL: Column list used by the publication does not cover the > replica identity. > > I modified the wording for this part of the docs. Few comments: 1) I felt no expressions are allowed in case of column filters. Only column names can be specified. The second part of the sentence confuses what is allowed and what is not allowed. Won't it be better to remove the second sentence and mention that only column names can be specified. + +Column list can contain only simple column references. Complex +expressions, function calls etc. are not allowed. + 2) tablename should be table name. + +A column list is specified per table following the tablename, and enclosed by +parenthesis. See for details. + We have used table name in the same page in other instances like: a) The row filter is defined per table. Use a WHERE clause after the table name for each published table that requires data to be filtered out. The WHERE clause must be enclosed by parentheses. b) The tables are matched between the publisher and the subscriber using the fully qualified table name. 3) One small whitespace issue: git am v2-0001-Column-List-replica-identity-rules.patch Applying: Column List replica identity rules. .git/rebase-apply/patch:30: trailing whitespace. if the publication publishes only INSERT operations. warning: 1 line adds whitespace errors. Regards, Vignesh
Re: Skipping schema changes in publication
On Mon, Aug 8, 2022 at 2:53 PM vignesh C wrote:
>
> On Mon, Aug 8, 2022 at 12:46 PM vignesh C wrote:
> >
> > On Fri, Jun 3, 2022 at 3:36 PM vignesh C wrote:
> > >
> > > On Thu, May 26, 2022 at 7:04 PM osumi.takami...@fujitsu.com
> > > wrote:
> > > >
> > > > On Monday, May 23, 2022 2:13 PM vignesh C wrote:
> > > > > Attached v7 patch which fixes the buildfarm warning for an unused
> > > > > warning in
> > > > > release mode as in [1].
> > > > Hi, thank you for the patches.
> > > >
> > > >
> > > > I'll share several review comments.
> > > >
> > > > For v7-0001.
> > > >
> > > > (1) I'll suggest some minor rewording.
> > > >
> > > > +
> > > > + The RESET clause will reset the publication to
> > > > the
> > > > + default state which includes resetting the publication options,
> > > > setting
> > > > + ALL TABLES flag to false and
> > > > + dropping all relations and schemas that are associated with the
> > > > publication.
> > > >
> > > > My suggestion is
> > > > "The RESET clause will reset the publication to the
> > > > default state. It resets the publication operations,
> > > > sets ALL TABLES flag to false and drops all relations
> > > > and schemas associated with the publication."
> > >
> > > I felt the existing looks better. I would prefer to keep it that way.
> > >
> > > > (2) typo and rewording
> > > >
> > > > +/*
> > > > + * Reset the publication.
> > > > + *
> > > > + * Reset the publication options, setting ALL TABLES flag to false and
> > > > drop
> > > > + * all relations and schemas that are associated with the publication.
> > > > + */
> > > >
> > > > The "setting" in this sentence should be "set".
> > > >
> > > > How about changing like below ?
> > > > FROM:
> > > > "Reset the publication options, setting ALL TABLES flag to false and
> > > > drop
> > > > all relations and schemas that are associated with the publication."
> > > > TO:
> > > > "Reset the publication operations, set ALL TABLES flag to false and drop
> > > > all relations and schemas associated with the publication."
> > >
> > > I felt the existing looks better. I would prefer to keep it that way.
> > >
> > > > (3) AlterPublicationReset
> > > >
> > > > Do we need to call CacheInvalidateRelcacheAll() or
> > > > InvalidatePublicationRels() at the end of
> > > > AlterPublicationReset() like AlterPublicationOptions() ?
> > >
> > > CacheInvalidateRelcacheAll should be called if we change all tables
> > > from true to false, else the cache will not be invalidated. Modified
> > >
> > > >
> > > > For v7-0002.
> > > >
> > > > (4)
> > > >
> > > > + if (stmt->for_all_tables)
> > > > + {
> > > > + boolisdefault =
> > > > CheckPublicationDefValues(tup);
> > > > +
> > > > + if (!isdefault)
> > > > + ereport(ERROR,
> > > > +
> > > > errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
> > > > + errmsg("adding ALL TABLES
> > > > requires the publication to have default publication options, no
> > > > tables/
> > > > + errhint("Use ALTER PUBLICATION
> > > > ... RESET to reset the publication"));
> > > >
> > > >
> > > > The errmsg string has three messages for user and is a bit long
> > > > (we have two sentences there connected by 'and').
> > > > Can't we make it concise and split it into a couple of lines for code
> > > > readability ?
> > > >
> > > > I'll suggest a change below.
> > > > FROM:
> > > > "adding ALL TABLES requires the publication to have default publication
> > > > options, no tables/schemas associated and ALL TABLES flag should not be
> > > > set"
> > > > TO:
&
Re: Skipping schema changes in publication
On Thu, Aug 18, 2022 at 12:33 PM Nitin Jadhav wrote: > > I spent some time on understanding the proposal and the patch. Here > are a few comments wrt the test cases. > > > +ALTER PUBLICATION testpub_reset ADD TABLE pub_sch1.tbl1; > > + > > +-- Verify that tables associated with the publication are dropped after > > RESET > > +\dRp+ testpub_reset > > +ALTER PUBLICATION testpub_reset RESET; > > +\dRp+ testpub_reset > > > > +ALTER PUBLICATION testpub_reset ADD ALL TABLES IN SCHEMA public; > > + > > +-- Verify that schemas associated with the publication are dropped after > > RESET > > +\dRp+ testpub_reset > > +ALTER PUBLICATION testpub_reset RESET; > > +\dRp+ testpub_reset > > The results for the above two cases are the same before and after the > reset. Is there any way to verify that? If you see the expected, first \dRp+ command includes: +Tables: +"pub_sch1.tbl1" The second \dRp+ does not include the Tables. We are trying to verify that after reset, the tables will be removed from the publication. The second test is similar to the first, the only difference here is that we test schema instead of tables. i.e we verify that the schemas will be removed from the publication. > --- > > > +-- Can't add EXCEPT TABLE to 'FOR ALL TABLES' publication > > +ALTER PUBLICATION testpub_reset ADD ALL TABLES EXCEPT TABLE pub_sch1.tbl1; > > + > > > > +-- Can't add EXCEPT TABLE to 'FOR TABLE' publication > > +ALTER PUBLICATION testpub_reset ADD ALL TABLES EXCEPT TABLE pub_sch1.tbl1; > > + > > > > +-- Can't add EXCEPT TABLE to 'FOR ALL TABLES IN SCHEMA' publication > > +ALTER PUBLICATION testpub_reset ADD ALL TABLES EXCEPT TABLE pub_sch1.tbl1; > > + > > I did not understand the objective of these tests. I think we need to > improve the comments. There are different publications like "ALL TABLES", "TABLE", "ALL TABLES IN SCHEMA" publications. Here we are trying to verify that except tables cannot be added to "ALL TABLES", "TABLE", "ALL TABLES IN SCHEMA" publications. If you see the expected file, you will see the following error: +-- Can't add EXCEPT TABLE to 'FOR ALL TABLES' publication +ALTER PUBLICATION testpub_reset ADD ALL TABLES EXCEPT TABLE pub_sch1.tbl1; +ERROR: adding ALL TABLES requires the publication to have default publication parameter values +DETAIL: ALL TABLES flag should not be set and no tables/schemas should be associated. +HINT: Use ALTER PUBLICATION ... RESET to reset the publication I felt the existing comment is ok. Let me know if you still feel any change is required. Regards, Vignesh
Re: Column Filtering in Logical Replication
On Mon, Aug 8, 2022 at 2:08 PM Peter Smith wrote: > > PSA patch version v1* for a new "Column Lists" pgdocs section > > This is just a first draft, but I wanted to post it as-is, with the > hope that I can get some feedback while continuing to work on it. Few comments: 1) Row filters mentions that "It has no effect on TRUNCATE commands.", the same is not present in case of column filters. We should keep the changes similarly for consistency. --- a/doc/src/sgml/ref/create_publication.sgml +++ b/doc/src/sgml/ref/create_publication.sgml @@ -90,8 +90,7 @@ CREATE PUBLICATION name When a column list is specified, only the named columns are replicated. If no column list is specified, all columns of the table are replicated - through this publication, including any columns added later. If a column - list is specified, it must include the replica identity columns. + through this publication, including any columns added later. 2) The document says that "if the table uses REPLICA IDENTITY FULL, specifying a column list is not allowed.": + publishes only INSERT operations. Furthermore, if the + table uses REPLICA IDENTITY FULL, specifying a column + list is not allowed. + Did you mean specifying a column list during create publication for REPLICA IDENTITY FULL table like below scenario: postgres=# create table t2(c1 int, c2 int, c3 int); CREATE TABLE postgres=# alter table t2 replica identity full ; ALTER TABLE postgres=# create publication pub1 for table t2(c1,c2); CREATE PUBLICATION If so, the document says specifying column list is not allowed, but creating a publication with column list on replica identity full was successful. Regards, Vignesh
Re: Include the dependent extension information in describe command.
On Sun, Aug 14, 2022 at 10:24 PM vignesh C wrote: > > On Sun, Aug 14, 2022 at 11:07 AM Tom Lane wrote: > > > > vignesh C writes: > > > Currently we do not include the dependent extension information for > > > index and materialized view in the describe command. I felt it would > > > be useful to include this information as part of the describe command > > > like: > > > \d+ idx_depends > > > Index "public.idx_depends" > > > Column | Type | Key? | Definition | Storage | Stats target > > > +-+--++-+-- > > > a | integer | yes | a | plain | > > > btree, for table "public.tbl_idx_depends" > > > Depends: > > > "plpgsql" > > > > > Attached a patch for the same. Thoughts? > > > > This seems pretty much useless noise to me. Can you point to > > any previous requests for such a feature? If we did do it, > > why would we do it in such a narrow fashion (ie, only dependencies > > of two specific kinds of objects on one other specific kind of > > object)? Why did you do it in this direction rather than > > the other one, ie show dependencies when examining the extension? > > While implementing logical replication of "index which depends on > extension", I found that this information was not available in any of > the \d describe commands. I felt having this information in the \d > describe command will be useful in validating the "depends on > extension" easily. Now that you pointed out, I agree that it will be > better to show the dependencies from the extension instead of handling > it in multiple places. I will change it to handle it from extension > and post an updated version soon for this. I have updated the patch to display "Objects depending on extension" as describe extension footer. The changes for the same are available in the v2 version patch attached. Thoughts? Regards, Vignesh From 3bdb382f38b889b303f7a7036d9a0fc5dbeb2be7 Mon Sep 17 00:00:00 2001 From: Vigneshwaran C Date: Thu, 28 Jul 2022 22:19:00 +0530 Subject: [PATCH v2] Include the objects depending on extension in describe extension Include the objects depending on extension in describe extension --- .../expected/create_transform.out | 3 - src/bin/psql/describe.c | 94 --- .../expected/test_extensions.out | 6 -- src/test/regress/expected/indexing.out| 17 src/test/regress/expected/matview.out | 16 src/test/regress/sql/indexing.sql | 7 ++ src/test/regress/sql/matview.sql | 6 ++ 7 files changed, 128 insertions(+), 21 deletions(-) diff --git a/contrib/hstore_plperl/expected/create_transform.out b/contrib/hstore_plperl/expected/create_transform.out index dc72395376..d060d6ff65 100644 --- a/contrib/hstore_plperl/expected/create_transform.out +++ b/contrib/hstore_plperl/expected/create_transform.out @@ -49,7 +49,6 @@ CREATE EXTENSION hstore_plperl; function hstore_to_plperl(internal) function plperl_to_hstore(internal) transform for hstore language plperl -(3 rows) ALTER EXTENSION hstore_plperl DROP TRANSFORM FOR hstore LANGUAGE plperl; \dx+ hstore_plperl @@ -58,7 +57,6 @@ Objects in extension "hstore_plperl" - function hstore_to_plperl(internal) function plperl_to_hstore(internal) -(2 rows) ALTER EXTENSION hstore_plperl ADD TRANSFORM FOR hstore LANGUAGE plperl; \dx+ hstore_plperl @@ -68,7 +66,6 @@ ALTER EXTENSION hstore_plperl ADD TRANSFORM FOR hstore LANGUAGE plperl; function hstore_to_plperl(internal) function plperl_to_hstore(internal) transform for hstore language plperl -(3 rows) DROP EXTENSION hstore CASCADE; NOTICE: drop cascades to extension hstore_plperl diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 327a69487b..003a3361ec 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -20,6 +20,7 @@ #include "catalog/pg_cast_d.h" #include "catalog/pg_class_d.h" #include "catalog/pg_default_acl_d.h" +#include "catalog/pg_extension_d.h" #include "common.h" #include "common/logging.h" #include "describe.h" @@ -45,7 +46,12 @@ static bool describeOneTSConfig(const char *oid, const char *nspname, const char *cfgname, const char *pnspname, const char *prsname); static void printACLColumn(PQExpBuffer buf, const char *colname); -static bool listOneExtensionContents(const char *extname, const char *oid); +static bool listOneExtensionContents(const char *extname, const char *oid, + printTableContent *const content, + PQExpBufferData *title, +
Re: Tab completion for "ALTER TYPE typename SET" and rearranged "Alter TYPE typename RENAME"
On Mon, Aug 15, 2022 at 10:42 AM Michael Paquier wrote:
>
> On Sun, Aug 14, 2022 at 07:56:00PM +0530, vignesh C wrote:
> > Modified the patch to list all the properties in case of "ALTER TYPE
> > typename SET (". I have included the properties in alphabetical order
> > as I notice that the ordering is in alphabetical order in few cases
> > ex: "ALTER SUBSCRIPTION SET (". The attached v2 patch has the
> > changes for the same. Thoughts?
>
> Seems fine here, so applied after tweaking a bit the comments, mostly
> for consistency with the area.
Thanks for pushing this patch.
Regards,
Vignesh
Re: Include the dependent extension information in describe command.
On Sun, Aug 14, 2022 at 11:07 AM Tom Lane wrote: > > vignesh C writes: > > Currently we do not include the dependent extension information for > > index and materialized view in the describe command. I felt it would > > be useful to include this information as part of the describe command > > like: > > \d+ idx_depends > > Index "public.idx_depends" > > Column | Type | Key? | Definition | Storage | Stats target > > +-+--++-+-- > > a | integer | yes | a | plain | > > btree, for table "public.tbl_idx_depends" > > Depends: > > "plpgsql" > > > Attached a patch for the same. Thoughts? > > This seems pretty much useless noise to me. Can you point to > any previous requests for such a feature? If we did do it, > why would we do it in such a narrow fashion (ie, only dependencies > of two specific kinds of objects on one other specific kind of > object)? Why did you do it in this direction rather than > the other one, ie show dependencies when examining the extension? While implementing logical replication of "index which depends on extension", I found that this information was not available in any of the \d describe commands. I felt having this information in the \d describe command will be useful in validating the "depends on extension" easily. Now that you pointed out, I agree that it will be better to show the dependencies from the extension instead of handling it in multiple places. I will change it to handle it from extension and post an updated version soon for this. Regards, Vignesh
Re: Tab completion for "ALTER TYPE typename SET" and rearranged "Alter TYPE typename RENAME"
On Sun, Aug 14, 2022 at 3:41 PM Michael Paquier wrote:
>
> On Sun, Aug 14, 2022 at 08:25:01AM +0530, vignesh C wrote:
> > Attached patch has the changes for the same. Thoughts?
> >
> > a) Add tab completion for "ALTER TYPE typename SET" was missing.
>
> Why not. I can also note that CREATE TYPE lists all the properties
> that can be set to a new type. We could bother adding these for ALTER
> TYPE, perhaps?
Modified the patch to list all the properties in case of "ALTER TYPE
typename SET (". I have included the properties in alphabetical order
as I notice that the ordering is in alphabetical order in few cases
ex: "ALTER SUBSCRIPTION SET (". The attached v2 patch has the
changes for the same. Thoughts?
Regards,
Vignesh
v2-0001-Tab-completion-for-ALTER-TYPE-typename-SET-and-re.patch
Description: Binary data
Tab completion for "ALTER TYPE typename SET" and rearranged "Alter TYPE typename RENAME"
Hi, This patch does a couple of things: a) Tab completion for "ALTER TYPE typename SET" was missing. Added tab completion for the same. b) Tab completion for "ALTER TYPE RENAME VALUE" was not along with tab completion of "ALTER TYPE" commands, it was present after "ALTER GROUP ", rearranged "ALTER TYPE RENAME VALUE", so that it is along with "ALTER TYPE" commands. Attached patch has the changes for the same. Thoughts? Regards, Vignesh v1-0001-Tab-completion-for-ALTER-TYPE-typename-SET-and-re.patch Description: Binary data
Include the dependent extension information in describe command.
Hi, Currently we do not include the dependent extension information for index and materialized view in the describe command. I felt it would be useful to include this information as part of the describe command like: \d+ idx_depends Index "public.idx_depends" Column | Type | Key? | Definition | Storage | Stats target +-+--++-+-- a | integer | yes | a | plain | btree, for table "public.tbl_idx_depends" Depends: "plpgsql" Attached a patch for the same. Thoughts? Regards, Vignesh v1-0001-Include-the-dependent-extension-information-for-i.patch Description: Binary data
Re: hash_xlog_split_allocate_page: failed to acquire cleanup lock
On Wed, Aug 10, 2022 at 2:52 PM Amit Kapila wrote:
>
> On Wed, Aug 10, 2022 at 10:58 AM Andres Freund wrote:
> >
> > Hi,
> >
> > On 2022-08-09 20:21:19 -0700, Mark Dilger wrote:
> > > > On Aug 9, 2022, at 7:26 PM, Andres Freund wrote:
> > > >
> > > > The relevant code triggering it:
> > > >
> > > > newbuf = XLogInitBufferForRedo(record, 1);
> > > > _hash_initbuf(newbuf, xlrec->new_bucket, xlrec->new_bucket,
> > > > xlrec->new_bucket_flag, true);
> > > > if (!IsBufferCleanupOK(newbuf))
> > > > elog(PANIC, "hash_xlog_split_allocate_page: failed to
> > > > acquire cleanup lock");
> > > >
> > > > Why do we just crash if we don't already have a cleanup lock? That
> > > > can't be
> > > > right. Or is there supposed to be a guarantee this can't happen?
> > >
> > > Perhaps the code assumes that when xl_hash_split_allocate_page record was
> > > written, the new_bucket field referred to an unused page, and so during
> > > replay it should also refer to an unused page, and being unused, that
> > > nobody
> > > will have it pinned. But at least in heap we sometimes pin unused pages
> > > just long enough to examine them and to see that they are unused. Maybe
> > > something like that is happening here?
> >
> > I don't think it's a safe assumption that nobody would hold a pin on such a
> > page during recovery. While not the case here, somebody else could have used
> > pg_prewarm to read it in.
> >
> > But also, the checkpointer or bgwriter could have it temporarily pinned, to
> > write it out, or another backend could try to write it out as a victim
> > buffer
> > and have it temporarily pinned.
> >
> >
> > static int
> > SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext
> > *wb_context)
> > {
> > ...
> > /*
> > * Pin it, share-lock it, write it. (FlushBuffer will do nothing
> > if the
> > * buffer is clean by the time we've locked it.)
> > */
> > PinBuffer_Locked(bufHdr);
> > LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
> >
> >
> > As you can see we acquire a pin without holding a lock on the page (and that
> > can't be changed!).
> >
>
> I think this could be the probable reason for failure though I didn't
> try to debug/reproduce this yet. AFAIU, this is possible during
> recovery/replay of WAL record XLOG_HASH_SPLIT_ALLOCATE_PAGE as via
> XLogReadBufferForRedoExtended, we can mark the buffer dirty while
> restoring from full page image. OTOH, because during normal operation
> we didn't mark the page dirty SyncOneBuffer would have skipped it due
> to check (if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))).
I'm trying to simulate the scenario in streaming replication using the below:
CREATE TABLE pvactst (i INT, a INT[], p POINT) with (autovacuum_enabled = off);
CREATE INDEX hash_pvactst ON pvactst USING hash (i);
INSERT INTO pvactst SELECT i, array[1,2,3], point(i, i+1) FROM
generate_series(1,1000) i;
With the above scenario, it will be able to replay allocation of page
for split operation. I will slightly change the above statements and
try to debug and see if we can make the background writer process to
pin this buffer and simulate the scenario. I will post my findings
once I'm done with the analysis.
Regards,
Vignesh
Re: Skipping schema changes in publication
On Mon, Aug 8, 2022 at 12:46 PM vignesh C wrote:
>
> On Fri, Jun 3, 2022 at 3:36 PM vignesh C wrote:
> >
> > On Thu, May 26, 2022 at 7:04 PM osumi.takami...@fujitsu.com
> > wrote:
> > >
> > > On Monday, May 23, 2022 2:13 PM vignesh C wrote:
> > > > Attached v7 patch which fixes the buildfarm warning for an unused
> > > > warning in
> > > > release mode as in [1].
> > > Hi, thank you for the patches.
> > >
> > >
> > > I'll share several review comments.
> > >
> > > For v7-0001.
> > >
> > > (1) I'll suggest some minor rewording.
> > >
> > > +
> > > + The RESET clause will reset the publication to the
> > > + default state which includes resetting the publication options,
> > > setting
> > > + ALL TABLES flag to false and
> > > + dropping all relations and schemas that are associated with the
> > > publication.
> > >
> > > My suggestion is
> > > "The RESET clause will reset the publication to the
> > > default state. It resets the publication operations,
> > > sets ALL TABLES flag to false and drops all relations
> > > and schemas associated with the publication."
> >
> > I felt the existing looks better. I would prefer to keep it that way.
> >
> > > (2) typo and rewording
> > >
> > > +/*
> > > + * Reset the publication.
> > > + *
> > > + * Reset the publication options, setting ALL TABLES flag to false and
> > > drop
> > > + * all relations and schemas that are associated with the publication.
> > > + */
> > >
> > > The "setting" in this sentence should be "set".
> > >
> > > How about changing like below ?
> > > FROM:
> > > "Reset the publication options, setting ALL TABLES flag to false and drop
> > > all relations and schemas that are associated with the publication."
> > > TO:
> > > "Reset the publication operations, set ALL TABLES flag to false and drop
> > > all relations and schemas associated with the publication."
> >
> > I felt the existing looks better. I would prefer to keep it that way.
> >
> > > (3) AlterPublicationReset
> > >
> > > Do we need to call CacheInvalidateRelcacheAll() or
> > > InvalidatePublicationRels() at the end of
> > > AlterPublicationReset() like AlterPublicationOptions() ?
> >
> > CacheInvalidateRelcacheAll should be called if we change all tables
> > from true to false, else the cache will not be invalidated. Modified
> >
> > >
> > > For v7-0002.
> > >
> > > (4)
> > >
> > > + if (stmt->for_all_tables)
> > > + {
> > > + boolisdefault =
> > > CheckPublicationDefValues(tup);
> > > +
> > > + if (!isdefault)
> > > + ereport(ERROR,
> > > +
> > > errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
> > > + errmsg("adding ALL TABLES
> > > requires the publication to have default publication options, no
> > > tables/
> > > + errhint("Use ALTER PUBLICATION
> > > ... RESET to reset the publication"));
> > >
> > >
> > > The errmsg string has three messages for user and is a bit long
> > > (we have two sentences there connected by 'and').
> > > Can't we make it concise and split it into a couple of lines for code
> > > readability ?
> > >
> > > I'll suggest a change below.
> > > FROM:
> > > "adding ALL TABLES requires the publication to have default publication
> > > options, no tables/schemas associated and ALL TABLES flag should not be
> > > set"
> > > TO:
> > > "adding ALL TABLES requires the publication defined not for ALL TABLES"
> > > "to have default publish actions without any associated tables/schemas"
> >
> > Added errdetail and split it
> >
> > > (5) typo
> > >
> > >
> > > +EXCEPT TABLE
> > > +
> > > +
> > > + This clause specifies a list of tables to exclude from the
> > > publication.
> > > + It can only be used with FOR ALL TABLES.
> > > +
&
Re: Skipping schema changes in publication
On Fri, Jun 3, 2022 at 3:36 PM vignesh C wrote:
>
> On Thu, May 26, 2022 at 7:04 PM osumi.takami...@fujitsu.com
> wrote:
> >
> > On Monday, May 23, 2022 2:13 PM vignesh C wrote:
> > > Attached v7 patch which fixes the buildfarm warning for an unused warning
> > > in
> > > release mode as in [1].
> > Hi, thank you for the patches.
> >
> >
> > I'll share several review comments.
> >
> > For v7-0001.
> >
> > (1) I'll suggest some minor rewording.
> >
> > +
> > + The RESET clause will reset the publication to the
> > + default state which includes resetting the publication options, setting
> > + ALL TABLES flag to false and
> > + dropping all relations and schemas that are associated with the
> > publication.
> >
> > My suggestion is
> > "The RESET clause will reset the publication to the
> > default state. It resets the publication operations,
> > sets ALL TABLES flag to false and drops all relations
> > and schemas associated with the publication."
>
> I felt the existing looks better. I would prefer to keep it that way.
>
> > (2) typo and rewording
> >
> > +/*
> > + * Reset the publication.
> > + *
> > + * Reset the publication options, setting ALL TABLES flag to false and drop
> > + * all relations and schemas that are associated with the publication.
> > + */
> >
> > The "setting" in this sentence should be "set".
> >
> > How about changing like below ?
> > FROM:
> > "Reset the publication options, setting ALL TABLES flag to false and drop
> > all relations and schemas that are associated with the publication."
> > TO:
> > "Reset the publication operations, set ALL TABLES flag to false and drop
> > all relations and schemas associated with the publication."
>
> I felt the existing looks better. I would prefer to keep it that way.
>
> > (3) AlterPublicationReset
> >
> > Do we need to call CacheInvalidateRelcacheAll() or
> > InvalidatePublicationRels() at the end of
> > AlterPublicationReset() like AlterPublicationOptions() ?
>
> CacheInvalidateRelcacheAll should be called if we change all tables
> from true to false, else the cache will not be invalidated. Modified
>
> >
> > For v7-0002.
> >
> > (4)
> >
> > + if (stmt->for_all_tables)
> > + {
> > + boolisdefault = CheckPublicationDefValues(tup);
> > +
> > + if (!isdefault)
> > + ereport(ERROR,
> > +
> > errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
> > + errmsg("adding ALL TABLES requires
> > the publication to have default publication options, no tables/
> > + errhint("Use ALTER PUBLICATION ...
> > RESET to reset the publication"));
> >
> >
> > The errmsg string has three messages for user and is a bit long
> > (we have two sentences there connected by 'and').
> > Can't we make it concise and split it into a couple of lines for code
> > readability ?
> >
> > I'll suggest a change below.
> > FROM:
> > "adding ALL TABLES requires the publication to have default publication
> > options, no tables/schemas associated and ALL TABLES flag should not be set"
> > TO:
> > "adding ALL TABLES requires the publication defined not for ALL TABLES"
> > "to have default publish actions without any associated tables/schemas"
>
> Added errdetail and split it
>
> > (5) typo
> >
> >
> > +EXCEPT TABLE
> > +
> > +
> > + This clause specifies a list of tables to exclude from the
> > publication.
> > + It can only be used with FOR ALL TABLES.
> > +
> > +
> > +
> > +
> >
> > Kindly change
> > FROM:
> > This clause specifies a list of tables to exclude from the publication.
> > TO:
> > This clause specifies a list of tables to be excluded from the publication.
> > or
> > This clause specifies a list of tables excluded from the publication.
>
> Modified
>
> > (6) Minor suggestion for an expression change
> >
> >Marks the publication as one that replicates changes for all tables
> > in
> > - the database, including tables created in the future.
> > + the database, including tables created in the f
Re: Logical replication - schema change not invalidating the relation cache
On Sat, Mar 12, 2022 at 1:29 PM vignesh C wrote: > > On Fri, Dec 3, 2021 at 3:21 PM vignesh C wrote: > > > > On Fri, Dec 3, 2021 at 1:13 PM Michael Paquier wrote: > > > > > > On Thu, Aug 26, 2021 at 09:00:39PM +0530, vignesh C wrote: > > > > The previous patch was failing because of the recent test changes made > > > > by commit 201a76183e2 which unified new and get_new_node, attached > > > > patch has the changes to handle the changes accordingly. > > > > > > Please note that the CF app is complaining about this patch, so a > > > rebase is required. I have moved it to next CF, waiting on author, > > > for now. > > > > Thanks for letting me know, I have rebased it on top of HEAD, the > > attached v2 version has the rebased changes. > > The patch was not applying on top of the HEAD, attached v3 version > which has the rebased changes. The patch needed to be rebased on top of HEAD because of commit "0c20dd33db1607d6a85ffce24238c1e55e384b49", attached a rebased v3 version for the changes of the same. Regards, Vignesh v3-0001-Fix-for-invalidating-logical-replication-relation.patch Description: Binary data
Re: Handle infinite recursion in logical replication setup
On Fri, Jul 29, 2022 at 10:51 AM vignesh C wrote: > > On Fri, Jul 29, 2022 at 8:31 AM Peter Smith wrote: > > > > Here are some comments for the patch v40-0001: > > > > == > > > > 1. Commit message > > > > It might be better to always use 'copy_data = true' in favour of > > 'copy_data = on' just for consistency with all the docs and the error > > messages. > > > > == > > Modified > > > 2. doc/src/sgml/ref/create_subscription.sgml > > > > @@ -386,6 +401,15 @@ CREATE SUBSCRIPTION > class="parameter">subscription_name > can have non-existent publications. > > > > > > + > > + If the subscription is created with origin = NONE and > > + copy_data = true, it will check if the publisher has > > + subscribed to the same table from other publishers and, if so, throw an > > + error to prevent possible non-local data from being copied. The user can > > + override this check and continue with the copy operation by specifying > > + copy_data = force. > > + > > > > 2a. > > It is interesting that you changed the note to say origin = NONE. > > Personally, I prefer it written as you did, but I think maybe this > > change does not belong in this patch. The suggestion for changing from > > "none" to NONE is being discussed elsewhere in this thread and > > probably all such changes should be done together (if at all) as a > > separate patch. Until then I think this patch 0001 should just stay > > consistent with whatever is already pushed on HEAD. > > Modified > > > 2b. > > "possible no-local data". Maybe the terminology "local/non-local" is a > > hangover from back when the subscription parameter was called local > > instead of origin. I'm not sure if you want to change this or not, and > > anyway I didn't have any better suggestions – so this comment is just > > to bring it to your attention. > > > > == > > Modified > > > 3. src/backend/commands/subscriptioncmds.c - DefGetCopyData > > > > + /* > > + * The set of strings accepted here should match up with the > > + * grammar's opt_boolean_or_string production. > > + */ > > + if (pg_strcasecmp(sval, "false") == 0 || > > + pg_strcasecmp(sval, "off") == 0) > > + return COPY_DATA_OFF; > > + if (pg_strcasecmp(sval, "true") == 0 || > > + pg_strcasecmp(sval, "on") == 0) > > + return COPY_DATA_ON; > > + if (pg_strcasecmp(sval, "force") == 0) > > + return COPY_DATA_FORCE; > > > > I understand the intention of the comment, but it is not strictly > > correct to say "should match up" because "force" is a new value. > > Perhaps the comment should be as suggested below. > > > > SUGGESTION > > The set of strings accepted here must include all those accepted by > > the grammar's opt_boolean_or_string production. > > > > ~~ > > Modified > > > > > 4. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed > > > > @@ -1781,6 +1858,122 @@ AlterSubscriptionOwner_oid(Oid subid, Oid > > newOwnerId) > > table_close(rel, RowExclusiveLock); > > } > > > > +/* > > + * Check and throw an error if the publisher has subscribed to the same > > table > > + * from some other publisher. This check is required only if "copy_data = > > on" > > + * and "origin = NONE" for CREATE SUBSCRIPTION and > > + * ALTER SUBSCRIPTION ... REFRESH statements to avoid the publisher from > > + * replicating data that has an origin. > > + * > > + * This check need not be performed on the tables that are already added as > > + * incremental sync for such tables will happen through WAL and the origin > > of > > + * the data can be identified from the WAL records. > > + * > > + * subrel_local_oids contains the list of relation oids that are already > > + * present on the subscriber. > > + */ > > +static void > > +check_pub_table_subscribed(WalReceiverConn *wrconn, List *publications, > > +CopyData copydata, char *origin, > > +Oid *subrel_local_oids, int subrel_count) > > > > 4a. > > "copy_data = on" -> "copy_data = true" (for consistency with the docs > > and the error messages) > > Modified > > > 4b. > > The same NONE/none review comment from #2a applies here too. Probably > > it should be written as none for now unle
Re: making relfilenodes 56 bits
On Wed, Jul 27, 2022 at 6:02 PM Dilip Kumar wrote:
>
> On Wed, Jul 27, 2022 at 3:27 PM vignesh C wrote:
> >
>
> > Thanks for the updated patch, Few comments:
> > 1) The format specifier should be changed from %u to INT64_FORMAT
> > autoprewarm.c -> apw_load_buffers
> > ...
> > if (fscanf(file, "%u,%u,%u,%u,%u\n", &blkinfo[i].database,
> >&blkinfo[i].tablespace, &blkinfo[i].filenumber,
> >&forknum, &blkinfo[i].blocknum) != 5)
> > ...
> >
> > 2) The format specifier should be changed from %u to INT64_FORMAT
> > autoprewarm.c -> apw_dump_now
> > ...
> > ret = fprintf(file, "%u,%u,%u,%u,%u\n",
> > block_info_array[i].database,
> > block_info_array[i].tablespace,
> > block_info_array[i].filenumber,
> > (uint32) block_info_array[i].forknum,
> > block_info_array[i].blocknum);
> > ...
> >
> > 3) should the comment "entry point for old extension version" be on
> > top of pg_buffercache_pages, as the current version will use
> > pg_buffercache_pages_v1_4
> > +
> > +Datum
> > +pg_buffercache_pages(PG_FUNCTION_ARGS)
> > +{
> > + return pg_buffercache_pages_internal(fcinfo, OIDOID);
> > +}
> > +
> > +/* entry point for old extension version */
> > +Datum
> > +pg_buffercache_pages_v1_4(PG_FUNCTION_ARGS)
> > +{
> > + return pg_buffercache_pages_internal(fcinfo, INT8OID);
> > +}
> >
> > 4) we could use the new style or ereport by removing the brackets
> > around errcode:
> > + if (fctx->record[i].relfilenumber > OID_MAX)
> > + ereport(ERROR,
> > +
> > (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> > +
> > errmsg("relfilenode" INT64_FORMAT " is too large to be represented as
> > an OID",
> > +
> > fctx->record[i].relfilenumber),
> > +
> > errhint("Upgrade the extension using ALTER EXTENSION pg_buffercache
> > UPDATE")));
> >
> > like:
> > ereport(ERROR,
> >
> > errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> >
> > errmsg("relfilenode" INT64_FORMAT " is too large to be represented as
> > an OID",
> >
> > fctx->record[i].relfilenumber),
> >
> > errhint("Upgrade the extension using ALTER EXTENSION pg_buffercache
> > UPDATE"));
> >
> > 5) Similarly in the below code too:
> > + /* check whether the relfilenumber is within a valid range */
> > + if ((relfilenumber) < 0 || (relfilenumber) > MAX_RELFILENUMBER)
> > + ereport(ERROR,
> > + (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> > +errmsg("relfilenumber " INT64_FORMAT
> > " is out of range",
> > + (relfilenumber;
> >
> >
> > 6) Similarly in the below code too:
> > +#define CHECK_RELFILENUMBER_RANGE(relfilenumber)
> > \
> > +do {
> > \
> > + if ((relfilenumber) < 0 || (relfilenumber) > MAX_RELFILENUMBER) \
> > + ereport(ERROR,
> > \
> > +
> > (errcode(ERRCODE_INVALID_PARAMETER_VALUE), \
> > +errmsg("relfilenumber " INT64_FORMAT
> > " is out of range", \
> > + (relfilenumber; \
> > +} while (0)
> > +
> >
> >
> > 7) This error code looks similar to CHECK_RELFILENUMBER_RANGE, can
> > this macro be used here too:
> > pg_filenode_relation(PG_FUNCTION_ARGS)
> > {
> > Oid reltablespace = PG_GETARG_OID(0);
> > - RelFileNumber relfilenumber = PG_GETARG_OID(1);
> > + RelFileNumber relfilenumber = PG_GETARG_INT64(1);
> > Oid heaprel;
> >
> > + /* check whether the relfilenumber is within a valid range */
> > + if ((relfilenumber) < 0 || (relfilenumber) > MAX_RELFILENUMBER)
> > + ereport(ERROR,
> > + (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> > +errmsg("relfilenumber " INT64_FORMAT
> > " is out of range",
> > +
Re: Handle infinite recursion in logical replication setup
WAL and > the origin of the data can be identified from the WAL records. > > == Modified > > 5. src/test/subscription/t/030_origin.pl > > + "Refresh publication when the publisher has subscribed for the new table" > > SUGGESTION (Just to mention origin = none somehow. Maybe you can > reword it better than this) > Refresh publication when the publisher has subscribed for the new > table, but the subscriber-side wants origin=none Modified Thanks for the comments, the attached v41 patch has the changes for the same. Regards, Vignesh From de95301eed4dec2a76e40effb9c8b8132c2c7e5d Mon Sep 17 00:00:00 2001 From: Vigneshwaran C Date: Thu, 28 Jul 2022 19:11:35 +0530 Subject: [PATCH v41] Check and throw an error if publication tables were also subscribing from other publishers and support force value for copy_data parameter. This patch does a couple of things: 1) Checks and throws an error if 'copy_data = true' and 'origin = none' but the publication tables were also replicated from other publishers. 2) Adds 'force' value for copy_data parameter. --- The steps below help to demonstrate how the new exception is useful: The initial copy phase has no way to know the origin of the row data, so if 'copy_data = true' in the step 4 below, then an error will be thrown to prevent any potentially non-local data from being copied: e.g. CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '' PUBLICATION pub_node1 WITH (copy_data = true, origin = none); ERROR: CREATE/ALTER SUBSCRIPTION with origin = none and copy_data = true is not allowed when the publisher might have replicated data. --- The following steps help to demonstrate how the 'copy_data = force' change will be useful: Let's take a scenario where the user wants to set up bidirectional logical replication between node1 and node2 where the same table on node1 has pre-existing data and node2 has no pre-existing data. e.g. node1: Table t1 (c1 int) has data 11, 12, 13, 14 node2: Table t1 (c1 int) has no pre-existing data The following steps are required in this case: step 1: node1=# CREATE PUBLICATION pub_node1 FOR TABLE t1; CREATE PUBLICATION step 2: node2=# CREATE PUBLICATION pub_node2 FOR TABLE t1; CREATE PUBLICATION step 3: node1=# CREATE SUBSCRIPTION sub_node1_node2 CONNECTION '' node1-# PUBLICATION pub_node2; CREATE SUBSCRIPTION step 4: node2=# CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '' node2-# PUBLICATION pub_node1; CREATE SUBSCRIPTION After the subscription is created on node2, node1 will be synced to node2 and the newly synced data will be sent to node2. This process of node1 sending data to node2 and node2 sending data to node1 will repeat infinitely. If table t1 has a unique key, this will lead to a unique key violation and replication won't proceed. This problem can be avoided by using origin and copy_data parameters as given below: Step 1 & Step 2 are same as above. step 3: Create a subscription on node1 to subscribe to node2: node1=# CREATE SUBSCRIPTION sub_node1_node2 CONNECTION '' node1-# PUBLICATION pub_node2 WITH (copy_data = false, origin = none); CREATE SUBSCRIPTION step 4: Create a subscription on node2 to subscribe to node1. Use 'copy_data = force' when creating a subscription to node1 so that the existing table data is copied during initial sync: node2=# CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '' node2-# PUBLICATION pub_node1 WITH (copy_data = force, origin = none); CREATE SUBSCRIPTION Author: Vignesh C Reviewed-By: Peter Smith, Amit Kapila, Jonathan Katz, Shi yu, Wang wei Discussion: https://www.postgresql.org/message-id/CALDaNm0gwjY_4HFxvvty01BOT01q_fJLKQ3pWP9=9orqubh...@mail.gmail.com --- doc/src/sgml/ref/alter_subscription.sgml | 14 +- doc/src/sgml/ref/create_subscription.sgml | 32 +- src/backend/commands/subscriptioncmds.c| 215 +++- src/test/regress/expected/subscription.out | 22 +- src/test/regress/sql/subscription.sql | 14 + src/test/subscription/t/030_origin.pl | 382 ++--- src/tools/pgindent/typedefs.list | 1 + 7 files changed, 613 insertions(+), 67 deletions(-) diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml index 64efc21f53..f4fb9c5282 100644 --- a/doc/src/sgml/ref/alter_subscription.sgml +++ b/doc/src/sgml/ref/alter_subscription.sgml @@ -161,12 +161,20 @@ ALTER SUBSCRIPTION name RENAME TO < -copy_data (boolean) +copy_data (enum) Specifies whether to copy pre-existing data in the publications - that are being subscribed to when the replication starts. - The default is true. + that are being subscribed to when t
Re: Handle infinite recursion in logical replication setup
On Thu, Jul 28, 2022 at 11:28 AM Peter Smith wrote: > > Hi Vignesh. > > FYI the v39* patch fails to apply [1]. Can you please rebase it? > > > [1] > === Applying patches on top of PostgreSQL commit ID > 5f858dd3bebd1f3845aef2bff7f4345bfb7b74b3 === > === applying patch > ./v39-0001-Check-and-throw-an-error-if-publication-tables-w.patch > patching file doc/src/sgml/ref/alter_subscription.sgml > patching file doc/src/sgml/ref/create_subscription.sgml > patching file src/backend/commands/subscriptioncmds.c > Hunk #10 FAILED at 886. > 1 out of 14 hunks FAILED -- saving rejects to file > src/backend/commands/subscriptioncmds.c.rej > patching file src/test/regress/expected/subscription.out > patching file src/test/regress/sql/subscription.sql > patching file src/test/subscription/t/030_origin.pl > patching file src/tools/pgindent/typedefs.list > > -- Please find the v40 patch attached which is rebased on top of head. Regards, Vignesh From e59c864801c066aff069deb528427788bd0cff0a Mon Sep 17 00:00:00 2001 From: Vigneshwaran C Date: Thu, 28 Jul 2022 19:11:35 +0530 Subject: [PATCH v40 1/2] Check and throw an error if publication tables were also subscribing from other publishers and support force value for copy_data parameter. This patch does a couple of things: 1) Checks and throws an error if 'copy_data = on' and 'origin = none' but the publication tables were also replicated from other publishers. 2) Adds 'force' value for copy_data parameter. --- The steps below help to demonstrate how the new exception is useful: The initial copy phase has no way to know the origin of the row data, so if 'copy_data = on' in the step 4 below, then an error will be thrown to prevent any potentially non-local data from being copied: e.g. CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '' PUBLICATION pub_node1 WITH (copy_data = on, origin = none); ERROR: CREATE/ALTER SUBSCRIPTION with origin = none and copy_data = on is not allowed when the publisher might have replicated data. --- The following steps help to demonstrate how the 'copy_data = force' change will be useful: Let's take a scenario where the user wants to set up bidirectional logical replication between node1 and node2 where the same table on node1 has pre-existing data and node2 has no pre-existing data. e.g. node1: Table t1 (c1 int) has data 11, 12, 13, 14 node2: Table t1 (c1 int) has no pre-existing data The following steps are required in this case: step 1: node1=# CREATE PUBLICATION pub_node1 FOR TABLE t1; CREATE PUBLICATION step 2: node2=# CREATE PUBLICATION pub_node2 FOR TABLE t1; CREATE PUBLICATION step 3: node1=# CREATE SUBSCRIPTION sub_node1_node2 CONNECTION '' node1-# PUBLICATION pub_node2; CREATE SUBSCRIPTION step 4: node2=# CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '' node2-# PUBLICATION pub_node1; CREATE SUBSCRIPTION After the subscription is created on node2, node1 will be synced to node2 and the newly synced data will be sent to node2. This process of node1 sending data to node2 and node2 sending data to node1 will repeat infinitely. If table t1 has a unique key, this will lead to a unique key violation and replication won't proceed. This problem can be avoided by using origin and copy_data parameters as given below: Step 1 & Step 2 are same as above. step 3: Create a subscription on node1 to subscribe to node2: node1=# CREATE SUBSCRIPTION sub_node1_node2 CONNECTION '' node1-# PUBLICATION pub_node2 WITH (copy_data = off, origin = none); CREATE SUBSCRIPTION step 4: Create a subscription on node2 to subscribe to node1. Use 'copy_data = force' when creating a subscription to node1 so that the existing table data is copied during initial sync: node2=# CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '' node2-# PUBLICATION pub_node1 WITH (copy_data = force, origin = none); CREATE SUBSCRIPTION Author: Vignesh C Reviewed-By: Peter Smith, Amit Kapila, Jonathan Katz, Shi yu, Wang wei Discussion: https://www.postgresql.org/message-id/CALDaNm0gwjY_4HFxvvty01BOT01q_fJLKQ3pWP9=9orqubh...@mail.gmail.com --- doc/src/sgml/ref/alter_subscription.sgml | 14 +- doc/src/sgml/ref/create_subscription.sgml | 32 +- src/backend/commands/subscriptioncmds.c| 215 +++- src/test/regress/expected/subscription.out | 22 +- src/test/regress/sql/subscription.sql | 14 + src/test/subscription/t/030_origin.pl | 382 ++--- src/tools/pgindent/typedefs.list | 1 + 7 files changed, 613 insertions(+), 67 deletions(-) diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml index 64efc21f53..f4fb9c5282 100644 --- a
Re: Handle infinite recursion in logical replication setup
On Thu, Jul 28, 2022 at 11:28 AM Peter Smith wrote: > > Hi Vignesh. > > FYI the v39* patch fails to apply [1]. Can you please rebase it? > > > [1] > === Applying patches on top of PostgreSQL commit ID > 5f858dd3bebd1f3845aef2bff7f4345bfb7b74b3 === > === applying patch > ./v39-0001-Check-and-throw-an-error-if-publication-tables-w.patch > patching file doc/src/sgml/ref/alter_subscription.sgml > patching file doc/src/sgml/ref/create_subscription.sgml > patching file src/backend/commands/subscriptioncmds.c > Hunk #10 FAILED at 886. > 1 out of 14 hunks FAILED -- saving rejects to file > src/backend/commands/subscriptioncmds.c.rej > patching file src/test/regress/expected/subscription.out > patching file src/test/regress/sql/subscription.sql > patching file src/test/subscription/t/030_origin.pl > patching file src/tools/pgindent/typedefs.list Thanks for reporting this, I will post an updated version for this soon. Regards, Vignesh
Re: Refactor to make use of a common function for GetSubscriptionRelations and GetSubscriptionNotReadyRelations.
On Wed, Jul 27, 2022 at 4:22 PM Michael Paquier wrote: > > On Wed, Jul 27, 2022 at 08:47:46AM +0530, vignesh C wrote: > > I feel this would be an overkill, I did not make any changes for this. > > Agreed. Using an extra layer of wrappers for that seems a bit too > much, so I have applied your v5 with a slight tweak to the comment. Thanks for pushing this patch. Regards, Vignesh
Re: making relfilenodes 56 bits
On Tue, Jul 26, 2022 at 1:32 PM Dilip Kumar wrote:
>
> On Thu, Jul 21, 2022 at 9:53 AM Thomas Munro wrote:
> >
> > On Wed, Jul 20, 2022 at 11:27 PM Dilip Kumar wrote:
> > > [v10 patch set]
> >
> > Hi Dilip, I'm experimenting with these patches and will hopefully have
> > more to say soon, but I just wanted to point out that this builds with
> > warnings and failed on 3/4 of the CI OSes on cfbot's last run. Maybe
> > there is the good kind of uninitialised data on Linux, and the bad
> > kind of uninitialised data on those other pesky systems?
>
> Here is the patch to fix the issue, basically, while asserting for the
> file existence it was not setting the relfilenumber in the
> relfilelocator before generating the path so it was just checking for
> the existence of the random path so it was asserting randomly.
Thanks for the updated patch, Few comments:
1) The format specifier should be changed from %u to INT64_FORMAT
autoprewarm.c -> apw_load_buffers
...
if (fscanf(file, "%u,%u,%u,%u,%u\n", &blkinfo[i].database,
&blkinfo[i].tablespace, &blkinfo[i].filenumber,
&forknum, &blkinfo[i].blocknum) != 5)
...
2) The format specifier should be changed from %u to INT64_FORMAT
autoprewarm.c -> apw_dump_now
...
ret = fprintf(file, "%u,%u,%u,%u,%u\n",
block_info_array[i].database,
block_info_array[i].tablespace,
block_info_array[i].filenumber,
(uint32) block_info_array[i].forknum,
block_info_array[i].blocknum);
...
3) should the comment "entry point for old extension version" be on
top of pg_buffercache_pages, as the current version will use
pg_buffercache_pages_v1_4
+
+Datum
+pg_buffercache_pages(PG_FUNCTION_ARGS)
+{
+ return pg_buffercache_pages_internal(fcinfo, OIDOID);
+}
+
+/* entry point for old extension version */
+Datum
+pg_buffercache_pages_v1_4(PG_FUNCTION_ARGS)
+{
+ return pg_buffercache_pages_internal(fcinfo, INT8OID);
+}
4) we could use the new style or ereport by removing the brackets
around errcode:
+ if (fctx->record[i].relfilenumber > OID_MAX)
+ ereport(ERROR,
+
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+
errmsg("relfilenode" INT64_FORMAT " is too large to be represented as
an OID",
+
fctx->record[i].relfilenumber),
+
errhint("Upgrade the extension using ALTER EXTENSION pg_buffercache
UPDATE")));
like:
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("relfilenode" INT64_FORMAT " is too large to be represented as
an OID",
fctx->record[i].relfilenumber),
errhint("Upgrade the extension using ALTER EXTENSION pg_buffercache
UPDATE"));
5) Similarly in the below code too:
+ /* check whether the relfilenumber is within a valid range */
+ if ((relfilenumber) < 0 || (relfilenumber) > MAX_RELFILENUMBER)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+errmsg("relfilenumber " INT64_FORMAT
" is out of range",
+ (relfilenumber;
6) Similarly in the below code too:
+#define CHECK_RELFILENUMBER_RANGE(relfilenumber)
\
+do {
\
+ if ((relfilenumber) < 0 || (relfilenumber) > MAX_RELFILENUMBER) \
+ ereport(ERROR,
\
+
(errcode(ERRCODE_INVALID_PARAMETER_VALUE), \
+errmsg("relfilenumber " INT64_FORMAT
" is out of range", \
+ (relfilenumber; \
+} while (0)
+
7) This error code looks similar to CHECK_RELFILENUMBER_RANGE, can
this macro be used here too:
pg_filenode_relation(PG_FUNCTION_ARGS)
{
Oid reltablespace = PG_GETARG_OID(0);
- RelFileNumber relfilenumber = PG_GETARG_OID(1);
+ RelFileNumber relfilenumber = PG_GETARG_INT64(1);
Oid heaprel;
+ /* check whether the relfilenumber is within a valid range */
+ if ((relfilenumber) < 0 || (relfilenumber) > MAX_RELFILENUMBER)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+errmsg("relfilenumber " INT64_FORMAT
" is out of range",
+ (relfilenumber;
8) I felt this include is not required:
diff --git a/src/backend/access/transam/varsup.c
b/src/backend/access/transam/varsup.c
index 849a7ce..a2f0d35 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -13,12 +13,16 @@
#include "postgres.h"
+#include
+
#include "access/clog.h"
#include "access/commit_ts.h"
9) should we change elog to ereport to use the New-style error reporting API
+ /* safety check, we should never get this far in a HS standby */
+ if (RecoveryInProgress())
+ elog(E
Re: Refactor to make use of a common function for GetSubscriptionRelations and GetSubscriptionNotReadyRelations.
On Mon, Jul 25, 2022 at 10:22 AM Amit Kapila wrote: > > On Mon, Jul 25, 2022 at 8:34 AM Peter Smith wrote: > > > > On Mon, Jul 25, 2022 at 11:08 AM Michael Paquier > > wrote: > > > > > > On Sun, Jul 24, 2022 at 09:52:16PM +0530, vignesh C wrote: > > > > Thanks for the comments, i have modified it by changing it to a > > > > boolean parameter. The attached v4 patch has the changes for the same. > > > > > > Okay, thanks for the patch. This looks good to me, so let's do as > > > Amit suggests. I'll apply that if there are no objections. > > > -- > > > > OK. I have no objections to just passing a boolean, but here are a > > couple of other small review comments for the v4-0001 patch: > > > > == > > > > 1. src/backend/catalog/pg_subscription.c > > > > @@ -533,65 +533,14 @@ HasSubscriptionRelations(Oid subid) > > } > > > > /* > > - * Get all relations for subscription. > > + * Get the relations for subscription. > > * > > - * Returned list is palloc'ed in current memory context. > > + * If only_not_ready is false, return all the relations for subscription. > > If > > + * true, return all the relations for subscription that are not in a ready > > + * state. Returned list is palloc'ed in current memory context. > > */ > > > > The function comment was describing the new boolean parameter in a > > kind of backwards way. It seems more natural to emphasise what true > > means. > > > > > > SUGGESTION > > Get the relations for the subscription. > > > > If only_not_ready is true, return only the relations that are not in a > > ready state, otherwise return all the subscription relations. The > > returned list is palloc'ed in the current memory context. > > > > This suggestion sounds good. Also, I don't much like "only" in the > parameter name. I think the explanation makes it clear. I have changed the parameter name to not_ready. The v5 patch attached at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm2sQD-bwMKavLyiogMBBrg3fx5PTaV5RyV8UiczR9K_tw%40mail.gmail.com Regards, Vignesh
Re: Refactor to make use of a common function for GetSubscriptionRelations and GetSubscriptionNotReadyRelations.
On Mon, Jul 25, 2022 at 8:34 AM Peter Smith wrote:
>
> On Mon, Jul 25, 2022 at 11:08 AM Michael Paquier wrote:
> >
> > On Sun, Jul 24, 2022 at 09:52:16PM +0530, vignesh C wrote:
> > > Thanks for the comments, i have modified it by changing it to a
> > > boolean parameter. The attached v4 patch has the changes for the same.
> >
> > Okay, thanks for the patch. This looks good to me, so let's do as
> > Amit suggests. I'll apply that if there are no objections.
> > --
>
> OK. I have no objections to just passing a boolean, but here are a
> couple of other small review comments for the v4-0001 patch:
>
> ==
>
> 1. src/backend/catalog/pg_subscription.c
>
> @@ -533,65 +533,14 @@ HasSubscriptionRelations(Oid subid)
> }
>
> /*
> - * Get all relations for subscription.
> + * Get the relations for subscription.
> *
> - * Returned list is palloc'ed in current memory context.
> + * If only_not_ready is false, return all the relations for subscription. If
> + * true, return all the relations for subscription that are not in a ready
> + * state. Returned list is palloc'ed in current memory context.
> */
>
> The function comment was describing the new boolean parameter in a
> kind of backwards way. It seems more natural to emphasise what true
> means.
>
>
> SUGGESTION
> Get the relations for the subscription.
>
> If only_not_ready is true, return only the relations that are not in a
> ready state, otherwise return all the subscription relations. The
> returned list is palloc'ed in the current memory context.
Modified
>
>
> 2.
>
> Perhaps this suggestion is overkill, but given that the param is not
> going to be a bitmask or enum anymore, IMO it means the calls are no
> longer very self-explanatory.The calling code will be more readable if
> the patch introduced some descriptive wrapper functions. e.g.
>
>
> List *
> GetSubscriptionAllRelations(Oid subid)
> {
> return GetSubscriptionRelations(subid, false);
> }
>
> List *
> GetSubscriptionNotReadyRelations(Oid subid)
> {
> return GetSubscriptionRelations(subid, true);
> }
I feel this would be an overkill, I did not make any changes for this.
Thanks for the comments, the attached v5 patch has the changes for the same.
Regards,
Vignesh
From 4172eeaef8f7890e6ff0148de65332264d039075 Mon Sep 17 00:00:00 2001
From: Vigneshwaran C
Date: Wed, 13 Jul 2022 11:54:59 +0530
Subject: [PATCH v5] Refactor to make use of a common function for
GetSubscriptionRelations and GetSubscriptionNotReadyRelations.
Most of the code is common between GetSubscriptionRelations and
GetSubscriptionNotReadyRelations. Added a parameter to GetSubscriptionRelations
which could provide the same functionality for GetSubscriptionRelations
and GetSubscriptionNotReadyRelations.
Author: Vignesh C
Reviewed-By: Michael Paquier, Peter Smith, Amit Kapila, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/flat/CALDaNm0eW-9g4G_EzHebnFT5zZoasWCS_EzZQ5BgnLZny9S%3Dpg%40mail.gmail.com
---
src/backend/catalog/pg_subscription.c | 70 +++--
src/backend/commands/subscriptioncmds.c | 4 +-
src/backend/replication/logical/tablesync.c | 2 +-
src/include/catalog/pg_subscription_rel.h | 3 +-
4 files changed, 14 insertions(+), 65 deletions(-)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 33ae3da8ae..a60fbaca26 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -533,65 +533,14 @@ HasSubscriptionRelations(Oid subid)
}
/*
- * Get all relations for subscription.
+ * Get the relations for the subscription.
*
- * Returned list is palloc'ed in current memory context.
+ * If not_ready is true, return only the relations that are not in a ready
+ * state, otherwise return all the subscription relations. The returned list is
+ * palloc'ed in the current memory context.
*/
List *
-GetSubscriptionRelations(Oid subid)
-{
- List *res = NIL;
- Relation rel;
- HeapTuple tup;
- ScanKeyData skey[1];
- SysScanDesc scan;
-
- rel = table_open(SubscriptionRelRelationId, AccessShareLock);
-
- ScanKeyInit(&skey[0],
-Anum_pg_subscription_rel_srsubid,
-BTEqualStrategyNumber, F_OIDEQ,
-ObjectIdGetDatum(subid));
-
- scan = systable_beginscan(rel, InvalidOid, false,
- NULL, 1, skey);
-
- while (HeapTupleIsValid(tup = systable_getnext(scan)))
- {
- Form_pg_subscription_rel subrel;
- SubscriptionRelState *relstate;
- Datum d;
- bool isnull;
-
- subrel = (Form_pg_subscription_rel) GETSTRUCT(tup);
-
- relstate = (SubscriptionRelState *) palloc(sizeof(SubscriptionRelState));
- relstate->relid = subrel->srrelid;
- relstate->state = subrel->srsubstate;
- d = SysCacheGetAttr(SU
Re: Handle infinite recursion in logical replication setup
On Mon, Jul 25, 2022 at 12:58 PM Peter Smith wrote: > > Firstly, I have some (case-sensitivity) questions about the previous > patch which was already pushed [1]. > > Q1. create_subscription docs > > I did not understand why the docs refer to slot_name = NONE, yet the > newly added option says origin = none/any. I think that saying origin > = NONE/ANY would be more consistent with the existing usage of NONE in > this documentation. > > ~~~ > > Q2. parse_subscription_options > > Similarly, in the code (parse_subscription_options), I did not > understand why the checks for special name values are implemented > differently: > > The new 'origin' code is using pg_strcmpcase to check special values > (none/any), and the old 'slot_name' code uses case-sensitive strcmp to > check the special value (none). > > FWIW, here I thought the new origin code is the correct one. > > == > > Now, here are some review comments for the patch v38-0001: > > 1. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed > > @@ -1781,6 +1858,121 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId) > table_close(rel, RowExclusiveLock); > } > > +/* > + * Check and throw an error if the publisher has subscribed to the same table > + * from some other publisher. This check is required only if copydata is ON > and > + * the origin is local for CREATE SUBSCRIPTION and > + * ALTER SUBSCRIPTION ... REFRESH statements to avoid replicating remote data > + * from the publisher. > + * > + * This check need not be performed on the tables that are already added as > + * incremental sync for such tables will happen through WAL and the origin of > + * the data can be identified from the WAL records. > + * > + * subrel_local_oids contains the list of relation oids that are already > + * present on the subscriber. > + */ > +static void > +check_pub_table_subscribed(WalReceiverConn *wrconn, List *publications, > +CopyData copydata, char *origin, > +Oid *subrel_local_oids, int subrel_count) > > 1a. > "copydata is ON" --> "copy_data = on" (because the comment is talking > about the CREATE/ALTER statements, so it seemed a bit confusing to > refer to the copydata function param instead of the copy_data > subscription parameter) Modified > 1b. > "the origin is local" ?? But, "local" was the old special name value. > Now it is "none", so I think this part needs minor rewording. Modified > > ~~~ > > 2. > > + if (copydata != COPY_DATA_ON || !origin || > + (pg_strcasecmp(origin, "none") != 0)) > + return; > > Should this be using the constant LOGICALREP_ORIGIN_NONE? Modified > ~~~ > > 3. > > + /* > + * Throw an error if the publisher has subscribed to the same table > + * from some other publisher. We cannot differentiate between the > + * origin and non-origin data that is present in the HEAP during the > + * initial sync. Identification of non-origin data can be done only > + * from the WAL by using the origin id. > + * > + * XXX: For simplicity, we don't check whether the table has any data > + * or not. If the table doesn't have any data then we don't need to > + * distinguish between local and non-local data so we can avoid > + * throwing an error in that case. > + */ > > 3a. > When the special origin value changed from "local" to "none" this > comment's first part seems to have got a bit lost in translation. Modified > SUGGESTION: > Throw an error if the publisher has subscribed to the same table from > some other publisher. We cannot know the origin of data during the > initial sync. Data origins can be found only from the WAL by looking > at the origin id. Modified > 3b. > I think referring to "local and non-local" data in the XXX part of > this comment also needs some minor rewording now that "local" is not a > special origin name anymore. Modified Thanks for the comments, the v39 patch shared at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm2POATc_jwQ-8MBJgGCVZGdUNhnTv8zkBuGzLaY03dM%3DA%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Tue, Jul 26, 2022 at 2:12 PM wangw.f...@fujitsu.com
wrote:
>
> On Sun, Jul 24, 2022 1:28 AM vignesh C wrote:
> > Added a note for the same and referred it to the conflicts section.
> >
> > Thanks for the comments, the attached v38 patch has the changes for the
> > same.
>
> Thanks for your patches.
>
> Two slight comments on the below message in the 0001 patch:
>
> The error message in the function check_pub_table_subscribed().
> + ereport(ERROR,
> +
> errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("could not replicate table \"%s.%s\"",
> + nspname, relname),
> + errdetail("CREATE/ALTER SUBSCRIPTION with
> origin = none and copy_data = on is not allowed when the publisher has
> subscribed same table."),
> + errhint("Use CREATE/ALTER SUBSCRIPTION with
> copy_data = off/force."));
>
> 1.
> I think it might be better to use true/false here than on/off.
> Just for consistency with another error message
> (in function parse_subscription_options) and the description of this parameter
> in the PG document.
Modified
> If you agree with this, please also kindly consider the attached
> "slight_modification.diff" file.
> (This is a slight modification for the second patch, just replace "off" with
> "false" in PG document.)
I have updated the documentation similarly
> 2.
> How about replacing "origin = none" and "copy_data = on" in the message with
> "%s"? I think this might be better for translation. Just like the following
> error message in the function parse_subscription_options:
> ```
> if (opts->copy_data &&
> IsSet(opts->specified_opts, SUBOPT_COPY_DATA))
> ereport(ERROR,
> (errcode(ERRCODE_SYNTAX_ERROR),
> errmsg("%s and %s are mutually
> exclusive options",
> "connect = false",
> "copy_data = true/force")));
> ```
Modified
Thanks for the comments, the v39 patch shared at [1] has the changes
for the same.
[1] -
https://www.postgresql.org/message-id/CALDaNm2POATc_jwQ-8MBJgGCVZGdUNhnTv8zkBuGzLaY03dM%3DA%40mail.gmail.com
Regards,
Vignesh
Re: Handle infinite recursion in logical replication setup
On Tue, Jul 26, 2022 at 1:35 PM shiy.f...@fujitsu.com wrote: > > On Sun, Jul 24, 2022 1:28 AM vignesh C wrote: > > > > Added a note for the same and referred it to the conflicts section. > > > > Thanks for the comments, the attached v38 patch has the changes for the > > same. > > > > Thanks for updating the patch. A comment on the test in 0001 patch. > > +# Alter subscription ... refresh publication should fail when a new table is > +# subscribing data from a different publication should fail > +($result, $stdout, $stderr) = $node_A->psql( > + 'postgres', " > +ALTER SUBSCRIPTION tap_sub_A2 REFRESH PUBLICATION"); > +like( > + $stderr, > + qr/ERROR: ( [A-Z0-9]+:)? could not replicate table "public.tab_new"/, > + "Create subscription with origin and copy_data having replicated > table in publisher" > +); > > The comment says "should fail" twice, the latter one can be removed. Modified > Besides, "Create subscription with origin and copy_data" should be changed to > "Alter subscription with origin and copy_data" I think. Modified to "Refresh publication" Thanks for the comments, the v39 patch shared at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm2POATc_jwQ-8MBJgGCVZGdUNhnTv8zkBuGzLaY03dM%3DA%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Tue, Jul 26, 2022 at 7:16 AM Peter Smith wrote: > > Here are some review comments for the patch v38-0002: > > == > > - terminology > > There seemed to be an inconsistent alternation of the terms > "primaries" and "nodes"... For example "Setting replication between > two primaries" versus "Adding a new node..." (instead of "Adding a new > primary..."?). I have included suggested minor rewording changes in > the review comments below, but please check in case I miss something. > Because I suggested changes to some titles maybe you will also want to > change the section ids too. > > ~~~ > > 1. Commit message > > The documentation was recently modified to remove the term > "bidirectional replication" and replace it all with "replication > between primaries", so this commit message (and also the patch name > itself) should be similarly modified. Modified > ~~~ > > 2. > + > +Replication between primaries is useful for creating a multi-master > +database environment for replicating write operations performed by any of > +the member nodes. The steps to create replication between primaries in > +various scenarios are given below. Note: User is responsible for > designing > +their schemas in a way to minimize the risk of conflicts. See > + for the details of > logical > +replication conflicts. The logical replication restrictions applies to > +the replication between primaries also. See > + for the details of > +logical replication restrictions. > + > > 2a. > "User" -> "The user" Modified > 2b. > "The logical replication restrictions applies to..." --> "The logical > replication restrictions apply to..." Modified > 2c. > These are important notes. Instead of just being part of the text > blurb, perhaps these should be rendered as SGML (or put them > both in a single if you want) Modified > ~~~ > > 3. Setting replication between two primaries > > + Setting replication between two primaries > + > +The following steps demonstrate how to setup replication between two > +primaries when there is no table data present on both nodes > +node1 and node2: > + > > SUGGESTED > The following steps demonstrate how to set up replication between two > primaries (node1 and node2) when there is no table data present on > both nodes:. Modified > ~~~ > > 4. > + > +Now the replication setup between two primaries node1 > +and node2 is complete. Any incremental changes from > +node1 will be replicated to node2, > +and any incremental changes from node2 will be > +replicated to node1. > + > > "between two primaries" -> "between primaries" Modified > ~~~ > > 5. Adding a new node when there is no table data on any of the nodes > > SUGGESTION (title) > Adding a new primary when there is no table data on any of the primaries Modified > ~~~ > > 6. > + > +The following steps demonstrate adding a new node > node3 > +to the existing node1 and node2 > when > +there is no t1 data on any of the nodes. This requires > > SUGGESTION > The following steps demonstrate adding a new primary (node3) to the > existing primaries (node1 and node2) when there is no t1 data on any > of the nodes. Modified > ~~~ > > 7. Adding a new node when table data is present on the existing nodes > > SUGGESTION (title) > Adding a new primary when table data is present on the existing primaries Modified > ~~~ > > 8. > + > + The following steps demonstrate adding a new node > node3 > + which has no t1 data to the existing > + node1 and node2 where > + t1 data is present. This needs similar steps; the > only > > SUGGESTION > The following steps demonstrate adding a new primary (node3) that has > no t1 data to the existing primaries (node1 and node2) where t1 data > is present. Modified > ~~~ > > 9. Adding a new node when table data is present on the new node > > SUGGESTION (title) > Adding a new primary that has existing table data Modified > ~~~ > > 10. > + > + > + Adding a new node when table data is present on the new node is not > + supported. > + > + > > SUGGESTION > Adding a new primary that has existing table data is not supported. Modified > ~~~ > > 11. Generic steps for adding a new node to an existing set of primaries > > SUGGESTION (title) > Generic steps for adding a new primary to an existing set of primaries Modified Thanks for the comments, the v39 patch shared at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm2POATc_jwQ-8MBJgGCVZGdUNhnTv8zkBuGzLaY03dM%3DA%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
changes for the same in the v39 patch attached. Regards, Vignesh From 29c2206e2440bee0ee9818b6a2af66dad5a5d5d4 Mon Sep 17 00:00:00 2001 From: Vigneshwaran C Date: Tue, 26 Jul 2022 22:59:15 +0530 Subject: [PATCH v39 1/2] Check and throw an error if publication tables were also subscribing from other publishers and support force value for copy_data parameter. This patch does a couple of things: 1) Checks and throws an error if 'copy_data = on' and 'origin = none' but the publication tables were also replicated from other publishers. 2) Adds 'force' value for copy_data parameter. --- The steps below help to demonstrate how the new exception is useful: The initial copy phase has no way to know the origin of the row data, so if 'copy_data = on' in the step 4 below, then an error will be thrown to prevent any potentially non-local data from being copied: e.g. CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '' PUBLICATION pub_node1 WITH (copy_data = on, origin = none); ERROR: CREATE/ALTER SUBSCRIPTION with origin = none and copy_data = on is not allowed when the publisher might have replicated data. --- The following steps help to demonstrate how the 'copy_data = force' change will be useful: Let's take a scenario where the user wants to set up bidirectional logical replication between node1 and node2 where the same table on node1 has pre-existing data and node2 has no pre-existing data. e.g. node1: Table t1 (c1 int) has data 11, 12, 13, 14 node2: Table t1 (c1 int) has no pre-existing data The following steps are required in this case: step 1: node1=# CREATE PUBLICATION pub_node1 FOR TABLE t1; CREATE PUBLICATION step 2: node2=# CREATE PUBLICATION pub_node2 FOR TABLE t1; CREATE PUBLICATION step 3: node1=# CREATE SUBSCRIPTION sub_node1_node2 CONNECTION '' node1-# PUBLICATION pub_node2; CREATE SUBSCRIPTION step 4: node2=# CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '' node2-# PUBLICATION pub_node1; CREATE SUBSCRIPTION After the subscription is created on node2, node1 will be synced to node2 and the newly synced data will be sent to node2. This process of node1 sending data to node2 and node2 sending data to node1 will repeat infinitely. If table t1 has a unique key, this will lead to a unique key violation and replication won't proceed. This problem can be avoided by using origin and copy_data parameters as given below: Step 1 & Step 2 are same as above. step 3: Create a subscription on node1 to subscribe to node2: node1=# CREATE SUBSCRIPTION sub_node1_node2 CONNECTION '' node1-# PUBLICATION pub_node2 WITH (copy_data = off, origin = none); CREATE SUBSCRIPTION step 4: Create a subscription on node2 to subscribe to node1. Use 'copy_data = force' when creating a subscription to node1 so that the existing table data is copied during initial sync: node2=# CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '' node2-# PUBLICATION pub_node1 WITH (copy_data = force, origin = none); CREATE SUBSCRIPTION Author: Vignesh C Reviewed-By: Peter Smith, Amit Kapila, Jonathan Katz, Shi yu Discussion: https://www.postgresql.org/message-id/CALDaNm0gwjY_4HFxvvty01BOT01q_fJLKQ3pWP9=9orqubh...@mail.gmail.com --- doc/src/sgml/ref/alter_subscription.sgml | 14 +- doc/src/sgml/ref/create_subscription.sgml | 32 +- src/backend/commands/subscriptioncmds.c| 215 +++- src/test/regress/expected/subscription.out | 22 +- src/test/regress/sql/subscription.sql | 14 + src/test/subscription/t/030_origin.pl | 382 ++--- src/tools/pgindent/typedefs.list | 1 + 7 files changed, 613 insertions(+), 67 deletions(-) diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml index 64efc21f53..f4fb9c5282 100644 --- a/doc/src/sgml/ref/alter_subscription.sgml +++ b/doc/src/sgml/ref/alter_subscription.sgml @@ -161,12 +161,20 @@ ALTER SUBSCRIPTION name RENAME TO < -copy_data (boolean) +copy_data (enum) Specifies whether to copy pre-existing data in the publications - that are being subscribed to when the replication starts. - The default is true. + that are being subscribed to when the replication starts. This + parameter may be either true, + false or force. The default is + true. + + + Refer to the for the + usage of force for copy_data + parameter and its interaction with the origin + parameter. Previously subscribed tables are not copied, even if a table's row diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
Re: Handle infinite recursion in logical replication setup
On Mon, Jul 25, 2022 at 2:24 PM vignesh C wrote: > > On Sun, Jul 24, 2022 at 10:21 PM Jonathan S. Katz > wrote: > > > > On 7/22/22 12:47 AM, Amit Kapila wrote: > > > On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz > > > wrote: > > > > >> 1. I'm concerned by calling this "Bidirectional replication" in the docs > > >> that we are overstating the current capabilities. I think this is > > >> accentuated int he opening paragraph: > > >> > > >> ==snip== > > >>Bidirectional replication is useful for creating a multi-master > > >> database > > >>environment for replicating read/write operations performed by any of > > >> the > > >>member nodes. > > >> ==snip== > > >> > > >> For one, we're not replicating reads, we're replicating writes. Amongst > > >> the writes, at this point we're only replicating DML. A reader could > > >> think that deploying can work for a full bidirectional solution. > > >> > > >> (Even if we're aspirationally calling this section "Bidirectional > > >> replication", that does make it sound like we're limited to two nodes, > > >> when we can support more than two). > > >> > > > > > > Right, I think the system can support N-Way replication. > > > > I did some more testing of the feature, i.e. doing 3-node and 4-node > > tests. While logical replication today can handle replicating between > > multiple nodes (N-way), the "origin = none" does require setting up > > subscribers between each of the nodes. > > > > For example, if I have 4 nodes A, B, C, D and I want to replicate the > > same table between all of them, I need to set up subscribers between all > > of them (A<=>B, A<=>C, A<=>D, B<=>C, B<=>D, C<=>D). However, each node > > can replicate between each other in a way that's convenient (vs. having > > to do something funky with partitions) so this is still a big step forward. > > > > This is a long way of saying that I do think it's fair to say we support > > "N-way" replication so long as you are set up in a mesh (vs. a ring, > > e.g. A=>B=>C=>D=>A). > > > > > Among the above "Replicating changes between primaries" sounds good to > > > me or simply "Replication between primaries". As this is a sub-section > > > on the Logical Replication page, I feel it is okay to not use Logical > > > in the title. > > > > Agreed, I think that's fine. > > > > >> At a minimum, I think we should reference the documentation we have in > > >> the logical replication section on conflicts. We may also want to advise > > >> that a user is responsible for designing their schemas in a way to > > >> minimize the risk of conflicts. > > >> > > > > > > This sounds reasonable to me. > > > > > > One more point about docs, it appears to be added as the last > > > sub-section on the Logical Replication page. Is there a reason for > > > doing so? I feel this should be third sub-section after describing > > > Publication and Subscription. > > > > When I first reviewed, I had not built the docs. Did so on this pass. > > > > I agree with the positioning argument, i.e. it should go after > > "Subscription" in the table of contents -- but it makes me question a > > couple of things: > > > > 1. The general ordering of the docs > > 2. How we describe that section (more on that in a sec) > > 3. If "row filters" should be part of "subscription" instead of its own > > section. > > > > If you look at the current order, "Quick setup" is the last section; one > > would think the "quick" portion goes first :) Given a lot of this is for > > the current docs, I may start a separate discussion on -docs for this part. > > > > For the time being, I agree it should be moved to the section after > > "Subscription". > > > > I think what this section describes is "Configuring Replication Between > > Nodes" as it covers a few different scenarios. > > > > I do think we need to iterate on these docs -- the examples with the > > commands are generally OK and easy to follow, but a few things I noticed: > > > > 1. The general description of the section needs work. We may want to > >
Re: Handle infinite recursion in logical replication setup
On Mon, Jul 25, 2022 at 12:58 PM Peter Smith wrote: > > Firstly, I have some (case-sensitivity) questions about the previous > patch which was already pushed [1]. > > Q1. create_subscription docs > > I did not understand why the docs refer to slot_name = NONE, yet the > newly added option says origin = none/any. I think that saying origin > = NONE/ANY would be more consistent with the existing usage of NONE in > this documentation. Both NONE and none are ok in the case of origin, if you want I can change it to NONE/ANY in case of origin to keep it consistent. Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Sun, Jul 24, 2022 at 10:21 PM Jonathan S. Katz wrote: > > On 7/22/22 12:47 AM, Amit Kapila wrote: > > On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz > > wrote: > > >> 1. I'm concerned by calling this "Bidirectional replication" in the docs > >> that we are overstating the current capabilities. I think this is > >> accentuated int he opening paragraph: > >> > >> ==snip== > >>Bidirectional replication is useful for creating a multi-master database > >>environment for replicating read/write operations performed by any of > >> the > >>member nodes. > >> ==snip== > >> > >> For one, we're not replicating reads, we're replicating writes. Amongst > >> the writes, at this point we're only replicating DML. A reader could > >> think that deploying can work for a full bidirectional solution. > >> > >> (Even if we're aspirationally calling this section "Bidirectional > >> replication", that does make it sound like we're limited to two nodes, > >> when we can support more than two). > >> > > > > Right, I think the system can support N-Way replication. > > I did some more testing of the feature, i.e. doing 3-node and 4-node > tests. While logical replication today can handle replicating between > multiple nodes (N-way), the "origin = none" does require setting up > subscribers between each of the nodes. > > For example, if I have 4 nodes A, B, C, D and I want to replicate the > same table between all of them, I need to set up subscribers between all > of them (A<=>B, A<=>C, A<=>D, B<=>C, B<=>D, C<=>D). However, each node > can replicate between each other in a way that's convenient (vs. having > to do something funky with partitions) so this is still a big step forward. > > This is a long way of saying that I do think it's fair to say we support > "N-way" replication so long as you are set up in a mesh (vs. a ring, > e.g. A=>B=>C=>D=>A). > > > Among the above "Replicating changes between primaries" sounds good to > > me or simply "Replication between primaries". As this is a sub-section > > on the Logical Replication page, I feel it is okay to not use Logical > > in the title. > > Agreed, I think that's fine. > > >> At a minimum, I think we should reference the documentation we have in > >> the logical replication section on conflicts. We may also want to advise > >> that a user is responsible for designing their schemas in a way to > >> minimize the risk of conflicts. > >> > > > > This sounds reasonable to me. > > > > One more point about docs, it appears to be added as the last > > sub-section on the Logical Replication page. Is there a reason for > > doing so? I feel this should be third sub-section after describing > > Publication and Subscription. > > When I first reviewed, I had not built the docs. Did so on this pass. > > I agree with the positioning argument, i.e. it should go after > "Subscription" in the table of contents -- but it makes me question a > couple of things: > > 1. The general ordering of the docs > 2. How we describe that section (more on that in a sec) > 3. If "row filters" should be part of "subscription" instead of its own > section. > > If you look at the current order, "Quick setup" is the last section; one > would think the "quick" portion goes first :) Given a lot of this is for > the current docs, I may start a separate discussion on -docs for this part. > > For the time being, I agree it should be moved to the section after > "Subscription". > > I think what this section describes is "Configuring Replication Between > Nodes" as it covers a few different scenarios. > > I do think we need to iterate on these docs -- the examples with the > commands are generally OK and easy to follow, but a few things I noticed: > > 1. The general description of the section needs work. We may want to > refine the description of the use cases, and in the warning, link to > instructions on how to take backups. > 2. We put the "case not supported" in the middle, not at the end. > 3. The "generic steps for adding a new node..." section uses a > convention for steps that is not found in the docs. We also don't > provide an example for this section, and this is the most complicated > scenario to set up. > > I may be able to propose some suggestions in a few days. > > > BTW, do you have any opinion on the idea of the first remaining patch > > where we accomplish two things: a) Checks and throws an error if > > 'copy_data = on' and 'origin = none' but the publication tables were > > also replicated from other publishers. b) Adds 'force' value for > > copy_data parameter to allow copying in such a case. The primary > > reason for this patch is to avoid loops or duplicate data in the > > initial phase. We can't skip copying based on origin as we can do > > while replicating changes from WAL. So, we detect that the publisher > > already has data from some other node and doesn't allow replication > > unless the user uses the 'force' option for copy_data. > > In general, I agree with the patch; but
Re: Refactor to make use of a common function for GetSubscriptionRelations and GetSubscriptionNotReadyRelations.
On Fri, Jul 22, 2022 at 11:11 AM Amit Kapila wrote:
>
> On Fri, Jul 22, 2022 at 3:47 AM Peter Smith wrote:
> >
> > I was in favour of enum mostly because I thought the bitmask of an
> > earlier patch was mis-used; IMO each bit should only be for
> > representing something as "on/set". So a bit for
> > SUBSCRIPTION_REL_STATE_READY makes sense, but a bit for
> > SUBSCRIPTION_REL_STATE_NOT_READY seemed strange/backwards to me. YMMV.
> >
> > So using a bitmask is fine, except I thought it should be implemented
> > so that one of the bits is for a "NOT" modifier (IIUC this is kind of
> > similar to what Michael [1] suggested above?). So "Not READY" would be
> > (SUBSCRIPTION_REL_STATE_MOD_NOT | SUBSCRIPTION_REL_STATE_READY)
> >
>
> Hmm, I think that sounds more complicated than what I expected. I
> suggest let's go with a simple idea of using a boolean not_ready which
> will decide whether to use the additional key to search. I feel we can
> extend it by using a bitmask or enum when we have a clear need for
> more states.
Thanks for the comments, i have modified it by changing it to a
boolean parameter. The attached v4 patch has the changes for the same.
Regards,
Vignesh
From e5edbf8dd47bbfcd7d8fcf3add8acab502e48665 Mon Sep 17 00:00:00 2001
From: Vigneshwaran C
Date: Wed, 13 Jul 2022 11:54:59 +0530
Subject: [PATCH v4] Refactor to make use of a common function for
GetSubscriptionRelations and GetSubscriptionNotReadyRelations.
Most of the code is common between GetSubscriptionRelations and
GetSubscriptionNotReadyRelations. Added a parameter to GetSubscriptionRelations
which could provide the same functionality for GetSubscriptionRelations
and GetSubscriptionNotReadyRelations.
Author: Vignesh C
Reviewed-By: Michael Paquier, Peter Smith, Amit Kapila, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/flat/CALDaNm0eW-9g4G_EzHebnFT5zZoasWCS_EzZQ5BgnLZny9S%3Dpg%40mail.gmail.com
---
src/backend/catalog/pg_subscription.c | 70 +++--
src/backend/commands/subscriptioncmds.c | 4 +-
src/backend/replication/logical/tablesync.c | 2 +-
src/include/catalog/pg_subscription_rel.h | 3 +-
4 files changed, 14 insertions(+), 65 deletions(-)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 33ae3da8ae..3d81449382 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -533,65 +533,14 @@ HasSubscriptionRelations(Oid subid)
}
/*
- * Get all relations for subscription.
+ * Get the relations for subscription.
*
- * Returned list is palloc'ed in current memory context.
+ * If only_not_ready is false, return all the relations for subscription. If
+ * true, return all the relations for subscription that are not in a ready
+ * state. Returned list is palloc'ed in current memory context.
*/
List *
-GetSubscriptionRelations(Oid subid)
-{
- List *res = NIL;
- Relation rel;
- HeapTuple tup;
- ScanKeyData skey[1];
- SysScanDesc scan;
-
- rel = table_open(SubscriptionRelRelationId, AccessShareLock);
-
- ScanKeyInit(&skey[0],
-Anum_pg_subscription_rel_srsubid,
-BTEqualStrategyNumber, F_OIDEQ,
-ObjectIdGetDatum(subid));
-
- scan = systable_beginscan(rel, InvalidOid, false,
- NULL, 1, skey);
-
- while (HeapTupleIsValid(tup = systable_getnext(scan)))
- {
- Form_pg_subscription_rel subrel;
- SubscriptionRelState *relstate;
- Datum d;
- bool isnull;
-
- subrel = (Form_pg_subscription_rel) GETSTRUCT(tup);
-
- relstate = (SubscriptionRelState *) palloc(sizeof(SubscriptionRelState));
- relstate->relid = subrel->srrelid;
- relstate->state = subrel->srsubstate;
- d = SysCacheGetAttr(SUBSCRIPTIONRELMAP, tup,
- Anum_pg_subscription_rel_srsublsn, &isnull);
- if (isnull)
- relstate->lsn = InvalidXLogRecPtr;
- else
- relstate->lsn = DatumGetLSN(d);
-
- res = lappend(res, relstate);
- }
-
- /* Cleanup */
- systable_endscan(scan);
- table_close(rel, AccessShareLock);
-
- return res;
-}
-
-/*
- * Get all relations for subscription that are not in a ready state.
- *
- * Returned list is palloc'ed in current memory context.
- */
-List *
-GetSubscriptionNotReadyRelations(Oid subid)
+GetSubscriptionRelations(Oid subid, bool only_not_ready)
{
List *res = NIL;
Relation rel;
@@ -607,10 +556,11 @@ GetSubscriptionNotReadyRelations(Oid subid)
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(subid));
- ScanKeyInit(&skey[nkeys++],
-Anum_pg_subscription_rel_srsubstate,
-BTEqualStrategyNumber, F_CHARNE,
-CharGetDatum(SUBREL_STATE_READY));
+ if (only_not_ready)
+ ScanKeyInit(&skey[nkeys++],
+ Anum_pg_subscription_rel_srsubstate,
+ BTEqualStrategyNumber, F_CHARNE,
+ CharGetDatum(SUBREL_STATE_READY));
scan = systable_beginscan(rel, InvalidOid, false
Re: Handle infinite recursion in logical replication setup
On Fri, Jul 22, 2022 at 10:17 AM Amit Kapila wrote: > > On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz wrote: > > > > Thanks for the work on this feature -- this is definitely very helpful > > towards supporting more types of use cases with logical replication! > > > > I've read through the proposed documentation and did some light testing > > of the patch. I have two general comments about the docs as they > > currently read: > > > > 1. I'm concerned by calling this "Bidirectional replication" in the docs > > that we are overstating the current capabilities. I think this is > > accentuated int he opening paragraph: > > > > ==snip== > > Bidirectional replication is useful for creating a multi-master database > > environment for replicating read/write operations performed by any of the > > member nodes. > > ==snip== > > > > For one, we're not replicating reads, we're replicating writes. Amongst > > the writes, at this point we're only replicating DML. A reader could > > think that deploying can work for a full bidirectional solution. > > > > (Even if we're aspirationally calling this section "Bidirectional > > replication", that does make it sound like we're limited to two nodes, > > when we can support more than two). > > > > Right, I think the system can support N-Way replication. > > > Perhaps "Logical replication between writers" or "Logical replication > > between primaries" or "Replicating changes between primaries", or > > something better. > > > > Among the above "Replicating changes between primaries" sounds good to > me or simply "Replication between primaries". As this is a sub-section > on the Logical Replication page, I feel it is okay to not use Logical > in the title. I have changed it to "Replication between primaries". > > 2. There is no mention of conflicts in the documentation, e.g. > > referencing the "Conflicts" section of the documentation. It's very easy > > to create a conflicting transaction that causes a subscriber to be > > unable to continue to apply transactions: > > > >-- DB 1 > >CREATE TABLE abc (id int); > >CREATE PUBLICATION node1 FOR ALL TABLES ; > > > >-- DB2 > >CREATE TABLE abc (id int); > >CREATE PUBLICATION node2 FOR ALL TABLES ; > >CREATE SUBSCRIPTION node2_node1 > > CONNECTION 'dbname=logi port=5433' > > PUBLICATION node1 > > WITH (copy_data = off, origin = none); > > > >-- DB1 > >CREATE SUBSCRIPTION node1_node2 > > CONNECTION 'dbname=logi port=5434' > > PUBLICATION node2 > > WITH (copy_data = off, origin = none); > >INSERT INTO abc VALUES (1); > > > >-- DB2 > >INSERT INTO abc VALUES (2); > > > >-- DB1 > >ALTER TABLE abc ADD PRIMARY KEY id; > >INSERT INTO abc VALUES (3); > > > >-- DB2 > >INSERT INTO abc VALUES (3); > > > >-- DB1 cannot apply the transactions > > > > At a minimum, I think we should reference the documentation we have in > > the logical replication section on conflicts. We may also want to advise > > that a user is responsible for designing their schemas in a way to > > minimize the risk of conflicts. > > > > This sounds reasonable to me. > > One more point about docs, it appears to be added as the last > sub-section on the Logical Replication page. Is there a reason for > doing so? I feel this should be third sub-section after describing > Publication and Subscription. I had initially kept it at the end since we demonstrated the various steps to create the replication between the primaries. Since it gives an introduction about the "Replication between primaries" and then states the steps, it looks ok to move it as suggested. I have modified this in the v38 patch attached at [1]. [1] - https://www.postgresql.org/message-id/CALDaNm01x0sLz8YzfCSjxcMFxM4NDQxcFzZa%2B4eesUmD40DdTg%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz wrote: > > Hi, > > On 7/21/22 6:34 AM, vignesh C wrote: > > On Thu, Jul 21, 2022 at 2:06 PM Amit Kapila wrote: > >> > >> On Wed, Jul 20, 2022 at 2:33 PM vignesh C wrote: > >>> > >>> Modified. Apart from this I have run pgperltidy on the perl file and > >>> renamed 032_origin.pl to 030_origin.pl as currently there is > >>> 029_on_error.pl, 031_column_list.pl and there is no 030_*.pl file. > >>> Thanks for the comment, the attached patch has the changes for the same. > >>> > >> > >> Pushed. Kindly rebase the remaining patches. > > > > Thanks for pushing the patch. > > The attached v37 version contains the rebased patch for the remaining > > patches. > > Thanks for the work on this feature -- this is definitely very helpful > towards supporting more types of use cases with logical replication! > > I've read through the proposed documentation and did some light testing > of the patch. I have two general comments about the docs as they > currently read: > > 1. I'm concerned by calling this "Bidirectional replication" in the docs > that we are overstating the current capabilities. I think this is > accentuated int he opening paragraph: > > ==snip== > Bidirectional replication is useful for creating a multi-master database > environment for replicating read/write operations performed by any of the > member nodes. > ==snip== > > For one, we're not replicating reads, we're replicating writes. Amongst > the writes, at this point we're only replicating DML. A reader could > think that deploying can work for a full bidirectional solution. I have changed read/write operations to write operations. I have also added a note saying "The logical replication restrictions applies to the replication between primaries also.", to clarify that non DML operations and other restrictions apply in this case too. > (Even if we're aspirationally calling this section "Bidirectional > replication", that does make it sound like we're limited to two nodes, > when we can support more than two). > > Perhaps "Logical replication between writers" or "Logical replication > between primaries" or "Replicating changes between primaries", or > something better. I have changed it to "Replication between primaries". > 2. There is no mention of conflicts in the documentation, e.g. > referencing the "Conflicts" section of the documentation. It's very easy > to create a conflicting transaction that causes a subscriber to be > unable to continue to apply transactions: > >-- DB 1 >CREATE TABLE abc (id int); >CREATE PUBLICATION node1 FOR ALL TABLES ; > >-- DB2 >CREATE TABLE abc (id int); >CREATE PUBLICATION node2 FOR ALL TABLES ; >CREATE SUBSCRIPTION node2_node1 > CONNECTION 'dbname=logi port=5433' > PUBLICATION node1 > WITH (copy_data = off, origin = none); > >-- DB1 >CREATE SUBSCRIPTION node1_node2 > CONNECTION 'dbname=logi port=5434' > PUBLICATION node2 > WITH (copy_data = off, origin = none); >INSERT INTO abc VALUES (1); > >-- DB2 >INSERT INTO abc VALUES (2); > >-- DB1 >ALTER TABLE abc ADD PRIMARY KEY id; >INSERT INTO abc VALUES (3); > >-- DB2 >INSERT INTO abc VALUES (3); > >-- DB1 cannot apply the transactions > > At a minimum, I think we should reference the documentation we have in > the logical replication section on conflicts. We may also want to advise > that a user is responsible for designing their schemas in a way to > minimize the risk of conflicts. Added a note for the same and referred it to the conflicts section. Thanks for the comments, the attached v38 patch has the changes for the same. Regards, Vignesh From 9207ff99cab50d4456862ba6e9f613d37234f15d Mon Sep 17 00:00:00 2001 From: Vigneshwaran C Date: Thu, 21 Jul 2022 14:56:57 +0530 Subject: [PATCH v38 1/2] Check and throw an error if publication tables were also subscribing from other publishers and support force value for copy_data parameter. This patch does a couple of things: 1) Checks and throws an error if 'copy_data = on' and 'origin = none' but the publication tables were also replicated from other publishers. 2) Adds 'force' value for copy_data parameter. --- The steps below help to demonstrate how the new exception is useful: The initial copy phase has no way to know the origin of the row data, so if 'cop
Re: making relfilenodes 56 bits
On Wed, Jul 20, 2022 at 4:57 PM Dilip Kumar wrote:
>
> On Mon, Jul 18, 2022 at 4:51 PM Dilip Kumar wrote:
> >
> > I was doing some more testing by setting the FirstNormalRelFileNumber
> > to a high value(more than 32 bits) I have noticed a couple of problems
> > there e.g. relpath is still using OIDCHARS macro which says max
> > relfilenumber file name can be only 10 character long which is no
> > longer true. So there we need to change this value to 20 and also
> > need to carefully rename the macros and other variable names used for
> > this purpose.
> >
> > Similarly there was some issue in macro in buf_internal.h while
> > fetching the relfilenumber. So I will relook into all those issues
> > and repost the patch soon.
>
> I have fixed these existing issues and there was also some issue in
> pg_dump.c which was creating problems in upgrading to the same version
> while using a higher range of the relfilenumber.
>
> There was also an issue where the user table from the old cluster's
> relfilenode could conflict with the system table of the new cluster.
> As a solution currently for system table object (while creating
> storage first time) we are keeping the low range of relfilenumber,
> basically we are using the same relfilenumber as OID so that during
> upgrade the normal user table from the old cluster will not conflict
> with the system tables in the new cluster. But with this solution
> Robert told me (in off list chat) a problem that in future if we want
> to make relfilenumber completely unique within a cluster by
> implementing the CREATEDB differently then we can not do that as we
> have created fixed relfilenodes for the system tables.
>
> I am not sure what exactly we can do to avoid that because even if we
> do something to avoid that in the new cluster the old cluster might
> be already using the non-unique relfilenode so after upgrading the new
> cluster will also get those non-unique relfilenode.
Thanks for the patch, my comments from the initial review:
1) Since we have changed the macros to inline functions, should we
change the function names similar to the other inline functions in the
same file like: ClearBufferTag, InitBufferTag & BufferTagsEqual:
-#define BUFFERTAGS_EQUAL(a,b) \
-( \
- RelFileLocatorEquals((a).rlocator, (b).rlocator) && \
- (a).blockNum == (b).blockNum && \
- (a).forkNum == (b).forkNum \
-)
+static inline void
+CLEAR_BUFFERTAG(BufferTag *tag)
+{
+ tag->rlocator.spcOid = InvalidOid;
+ tag->rlocator.dbOid = InvalidOid;
+ tag->rlocator.relNumber = InvalidRelFileNumber;
+ tag->forkNum = InvalidForkNumber;
+ tag->blockNum = InvalidBlockNumber;
+}
2) We could move this macros along with the other macros at the top of the file:
+/*
+ * The freeNext field is either the index of the next freelist entry,
+ * or one of these special values:
+ */
+#define FREENEXT_END_OF_LIST (-1)
+#define FREENEXT_NOT_IN_LIST (-2)
3) typo thn should be then:
+ * can raise it as necessary if we end up with more mapped relations. For
+ * now, we just pick a round number that is modestly larger thn the expected
+ * number of mappings.
+ */
4) There is one whitespace issue:
git am v10-0004-Widen-relfilenumber-from-32-bits-to-56-bits.patch
Applying: Widen relfilenumber from 32 bits to 56 bits
.git/rebase-apply/patch:1500: space before tab in indent.
(relfilenumber; \
warning: 1 line adds whitespace errors.
Regards,
Vignesh
Re: Handle infinite recursion in logical replication setup
On Thu, Jul 21, 2022 at 2:06 PM Amit Kapila wrote: > > On Wed, Jul 20, 2022 at 2:33 PM vignesh C wrote: > > > > Modified. Apart from this I have run pgperltidy on the perl file and > > renamed 032_origin.pl to 030_origin.pl as currently there is > > 029_on_error.pl, 031_column_list.pl and there is no 030_*.pl file. > > Thanks for the comment, the attached patch has the changes for the same. > > > > Pushed. Kindly rebase the remaining patches. Thanks for pushing the patch. The attached v37 version contains the rebased patch for the remaining patches. Regards, Vignesh From c162bfe2367f71b6e9a2e13ee00b68b61d08840e Mon Sep 17 00:00:00 2001 From: Vigneshwaran C Date: Mon, 27 Jun 2022 18:44:18 +0530 Subject: [PATCH v37 2/2] Document bidirectional logical replication steps in various scenarios. Document the steps for the following: a) Setting bidirectional replication between two nodes. b) Adding a new node when there is no table data on any of the nodes. c) Adding a new node when table data is present on the existing nodes. d) Generic steps for adding a new node to an existing set of nodes. Author: Vignesh C Reviewed-By: Peter Smith, Amit Kapila, Shi yu Discussion: https://www.postgresql.org/message-id/CALDaNm0gwjY_4HFxvvty01BOT01q_fJLKQ3pWP9=9orqubh...@mail.gmail.com --- doc/src/sgml/logical-replication.sgml | 301 ++ doc/src/sgml/ref/create_subscription.sgml | 5 +- 2 files changed, 305 insertions(+), 1 deletion(-) diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index bdf1e7b727..c94b3bfd27 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -1479,4 +1479,305 @@ CREATE SUBSCRIPTION mysub CONNECTION 'dbname=foo host=bar user=repuser' PUBLICAT incremental changes to those tables. + + + Bidirectional logical replication + + +Bidirectional replication is useful for creating a multi-master database +environment for replicating read/write operations performed by any of the +member nodes. The steps to create a bidirectional replication in various +scenarios are given below. + + + + + Setting up bidirectional logical replication requires multiple steps to be + performed on various nodes. Because not all operations are transactional, + the user is advised to take backups. + + + + + Setting bidirectional replication between two nodes + +The following steps demonstrate how to create a two-node bidirectional +replication when there is no table data present on both nodes +node1 and node2: + + + +Create a publication on node1: + +node1=# CREATE PUBLICATION pub_node1 FOR TABLE t1; +CREATE PUBLICATION + + + +Create a publication on node2: + +node2=# CREATE PUBLICATION pub_node2 FOR TABLE t1; +CREATE PUBLICATION + + + +Lock the table t1 on node1 and +node2 in EXCLUSIVE mode until the +setup is completed. + + + +Create a subscription on node2 to subscribe to +node1: + +node2=# CREATE SUBSCRIPTION sub_node2_node1 +node2-# CONNECTION 'dbname=foo host=node1 user=repuser' +node2-# PUBLICATION pub_node1 +node2-# WITH (copy_data = off, origin = none); +CREATE SUBSCRIPTION + + + +Create a subscription on node1 to subscribe to +node2: + +node1=# CREATE SUBSCRIPTION sub_node1_node2 +node1-# CONNECTION 'dbname=foo host=node2 user=repuser' +node1-# PUBLICATION pub_node2 +node1-# WITH (copy_data = off, origin = none); +CREATE SUBSCRIPTION + + + +Now the bidirectional logical replication setup is complete between +node1 and node2. Any incremental +changes from node1 will be replicated to +node2, and any incremental changes from +node2 will be replicated to node1. + + + + + Adding a new node when there is no table data on any of the nodes + +The following steps demonstrate adding a new node node3 +to the existing node1 and node2 when +there is no t1 data on any of the nodes. This requires +creating subscriptions on node1 and +node2 to replicate the data from +node3 and creating subscriptions on +node3 to replicate data from node1 +and node2. Note: These steps assume that the +bidirectional logical replication between node1 and +node2 is already completed. + + + +Create a publication on node3: + +node3=# CREATE PUBLICATION pub_node3 FOR TABLE t1; +CREATE PUBLICATION + + + +Lock table t1 on all the nodes node1, +node2 and node3 in +EXCLUSIVE mode until the setup is completed. + + + +Create a subscription on node1 to subscribe to +node3: + +node1=# CREATE SUBSCRIPTION sub_node1_node3 +node1-# CONNECTION 'dbname=foo host=node3 user=repuser' +node1-# PUBLICATION pub_node3 +node1-# WITH (copy_data = off, origin = none); +CREATE SUBSCRIPTION + + + +Create a subscription on
Re: Handle infinite recursion in logical replication setup
On Wed, Jul 20, 2022 at 10:38 AM Peter Smith wrote:
>
> On Tue, Jul 19, 2022 at 11:34 PM Amit Kapila wrote:
> >
> > On Mon, Jul 18, 2022 at 9:46 PM vignesh C wrote:
> > >
> > > I have updated the patch to handle the origin value case
> > > insensitively. The attached patch has the changes for the same.
> > >
> >
> > Thanks, the patch looks mostly good to me. I have made a few changes
> > in 0001 patch which are as follows: (a) make a comparison of origin
> > names in maybe_reread_subscription similar to slot names as in future
> > we may support origin names other than 'any' and 'none', (b) made
> > comment changes at few places and minor change in one of the error
> > message, (c) ran pgindent and slightly changed the commit message.
> >
> > I am planning to push this day after tomorrow unless there are any
> > comments/suggestions.
>
> FYI, the function name in the comment is not same as the function name here:
>
> +/*
> + * IsReservedName
> + * True iff name is either "none" or "any".
> + */
> +static bool
> +IsReservedOriginName(const char *name)
Modified. Apart from this I have run pgperltidy on the perl file and
renamed 032_origin.pl to 030_origin.pl as currently there is
029_on_error.pl, 031_column_list.pl and there is no 030_*.pl file.
Thanks for the comment, the attached patch has the changes for the same.
Regards,
Vignesh
From 69f9b55aa28a0325e0d85c8f8bed407c7218e813 Mon Sep 17 00:00:00 2001
From: Vigneshwaran C
Date: Sat, 9 Jul 2022 13:20:34 +0530
Subject: [PATCH v36] Allow users to skip logical replication of data having
origin.
This patch adds a new SUBSCRIPTION parameter "origin". It specifies
whether the subscription will request the publisher to only send changes
that don't have an origin or send changes regardless of origin. Setting it
to "none" means that the subscription will request the publisher to only
send changes that have no origin associated. Setting it to "any" means
that the publisher sends changes regardless of their origin. The default
is "any".
Usage:
CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=postgres port='
PUBLICATION pub1 WITH (origin = none);
This can be used to avoid loops (infinite replication of the same data)
among replication nodes.
This feature allows filtering only the replication data originating from
WAL but for initial sync (initial copy of table data) we don't have such a
facility as we can only distinguish the data based on origin from WAL. As
a follow-up patch, we are planning to forbid the initial sync if we notice
that the publication tables were also replicated from other publishers to
avoid duplicate data or loops.
Author: Vignesh C, Amit Kapila
Reviewed-By: Peter Smith, Amit Kapila, Dilip Kumar, Ashutosh Bapat, Hayato Kuroda, Shi yu, Alvaro Herrera
Discussion: https://postgr.es/m/CALDaNm0gwjY_4HFxvvty01BOT01q_fJLKQ3pWP9=9orqubh...@mail.gmail.com
---
contrib/test_decoding/expected/replorigin.out | 10 ++
contrib/test_decoding/sql/replorigin.sql | 5 +
doc/src/sgml/catalogs.sgml| 14 ++
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 15 ++
src/backend/catalog/pg_subscription.c | 8 +
src/backend/catalog/system_views.sql | 4 +-
src/backend/commands/subscriptioncmds.c | 43 -
.../libpqwalreceiver/libpqwalreceiver.c | 5 +
src/backend/replication/logical/origin.c | 22 ++-
src/backend/replication/logical/worker.c | 2 +
src/backend/replication/pgoutput/pgoutput.c | 27 ++-
src/bin/pg_dump/pg_dump.c | 15 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 22 +++
src/bin/psql/describe.c | 7 +-
src/bin/psql/tab-complete.c | 7 +-
src/include/catalog/pg_subscription.h | 17 ++
src/include/replication/pgoutput.h| 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/subscription.out| 142 +---
src/test/regress/sql/subscription.sql | 10 ++
src/test/subscription/t/030_origin.pl | 155 ++
23 files changed, 462 insertions(+), 77 deletions(-)
create mode 100644 src/test/subscription/t/030_origin.pl
diff --git a/contrib/test_decoding/expected/replorigin.out b/contrib/test_decoding/expected/replorigin.out
index 2e9ef7c823..49ffaeea2d 100644
--- a/contrib/test_decoding/expected/replorigin.out
+++ b/contrib/test_decoding/expected/replorigin.out
@@ -56,6 +56,16 @@ SELECT pg_replication_origin_drop('regress_test_decoding: temp');
SELECT pg_replication_origin_drop('regress_test_decoding: temp');
ERROR: replication origin
Re: Handle infinite recursion in logical replication setup
On Thu, Jul 14, 2022 at 6:42 PM Dilip Kumar wrote:
>
> On Thu, 14 Jul 2022 at 6:34 PM, vignesh C wrote:
>>
>> On Thu, Jul 14, 2022 at 11:26 AM Dilip Kumar wrote:
>> >
>> > On Wed, Jul 13, 2022 at 4:49 PM Dilip Kumar wrote:
>> > >
>> > > On Tue, Jul 12, 2022 at 2:58 PM vignesh C wrote:
>> > > >
>> > > > On Tue, Jul 12, 2022 at 9:51 AM Amit Kapila
>> > > > wrote:
>> > >
>> > > I find one thing confusing about this patch. Basically, this has two
>> > > option 'local' and 'any', so I would assume that all the local server
>> > > changes should be covered under the 'local' but now if we set some
>> > > origin using 'select pg_replication_origin_session_setup('aa');' then
>> > > changes from that session will be ignored because it has an origin id.
>> > > I think actually the name is creating confusion, because by local it
>> > > seems like a change which originated locally and the document is also
>> > > specifying the same.
>> > >
>> > > + If local, the subscription will request the
>> > > publisher
>> > > + to only send changes that originated locally. If
>> > > any,
>> > >
>> > > I think if we want to keep the option local then we should look up all
>> > > the origin in the replication origin catalog and identify whether it
>> > > is a local origin id or remote origin id and based on that filter out
>> > > the changes.
>> >
>> > On the other hand if we are interested in receiving the changes which
>> > are generated without any origin then I think we should change 'local'
>> > to 'none' and then in future we can provide a new option which can
>> > send the changes generated by all the local origin? I think other
>> > than this the patch LGTM.
>>
>> Thanks for the comment. The attached v33 patch has the changes to
>> specify origin as 'none' instead of 'local' which will not publish the
>> data having any origin.
>
>
> I think the ‘none’ might have problem from expand ability pov? what if in
> future we support the actual origin name and than what none mean? no origin
> or origin name none? Should we just give origin name empty name ‘’? Or is
> there some other issue?
Currently there is no restriction in the name we can specify for
origin, ex any, none, local, etc all are allowed as origin name. How
about extending it with another parameter "origin_name" when we
support filtering of a particular origin like:
1) origin = name, origin_name = 'orig1' -- Means that the publisher
will filter the changes having origin name as 'orig1' and send the
other changes.
2) origin = any -- Means that the publisher sends all changes
regardless of their origin.
3) origin = none -- Means that the subscription will request the
publisher to only send changes that have no origin associated.
When we need to specify filtering of a particular origin name we will
have to use both origin and origin_name option, like origin = name,
origin_name = 'orig1' as in the first example.
I'm not sure if there is a simpler way to do this with only a single
option as both 'none' and 'any' can be specified as the origin name.
Thoughts?
Regards,
Vignesh
Re: Refactor to make use of a common function for GetSubscriptionRelations and GetSubscriptionNotReadyRelations.
On Thu, Jul 14, 2022 at 4:34 AM Peter Smith wrote:
>
> On Wed, Jul 13, 2022 at 7:55 PM vignesh C wrote:
> >
> > On Wed, Jul 13, 2022 at 1:13 PM Michael Paquier wrote:
> > >
> > > On Wed, Jul 13, 2022 at 12:22:06PM +0530, vignesh C wrote:
> > > > Most of the code is common between GetSubscriptionRelations and
> > > > GetSubscriptionNotReadyRelations. Added a parameter to
> > > > GetSubscriptionRelations which could provide the same functionality as
> > > > the existing GetSubscriptionRelations and
> > > > GetSubscriptionNotReadyRelations. Attached patch has the changes for
> > > > the same. Thoughts?
> > >
> > > Right. Using all_rels to mean that we'd filter relations that are not
> > > ready is a bit confusing, though. Perhaps this could use a bitmask as
> > > argument.
> >
> > The attached v2 patch has the modified version which includes the
> > changes to make the argument as bitmask.
> >
>
> By using a bitmask I think there is an implication that the flags can
> be combined...
>
> Perhaps it is not a problem today, but later you may want more flags. e.g.
> #define SUBSCRIPTION_REL_STATE_READY 0x02 /* READY relations */
>
> then the bitmask idea falls apart because IIUC you have no intentions
> to permit things like:
> (SUBSCRIPTION_REL_STATE_NOT_READY | SUBSCRIPTION_REL_STATE_READY)
>
> IMO using an enum might be a better choice for that parameter.
Changed it to enum so that it can be extended to support other
subscription relations like ready state subscription relations later
easily. The attached v3 patch has the changes for the same.
Regards,
Vignesh
From e0186ae685acb6334b711ea821b287be61d6cbd3 Mon Sep 17 00:00:00 2001
From: Vigneshwaran C
Date: Wed, 13 Jul 2022 11:54:59 +0530
Subject: [PATCH v3] Refactor to make use of a common function for
GetSubscriptionRelations and GetSubscriptionNotReadyRelations.
Most of the code is common between GetSubscriptionRelations and
GetSubscriptionNotReadyRelations. Added a parameter to GetSubscriptionRelations
which could provide the same functionality for GetSubscriptionRelations
and GetSubscriptionNotReadyRelations.
---
src/backend/catalog/pg_subscription.c | 71 -
src/backend/commands/subscriptioncmds.c | 5 +-
src/backend/replication/logical/tablesync.c | 3 +-
src/include/catalog/pg_subscription_rel.h | 19 +-
src/tools/pgindent/typedefs.list| 1 +
5 files changed, 34 insertions(+), 65 deletions(-)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 8856ce3b50..46eedbc062 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -525,65 +525,15 @@ HasSubscriptionRelations(Oid subid)
}
/*
- * Get all relations for subscription.
+ * Get the relations for subscription.
*
- * Returned list is palloc'ed in current memory context.
+ * If subrel_options is SUBSCRIPTION_REL_ALL, return all the relations for
+ * subscription. If SUBSCRIPTION_REL_NOT_READY, return all the relations for
+ * subscription that are not in a ready state. Returned list is palloc'ed in
+ * current memory context.
*/
List *
-GetSubscriptionRelations(Oid subid)
-{
- List *res = NIL;
- Relation rel;
- HeapTuple tup;
- ScanKeyData skey[1];
- SysScanDesc scan;
-
- rel = table_open(SubscriptionRelRelationId, AccessShareLock);
-
- ScanKeyInit(&skey[0],
-Anum_pg_subscription_rel_srsubid,
-BTEqualStrategyNumber, F_OIDEQ,
-ObjectIdGetDatum(subid));
-
- scan = systable_beginscan(rel, InvalidOid, false,
- NULL, 1, skey);
-
- while (HeapTupleIsValid(tup = systable_getnext(scan)))
- {
- Form_pg_subscription_rel subrel;
- SubscriptionRelState *relstate;
- Datum d;
- bool isnull;
-
- subrel = (Form_pg_subscription_rel) GETSTRUCT(tup);
-
- relstate = (SubscriptionRelState *) palloc(sizeof(SubscriptionRelState));
- relstate->relid = subrel->srrelid;
- relstate->state = subrel->srsubstate;
- d = SysCacheGetAttr(SUBSCRIPTIONRELMAP, tup,
- Anum_pg_subscription_rel_srsublsn, &isnull);
- if (isnull)
- relstate->lsn = InvalidXLogRecPtr;
- else
- relstate->lsn = DatumGetLSN(d);
-
- res = lappend(res, relstate);
- }
-
- /* Cleanup */
- systable_endscan(scan);
- table_close(rel, AccessShareLock);
-
- return res;
-}
-
-/*
- * Get all relations for subscription that are not in a ready state.
- *
- * Returned list is palloc'ed in current memory context.
- */
-List *
-GetSubscriptionNotReadyRelations(Oid subid)
+GetSubscriptionRelations(Oid subid, SubscriptionRelationState subrel_state)
{
List *res = NIL;
Relation rel;
@@ -599,10 +549,11 @@ GetSubscriptionNotReadyRelations(Oid subid)
BTEqualStrategyNumber, F_OIDEQ,
Obje
Re: Refactor to make use of a common function for GetSubscriptionRelations and GetSubscriptionNotReadyRelations.
On Wed, Jul 13, 2022 at 1:13 PM Michael Paquier wrote:
>
> On Wed, Jul 13, 2022 at 12:22:06PM +0530, vignesh C wrote:
> > Most of the code is common between GetSubscriptionRelations and
> > GetSubscriptionNotReadyRelations. Added a parameter to
> > GetSubscriptionRelations which could provide the same functionality as
> > the existing GetSubscriptionRelations and
> > GetSubscriptionNotReadyRelations. Attached patch has the changes for
> > the same. Thoughts?
>
> Right. Using all_rels to mean that we'd filter relations that are not
> ready is a bit confusing, though. Perhaps this could use a bitmask as
> argument.
The attached v2 patch has the modified version which includes the
changes to make the argument as bitmask.
Regards,
Vignesh
From 1f67e4b5ff2d304323f262e8acf74d48893ceb06 Mon Sep 17 00:00:00 2001
From: Vigneshwaran C
Date: Wed, 13 Jul 2022 11:54:59 +0530
Subject: [PATCH v2] Refactor to make use of a common function for
GetSubscriptionRelations and GetSubscriptionNotReadyRelations.
Most of the code is common between GetSubscriptionRelations and
GetSubscriptionNotReadyRelations. Added a parameter to GetSubscriptionRelations
which could provide the same functionality for GetSubscriptionRelations
and GetSubscriptionNotReadyRelations.
---
src/backend/catalog/pg_subscription.c | 71 -
src/backend/commands/subscriptioncmds.c | 5 +-
src/backend/replication/logical/tablesync.c | 3 +-
src/include/catalog/pg_subscription.h | 3 +
src/include/catalog/pg_subscription_rel.h | 3 +-
5 files changed, 20 insertions(+), 65 deletions(-)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 8856ce3b50..cd7dc00c79 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -525,65 +525,15 @@ HasSubscriptionRelations(Oid subid)
}
/*
- * Get all relations for subscription.
+ * Get the relations for subscription.
*
- * Returned list is palloc'ed in current memory context.
+ * If subrel_options is not set, return all the relations for subscription. If
+ * SUBSCRIPTION_REL_STATE_NOT_READY bit is set in subrel_options, return all
+ * the relations for subscription that are not in a ready state. Returned list
+ * is palloc'ed in current memory context.
*/
List *
-GetSubscriptionRelations(Oid subid)
-{
- List *res = NIL;
- Relation rel;
- HeapTuple tup;
- ScanKeyData skey[1];
- SysScanDesc scan;
-
- rel = table_open(SubscriptionRelRelationId, AccessShareLock);
-
- ScanKeyInit(&skey[0],
-Anum_pg_subscription_rel_srsubid,
-BTEqualStrategyNumber, F_OIDEQ,
-ObjectIdGetDatum(subid));
-
- scan = systable_beginscan(rel, InvalidOid, false,
- NULL, 1, skey);
-
- while (HeapTupleIsValid(tup = systable_getnext(scan)))
- {
- Form_pg_subscription_rel subrel;
- SubscriptionRelState *relstate;
- Datum d;
- bool isnull;
-
- subrel = (Form_pg_subscription_rel) GETSTRUCT(tup);
-
- relstate = (SubscriptionRelState *) palloc(sizeof(SubscriptionRelState));
- relstate->relid = subrel->srrelid;
- relstate->state = subrel->srsubstate;
- d = SysCacheGetAttr(SUBSCRIPTIONRELMAP, tup,
- Anum_pg_subscription_rel_srsublsn, &isnull);
- if (isnull)
- relstate->lsn = InvalidXLogRecPtr;
- else
- relstate->lsn = DatumGetLSN(d);
-
- res = lappend(res, relstate);
- }
-
- /* Cleanup */
- systable_endscan(scan);
- table_close(rel, AccessShareLock);
-
- return res;
-}
-
-/*
- * Get all relations for subscription that are not in a ready state.
- *
- * Returned list is palloc'ed in current memory context.
- */
-List *
-GetSubscriptionNotReadyRelations(Oid subid)
+GetSubscriptionRelations(Oid subid, bits32 subrel_state_options)
{
List *res = NIL;
Relation rel;
@@ -599,10 +549,11 @@ GetSubscriptionNotReadyRelations(Oid subid)
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(subid));
- ScanKeyInit(&skey[nkeys++],
-Anum_pg_subscription_rel_srsubstate,
-BTEqualStrategyNumber, F_CHARNE,
-CharGetDatum(SUBREL_STATE_READY));
+ if ((subrel_state_options & SUBSCRIPTION_REL_STATE_NOT_READY) != 0)
+ ScanKeyInit(&skey[nkeys++],
+ Anum_pg_subscription_rel_srsubstate,
+ BTEqualStrategyNumber, F_CHARNE,
+ CharGetDatum(SUBREL_STATE_READY));
scan = systable_beginscan(rel, InvalidOid, false,
NULL, nkeys, skey);
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index bdc1208724..d9fb235ce5 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -785,7 +785,7 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data,
pubrel_names = fetch_table_list(wrconn, sub->publications);
/* Get local table list. */
- subrel_states = GetSubscriptionRelations(sub->oid);
+ subrel_states = GetSubscriptionRelations(sub->oid, 0);
Refactor to make use of a common function for GetSubscriptionRelations and GetSubscriptionNotReadyRelations.
Hi,
Most of the code is common between GetSubscriptionRelations and
GetSubscriptionNotReadyRelations. Added a parameter to
GetSubscriptionRelations which could provide the same functionality as
the existing GetSubscriptionRelations and
GetSubscriptionNotReadyRelations. Attached patch has the changes for
the same. Thoughts?
Regards,
Vignesh
From 5fb15291abb489cb4469f43afa95106e421002c0 Mon Sep 17 00:00:00 2001
From: Vigneshwaran C
Date: Wed, 13 Jul 2022 11:54:59 +0530
Subject: [PATCH v1] Refactor to make use of a common function for
GetSubscriptionRelations and GetSubscriptionNotReadyRelations.
Most of the code is common between GetSubscriptionRelations and
GetSubscriptionNotReadyRelations. Added a parameter to GetSubscriptionRelations
which could provide the same functionality for GetSubscriptionRelations
and GetSubscriptionNotReadyRelations.
---
src/backend/catalog/pg_subscription.c | 68 +++--
src/backend/commands/subscriptioncmds.c | 4 +-
src/backend/replication/logical/tablesync.c | 2 +-
src/include/catalog/pg_subscription_rel.h | 3 +-
4 files changed, 13 insertions(+), 64 deletions(-)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 8856ce3b50..5757e2b338 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -525,65 +525,14 @@ HasSubscriptionRelations(Oid subid)
}
/*
- * Get all relations for subscription.
+ * Get the relations for subscription.
*
+ * If all_rels is true, return all the relations for subscription. If false,
+ * return all the relations for subscription that are not in a ready state.
* Returned list is palloc'ed in current memory context.
*/
List *
-GetSubscriptionRelations(Oid subid)
-{
- List *res = NIL;
- Relation rel;
- HeapTuple tup;
- ScanKeyData skey[1];
- SysScanDesc scan;
-
- rel = table_open(SubscriptionRelRelationId, AccessShareLock);
-
- ScanKeyInit(&skey[0],
-Anum_pg_subscription_rel_srsubid,
-BTEqualStrategyNumber, F_OIDEQ,
-ObjectIdGetDatum(subid));
-
- scan = systable_beginscan(rel, InvalidOid, false,
- NULL, 1, skey);
-
- while (HeapTupleIsValid(tup = systable_getnext(scan)))
- {
- Form_pg_subscription_rel subrel;
- SubscriptionRelState *relstate;
- Datum d;
- bool isnull;
-
- subrel = (Form_pg_subscription_rel) GETSTRUCT(tup);
-
- relstate = (SubscriptionRelState *) palloc(sizeof(SubscriptionRelState));
- relstate->relid = subrel->srrelid;
- relstate->state = subrel->srsubstate;
- d = SysCacheGetAttr(SUBSCRIPTIONRELMAP, tup,
- Anum_pg_subscription_rel_srsublsn, &isnull);
- if (isnull)
- relstate->lsn = InvalidXLogRecPtr;
- else
- relstate->lsn = DatumGetLSN(d);
-
- res = lappend(res, relstate);
- }
-
- /* Cleanup */
- systable_endscan(scan);
- table_close(rel, AccessShareLock);
-
- return res;
-}
-
-/*
- * Get all relations for subscription that are not in a ready state.
- *
- * Returned list is palloc'ed in current memory context.
- */
-List *
-GetSubscriptionNotReadyRelations(Oid subid)
+GetSubscriptionRelations(Oid subid, bool all_rels)
{
List *res = NIL;
Relation rel;
@@ -599,10 +548,11 @@ GetSubscriptionNotReadyRelations(Oid subid)
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(subid));
- ScanKeyInit(&skey[nkeys++],
-Anum_pg_subscription_rel_srsubstate,
-BTEqualStrategyNumber, F_CHARNE,
-CharGetDatum(SUBREL_STATE_READY));
+ if (!all_rels)
+ ScanKeyInit(&skey[nkeys++],
+ Anum_pg_subscription_rel_srsubstate,
+ BTEqualStrategyNumber, F_CHARNE,
+ CharGetDatum(SUBREL_STATE_READY));
scan = systable_beginscan(rel, InvalidOid, false,
NULL, nkeys, skey);
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index bdc1208724..f9563304af 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -785,7 +785,7 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data,
pubrel_names = fetch_table_list(wrconn, sub->publications);
/* Get local table list. */
- subrel_states = GetSubscriptionRelations(sub->oid);
+ subrel_states = GetSubscriptionRelations(sub->oid, true);
/*
* Build qsorted array of local table oids for faster lookup. This can
@@ -1457,7 +1457,7 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
* the apply and tablesync workers and they can't restart because of
* exclusive lock on the subscription.
*/
- rstates = GetSubscriptionNotReadyRelations(subid);
+ rstates = GetSubscriptionRelations(subid, false);
foreach(lc, rstates)
{
SubscriptionRelState *rstate = (SubscriptionRelState *) lfirst(lc);
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 670c6fcada..4b5b388f6c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -1479,7 +1479,7 @@ FetchTableStates(bool *started_
Re: Handle infinite recursion in logical replication setup
On Mon, Jul 11, 2022 at 5:03 PM Amit Kapila wrote:
>
> On Sat, Jul 9, 2022 at 8:11 PM vignesh C wrote:
> >
>
> Thanks, a few more comments on v30_0002*
> 1.
> +/*
> + * Represents whether copy_data parameter is specified with off, on or force.
>
> A comma is required after on.
Modified
> 2.
> qsort(subrel_local_oids, list_length(subrel_states),
> sizeof(Oid), oid_cmp);
>
> + check_pub_table_subscribed(wrconn, sub->publications, copy_data,
> +sub->origin, subrel_local_oids,
> +list_length(subrel_states));
>
> We can avoid using list_length by using an additional variable in this case.
Modified
> 3.
> errmsg("table: \"%s.%s\" might have replicated data in the publisher",
> +nspname, relname),
>
> Why ':' is used after the table in the above message? I don't see such
> a convention at other places in the code. Also, having might in the
> error messages makes it less clear, so, can we slightly change the
> message as in the attached patch?
Modified as suggested
> 4. I have made some additional changes in the comments, kindly check
> the attached and merge those if you are okay.
I have merged the changes
> 5.
> +$node_C->safe_psql(
> + 'postgres', "
> +DELETE FROM tab_full");
> +$node_B->safe_psql(
> + 'postgres', "
> +DELETE FROM tab_full where a = 13");
>
> Don't we need to wait for these operations to replicate?
Modified to include wait
Thanks for the comments, the v31 patch attached at [1] has the changes
for the same.
[1] -
https://www.postgresql.org/message-id/CALDaNm2-D860yULtcmZAzDbdiof-Dg6Y_YaY4owbO6Rj%3DXEHMw%40mail.gmail.com
Regards,
Vignesh
Re: explain analyze rows=%.0f
On Thu, Jun 23, 2022 at 2:25 AM Ibrar Ahmed wrote:
>
>
>
> On Thu, Jun 23, 2022 at 1:04 AM David G. Johnston
> wrote:
>>
>> On Wed, Jun 22, 2022 at 12:11 PM Ibrar Ahmed wrote:
>>>
>>> On Thu, Jun 23, 2022 at 12:01 AM Tom Lane wrote:
Robert Haas writes:
> On Jun 2, 2009, at 9:41 AM, Simon Riggs wrote:
>> You're right that the number of significant digits already exceeds the
>> true accuracy of the computation. I think what Robert wants to see is
>> the exact value used in the calc, so the estimates can be checked more
>> thoroughly than is currently possible.
> Bingo.
Uh, the planner's estimate *is* an integer. What was under discussion
(I thought) was showing some fractional digits in the case where EXPLAIN
ANALYZE is outputting a measured row count that is an average over
multiple loops, and therefore isn't necessarily an integer. In that
case the measured value can be considered arbitrarily precise --- though
I think in practice one or two fractional digits would be plenty.
regards, tom lane
>>> Hi,
>>> I was looking at the TODO list and found that the issue requires
>>> a quick fix. Attached is a patch which shows output like this.
>>
>>
>> Quick code review:
>>
>> + "actual rows=%.0f loops=%.0f": " rows=%.2f loops=%.0f",
>>
>> The leading space before the else block "rows" does not belong.
>>
>> There should be a space after the colon.
>>
> Thanks, David for your quick response. I have updated the patch.
>
>>
>> The word "actual" that you are dropping in the else block seems like it
>> should belong - it is a header for the entire section not just a modifier
>> for the word "rows". This is evidenced by the timing block verbiage where
>> rows is standalone and the word actual comes before time. In short, only
>> the format specifier should change under the current scheme. Both sections.
>>
>> - WRITE_FLOAT_FIELD(rows, "%.0f");
>> + WRITE_FLOAT_FIELD(rows, "%.2f");
>>
>> This one looks suspicious, though I haven't dug into the code to see exactly
>> what all is being touched. That it doesn't have an nloops condition like
>> everything else stands out.
>>
> I was also thinking about that, but I don't see any harm when we ultimately
> truncating that decimal
> at a latter stage of code in case of loop = 1.
Thanks for the patch.
1) There are some existing regression tests that are failing, you
should update the expect files accordingly for the same:
--- /home/vignesh/postgres/src/test/regress/expected/select_parallel.out
2022-05-18 20:51:46.874818044 +0530
+++ /home/vignesh/postgres/src/test/regress/results/select_parallel.out
2022-07-07 15:27:34.450440922 +0530
@@ -545,17 +545,17 @@
explain (analyze, timing off, summary off, costs off)
select count(*) from tenk1, tenk2 where tenk1.hundred > 1
and tenk2.thousand=0;
-QUERY PLAN
---
+ QUERY PLAN
+-
Aggregate (actual rows=1 loops=1)
-> Nested Loop (actual rows=98000 loops=1)
-> Seq Scan on tenk2 (actual rows=10 loops=1)
Filter: (thousand = 0)
Rows Removed by Filter: 9990
- -> Gather (actual rows=9800 loops=10)
+ -> Gather (actual rows=9800.00 loops=10)
Workers Planned: 4
Workers Launched: 4
- -> Parallel Seq Scan on tenk1 (actual rows=1960 loops=50)
+ -> Parallel Seq Scan on tenk1 (actual rows=1960.00 loops=50)
Filter: (hundred > 1)
test select_parallel ... FAILED 744 ms
partition_prune ... FAILED 861 ms
explain ... FAILED 134 ms
memoize ... FAILED 250 ms
2) This change is not required as part of this patch:
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -122,7 +122,7 @@ boolbsysscan = false;
* lookups as fast as possible.
*/
static FullTransactionId XactTopFullTransactionId = {InvalidTransactionId};
-static int nParallelCurrentXids = 0;
+static int nParallelCurrentXids = 0;
static TransactionId *ParallelCurrentXids;
Regards,
Vignesh
Re: Handle infinite recursion in logical replication setup
On Tue, Jul 5, 2022 at 6:14 AM Peter Smith wrote: > > I checked again the v26* patch set. > > I had no more comments for v26-0001, v26-0002, or v26-0004, but below > are some comments for v26-0003. > > > v26-0003 > > > 3.1 src/backend/catalog/pg_subscription.c > > 3.1.a > +/* > + * Get all relations for subscription that are in a ready state. > + * > + * Returned list is palloc'ed in current memory context. > + */ > +List * > +GetSubscriptionReadyRelations(Oid subid) > > "subscription" -> "the subscription" > > Also, It might be better to say in the function header what kind of > structures are in the returned List. E.g. Firstly, I'd assumed it was > the same return type as the other function > GetSubscriptionReadyRelations, but it isn’t. This function has been removed and GetSubscriptionRelations is used with slight changes. > 3.1.b > I think there might be a case to be made for *combining* those > SubscriptionRelState and SubscripotionRel structs into a single common > struct. Then all those GetSubscriptionNotReadyRelations and > GetSubscriptionNotReadyRelations (also GetSubscriptionRelations?) can > be merged to be just one common function that takes a parameter to say > do you want to return the relation List of ALL / READY / NOT_READY? > What do you think? I have used a common function that can get all relations, ready relations and not ready relations instead of the 3 functions. > == > > 3.2 src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed > > +/* > + * Check and throw an error if the publisher has subscribed to the same table > + * from some other publisher. This check is required only if copydata is ON > and > + * the origin is local. > + */ > +static void > +check_pub_table_subscribed(WalReceiverConn *wrconn, List *publications, > +CopyData copydata, char *origin, Oid subid) > > The function comment probably should say something about why the new > READY logic was added in v26. Added a comment for the same. > ~~~ > > 3.3 > > 3.3.a > + /* > + * The subid will be valid only for ALTER SUBSCRIPTION ... REFRESH > + * PUBLICATION. Get the ready relations for the subscription only in case > + * of ALTER SUBSCRIPTION case as there will be no relations in ready state > + * while the subscription is created. > + */ > + if (subid != InvalidOid) > + subreadyrels = GetSubscriptionReadyRelations(subid); > > The word "case" is 2x in the same sentence. I also paraphrased my > understanding of this comment below. Maybe it is simpler? > > SUGGESTION > Get the ready relations for the subscription. The subid will be valid > only for ALTER SUBSCRIPTION ... REFRESH because there will be no > relations in ready state while the subscription is created. Modified > 3.3.b > + if (subid != InvalidOid) > + subreadyrels = GetSubscriptionReadyRelations(subid); > > SUGGESTION > if (OidIsValid(subid)) Modified > == > > 3.4 src/test/subscription/t/032_localonly.pl > > Now all the test cases are re-using the same data (e.g. 11,12,13) and > you are deleting the data between the tests. I guess it is OK, but IMO > the tests are easier to read when each test part was using unique data > (e.g. 11,12,13, then 12,22,32, then 13,23,33 etc) Modified Thanks for the comments, the v29 patch attached at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm1T5utq60qVx%3DRN60rHcg7wt2psM7PpCQ2fDiB-R8oLGg%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Mon, Jul 4, 2022 at 7:44 PM Alvaro Herrera wrote: > > > From d8f8844f877806527b6f3f45320b6ba55a8e3154 Mon Sep 17 00:00:00 2001 > > From: Vigneshwaran C > > Date: Thu, 26 May 2022 19:29:33 +0530 > > Subject: [PATCH v27 1/4] Add a missing test to verify only-local parameter > > in > > test_decoding plugin. > > > > Add a missing test to verify only-local parameter in test_decoding plugin. > > I don't get it. replorigin.sql already has some lines to test > local-only. What is your patch adding that is new? Maybe instead of > adding some more lines at the end of the script, you should add lines > where this stuff is already being tested. But that assumes that there > is something new that is being tested; if so what is it? The test is to check that remote origin data (i.e. replication origin being set) will be filtered when only-local parameter is set. I felt that this scenario is not covered, unless I'm missing something. I have moved the test as suggested. I have changed this in the v28 patch attached at [1]. [1] - https://www.postgresql.org/message-id/CALDaNm3MNK2hMYroTiHGS9HkSxiA-az1QC1mpa0YwDZ8nGmmZg%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Mon, Jul 4, 2022 at 9:23 AM Amit Kapila wrote:
>
> On Mon, Jul 4, 2022 at 3:59 AM Peter Smith wrote:
> >
> > On Mon, Jul 4, 2022 at 12:59 AM vignesh C wrote:
> > ...
> > > > 2.
> > > > /* ALTER SUBSCRIPTION SET ( */
> > > > else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
> > > > TailMatches("SET", "("))
> > > > - COMPLETE_WITH("binary", "slot_name", "streaming",
> > > > "synchronous_commit", "disable_on_error");
> > > > + COMPLETE_WITH("binary", "origin", "slot_name", "streaming",
> > > > "synchronous_commit", "disable_on_error");
> > > > /* ALTER SUBSCRIPTION SKIP ( */
> > > > else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
> > > > TailMatches("SKIP", "("))
> > > > COMPLETE_WITH("lsn");
> > > > @@ -3152,7 +3152,7 @@ psql_completion(const char *text, int start, int
> > > > end)
> > > > /* Complete "CREATE SUBSCRIPTION ... WITH ( " */
> > > > else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH",
> > > > "("))
> > > > COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
> > > > - "enabled", "slot_name", "streaming",
> > > > + "enabled", "origin", "slot_name", "streaming",
> > > > "synchronous_commit", "two_phase", "disable_on_error");
> > > >
> > > > Why do you choose to add a new option in-between other parameters
> > > > instead of at the end which we normally do? The one possible reason I
> > > > can think of is that all the parameters at the end are boolean so you
> > > > want to add this before those but then why before slot_name, and again
> > > > I don't see such a rule being followed for other parameters.
> > >
> > > I was not sure if it should be maintained in alphabetical order,
> > > anyway since the last option "disable_on_error" is at the end, I have
> > > changed it to the end.
> > >
> >
> > Although it seems it is not a hard rule, mostly the COMPLETE_WITH are
> > coded using alphabetical order. Anyway, I think that was a clear
> > intention here too since 13 of 14 parameters were already in
> > alphabetical order; it is actually only that "disable_on_error"
> > parameter that was misplaced; not the new "origin" parameter.
> >
>
> Agreed, but let's not change disable_on_error as part of this patch.
I have changed this in the v28 patch attached at [1].
[1] -
https://www.postgresql.org/message-id/CALDaNm3MNK2hMYroTiHGS9HkSxiA-az1QC1mpa0YwDZ8nGmmZg%40mail.gmail.com
Regards,
Vignesh
Re: Handle infinite recursion in logical replication setup
On Mon, Jul 4, 2022 at 1:46 PM shiy.f...@fujitsu.com
wrote:
>
> On Sun, Jul 3, 2022 11:00 PM vignesh C wrote:
> >
> > Thanks for the comments, the attached v27 patch has the changes for the
> > same.
> >
>
> Thanks for updating the patch.
>
> A comment on 0003 patch:
> + /*
> +* No need to throw an error for the tables that are in ready
> state,
> +* as the walsender will send the changes from WAL in case of
> tables
> +* in ready state.
> +*/
> + if (isreadytable)
> + continue;
> +
> ...
> + ereport(ERROR,
> +
> errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("table: \"%s.%s\" might have
> replicated data in the publisher",
> + nspname, relname),
> + errdetail("CREATE/ALTER SUBSCRIPTION with
> origin = local and copy_data = on is not allowed when the publisher might
> have replicated data."),
> + errhint("Use CREATE/ALTER SUBSCRIPTION with
> copy_data = off/force."));
> +
> + ExecClearTuple(slot);
>
> I think we should call ExecClearTuple() before getting next tuple, so it
> should
> be called if the table is in ready state. How about modifying it to:
> if (!isreadytable)
> ereport(ERROR,
>
> errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> errmsg("table: \"%s.%s\" might have
> replicated data in the publisher",
>nspname, relname),
> errdetail("CREATE/ALTER SUBSCRIPTION
> with origin = local and copy_data = on is not allowed when the publisher
> might have replicated data."),
> errhint("Use CREATE/ALTER
> SUBSCRIPTION with copy_data = off/force."));
>
> ExecClearTuple(slot);
Thanks for the comment, I have modified this in the v28 patch attached at [1].
[1] -
https://www.postgresql.org/message-id/CALDaNm3MNK2hMYroTiHGS9HkSxiA-az1QC1mpa0YwDZ8nGmmZg%40mail.gmail.com
Regards,
Vignesh
Re: Support logical replication of DDLs
On Sat, Jul 2, 2022 at 8:51 AM Amit Kapila wrote: > > On Fri, Jul 1, 2022 at 10:22 PM vignesh C wrote: > > > > On Wed, Jun 29, 2022 at 3:25 PM houzj.f...@fujitsu.com > > wrote: > > > > > > > Thanks for the updated patch. > > Few comments on 0002 patch: > > 1) When we create a subscription for a publication with the existing > > default PUBLISH parameter having default value as > > 'insert,update,delete,truncate', we do an initial table sync to get > > the initial table data from the publisher to the subscriber. But in > > case of a publication created with 'ddl', the subscription expects the > > existing initial tables present in the publisher to be created > > beforehand in the subscriber. Should this be the default behavior? > > Should we do a ddl dump for all the tables and restore the ddl to the > > subscription while creating the subscription? Or is this planned as an > > option for the later version. > > > > The idea is to develop initial sync (for ddl replication) as a > separate patch. But both need to be integrated at some point. Yes, that approach makes sense. > > > > 3) SYNTAX Support: > > Currently creation of "FOR TABLE" publication with ddl is supported. > > Should we allow support of ddl for "FOR TABLE" publication. > > > > The above comment is unclear to me. It seems to me in the first > sentence, you are saying that the "FOR TABLE" syntax is supported and > in the second sentence, you are asking to allow support of it? I think > at this stage, the focus is to build the core part of the feature > (allow ddl replication and deparsing support), and then we can discuss > more on Syntax. Having said that, it will be good if we can support > table-level DDL replication as well in the patch as you seem to be > suggesting. I initially thought that supporting "FOR TABLE" publication for ddl might not be useful as currently the create subscription fails with table does not exist error. Now that the initial sync for ddl replication will also be implemented as mentioned in [1], this issue will be handled. I agree with supporting table-level DDL replication. [1] - https://www.postgresql.org/message-id/CAA4eK1%2B5zJAT_RYOAEOq8M33s196kR5sDyLQLUXd8Rnqr%2BiB0Q%40mail.gmail.com Regards, Vignesh
Re: Support logical replication of DDLs
On Wed, Jun 29, 2022 at 3:25 PM houzj.f...@fujitsu.com
wrote:
>
> On Wednesday, June 29, 2022 11:07 AM Amit Kapila
> wrote:
> >
> > On Tue, Jun 28, 2022 at 5:43 PM Amit Kapila
> > wrote:
> > >
> > > 5.
> > > +static ObjTree *
> > > +deparse_CreateStmt(Oid objectId, Node *parsetree)
> > > {
> > > ...
> > > + tmp = new_objtree_VA("TABLESPACE %{tablespace}I", 0); if
> > > + (node->tablespacename) append_string_object(tmp, "tablespace",
> > > + node->tablespacename); else { append_null_object(tmp, "tablespace");
> > > + append_bool_object(tmp, "present", false); }
> > > + append_object_object(createStmt, "tablespace", tmp);
> > > ...
> > > }
> > >
> > > Why do we need to append the objects (tablespace, with clause, etc.)
> > > when they are not present in the actual CREATE TABLE statement? The
> > > reason to ask this is that this makes the string that we want to send
> > > downstream much longer than the actual statement given by the user on
> > > the publisher.
> > >
> >
> > After thinking some more on this, it seems the user may want to optionally
> > change some of these attributes, for example, on the subscriber, it may
> > want to
> > associate the table with a different tablespace. I think to address that we
> > can
> > append these additional attributes optionally, say via an additional
> > parameter
> > (append_all_options/append_all_attributes or something like that) in exposed
> > APIs like deparse_utility_command().
>
> I agree and will research this part.
>
> And here is the new version patch set.
> Most of changes are in the deparser which include:
>
> support CREATE PARTITIONED TABLE
> support ALTER ATTACH/DETACH PARTITION
> support CREATE/ALTER TABLE with ACCESS METHOD
> support CREATE TABLE OF
> support CREATE/ALTER TABLE with GENERATED COLUMN
> support CREATE/ALTER TABLE with DENTITY COLUMN
> support CREATE/ALTER TABLE with COMPRESSION METHOD
> support ALTER COLUMN numofcol SET STATISTICS (mentioned by sawada-san [1])
> support ALTER SCHEMA
> support CRAETE/DROP INDEX
>
> Note that, for ATTACH/DETACH PARTITION, I haven't added extra logic on
> subscriber to handle the case where the table on publisher is a PARTITIONED
> TABLE while the target table on subscriber side is NORMAL table. We will
> research this more and improve this later.
>
> Besides, the new version event trigger won't WAL log the DDL whose target
> table is a temporary table so that the problem reported by Vignesh[2] is
> fixed.
>
> About the recent comment from Amit[3] and Vignesh[4], I will investigate the
> comments and address them in next version.
>
> [1]
> https://www.postgresql.org/message-id/CAD21AoBVCoPPRKvU_5-%3DwEXsa92GsNJFJOcYyXzvoSEJCx5dKw%40mail.gmail.com
> [2]
> https://www.postgresql.org/message-id/CALDaNm33W35pcBE3zOpJhwnYBdBoZDpKxssemAN21NwVhJuong%40mail.gmail.com
> [3]
> https://www.postgresql.org/message-id/CAA4eK1K88SMoBq%3DDRA4XU-F3FG6qyzCjGMMKsPpcRBPRcrELrw%40mail.gmail.com
> [4]
> https://www.postgresql.org/message-id/CALDaNm3rEA_zmnDMOCT7NqK4aAffhAgooLf8rXjUN%3DYwA8ASFw%40mail.gmail.com
Thanks for the updated patch.
Few comments on 0002 patch:
1) When we create a subscription for a publication with the existing
default PUBLISH parameter having default value as
'insert,update,delete,truncate', we do an initial table sync to get
the initial table data from the publisher to the subscriber. But in
case of a publication created with 'ddl', the subscription expects the
existing initial tables present in the publisher to be created
beforehand in the subscriber. Should this be the default behavior?
Should we do a ddl dump for all the tables and restore the ddl to the
subscription while creating the subscription? Or is this planned as an
option for the later version. If we could do this as part of ddl
logical replication, it will help in reducing the steps further for
logical replication setup. If this will not be supported in this patch
series, probably we could document the behavior and add comments for
this at an appropriate place.
2) When a publication is created with ddl enabled, event triggers
will be created implicitly. Currently these event triggers are also
getting dumped. These should not be dumped. Currently while the
publication is restored, the event trigger will be created and also
the explicit event triggers which were dumped will also get created
resulting in multiple event triggers. The event trigger should not be
dumped.
@@ -4016,6 +4026,15 @@ dumpPublication(Archive *fout, const
PublicationInfo *pubinfo)
first = false;
}
+ if (pubinfo->pubddl)
+ {
+ if (!first)
+ appendPQExpBufferStr(query, ", ");
+
+ appendPQExpBufferStr(query, "ddl");
+ first = false;
+ }
+
3) SYNTAX Support:
Currently creation of "FOR TABLE" publication with ddl is supported.
Should we allow support of ddl for "FOR TABLE" publication. The
creation of the subscription will fail saying the table does no
Re: Support logical replication of DDLs
On Tue, Jun 21, 2022 at 5:49 PM houzj.f...@fujitsu.com wrote: > > On Monday, June 20, 2022 11:32 AM houzj.f...@fujitsu.com > wrote: > > > > On Saturday, June 18, 2022 3:38 AM Zheng Li wrote: > > > On Wed, Jun 15, 2022 at 12:00 AM Amit Kapila > > > wrote: > > > > > > > > On Wed, Jun 15, 2022 at 5:44 AM Zheng Li wrote: > > > > > > > > > > > > > > > While I agree that the deparser is needed to handle the potential > > > > > syntax differences between the pub/sub, I think it's only relevant > > > > > for the use cases where only a subset of tables in the database are > > > > > replicated. For other use cases where all tables, functions and > > > > > other objects need to be replicated, (for example, creating a > > > > > logical replica for major version upgrade) there won't be any syntax > > > > > difference to handle and the schemas are supposed to match exactly > > > > > between the pub/sub. In other words the user seeks to create an > > > > > identical replica of the source database and the DDLs should be > > > > > replicated as is in this case. > > > > > > > > > > > > > I think even for database-level replication we can't assume that > > > > source and target will always have the same data in which case "Create > > > > Table As ..", "Alter Table .. " kind of statements can't be replicated > > > > as it is because that can lead to different results. > > > "Create Table As .." is already handled by setting the skipData flag of > > > the > > > statement parsetreee before replay: > > > > > > /* > > > * Force skipping data population to avoid data inconsistency. > > > * Data should be replicated from the publisher instead. > > > */ > > > castmt->into->skipData = true; > > > > > > "Alter Table .. " that rewrites with volatile expressions can also be > > > handled > > > without any syntax change, by enabling the table rewrite replication and > > > converting the rewrite inserts to updates. ZJ's patch introduced this > > > solution. > > > I've also adopted this approach in my latest patch > > > 0012-Support-replication-of-ALTER-TABLE-commands-that-rew.patch > > > > > > > The other point > > > > is tomorrow we can extend the database level option/syntax to exclude > > > > a few objects (something like [1]) as well in which case we again need > > > > to filter at the publisher level > > > > > > I think for such cases it's not full database replication and we could > > > treat it as > > > table level DDL replication, i.e. use the the deparser format. > > > > Hi, > > > > Here are some points in my mind about the two approaches discussed here. > > > > 1) search_patch vs schema qualify > > > > Again, I still think it will bring more flexibility and security by schema > > qualify the > > objects in DDL command as mentioned before[1]. > > > > Besides, a schema qualified DDL is also more appropriate for other use > > cases(e.g. a table-level replication). As it's possible the schema is > > different > > between pub/sub and it's easy to cause unexpected and undetectable failure > > if > > we just log the search_path. > > > > It makes more sense to me to have the same style WAL log(schema qualified) > > for > > both database level or table level replication as it will bring more > > flexibility. > > > > > > > "Create Table As .." is already handled by setting the skipData flag of > > > the > > > statement parsetreee before replay: > > > > 2) About the handling of CREATE TABLE AS: > > > > I think it's not a appropriate approach to set the skipdata flag on > > subscriber > > as it cannot handle EXECUTE command in CTAS. > > > > CREATE TABLE q5_prep_results AS EXECUTE q5(200, 'DT'); > > > > The Prepared statement is a temporary object which we don't replicate. So if > > you directly execute the original SQL on subscriber, even if you set > > skipdata > > it will fail. > > > > I think it difficult to make this work as you need handle the create/drop of > > this prepared statement. And even if we extended subscriber's code to make > > it > > work, it doesn't seems like a standard and elegant approach. > > > > > > > "Alter Table .. " that rewrites with volatile expressions can also be > > > handled > > > without any syntax change, by enabling the table rewrite replication and > > > converting the rewrite inserts to updates. ZJ's patch introduced this > > > solution. > > > > 3) About the handling of ALTER TABLE rewrite. > > > > The approach I proposed before is based on the event trigger + deparser > > approach. We were able to improve that approach as we don't need to > > replicate > > the rewrite in many cases. For example: we don't need to replicate rewrite > > dml > > if there is no volatile/mutable function. We should check and filter these > > case > > at publisher (e.g. via deparser) instead of checking that at subscriber. > > > > Besides, as discussed, we need to give warning or error for the cases when > > DDL > > contains volatile function which would be executed[2]. We should check this > > at >
Re: Handle infinite recursion in logical replication setup
On Thu, Jun 23, 2022 at 7:05 AM shiy.f...@fujitsu.com
wrote:
>
> On Mon, Jun 20, 2022 7:55 PM vignesh C wrote:
> >
> > Thanks for the comment, the v22 patch attached has the changes for the
> > same.
>
> Thanks for updating the patch, here are some comments on 0003 patch.
>
> 1. 032_origin.pl
> +###
> +# Join 3rd node (node_C) to the existing 2 nodes(node_A & node_B)
> bidirectional
> +# replication setup when the existing nodes (node_A & node_B) and the new
> node
> +# (node_C) does not have any data.
> +###
>
> "does not have any data" should be "do not have any data" I think.
I felt the existing is ok, I did not find it gramatically wrong. I did
not make any changes for this.
> 2.
> The comment for 032_origin.pl:
>
> # Test the CREATE SUBSCRIPTION 'origin' parameter.
>
> After applying this patch, this file tests no only 'origin' parameter, but
> also
> "copy_data" parameter, so should we modify this comment?
Modified
> Besides, should we change the file name in this patch? It looks more like test
> cases for bidirectional logical replication.
I felt let's keep it as origin, as most of the tests are based on
'origin' parameter, also later features will be able to filter a
particular origin, those tests can easily fit in the same file.
> 3. subscriptioncmds.c
> /* Set default values for the boolean supported options. */
> ...
> if (IsSet(supported_opts, SUBOPT_CREATE_SLOT))
> opts->create_slot = true;
> if (IsSet(supported_opts, SUBOPT_COPY_DATA))
> - opts->copy_data = true;
> + opts->copy_data = COPY_DATA_ON;
> if (IsSet(supported_opts, SUBOPT_REFRESH))
> opts->refresh = true;
> if (IsSet(supported_opts, SUBOPT_BINARY))
>
> "copy_data" option is not Boolean now, which is inconsistent with the comment.
> So maybe we can change the comment here? ("the boolean supported options" ->
> "the supported options")
Modified
Thanks for the comments, the v23 patch attached at [1] has the changes
for the same.
[1] -
https://www.postgresql.org/message-id/CALDaNm1-ZrG%3DhaAoiB2yFKYc%2Bckcd1NLaU8QB3SWs32wPsph4w%40mail.gmail.com
Regards,
Vignesh
Re: Support logical replication of DDLs
On Tue, Jun 21, 2022 at 5:49 PM houzj.f...@fujitsu.com wrote: > > On Monday, June 20, 2022 11:32 AM houzj.f...@fujitsu.com > wrote: > > > > On Saturday, June 18, 2022 3:38 AM Zheng Li wrote: > > > On Wed, Jun 15, 2022 at 12:00 AM Amit Kapila > > > wrote: > > > > > > > > On Wed, Jun 15, 2022 at 5:44 AM Zheng Li wrote: > > > > > > > > > > > > > > > While I agree that the deparser is needed to handle the potential > > > > > syntax differences between the pub/sub, I think it's only relevant > > > > > for the use cases where only a subset of tables in the database are > > > > > replicated. For other use cases where all tables, functions and > > > > > other objects need to be replicated, (for example, creating a > > > > > logical replica for major version upgrade) there won't be any syntax > > > > > difference to handle and the schemas are supposed to match exactly > > > > > between the pub/sub. In other words the user seeks to create an > > > > > identical replica of the source database and the DDLs should be > > > > > replicated as is in this case. > > > > > > > > > > > > > I think even for database-level replication we can't assume that > > > > source and target will always have the same data in which case "Create > > > > Table As ..", "Alter Table .. " kind of statements can't be replicated > > > > as it is because that can lead to different results. > > > "Create Table As .." is already handled by setting the skipData flag of > > > the > > > statement parsetreee before replay: > > > > > > /* > > > * Force skipping data population to avoid data inconsistency. > > > * Data should be replicated from the publisher instead. > > > */ > > > castmt->into->skipData = true; > > > > > > "Alter Table .. " that rewrites with volatile expressions can also be > > > handled > > > without any syntax change, by enabling the table rewrite replication and > > > converting the rewrite inserts to updates. ZJ's patch introduced this > > > solution. > > > I've also adopted this approach in my latest patch > > > 0012-Support-replication-of-ALTER-TABLE-commands-that-rew.patch > > > > > > > The other point > > > > is tomorrow we can extend the database level option/syntax to exclude > > > > a few objects (something like [1]) as well in which case we again need > > > > to filter at the publisher level > > > > > > I think for such cases it's not full database replication and we could > > > treat it as > > > table level DDL replication, i.e. use the the deparser format. > > > > Hi, > > > > Here are some points in my mind about the two approaches discussed here. > > > > 1) search_patch vs schema qualify > > > > Again, I still think it will bring more flexibility and security by schema > > qualify the > > objects in DDL command as mentioned before[1]. > > > > Besides, a schema qualified DDL is also more appropriate for other use > > cases(e.g. a table-level replication). As it's possible the schema is > > different > > between pub/sub and it's easy to cause unexpected and undetectable failure > > if > > we just log the search_path. > > > > It makes more sense to me to have the same style WAL log(schema qualified) > > for > > both database level or table level replication as it will bring more > > flexibility. > > > > > > > "Create Table As .." is already handled by setting the skipData flag of > > > the > > > statement parsetreee before replay: > > > > 2) About the handling of CREATE TABLE AS: > > > > I think it's not a appropriate approach to set the skipdata flag on > > subscriber > > as it cannot handle EXECUTE command in CTAS. > > > > CREATE TABLE q5_prep_results AS EXECUTE q5(200, 'DT'); > > > > The Prepared statement is a temporary object which we don't replicate. So if > > you directly execute the original SQL on subscriber, even if you set > > skipdata > > it will fail. > > > > I think it difficult to make this work as you need handle the create/drop of > > this prepared statement. And even if we extended subscriber's code to make > > it > > work, it doesn't seems like a standard and elegant approach. > > > > > > > "Alter Table .. " that rewrites with volatile expressions can also be > > > handled > > > without any syntax change, by enabling the table rewrite replication and > > > converting the rewrite inserts to updates. ZJ's patch introduced this > > > solution. > > > > 3) About the handling of ALTER TABLE rewrite. > > > > The approach I proposed before is based on the event trigger + deparser > > approach. We were able to improve that approach as we don't need to > > replicate > > the rewrite in many cases. For example: we don't need to replicate rewrite > > dml > > if there is no volatile/mutable function. We should check and filter these > > case > > at publisher (e.g. via deparser) instead of checking that at subscriber. > > > > Besides, as discussed, we need to give warning or error for the cases when > > DDL > > contains volatile function which would be executed[2]. We should check this > > at >
Re: Handle infinite recursion in logical replication setup
On Mon, Jun 20, 2022 at 3:16 PM vignesh C wrote: > > On Mon, Jun 20, 2022 at 2:37 PM Dilip Kumar wrote: > > > > On Thu, Jun 16, 2022 at 3:48 PM vignesh C wrote: > > > > > > On Wed, Jun 15, 2022 at 12:09 PM Peter Smith > > > wrote: > > > > > > > > > Thanks for the comments, the attached v21 patch has the changes for the > > > same. > > > > I have done some basic review of v21 and I have a few comments, > > > > 1. > > +/* > > + * The subscription will request the publisher to only send changes that > > + * originated locally. > > + */ > > +#define LOGICALREP_ORIGIN_LOCAL "local" > > + > > +/* > > + * The subscription will request the publisher to send any changes > > regardless > > + * of their origin > > + */ > > +#define LOGICALREP_ORIGIN_ANY "any" > > > > Are we planning to extend this to more options or are we planning to > > support the actual origin name here? If not then why isn't it just > > bool? I think the comments and the patch commit message should > > explain the details behind it if it has been already discussed and > > concluded. > > Currently we only support local and any. But this was designed to > accept string instead of boolean type, so that it can be extended > later to support filtering of origin names specified by the user in > the later versions. The same was also discussed in pg unconference as > mentioned in [1]. I will add it to the commit message and a comment > for the same in the next version. Thanks for the comment, the v22 patch attached at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm1h-9UNi_Jo_K%2BPK34tXBmV7fhj5C_nB8YzGA9rmUwHEA%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Mon, Jun 20, 2022 at 2:37 PM Dilip Kumar wrote:
>
> On Thu, Jun 16, 2022 at 3:48 PM vignesh C wrote:
> >
> > On Wed, Jun 15, 2022 at 12:09 PM Peter Smith wrote:
> > >
>
> > Thanks for the comments, the attached v21 patch has the changes for the
> > same.
>
> I have done some basic review of v21 and I have a few comments,
>
> 1.
> +/*
> + * The subscription will request the publisher to only send changes that
> + * originated locally.
> + */
> +#define LOGICALREP_ORIGIN_LOCAL "local"
> +
> +/*
> + * The subscription will request the publisher to send any changes regardless
> + * of their origin
> + */
> +#define LOGICALREP_ORIGIN_ANY "any"
>
> Are we planning to extend this to more options or are we planning to
> support the actual origin name here? If not then why isn't it just
> bool? I think the comments and the patch commit message should
> explain the details behind it if it has been already discussed and
> concluded.
Currently we only support local and any. But this was designed to
accept string instead of boolean type, so that it can be extended
later to support filtering of origin names specified by the user in
the later versions. The same was also discussed in pg unconference as
mentioned in [1]. I will add it to the commit message and a comment
for the same in the next version.
> 2.
> +/*
> + * Check and throw an error if the publisher has subscribed to the same table
> + * from some other publisher. This check is required only if copydata is ON
> and
> + * the origin is local.
> + */
>
> I think it should also explain why this combination is not allowed and
> if it is already explained in code
> then this code can add comments to refer to that part of the code.
In the same function, the reason for this is mentioned detailly just
above the place where error is thrown. I think that should be enough.
Have a look and let me know if that is not sufficient:
+ /*
+* Throw an error if the publisher has subscribed to
the same table
+* from some other publisher. We cannot differentiate
between the
+* local and non-local data that is present in the
HEAP during the
+* initial sync. Identification of local data can be
done only from
+* the WAL by using the origin id. XXX: For
simplicity, we don't check
+* whether the table has any data or not. If the table
doesn't have
+* any data then we don't need to distinguish between local and
+* non-local data so we can avoid throwing error in that case.
+*/
+ if (!slot_attisnull(slot, 3))
+ ereport(ERROR,
+
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("table:%s.%s might have
replicated data in the publisher",
+ nspname, relname),
+ errdetail("CREATE/ALTER
SUBSCRIPTION with origin = local and copy_data = on is not allowed
when the publisher might have replicated data."),
+ errhint("Use CREATE/ALTER
SUBSCRIPTION with copy_data = off/force."));
[1] -
https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Logical_Replication_Origin_Filtering_and_Consistency
Regards,
Vignesh
Re: Handle infinite recursion in logical replication setup
On Wed, Jun 15, 2022 at 12:13 PM Peter Smith wrote: > > Here are some review comments for patch v20-0004. > > == > > 1. General > > I thought that it is better to refer to the > subscription/publications/table "on" the node, rather than "in" the > node. Most of the review comments below are related to this point. Modified > == > > 2. Commit message > > a) Creating a two-node bidirectional replication when there is no data > in both nodes. > b) Adding a new node when there is no data in any of the nodes. > c) Adding a new node when data is present in the existing nodes. > > "in both nodes" -> "on both nodes" > "in any of the nodes" -> "on any of the nodes" > "in the existing nodes" -> "on the existing nodes" Modified > == > > 3. doc/src/sgml/logical-replication.sgml - Setting bidirectional > replication between two nodes > > 3a. > + > +The following steps demonstrate how to create a two-node bidirectional > +replication when there is no table data present in both nodes > +node1 and node2: > + > -> "on both nodes" Modified > 3b. > +Create a publication in node1: > -> "on" Modified > 3c. > +Create a publication in node2: > -> "on" Modified > 3d. > + > +Lock the table t1 in node1 and > +node2 in EXCLUSIVE mode until the > +setup is completed. > + > -> "on node1" Modified > 3e. > +Create a subscription in node2 to subscribe to > -> "on" Modified > 3f. > +Create a subscription in node1 to subscribe to > +node2: > -> "on" Modified > ~~~ > > 4. doc/src/sgml/logical-replication.sgml - Adding a new node when > there is no data in any of the nodes > > 4a. > + Adding a new node when there is no data in any of the nodes > SUGGESTION > Adding a new node when there is no table data on any of the nodes Modified > 4b. > + > +The following steps demonstrate adding a new node > node3 > +to the existing node1 and node2 > when > +there is no t1 data in any of the nodes. This requires > +creating subscriptions in node1 and > +node2 to replicate the data from > +node3 and creating subscriptions in > +node3 to replicate data from node1 > +and node2. Note: These steps assume that the > +bidirectional logical replication between node1 and > +node2 is already completed. > + > > "data in any of the nodes" -> "data on any of the nodes" > "creating subscriptions in node1" -> "creating > subscriptions on node1" > "creating subscriptions in node3" -> "creating > subscriptions on node3" Modified > 4c. > +Create a publication in node3: > -> "on" Modified > 4d. > +Lock table t1 in all the nodes > -> "on" Modified > 4e. > +Create a subscription in node1 to subscribe to > +node3: > -> "on" Modified > 4f. > +Create a subscription in node2 to subscribe to > +node3: > -> "on" Modified > 4g. > +Create a subscription in node3 to subscribe to > +node1: > -> "on" Modified > 4h. > +Create a subscription in node3 to subscribe to > +node2: Modified > 4i. > +node3. Incremental changes made in any node will be > +replicated to the other two nodes. > "in any node" -> "on any node" Modified > ~~~ > > 5. doc/src/sgml/logical-replication.sgml - Adding a new node when data > is present in the existing nodes > > 5a. > + Adding a new node when data is present in the existing > nodes > SUGGESTION > Adding a new node when table data is present on the existing nodes Modified > 5b. > + during initial data synchronization. Note: These steps assume that the > + bidirectional logical replication between node1 and > + node2 is already completed, and the pre-existing data > + in table t1 is already synchronized in both those > + nodes. > + > "in both those nodes" -> "on both those nodes" Modified > 5c. > +Create a publication in node3 > -> "on" Modified > 5d. > +Lock table t1 in node2 and > -> "on" Modified > 5e. > +Create a subscription in node1 to subscribe to > +node3: > -> "on" Modified > 5f. > +Create a subscription in node2 to subscribe to > +node3: > -> "on" Modified > 5g. > +Create a subscription in node3 to subscribe to > +node1. Use copy_data = force so > that > +the existing table data is copied during initial sync: > -> "on" Modified > > 5h. > +Create a subscription in node3 to subscribe to > +node2. Use copy_data = off > -> "on" Modified > 5i. > +node3. Incremental changes made in any node will be > +replicated to the other two nodes. > "in any node" -> "on any node" Modified > ~~~ > > 6. doc/src/sgml/logical-replication.sgml - Adding a new node when data > is present in the new node > > + Adding a new node when data is present in the new node > SUGGESTION > Adding a new node when table data is present on the new node Modified > ~~~ > > 7. doc/src/sgml/logical-replication.sgml - Generic steps for adding a > new node to an existing set of nodes > > 7a. > + >
Re: Handle infinite recursion in logical replication setup
On Wed, Jun 15, 2022 at 12:11 PM Peter Smith wrote:
>
> Here are some review comments for patch v20-0003.
>
> ==
>
> 1. Commit message
>
> In case, table t1 has a unique key, it will lead to a unique key
> violation and replication won't proceed.
>
> SUGGESTION
> If table t1 has a unique key, this will lead to a unique key
> violation, and replication won't proceed.
Modified
> ~~~
>
> 2. Commit message
>
> This problem can be solved by using...
>
> SUGGESTION
> This problem can be avoided by using...
Modified
> ~~~
>
> 3. Commit message
>
> step 3: Create a subscription in node1 to subscribe to node2. Use
> 'copy_data = on' so that the existing table data is copied during
> initial sync:
> node1=# CREATE SUBSCRIPTION sub_node1_node2 CONNECTION ''
> node1-# PUBLICATION pub_node2 WITH (copy_data = off, origin = local);
> CREATE SUBSCRIPTION
>
> This is wrong information. The table on node2 has no data, so talking
> about 'copy_data = on' is inappropriate here.
Modified
> ==
>
> 4. Commit message
>
> IMO it might be better to refer to subscription/publication/table "on"
> nodeXXX, instead of saying "in" nodeXXX.
>
> 4a.
> "the publication tables were also subscribing data in the publisher
> from other publishers." -> "the publication tables were also
> subscribing from other publishers.
Modified
> 4b.
> "After the subscription is created in node2" -> "After the
> subscription is created on node2"
Modified
> 4c.
> "step 3: Create a subscription in node1 to subscribe to node2." ->
> "step 3: Create a subscription on node1 to subscribe to node2."
Modified
> 4d.
> "step 4: Create a subscription in node2 to subscribe to node1." ->
> "step 4: Create a subscription on node2 to subscribe to node1."
Modified
> ==
>
> 5. doc/src/sgml/ref/create_subscription.sgml
>
> @@ -383,6 +397,15 @@ CREATE SUBSCRIPTION class="parameter">subscription_name can have non-existent publications.
>
>
> +
> + If the subscription is created with origin = local and
> + copy_data = on, it will check if the publisher tables
> are
> + being subscribed to any other publisher and, if so, then throw an error to
> + prevent possible non-local data from being copied. The user can override
> + this check and continue with the copy operation by specifying
> + copy_data = force.
> +
>
> Perhaps it is better here to say 'copy_data = true' instead of
> 'copy_data = on', simply because the value 'true' was mentioned
> earlier on this page (but this would be the first mention of 'on').
Modified
> ==
>
> 6. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> + errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force."));
>
> Saying "off or force" is not consistent with the other message wording
> in this patch, which used "/" for multiple enums.
> (e.g. "connect = false", "copy_data = true/force").
>
> So perhaps this errhint should be worded similarly:
> "Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force."
Modified
Thanks for the comments, the v21 patch attached at [1] has the changes
for the same.
[1] -
https://www.postgresql.org/message-id/CALDaNm3%2B6cey0rcDft1ZUCjSUtLDM0xmU_Q%2BYhcsBrqe1RH8%3Dw%40mail.gmail.com
Regards,
Vignesh
Re: Handle infinite recursion in logical replication setup
On Fri, Jun 10, 2022 at 2:09 PM Peter Smith wrote: > > Below are some review comments for the patch v18-0004 > > == > > 1. Commit message > > Document the steps for the following: > a) Create a two-node bidirectional replication when there is no data in both > the > nodes. > b) Adding a new node when there is no data in any of the nodes. > c) Adding a new node when data is present in the existing nodes. > d) Generic steps to add a new node to the existing set of nodes. > > SUGGESTION (minor changes to make the tense consistent) > Documents the steps for the following: > a) Creating a two-node bidirectional replication when there is no data > in both nodes. > b) Adding a new node when there is no data in any of the nodes. > c) Adding a new node when data is present in the existing nodes. > d) Generic steps for adding a new node to an existing set of nodes. Modified > == > > 2. doc/src/sgml/logical-replication.sgml - blurb > > + > +Bidirectional replication is useful in creating a multi-master database > +which helps in performing read/write operations from any of the nodes. > > SUGGESTION > Bidirectional replication is useful for creating a multi-master > database environment for replicating read/write operations performed > by any of the member nodes. Modified > ~~~ > > 3. doc/src/sgml/logical-replication.sgml - warning > > + > + > + Setting up bidirectional logical replication across nodes > requires multiple > + steps to be performed on various nodes. Because not all operations are > + transactional, the user is advised to take backups. > + > + > > SUGGESTION (but keep your formatting) > Setting up bidirectional logical replication requires multiple steps > to be performed on various nodes. Because... Modified > ~~~ > > 4. doc/src/sgml/logical-replication.sgml - Setting bidirectional > replication between two nodes > > + > +The steps to create a two-node bidirectional replication when there is no > +data in both the nodes node1 and > +node2 are given below: > + > > SUGGESTION (but keep your formatting) > The following steps demonstrate how to create a two-node bidirectional > replication when there is no table data present in both nodes node1 > and node2: Modified > ~~~ > > 5. doc/src/sgml/logical-replication.sgml - Setting bidirectional > replication between two nodes > > + > +Lock the required tables of node1 and > +node2 in EXCLUSIVE mode until the > +setup is completed. > + > > Instead of "the required tables" shouldn't this just say "table t1"? Modified > ~~~ > > 6. doc/src/sgml/logical-replication.sgml - Setting bidirectional > replication between two nodes > > + > +Now the bidirectional logical replication setup is complete between > +node1, node2 and > +node2. Any subsequent changes in one node will > +replicate the changes to the other nodes. > + > > SUGGESTION (for 2nd sentence, and keep your formatting) > Any incremental changes from node1 will be replicated to node2, and > any incremental changes from node2 will be replicated to node1. Modified > ~~~ > > 7. doc/src/sgml/logical-replication.sgml - Adding a new node when > there is no data in any of the nodes > > + > +Adding a new node node3 to the existing > +node1 and node2 when there is no > data > +in any of the nodes requires setting up subscription in > +node1 and node2 to replicate the > data > +from node3 and setting up subscription in > +node3 to replicate data from node1 > +and node2. > + > > SUGGESTION (but keep your formatting) > The following steps demonstrate adding a new node node3 to the > existing node1 and node2 when there is no t1 data in any of the nodes. > This requires creating subscriptions in node1 and node2 to replicate > the data from node3 and creating subscriptions in node3 to replicate > data from node1 and node2. Modified > ~~~ > > 8. doc/src/sgml/logical-replication.sgml - Adding a new node when > there is no data in any of the nodes > > + > + > + It is assumed that the bidirectional logical replication between > + node1 and node2 is already > completed. > + > + > > IMO this note should just be a text note in the previous paragraph > instead of an SGML . e.g. "Note: These steps assume that..." Modified > ~~~ > > 9. doc/src/sgml/logical-replication.sgml - Adding a new node when > there is no data in any of the nodes > > + > +Lock the required tables of all the nodes node1, > +node2 and node3 in > +EXCLUSIVE mode until the setup is completed. > + > > Instead of "the required tables" shouldn't this just say "table t1"? Modified > ~~~ > > 10. doc/src/sgml/logical-replication.sgml - Adding a new node when > there is no data in any of the nodes > > + > +Now the bidirectional logical replication setup is complete between > +node1, node2 and > +node2. Any subsequent changes in one node will > +replicate the
Re: Handle infinite recursion in logical replication setup
On Fri, Jun 3, 2022 at 11:01 AM Peter Smith wrote:
>
> Please find below my review comments for patch v17-0002:
>
> ==
>
> 1. Commit message
>
> This patch adds a new SUBSCRIPTION parameter "origin". It Specifies whether
> the subscription will request the publisher to only send changes that
> originated locally, or to send any changes regardless of origin.
>
> "It Specifies" -> "It specifies"
Modified
> ~~~
>
> 2. Commit message
>
> Usage:
> CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=postgres port='
> PUBLICATION pub1 with (origin = local);
>
> "with" -> "WITH"
Modified
> ==
>
> 3. doc/src/sgml/catalogs.sgml
>
> +
> + Possible origin values are local,
> + any, or NULL if none is
> specified.
> + If local, the subscription will request the
> + publisher to only send changes that originated locally. If
> + any, the publisher sends any changes regardless of
> + their origin.
> +
>
> Should this also mention that NULL (default) is equivalent to 'any'?
>
> SUGGESTION
> If any (or NULL), the publisher
> sends any changes regardless of their origin.
Modified
> ==
>
> 4. src/backend/catalog/pg_subscription.c
>
> @@ -72,6 +72,16 @@ GetSubscription(Oid subid, bool missing_ok)
> sub->twophasestate = subform->subtwophasestate;
> sub->disableonerr = subform->subdisableonerr;
>
> + datum = SysCacheGetAttr(SUBSCRIPTIONOID,
> + tup,
> + Anum_pg_subscription_suborigin,
> + &isnull);
> +
> + if (!isnull)
> + sub->origin = TextDatumGetCString(datum);
> + else
> + sub->origin = NULL;
> +
> /* Get conninfo */
> datum = SysCacheGetAttr(SUBSCRIPTIONOID,
> tup,
>
> Missing comment like the nearby code has:
> /* Get origin */
Modified
> ==
>
> 5. src/backend/replication/logical/worker.c
>
> +/* Macro for comparing string fields that might be NULL */
> +#define equalstr(a, b) \
> + (((a) != NULL && (b) != NULL) ? (strcmp(a, b) == 0) : (a) == (b))
> +
>
> Should that have some extra parens for the macro args?
> e.g. "strcmp((a), (b))"
Modified
> ~~~
>
> 6. src/backend/replication/logical/worker.c - maybe_reread_subscription
>
> @@ -3059,6 +3063,7 @@ maybe_reread_subscription(void)
> strcmp(newsub->slotname, MySubscription->slotname) != 0 ||
> newsub->binary != MySubscription->binary ||
> newsub->stream != MySubscription->stream ||
> + equalstr(newsub->origin, MySubscription->origin) ||
> newsub->owner != MySubscription->owner ||
> !equal(newsub->publications, MySubscription->publications))
> {
>
> Is that right? Shouldn't it say !equalstr(...)?
Modified
> ==
>
> 7. src/backend/replication/pgoutput/pgoutput.c - parse_output_parameters
>
> @@ -380,6 +382,16 @@ parse_output_parameters(List *options, PGOutputData
> *data)
>
> data->two_phase = defGetBoolean(defel);
> }
> + else if (strcmp(defel->defname, "origin") == 0)
> + {
> + if (origin_option_given)
> + ereport(ERROR,
> + (errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("conflicting or redundant options")));
> + origin_option_given = true;
> +
> + data->origin = defGetString(defel);
> + }
>
> Should this function also be validating that the origin parameter
> value is only permitted to be one of "local" or "any"?
Modified
> ~~~
>
> 8. src/backend/replication/pgoutput/pgoutput.c - pgoutput_origin_filter
>
> @@ -1698,12 +1710,20 @@ pgoutput_message(LogicalDecodingContext *ctx,
> ReorderBufferTXN *txn,
> }
>
> /*
> - * Currently we always forward.
> + * Return true if the data source (origin) is remote and user has requested
> + * only local data, false otherwise.
> */
> static bool
> pgoutput_origin_filter(LogicalDecodingContext *ctx,
> RepOriginId origin_id)
> {
> + PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
> +
> + if (data->origin &&
> + (strcmp(data->origin, "local") == 0) &&
> + origin_id != InvalidRepOriginId)
> + return true;
> +
> return false;
> }
>
> 8a.
> Could rewrite the condition by putting the strcmp last so you can
> avoid doing unnecessary strcmp.
>
> e.g
> + if (data->origin &&
> + origin_id != InvalidRepOriginId &&
> + strcmp(data->origin, "local" == 0)
I have avoided the strcmp by caching locally as suggested in 8b. I
have not made any change for this.
>
> 8b.
> I also wondered if it might be worth considering caching the origin
> parameter value when it was parsed so that you can avoid doing any
> strcmp at all during this function. Because otherwise this might be
> called millions of times, right?
Modified
> ==
>
> 9. src/include/catalog/pg_subscription.h
>
> @@ -87,6 +87,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
> BKI_SHARED_RELATION BKI_ROW
>
> /* List of publications subscribed to */
> text subpublications[1] BKI_FORCE_NOT_NULL;
> +
> + /* Publish the data originated from the specified origin */
> + text suborigin;
> #endif
> } FormData_pg_subscription;
>
> SUGGESTION (for the comment text)
> /* Only publish data originating from the specified origin */
Modified
> ~~~
>
> 10 src/in
Re: Skipping schema changes in publication
On Tue, May 31, 2022 at 11:51 AM Peter Smith wrote:
>
> Here are my review comments for patch v7-0002.
>
> ==
>
> 1. doc/src/sgml/logical-replication.sgml
>
> @@ -1167,8 +1167,9 @@ CONTEXT: processing remote data for replication
> origin "pg_16395" during "INSER
>
> To add tables to a publication, the user must have ownership rights on the
> table. To add all tables in schema to a publication, the user must be a
> - superuser. To create a publication that publishes all tables or
> all tables in
> - schema automatically, the user must be a superuser.
> + superuser. To add all tables to a publication, the user must be a
> superuser.
> + To create a publication that publishes all tables or all tables in schema
> + automatically, the user must be a superuser.
>
>
> I felt that maybe this whole paragraph should be rearranged. Put the
> "create publication" parts before the "alter publication" parts;
> Re-word the sentences more similarly. I also felt the ALL TABLES and
> ALL TABLES IN SCHEMA etc should be written uppercase/literal since
> that is what was meant.
>
> SUGGESTION
> To create a publication using FOR ALL TABLES or FOR ALL TABLES IN
> SCHEMA, the user must be a superuser. To add ALL TABLES or ALL TABLES
> IN SCHEMA to a publication, the user must be a superuser. To add
> tables to a publication, the user must have ownership rights on the
> table.
Modified
> ~~~
>
> 2. doc/src/sgml/ref/alter_publication.sgml
>
> @@ -82,8 +88,8 @@ ALTER PUBLICATION class="parameter">name RESET
>
>
> You must own the publication to use ALTER PUBLICATION.
> - Adding a table to a publication additionally requires owning that table.
> - The ADD ALL TABLES IN SCHEMA,
> + Adding a table to or excluding a table from a publication additionally
> + requires owning that table. The ADD ALL TABLES IN
> SCHEMA,
> SET ALL TABLES IN SCHEMA to a publication and
>
> Isn't this missing some information that says ADD ALL TABLES requires
> the invoking user to be a superuser?
Modified
> ~~~
>
> 3. doc/src/sgml/ref/alter_publication.sgml - examples
>
> +
> + Alter publication production_publication to
> publish
> + all tables except users and
> + departments tables:
> +
> +ALTER PUBLICATION production_publication ADD ALL TABLES EXCEPT users,
> departments;
> +
> +
>
> I didn't think it needs to say "tables" 2x (e.g. remove the last "tables")
Modified
> ~~~
>
> 4. doc/src/sgml/ref/create_publication.sgml - examples
>
> +
> + Create a publication that publishes all changes in all the tables except
> for
> + the changes of users and
> + departments tables:
> +
> +CREATE PUBLICATION mypublication FOR ALL TABLES EXCEPT users, departments;
> +
> +
>
> I didn't think it needs to say "tables" 2x (e.g. remove the last "tables")
Modified
> ~~~
>
> 5. src/backend/catalog/pg_publication.c
>
> foreach(lc, ancestors)
> {
> Oid ancestor = lfirst_oid(lc);
> - List*apubids = GetRelationPublications(ancestor);
> - List*aschemaPubids = NIL;
> + List*apubids = GetRelationPublications(ancestor, false);
> + List*aschemapubids = NIL;
> + List*aexceptpubids = NIL;
>
> level++;
>
> - if (list_member_oid(apubids, puboid))
> + /* check if member of table publications */
> + if (!list_member_oid(apubids, puboid))
> {
> - topmost_relid = ancestor;
> -
> - if (ancestor_level)
> - *ancestor_level = level;
> - }
> - else
> - {
> - aschemaPubids = GetSchemaPublications(get_rel_namespace(ancestor));
> - if (list_member_oid(aschemaPubids, puboid))
> + /* check if member of schema publications */
> + aschemapubids = GetSchemaPublications(get_rel_namespace(ancestor));
> + if (!list_member_oid(aschemapubids, puboid))
> {
> - topmost_relid = ancestor;
> -
> - if (ancestor_level)
> - *ancestor_level = level;
> + /*
> + * If the publication is all tables publication and the table
> + * is not part of exception tables.
> + */
> + if (puballtables)
> + {
> + aexceptpubids = GetRelationPublications(ancestor, true);
> + if (list_member_oid(aexceptpubids, puboid))
> + goto next;
> + }
> + else
> + goto next;
> }
> }
>
> + topmost_relid = ancestor;
> +
> + if (ancestor_level)
> + *ancestor_level = level;
> +
> +next:
> list_free(apubids);
> - list_free(aschemaPubids);
> + list_free(aschemapubids);
> + list_free(aexceptpubids);
> }
>
>
> I felt those negative (!) conditions and those goto are making this
> logic hard to understand. Can’t it be simplified more than this? Even
> just having another bool flag might help make it easier.
>
> e.g. Perhaps something a bit like this (but add some comments)
>
> foreach(lc, ancestors)
> {
> Oid ancestor = lfirst_oid(lc);
> List*apubids = GetRelationPublications(ancestor);
> List*aschemaPubids = NIL;
> List*aexceptpubids = NIL;
> bool set_top = false;
> level++;
>
> set_top = list_member_oid(apubids, puboid);
> if (!set_top)
> {
> aschemaPubids = GetSchemaPublications(get_rel_namespace(ancestor));
> set_top = list_
Re: Skipping schema changes in publication
'On Mon, May 30, 2022 at 1:51 PM Peter Smith wrote: > > Here are some minor review comments for v7-0001. > > == > > 1. General > > Probably the commit message and all the PG docs and code comments > should be changed to refer to "publication parameters" instead of > (currently) "publication options". This is because these things are > really called "publication_parameters" in the PG docs [1]. > > All the following review comments are just examples of this suggestion. Modified > ~~~ > > 2. Commit message > > "includes resetting the publication options," -> "includes resetting > the publication parameters," Modified > ~~~ > > 3. doc/src/sgml/ref/alter_publication.sgml > > + > + The RESET clause will reset the publication to the > + default state which includes resetting the publication options, setting > + ALL TABLES flag to false and > + dropping all relations and schemas that are associated with the > publication. > > > > "resetting the publication options," -> "resetting the publication > parameters," Modified > ~~~ > > 4. src/backend/commands/publicationcmds.c > > @@ -53,6 +53,14 @@ > #include "utils/syscache.h" > #include "utils/varlena.h" > > +/* CREATE PUBLICATION default values for flags and options */ > +#define PUB_DEFAULT_ACTION_INSERT true > +#define PUB_DEFAULT_ACTION_UPDATE true > +#define PUB_DEFAULT_ACTION_DELETE true > +#define PUB_DEFAULT_ACTION_TRUNCATE true > +#define PUB_DEFAULT_VIA_ROOT false > +#define PUB_DEFAULT_ALL_TABLES false > > "flags and options" -> "flags and publication parameters" Modified > ~~~ > > 5. src/backend/commands/publicationcmds.c > > +/* > + * Reset the publication. > + * > + * Reset the publication options, setting ALL TABLES flag to false and drop > + * all relations and schemas that are associated with the publication. > + */ > +static void > +AlterPublicationReset(ParseState *pstate, AlterPublicationStmt *stmt, > + Relation rel, HeapTuple tup) > > "Reset the publication options," -> "Reset the publication parameters," Modified > ~~~ > > 6. src/test/regress/sql/publication.sql > > +-- Verify that publish options and publish_via_partition_root option are > reset > +\dRp+ testpub_reset > +ALTER PUBLICATION testpub_reset RESET; > +\dRp+ testpub_reset > > SUGGESTION > -- Verify that 'publish' and 'publish_via_partition_root' publication > parameters are reset Modified, I have split this into two tests as it will help the 0002 patch to add few tests with the existing steps for 'publish' and 'publish_via_partition_root' publication parameter. Thanks for the comments. the v8 patch attached at [1] has the fixes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm0sAU4s1KTLOEWv%3DrYo5dQK6uFTJn_0FKj3XG1Nv4D-qw%40mail.gmail.com Regards, Vignesh
Re: Skipping schema changes in publication
On Thu, May 26, 2022 at 7:04 PM osumi.takami...@fujitsu.com
wrote:
>
> On Monday, May 23, 2022 2:13 PM vignesh C wrote:
> > Attached v7 patch which fixes the buildfarm warning for an unused warning in
> > release mode as in [1].
> Hi, thank you for the patches.
>
>
> I'll share several review comments.
>
> For v7-0001.
>
> (1) I'll suggest some minor rewording.
>
> +
> + The RESET clause will reset the publication to the
> + default state which includes resetting the publication options, setting
> + ALL TABLES flag to false and
> + dropping all relations and schemas that are associated with the
> publication.
>
> My suggestion is
> "The RESET clause will reset the publication to the
> default state. It resets the publication operations,
> sets ALL TABLES flag to false and drops all relations
> and schemas associated with the publication."
I felt the existing looks better. I would prefer to keep it that way.
> (2) typo and rewording
>
> +/*
> + * Reset the publication.
> + *
> + * Reset the publication options, setting ALL TABLES flag to false and drop
> + * all relations and schemas that are associated with the publication.
> + */
>
> The "setting" in this sentence should be "set".
>
> How about changing like below ?
> FROM:
> "Reset the publication options, setting ALL TABLES flag to false and drop
> all relations and schemas that are associated with the publication."
> TO:
> "Reset the publication operations, set ALL TABLES flag to false and drop
> all relations and schemas associated with the publication."
I felt the existing looks better. I would prefer to keep it that way.
> (3) AlterPublicationReset
>
> Do we need to call CacheInvalidateRelcacheAll() or
> InvalidatePublicationRels() at the end of
> AlterPublicationReset() like AlterPublicationOptions() ?
CacheInvalidateRelcacheAll should be called if we change all tables
from true to false, else the cache will not be invalidated. Modified
>
> For v7-0002.
>
> (4)
>
> + if (stmt->for_all_tables)
> + {
> + boolisdefault = CheckPublicationDefValues(tup);
> +
> + if (!isdefault)
> + ereport(ERROR,
> +
> errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
> + errmsg("adding ALL TABLES requires
> the publication to have default publication options, no tables/
> + errhint("Use ALTER PUBLICATION ...
> RESET to reset the publication"));
>
>
> The errmsg string has three messages for user and is a bit long
> (we have two sentences there connected by 'and').
> Can't we make it concise and split it into a couple of lines for code
> readability ?
>
> I'll suggest a change below.
> FROM:
> "adding ALL TABLES requires the publication to have default publication
> options, no tables/schemas associated and ALL TABLES flag should not be set"
> TO:
> "adding ALL TABLES requires the publication defined not for ALL TABLES"
> "to have default publish actions without any associated tables/schemas"
Added errdetail and split it
> (5) typo
>
>
> +EXCEPT TABLE
> +
> +
> + This clause specifies a list of tables to exclude from the publication.
> + It can only be used with FOR ALL TABLES.
> +
> +
> +
> +
>
> Kindly change
> FROM:
> This clause specifies a list of tables to exclude from the publication.
> TO:
> This clause specifies a list of tables to be excluded from the publication.
> or
> This clause specifies a list of tables excluded from the publication.
Modified
> (6) Minor suggestion for an expression change
>
>Marks the publication as one that replicates changes for all tables in
> - the database, including tables created in the future.
> + the database, including tables created in the future. If
> + EXCEPT TABLE is specified, then exclude replicating
> + the changes for the specified tables.
>
>
> I'll suggest a minor rewording.
> FROM:
> ...exclude replicating the changes for the specified tables
> TO:
> ...exclude replication changes for the specified tables
I felt the existing is better.
> (7)
> (7-1)
>
> +/*
> + * Check if the publication has default values
> + *
> + * Check the following:
> + * a) Publication is not set with "FOR ALL TABLES"
> + * b) Publication is having default options
> + * c) Publication is not associated with schemas
> + * d) Publication is not assoc
Re: Handle infinite recursion in logical replication setup
On Fri, May 27, 2022 at 2:08 PM Peter Smith wrote: > > On Fri, May 27, 2022 at 5:04 PM shiy.f...@fujitsu.com > wrote: > > > > On Wed, May 25, 2022 7:55 PM vignesh C wrote: > > > > > > The attached v16 patch has the changes for the same. > > > > > > > Thanks for updating the patch. > > > > Some comments for the document in 0002 patch. > > > > 1. > > + > > +Lock the required tables in node1 and > > +node2 till the setup is completed. > > + > > + > > + > > +Create a publication in node1: > > + > > +node1=# CREATE PUBLICATION pub_node1 FOR TABLE t1; > > +CREATE PUBLICATION > > + > > > > If the table is locked in the very beginning, we will not be able to create > > the > > publication (because the locks have conflict). Maybe we should switch the > > order > > of creating publication and locking tables here. > > > > I agree. It seems some of the locking instructions in the earlier > sections 31.11.1 - 31.11.4 are contradictory to the later "generic" > steps given in "31.11.5. Generic steps to add a new node to the > existing set of nodes". I'm assuming the generic steps are the > "correct" steps > > e.g. generic steps say get the lock on new node tables AFTER the > publication of new node. > e.g. generic steps say do NOT get a table lock on the node one you are > (first) joining to. Yes, the generic steps are correct. modified > ~~~ > > Furthermore, the generic steps are describing attaching a new N+1th > node to some (1 ... N) other nodes. > > So I think in the TWO-node case (section 31.11.1) node2 should be > treated as the "new" node that you are attaching to the "first" node1. > IMO the section 31.11.1 example should be reversed so that it obeys > the "generic" pattern. > e.g. It should be doing the CREATE PUBLICATION pub_node2 first (since > that is the "new" node) In generic steps we mention the publication must be created in a new node and we don't say about the existing nodes, because the publication would be existing already. I feel the existing order of publication creation in the TWO-node case (section 31.11.1) is ok. Thanks for the comments, the v17 patch attached at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm1rMihO7daiFyLdxkqArrC%2BdtM61oPXc-XrTYBYnJg3nw%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Fri, May 27, 2022 at 12:34 PM shiy.f...@fujitsu.com wrote: > > On Wed, May 25, 2022 7:55 PM vignesh C wrote: > > > > The attached v16 patch has the changes for the same. > > > > Thanks for updating the patch. > > Some comments for the document in 0002 patch. > > 1. > + > +Lock the required tables in node1 and > +node2 till the setup is completed. > + > + > + > +Create a publication in node1: > + > +node1=# CREATE PUBLICATION pub_node1 FOR TABLE t1; > +CREATE PUBLICATION > + > > If the table is locked in the very beginning, we will not be able to create > the > publication (because the locks have conflict). Maybe we should switch the > order > of creating publication and locking tables here. Modified > 2. > In the case of "Adding a new node when data is present in the new node", we > need > to truncate table t1 in node3, but the truncate operation would be blocked > because the table has be locked before. Maybe we need some changes for it. This section contents have been removed because this is not feasible currently Thanks for the comments, the v17 patch attached at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm1rMihO7daiFyLdxkqArrC%2BdtM61oPXc-XrTYBYnJg3nw%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Fri, May 27, 2022 at 10:56 AM Peter Smith wrote: > > Here are some more review comments for v15-0002 I found while working > through the documented examples. These are all for > doc/src/sgml/logical-replication.sgml. The section numbers (e.g. > 31.11.1) are the assigned numbers after HTML rendering. > > The general comment is that the sections appeared at first to be > independent of each other, but they are not really - sometimes > existing data and existing pub/sub are assumed to remain. Therefore, I > think some more text needs to be added to clarify the initial state > for each of these sections. > > == > > 1. Section id. > > > > IMO this section ID should be renamed > "logical-replication-bidirectional" then it will be more consistent > filenames with all the other logical replication section names. Modified > ~~~ > > 2. Section 31.11.2. Adding a new node when there is no data in any of the > nodes > > 2a. > This seems like a continuation of 31.11.1 because pub/sub of > node1,node2 is assumed to exist still. Added a note > 2b. > The bottom of this section should say after these steps any subsequent > incremental data changes on any node will be replicated to all nodes > (node1, node2, node3) Modified > ~~~ > > 3. Section 31.11.3. Adding a new node when data is present in the existing > nodes > > 3a. > This seems to be a continuation of 31.11.1 because pub/sub (and > initial data) of node1/node2 is assumed to exist. Added a note > 3b. > > The bottom of this section should say after these steps that any > initial data of node1/node2 will be seen in node3, and any subsequent > incremental data changes on any node will be replicated to all nodes > (node1, node2, node3) Modified > ~~~ > > 4. Section 31.11.4. Adding a new node when data is present in the new node > > 4a. > This seems to be a continuation of 31.11.1 because pub/sub (and > initial data) of node1/node2 is assumed to exist. This section contents have been removed because this is not feasible currently > 4b. > It is not made very clear up-front if the tables on node1 and node2 > are empty or not. Apparently, they are considered NOT empty because > later in the example you are using "force" when you create the > subscription on node3. This section contents have been removed because this is not feasible currently > 4c. > The SQL wrapping here is strangely different from others (put the > subscription name on 1st line) > node3=# CREATE SUBSCRIPTION > node3-# sub_node3_node1 CONNECTION 'dbname=foo host=node1 user=repuser' > node3-# PUBLICATION pub_node1 > node3-# WITH (copy_data = force, only_local = on); > CREATE SUBSCRIPTION This section contents have been removed because this is not feasible currently > 4d. > The SQL wrapping here is strangely different from others (put the > subscription name on 1st line) > node3=# CREATE SUBSCRIPTION > node3-# sub_node3_node2 CONNECTION 'dbname=foo host=node2 user=repuser' > node3-# PUBLICATION pub_node2 > node3-# WITH (copy_data = off, only_local = on); > CREATE SUBSCRIPTION This section contents have been removed because this is not feasible currently > 4e. > The bottom of this section should say after this that any initial data > of node1/node2 will be seen in node3, and any subsequent incremental > data changes on any node will be replicated to all nodes (node1, > node2, node3) This section contents have been removed because this is not feasible currently > ~~~ > > 5. Section 31.11.5. Generic steps to add a new node to the existing set of > nodes > > 5a > Create subscriptions on the new node pointing to publication on the > first node with only_local option specified as on and copy_data option > specified as "force". > > -> that should say "Create a subscription" (singular) Modified > 5b. > Create subscriptions on the new node pointing to publications on the > remaining node with only_local option specified as on and copy_data > option specified as off. > > -> that should say "on the remaining node" (plural) Modified Thanks for the comments, the v17 patch attached at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm1rMihO7daiFyLdxkqArrC%2BdtM61oPXc-XrTYBYnJg3nw%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Fri, May 27, 2022 at 7:54 AM Amit Kapila wrote: > > On Wed, Apr 6, 2022 at 9:36 AM Amit Kapila wrote: > > > > On Tue, Apr 5, 2022 at 7:06 PM Ashutosh Bapat > > wrote: > > > > > > On Tue, Apr 5, 2022 at 6:21 AM Peter Smith wrote: > > > > > > > > Below are some other name ideas. Maybe they are not improvements, but > > > > it might help other people to come up with something better. > > > > > > > > subscribe_publocal_only = true/false > > > > origin = pub_only/any > > > > adjacent_data_only = true/false > > > > direct_subscriptions_only = true/false > > > > ... > > > > > > FWIW, The subscriber wants "changes originated on publisher". From > > > that angle origin = publisher/any looks attractive. It also leaves > > > open the possibility that the subscriber may ask changes from a set of > > > origins or even non-publisher origins. > > > > > > > So, how are you imagining extending it for multiple origins? I think > > we can have multiple origins data on a node when there are multiple > > subscriptions pointing to the different or same node. The origin names > > are internally generated names but are present in > > pg_replication_origin. So, are we going to ask the user to check that > > table and specify it as an option? But, note, that the user may not be > > able to immediately recognize which origin data is useful for her. > > > > I still don't have a very clear answer for the usability aspect but it > seems this was discussed in PGCon-Unconference [1] (Section: Origin > filter) and there also it was suggested to allow users to specify > multiple origin names. So, probably Ashutosh's idea makes sense and we > should use "origin = publisher/any or origin=local/any". Among these, > I would prefer later. I have changed it to origin = local/any. The v17 patch attached at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm1rMihO7daiFyLdxkqArrC%2BdtM61oPXc-XrTYBYnJg3nw%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Thu, May 26, 2022 at 4:40 PM Amit Kapila wrote:
>
> On Fri, May 20, 2022 at 3:31 PM vignesh C wrote:
> >
> > On Wed, May 18, 2022 at 4:22 PM Amit Kapila wrote:
> > >
> > > 5.
> > > * It is quite possible that subscriber has not yet pulled data to
> > > + * the tables, but in ideal cases the table data will be subscribed.
> > > + * To keep the code simple it is not checked if the subscriber table
> > > + * has pulled the data or not.
> > > + */
> > > + if (copydata == COPY_DATA_ON && local_only && !slot_attisnull(slot, 3))
> > >
> > > Sorry, but I don't understand what you intend to say by the above
> > > comment. Can you please explain?
> >
> > When the user specifies copy_data as on, we should check if the
> > publisher has the publication tables being subscribed from a remote
> > publisher. If so throw an error as it remote origin data present.
> > Ex:
> > Node1 - pub1 for table t1 -- no data
> > Node2 - Sub1 subscribing to data from pub1
> > Node2 - pub2 for table t1 -- no data
> > Node3 - create subscription to Node2 with copy_data = ON
> >
> > In this case even though the table does not have any remote origin
> > data, as Node2 is subscribing to data from Node1, throw an error.
> > We throw an error irrespective of data present in Node1 or not to keep
> > the code simple.
> >
>
> I think we can change the contents of comments something like: "Throw
> an error if the publisher has subscribed to the same table from some
> other publisher. We cannot differentiate between the local and
> non-local data that is present in the HEAP during the initial sync.
> Identification of local data can be done only from the WAL by using
> the origin id. XXX: For simplicity, we don't check whether the table
> has any data or not. If the table doesn't have any data then we don't
> need to distinguish between local and non-local data so we can avoid
> throwing error in that case."
Modified
> Few more comments:
> ==
> Patch 0002
> ==
> 1.
> + if (copydata == COPY_DATA_ON && only_local && !slot_attisnull(slot, 3))
> + ereport(ERROR,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("CREATE/ALTER SUBSCRIPTION with only_local and copy_data as
> true is not allowed when the publisher might have replicated data,
> table:%s.%s might have replicated data in the publisher",
> +nspname, relname),
> + errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force"));
>
> Can we split the error message? errmsg: table:%s.%s has replicated
> data in the publisher; errdetail:CREATE/ALTER SUBSCRIPTION with
> only_local and copy_data as true is not allowed when the publisher has
> replicated data
Modified
> 2.
> +
> +Lock the required tables in the new node until the setup is complete.
> +
> +
> +Create subscriptions on existing nodes pointing to publication on
> +the new node with only_local option specified as
> +on and copy_data specified as
> +on.
> +
> +
> +Wait for data to be copied from the new node to existing nodes.
> +
> +
> +Alter the publication in new node so that the truncate operation is not
> +replicated to the subscribers.
> +
>
> Here and at other places, we should specify that the lock mode should
> to acquire the lock on table should be EXCLUSIVE so that no concurrent
> DML is allowed on it. Also, it is better if somewhere we explain why
> and which nodes need locks?
Modified
> Patch 0001:
> ==
> 1.
> +$node_A->append_conf(
> + 'postgresql.conf', qq(
> +max_prepared_transactions = 10
> +logical_decoding_work_mem = 64kB
> +));
>
> I don't see why you need to set these parameters. There is no test
> case that needs these parameters. Please remove these from here and
> all other similar places in 032_onlylocal.pl.
Removed it.
Thanks for the comments, the v17 patch attached at [1] has the changes
for the same.
[1] -
https://www.postgresql.org/message-id/CALDaNm1rMihO7daiFyLdxkqArrC%2BdtM61oPXc-XrTYBYnJg3nw%40mail.gmail.com
Regards,
Vignesh
Re: Handle infinite recursion in logical replication setup
On Thu, May 26, 2022 at 2:06 PM Peter Smith wrote: > > This post completes my review of patch v15*. > > == > > 1. A general comment affecting both patches > > I realised the things defined in the subscription WITH are referred to > in the PG DOCS as "parameters" (not "options"). See CREATE > SUBSCRIPTION [1]. This is already reflected in some of my review > comments for the v15-0002 below., but I'd already posted my v15-0001 > review comments before noticing this. > > Please also go through the v15-0001 patch to make all the necessary > terminology changes (including in the v15-0001 commit message). Modified > = > > Please find below review comments for patch v15-0002. > > 2. Commit message > > This patch does couple of things: > change 1) Added force option for copy_data. > change 2) Check and throw an error if the publication tables were also > subscribing data in the publisher from other publishers. > > -- > > 2a. > "couple" -> "a couple" > "Added force" -> "Adds force ..." > "Check and throw" -> "Checks and throws ..." Modified > 2b. > Isn't "Change 2)" only applicable when copy_data and only_local are > on? It should say so here. Modified > ~~~ > > 3. Commit message > > node1: Table t1 (c1 int) has data 1, 2, 3, 4 > node2: Table t1 (c1 int) has data 5, 6, 7, 8 > > -- > > I feel these kinds of replication examples are always much easier to > visualize when the data somehow indicates where it came from. E.g. > consider using different data in your example like: > > node1: Table t1 (c1 int) has data 11, 12, 13, 14 > node2: Table t1 (c1 int) has data 21, 22, 23, 24 Modified > ~~~ > > 4. Commit message > > Examples in this message exceed 80 chars. Please wrap the SQL appropriately Modified > ~~~ > > 5. Commit message > > After this the data will be something like this: > node1: > 1, 2, 3, 4, 5, 6, 7, 8 > > node2: > 1, 2, 3, 4, 5, 6, 7, 8, 5, 6, 7, 8 > > So, you can see that data on node2 (5, 6, 7, 8) is duplicated. > > -- > > Yeah, but is that entirely correct though? AFAIK the scenario > described by this example is going to recurse forever. Modified > ~~~ > > 6. Commit message > > This problem can be solved by using only_local and copy_data option as > given below > > -- > > "option" -> "parameters" Modified > ~~~ > > 7. Commit message > > There are other minor review comments but it's too messy to report in > detail here. I suggest checking all the ":" (some were missing) and > the letter case of/within the sentences. I also suggest > cutting/pasting all this text into MSWord or some grammar checker to > correct anything else reported. Modified > ~~~ > > 8. Commit message > > CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '' > PUBLICATION pub_node1 WITH (copy_data = force, only_local = on); > ERROR: CREATE/ALTER SUBSCRIPTION with only_local and copy_data as > true is not allowed when the publisher might have replicated data > > -- > > Looks like a typo: e.g. If copy_data is "force" then how is the error > happening? Modified > ~~~ > > 9. Commit message > > The examples seem confused/muddled in some way IMO. > > IIUC the new ERROR is like a new integrity check for when the > subscription is using the only_local = on, along with copy_data = on. > > And so then the copy_data = force was added as a way to override that > previous error being reported. > > So unless you know about the possible error then the "force" option > makes no sense. So I think you need to change the order of the > explanations in the commit message to describe the new error first. Modified > == > > 10. doc/src/sgml/logical-replication.sgml > > + > + > + Bidirectional logical replication > + > + > + Setting bidirectional replication between two nodes > + > +Bidirectional replication is useful in creating a multi-master database > +which helps in performing read/write operations from any of the nodes. > +Setting up bidirectional logical replication between two nodes requires > +creation of a publication in all the nodes, creating subscriptions in > +each of the nodes that subscribes to data from all the nodes. The steps > +to create a two-node bidirectional replication when there is no data in > +both the nodes are given below: > + > > This description is confusing. It seems to be trying to say something > about a multi-master system (e.g. you say "all the nodes") yet this is > all written inside the section about "... two nodes", so saying "all > the nodes" makes no sense here. > > I think you need to split this information and put some kind of blurb > text at the top level (section 31.11), and then this section (for "two > nodes") should ONLY be talking about the case when there are just TWO > nodes. Modified > ~~~ > > 11. doc/src/sgml/logical-replication.sgml > > + > +Lock the required tables in node1 and > +node2 till the setup is completed. > + > > "till" -> "until" (make this same change in multiple places) Modified > ~~~ > > 12. doc/src/
Re: Handle infinite recursion in logical replication setup
On Tue, May 24, 2022 at 8:06 AM shiy.f...@fujitsu.com wrote: > > On Tue, May 24, 2022 1:34 AM vignesh C wrote: > > > > Thanks for the comments, the attached v15 patch has the fixes for the same. > > > > Thanks for updating the patch. I have a comment on the document in 0002 patch. > > @@ -300,6 +310,11 @@ CREATE SUBSCRIPTION class="parameter">subscription_namereplication from the publisher. The default is >false. > > + > + There is some interaction between the only_local > + option and copy_data option. Refer to the > + for details. > + > > > > This change is related to "only_local" option and "copy_data" option, so I > think > it should be put together with "only_local", instead of "disable_on_error". Modified. The v16 patch attached at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm2_Ytder-6C68ia%3Dm39cmknAhxf2KGkeNAtxt84MxMT3w%40mail.gmail.com Regards, Vignesh
Re: Skipping schema changes in publication
On Sat, May 21, 2022 at 11:06 AM vignesh C wrote:
>
> On Fri, May 20, 2022 at 11:23 AM Peter Smith wrote:
> >
> > Below are my review comments for v6-0002.
> >
> > ==
> >
> > 1. Commit message.
> > The psql \d family of commands to display excluded tables.
> >
> > SUGGESTION
> > The psql \d family of commands can now display excluded tables.
>
> Modified
>
> > ~~~
> >
> > 2. doc/src/sgml/ref/alter_publication.sgml
> >
> > @@ -22,6 +22,7 @@ PostgreSQL documentation
> >
> >
> > ALTER PUBLICATION name
> > ADD publication_object [,
> > ...]
> > +ALTER PUBLICATION name
> > ADD ALL TABLES [ EXCEPT [ TABLE ] exception_object [, ... ] ]
> >
> > The "exception_object" font is wrong. Should look the same as
> > "publication_object"
>
> Modified
>
> > ~~~
> >
> > 3. doc/src/sgml/ref/alter_publication.sgml - Examples
> >
> > @@ -214,6 +220,14 @@ ALTER PUBLICATION sales_publication ADD ALL
> > TABLES IN SCHEMA marketing, sales;
> >
> >
> >
> > +
> > + Alter publication production_publication to
> > publish
> > + all tables except users and
> > + departments tables:
> > +
> > +ALTER PUBLICATION production_publication ADD ALL TABLES EXCEPT TABLE
> > users, departments;
> > +
> >
> > Consider using "EXCEPT" instead of "EXCEPT TABLE" because that will
> > show TABLE keyword is optional.
>
> Modified
>
> > ~~~
> >
> > 4. doc/src/sgml/ref/create_publication.sgml
> >
> > An SGML tag error caused building the docs to fail. My fix was
> > previously reported [1].
>
> Modified
>
> > ~~~
> >
> > 5. doc/src/sgml/ref/create_publication.sgml
> >
> > @@ -22,7 +22,7 @@ PostgreSQL documentation
> >
> >
> > CREATE PUBLICATION name
> > -[ FOR ALL TABLES
> > +[ FOR ALL TABLES [ EXCEPT [ TABLE ] exception_object [, ... ] ]
> >
> > The "exception_object" font is wrong. Should look the same as
> > "publication_object"
>
> Modified
>
> > ~~~
> >
> > 6. doc/src/sgml/ref/create_publication.sgml - Examples
> >
> > @@ -351,6 +366,15 @@ CREATE PUBLICATION production_publication FOR
> > TABLE users, departments, ALL TABL
> > CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing,
> > sales;
> >
> >
> > +
> > + Create a publication that publishes all changes in all the tables
> > except for
> > + the changes of users and
> > + departments table:
> > +
> > +CREATE PUBLICATION mypublication FOR ALL TABLE EXCEPT TABLE users,
> > departments;
> > +
> > +
> > +
> >
> > 6a.
> > Typo: "FOR ALL TABLE" -> "FOR ALL TABLES"
>
> Modified
>
> > 6b.
> > Consider using "EXCEPT" instead of "EXCEPT TABLE" because that will
> > show TABLE keyword is optional.
>
> Modified
>
> > ~~~
> >
> > 7. src/backend/catalog/pg_publication.c - GetTopMostAncestorInPublication
> >
> > @@ -316,18 +316,25 @@ GetTopMostAncestorInPublication(Oid puboid, List
> > *ancestors, int *ancestor_level
> > }
> > else
> > {
> > - aschemaPubids = GetSchemaPublications(get_rel_namespace(ancestor));
> > - if (list_member_oid(aschemaPubids, puboid))
> > + List*aschemapubids = NIL;
> > + List*aexceptpubids = NIL;
> > +
> > + aschemapubids = GetSchemaPublications(get_rel_namespace(ancestor));
> > + aexceptpubids = GetRelationPublications(ancestor, true);
> > + if (list_member_oid(aschemapubids, puboid) ||
> > + (puballtables && !list_member_oid(aexceptpubids, puboid)))
> > {
> >
> > You could re-write this as multiple conditions instead of one. That
> > could avoid always assigning the 'aexceptpubids', so it might be a
> > more efficient way to write this logic.
>
> Modified
>
> > ~~~
> >
> > 8. src/backend/catalog/pg_publication.c - CheckPublicationDefValues
> >
> > +/*
> > + * Check if the publication has default values
> > + *
> > + * Check the following:
> > + * Publication is having default options
> > + * Publication is not associated with relations
> > + * Publication is not associated with schemas
> > + * Publication is not set with "FOR ALL TABLES"
> > + */
> > +static bool
> > +CheckPublicationDefValues(HeapTuple t
Re: Skipping schema changes in publication
On Fri, May 20, 2022 at 11:23 AM Peter Smith wrote:
>
> Below are my review comments for v6-0002.
>
> ==
>
> 1. Commit message.
> The psql \d family of commands to display excluded tables.
>
> SUGGESTION
> The psql \d family of commands can now display excluded tables.
Modified
> ~~~
>
> 2. doc/src/sgml/ref/alter_publication.sgml
>
> @@ -22,6 +22,7 @@ PostgreSQL documentation
>
>
> ALTER PUBLICATION name
> ADD publication_object [,
> ...]
> +ALTER PUBLICATION name
> ADD ALL TABLES [ EXCEPT [ TABLE ] exception_object [, ... ] ]
>
> The "exception_object" font is wrong. Should look the same as
> "publication_object"
Modified
> ~~~
>
> 3. doc/src/sgml/ref/alter_publication.sgml - Examples
>
> @@ -214,6 +220,14 @@ ALTER PUBLICATION sales_publication ADD ALL
> TABLES IN SCHEMA marketing, sales;
>
>
>
> +
> + Alter publication production_publication to
> publish
> + all tables except users and
> + departments tables:
> +
> +ALTER PUBLICATION production_publication ADD ALL TABLES EXCEPT TABLE
> users, departments;
> +
>
> Consider using "EXCEPT" instead of "EXCEPT TABLE" because that will
> show TABLE keyword is optional.
Modified
> ~~~
>
> 4. doc/src/sgml/ref/create_publication.sgml
>
> An SGML tag error caused building the docs to fail. My fix was
> previously reported [1].
Modified
> ~~~
>
> 5. doc/src/sgml/ref/create_publication.sgml
>
> @@ -22,7 +22,7 @@ PostgreSQL documentation
>
>
> CREATE PUBLICATION name
> -[ FOR ALL TABLES
> +[ FOR ALL TABLES [ EXCEPT [ TABLE ] exception_object [, ... ] ]
>
> The "exception_object" font is wrong. Should look the same as
> "publication_object"
Modified
> ~~~
>
> 6. doc/src/sgml/ref/create_publication.sgml - Examples
>
> @@ -351,6 +366,15 @@ CREATE PUBLICATION production_publication FOR
> TABLE users, departments, ALL TABL
> CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing,
> sales;
>
>
> +
> + Create a publication that publishes all changes in all the tables except
> for
> + the changes of users and
> + departments table:
> +
> +CREATE PUBLICATION mypublication FOR ALL TABLE EXCEPT TABLE users,
> departments;
> +
> +
> +
>
> 6a.
> Typo: "FOR ALL TABLE" -> "FOR ALL TABLES"
Modified
> 6b.
> Consider using "EXCEPT" instead of "EXCEPT TABLE" because that will
> show TABLE keyword is optional.
Modified
> ~~~
>
> 7. src/backend/catalog/pg_publication.c - GetTopMostAncestorInPublication
>
> @@ -316,18 +316,25 @@ GetTopMostAncestorInPublication(Oid puboid, List
> *ancestors, int *ancestor_level
> }
> else
> {
> - aschemaPubids = GetSchemaPublications(get_rel_namespace(ancestor));
> - if (list_member_oid(aschemaPubids, puboid))
> + List*aschemapubids = NIL;
> + List*aexceptpubids = NIL;
> +
> + aschemapubids = GetSchemaPublications(get_rel_namespace(ancestor));
> + aexceptpubids = GetRelationPublications(ancestor, true);
> + if (list_member_oid(aschemapubids, puboid) ||
> + (puballtables && !list_member_oid(aexceptpubids, puboid)))
> {
>
> You could re-write this as multiple conditions instead of one. That
> could avoid always assigning the 'aexceptpubids', so it might be a
> more efficient way to write this logic.
Modified
> ~~~
>
> 8. src/backend/catalog/pg_publication.c - CheckPublicationDefValues
>
> +/*
> + * Check if the publication has default values
> + *
> + * Check the following:
> + * Publication is having default options
> + * Publication is not associated with relations
> + * Publication is not associated with schemas
> + * Publication is not set with "FOR ALL TABLES"
> + */
> +static bool
> +CheckPublicationDefValues(HeapTuple tup)
>
> 8a.
> Remove the tab. Replace with spaces.
Modified
> 8b.
> It might be better if this comment order is the same as the logic order.
> e.g.
>
> * Check the following:
> * Publication is not set with "FOR ALL TABLES"
> * Publication is having default options
> * Publication is not associated with schemas
> * Publication is not associated with relations
Modified
> ~~~
>
> 9. src/backend/catalog/pg_publication.c - AlterPublicationSetAllTables
>
> +/*
> + * Reset the publication.
> + *
> + * Reset the publication options, publication relations and
> publication schemas.
> + */
> +static void
> +AlterPublicationSetAllTables(Relation rel, HeapTuple tup)
>
> The function comment and the function name do not seem to match here;
> something looks like a cut/paste error ??
Modified
> ~~~
>
> 10. src/backend/catalog/pg_publication.c - AlterPublicationSetAllTables
>
> + /* set all tables option */
> + values[Anum_pg_publication_puballtables - 1] = BoolGetDatum(true);
> + replaces[Anum_pg_publication_puballtables - 1] = true;
>
> SUGGEST (comment)
> /* set all ALL TABLES flag */
Modified
> ~~~
>
> 11. src/backend/catalog/pg_publication.c - AlterPublication
>
> @@ -1501,6 +1579,20 @@ AlterPublication(ParseState *pstate,
> AlterPublicationStmt *stmt)
> aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_PUBLICATION,
> stmt->pubn
Re: Skipping schema changes in publication
On Fri, May 20, 2022 at 5:49 AM Peter Smith wrote:
>
> FYI, although the v6-0002 patch applied cleanly, I found that the SGML
> was malformed and so the pg docs build fails.
>
> ~~~
> e.g.
>
> [postgres@CentOS7-x64 sgml]$ make STYLE=website html
> { \
> echo ""; \
> echo ""; \
> } > version.sgml
> '/usr/bin/perl' ./mk_feature_tables.pl YES
> ../../../src/backend/catalog/sql_feature_packages.txt
> ../../../src/backend/catalog/sql_features.txt >
> features-supported.sgml
> '/usr/bin/perl' ./mk_feature_tables.pl NO
> ../../../src/backend/catalog/sql_feature_packages.txt
> ../../../src/backend/catalog/sql_features.txt >
> features-unsupported.sgml
> '/usr/bin/perl' ./generate-errcodes-table.pl
> ../../../src/backend/utils/errcodes.txt > errcodes-table.sgml
> '/usr/bin/perl' ./generate-keywords-table.pl . > keywords-table.sgml
> /usr/bin/xmllint --path . --noout --valid postgres.sgml
> ref/create_publication.sgml:171: parser error : Opening and ending tag
> mismatch: varlistentry line 166 and listitem
>
>^
> ref/create_publication.sgml:172: parser error : Opening and ending tag
> mismatch: variablelist line 60 and varlistentry
>
> ^
> ref/create_publication.sgml:226: parser error : Opening and ending tag
> mismatch: refsect1 line 57 and variablelist
>
> ^
> ...
>
> I will work around it locally, but for future patches please check the
> SGML builds ok before posting.
Thanks for reporting this, I have made the changes for this.
The v7 patch attached at [1] has the changes for the same.
[1] -
https://www.postgresql.org/message-id/CALDaNm3EpX3%2BRu%3DSNaYi%3DUW5ZLE6nNhGRHZ7a8-fXPZ_-gLdxQ%40mail.gmail.com
Regards,
Vignesh
Re: Skipping schema changes in publication
On Thu, May 19, 2022 at 1:49 PM Peter Smith wrote:
>
> Below are my review comments for v6-0001.
>
> ==
>
> 1. General.
>
> The patch failed 'publication' tests in the make check phase.
>
> Please add this work to the commit-fest so that the 'cfbot' can report
> such errors sooner.
Added commitfest entry
> ~~~
>
> 2. src/backend/commands/publicationcmds.c - AlterPublicationReset
>
> +/*
> + * Reset the publication.
> + *
> + * Reset the publication options, publication relations and
> publication schemas.
> + */
> +static void
> +AlterPublicationReset(ParseState *pstate, AlterPublicationStmt *stmt,
> + Relation rel, HeapTuple tup)
>
> SUGGESTION (Make the comment similar to the sgml text instead of
> repeating "publication" 4x !)
> /*
> * Reset the publication options, set the ALL TABLES flag to false, and
> * drop all relations and schemas that are associated with the publication.
> */
Modified
> ~~~
>
> 3. src/test/regress/expected/publication.out
>
> make check failed. The diff is below:
>
> @@ -1716,7 +1716,7 @@
> -- Verify that only superuser can reset a publication
> ALTER PUBLICATION testpub_reset OWNER TO regress_publication_user2;
> SET ROLE regress_publication_user2;
> -ALTER PUBLICATION testpub_reset RESET; -- fail
> +ALTER PUBLICATION testpub_reset RESET; -- fail - must be superuser
> ERROR: must be superuser to RESET publication
> SET ROLE regress_publication_user;
> DROP PUBLICATION testpub_reset;
It passed for me locally because the change was present in the 002
patch. I have moved the change to 001.
The attached v7 patch has the changes for the same.
[1] - https://commitfest.postgresql.org/38/3646/
Regards,
Vignesh
From 1855e00f2d6cc19427c55eec2f1e40ecc8f8c1cc Mon Sep 17 00:00:00 2001
From: Vigneshwaran C
Date: Sat, 14 May 2022 13:13:46 +0530
Subject: [PATCH v7 1/2] Add RESET clause to Alter Publication which will reset
the publication with default values.
This patch adds a new RESET clause to ALTER PUBLICATION which will reset
the publication to the default state which includes resetting the publication
options, setting ALL TABLES flag to false and dropping the relations and
schemas that are associated with the publication.
Usage:
ALTER PUBLICATION pub1 RESET;
---
doc/src/sgml/ref/alter_publication.sgml | 38 ++--
src/backend/commands/publicationcmds.c| 100 --
src/backend/parser/gram.y | 9 ++
src/bin/psql/tab-complete.c | 2 +-
src/include/nodes/parsenodes.h| 3 +-
src/test/regress/expected/publication.out | 69 +++
src/test/regress/sql/publication.sql | 37
7 files changed, 242 insertions(+), 16 deletions(-)
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index e2cce49471..47bd15f1fa 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -27,6 +27,7 @@ ALTER PUBLICATION name DROP name SET ( publication_parameter [= value] [, ... ] )
ALTER PUBLICATION name OWNER TO { new_owner | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION name RENAME TO new_name
+ALTER PUBLICATION name RESET
where publication_object is one of:
@@ -65,20 +66,33 @@ ALTER PUBLICATION name RENAME TO
- The remaining variants change the owner and the name of the publication.
+ The OWNER clause will change the owner of the publication.
+
+
+
+ The RENAME clause will change the name of the publication.
+
+
+
+ The RESET clause will reset the publication to the
+ default state which includes resetting the publication options, setting
+ ALL TABLES flag to false and
+ dropping all relations and schemas that are associated with the publication.
You must own the publication to use ALTER PUBLICATION.
Adding a table to a publication additionally requires owning that table.
- The ADD ALL TABLES IN SCHEMA and
- SET ALL TABLES IN SCHEMA to a publication requires the
- invoking user to be a superuser. To alter the owner, you must also be a
- direct or indirect member of the new owning role. The new owner must have
- CREATE privilege on the database. Also, the new owner
- of a FOR ALL TABLES or FOR ALL TABLES IN
- SCHEMA publication must be a superuser. However, a superuser can
- change the ownership of a publication regardless of these restrictions.
+ The ADD ALL TABLES IN SCHEMA,
+ SET ALL TABLES IN SCHEMA to a publication and
+ RESET of publication requires the invoking user to be a
+ superuser. To alter the owner, you must also be a direct or indirect member
+ of the new owning role. The new owner must have CREATE
+ privilege on the database. Also, the new owner of a
+ FOR ALL TABLES or
+ FOR ALL TABLES IN SCHEMA publication must be a superuser.
+ However, a superuser can change the ownership of a publication regardless of
+ these restrictions.
@@ -207,6 +221,12 @@ ALTER PUBLICATION sales_publ
Re: Handle infinite recursion in logical replication setup
On Thu, May 19, 2022 at 2:12 PM shiy.f...@fujitsu.com wrote: > > On Wed, May 4, 2022 2:47 PM vignesh C wrote: > > > > Thanks for the comments, the attached v13 patch has the changes for the > > same. > > > > Here are some comments on v13-0002 patch. > > 1) > +* Throw an error so that the user can take care of the > initial data > +* copying and then create subscription with copy_data off. > > Should "with copy_data off" be changed to "with copy_data off or force"? Modified > 2) > case ALTER_SUBSCRIPTION_ADD_PUBLICATION: > case ALTER_SUBSCRIPTION_DROP_PUBLICATION: > ... > /* > * See ALTER_SUBSCRIPTION_REFRESH for > details why this is > * not allowed. > */ > if (sub->twophasestate == > LOGICALREP_TWOPHASE_STATE_ENABLED && opts.copy_data) > > I think we need some changes here, too. Should it be modified to: > if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && > IS_COPY_DATA_ON_OR_FORCE(opts.copy_data)) Modified Thanks for the comments, the v14 patch attached at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm0xuYy35vOudVHBjov3fQ%3DjBRHJHKUUN9VarqO%3DYqtaxg%40mail.gmail.com Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Wed, May 18, 2022 at 4:22 PM Amit Kapila wrote: > > On Wed, May 18, 2022 at 10:29 AM Amit Kapila wrote: > > > > On Wed, May 4, 2022 at 12:17 PM vignesh C wrote: > > > > > > Thanks for the comments, the attached v13 patch has the changes for the > > > same. > > > > > > > Few comments on v13-0001 > > == > > > > Few comments on v13-0002 > === > 1. > The steps > + to create a two-node bidirectional replication are given below: > + > > The steps given after this will be successful only when there is no > data in any of the nodes and that is not clear by reading docs. Modified > 2. > + > + Adding a new node when data is present in the existing > nodes > + > + Adding a new node node3 to the existing > + node1 and node2 when data is > present > + in existing nodes node1 and node2 > + needs similar steps. The only change required here is that > + node3 should create a subscription with > + copy_data = force to one of the existing nodes to > + receive the existing data during initial data synchronization. > + > > I think the steps for these require the user to lock the required > tables/database (or in some other way hold the operations on required > tables) for node-2 till the time setup is complete, otherwise, node-3 > might miss some data. It seems the same is also missing in the > section: "Adding a new node when data is present in the new node". Modified. Mentioned that lock is required on new node-3 and node-2. I have mentioned node-3 also should be locked so that no operation is happened in node-3, else there can be some dml operation after truncate which is sent to node-1 and the same data is synced when create subscription in node-3 subscribing to the publisher in node-1. > 3. > + > + > + Adding a new node when data is present in the new node > ... > ... > > + > +Create a subscription in node1 to subscribe to > +node3. Use copy_data specified as > +force so that the existing table data is copied during > +initial sync: > + > +node1=# CREATE SUBSCRIPTION sub_node1_node3 > +node1-# CONNECTION 'dbname=foo host=node3 user=repuser' > +node1-# PUBLICATION pub_node3 > +node1-# WITH (copy_data = force, local_only = on); > +CREATE SUBSCRIPTION > + > + > + > +Create a subscription in node2 to subscribe to > +node3. Use copy_data specified as > +force so that the existing table data is copied during > +initial sync: > + > +node2=# CREATE SUBSCRIPTION sub_node2_node3 > +node2-# CONNECTION 'dbname=foo host=node3 user=repuser' > +node2-# PUBLICATION pub_node3 > +node2-# WITH (copy_data = force, local_only = on); > +CREATE SUBSCRIPTION > + > > Why do we need to use "copy_data = force" here? AFAIU, unless, we > create any subscription on node-3, we don't need the 'force' option. Modified to on. > 4. We should have a generic section to explain how users can add a new > node using the new options to the existing set of nodes in all cases. > For example, say when the existing set of nodes has some data and the > new node also has some pre-existing data. I think the basic steps are > something like: a. create a required publication(s) on the new node. > (b) create subscriptions on existing nodes pointing to publication on > the new node with the local_only option as true and copy_data = on. > (c) wait for data to be copied from the new node to existing nodes. > (d) Truncate the data on the new node. (e) create subscriptions > corresponding to each of the existing publisher nodes on the new node > with local_only as true and copy_data = force for one of the nodes. > (f) One needs to ensure that there is no new activity on required > tables/database (either by locking or in some other way) in all the > nodes for which copy_data option is kept as false while creating > subscriptions in the previous step to avoid any data loss. Added > We also need to mention in Notes that as all operations are not > transactional, user is advised to take backup of existing data to > avoid any inconsistency. Modified > 5. > * It is quite possible that subscriber has not yet pulled data to > + * the tables, but in ideal cases the table data will be subscribed. > + * To keep the code simple it is not checked if the subscriber table > + * has pulled the data or not. > + */ > + if (copydata == COPY_DATA_ON && local_only && !slot_attisnull(slot, 3)) > > Sorry, but I don't understand what you intend to say by the above > comment. Can you please explain? When the user specif
Re: Handle infinite recursion in logical replication setup
On Wed, May 18, 2022 at 1:40 PM shiy.f...@fujitsu.com wrote: > > On Wed, May 4, 2022 2:47 PM vignesh C wrote: > > > > Thanks for the comments, the attached v13 patch has the changes for the > > same. > > > > Thanks for your patch. Here are some comments on v13-0001 patch. > > 1) > @@ -152,6 +152,18 @@ CREATE SUBSCRIPTION class="parameter">subscription_name > > > + > +local_only (boolean) > + > + > + Specifies whether the subscription will request the publisher to > send > + locally originated changes at the publisher node, or send any > + publisher node changes regardless of their origin. The default is > + false. > + > + > + > + > > slot_name (string) > > > I think this change should be put after "The following parameters control the > subscription's replication behavior after it has been created", thoughts? Modified > 2) > A new column "sublocalonly" is added to pg_subscription, so maybe we need add > it > to pg_subscription document, too. (in doc/src/sgml/catalogs.sgml) Modified > 3) > /* > * Currently we always forward. > */ > static bool > pgoutput_origin_filter(LogicalDecodingContext *ctx, >RepOriginId origin_id) > > Should we modify the comment of pgoutput_origin_filter()? It doesn't match the > code. Modified Thanks for the comments, the v14 patch attached at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm0xuYy35vOudVHBjov3fQ%3DjBRHJHKUUN9VarqO%3DYqtaxg%40mail.gmail.com Regards, Vignesh
Re: Skipping schema changes in publication
On Tue, May 17, 2022 at 7:35 AM Peter Smith wrote: > > Below are my review comments for v5-0002. > > There may be an overlap with comments recently posted by Osumi-san [1]. > > (I also have review comments for v5-0002; will post them tomorrow) > > == > > 1. General > > Is it really necessary to have to say "EXCEPT TABLE" instead of just > "EXCEPT". It seems unnecessarily verbose and redundant when you write > "FOR ALL TABLES EXCEPT TABLE...". > > If you want to keep this TABLE keyword (maybe you have plans for other > kinds of except?) then IMO perhaps at least it can be the optional > default except type. e.g. EXCEPT [TABLE]. I have made TABLE optional. > ~~~ > > 2. General > > (I was unsure whether to even mention this one). > > I understand the "EXCEPT" is chosen as the user-facing syntax, but it > still seems strange when reading the patch to see attribute members > and column names called 'except'. I think the problem is that "except" > is not a verb, so saying except=t/f just does not make much sense. > Sometimes I feel that for all the internal usage > (code/comments/catalog) using "skip" and "skip-list" etc would be a > much better choice of names. OTOH I can see that having consistency > with the outside syntax might also be good. Anyway, please consider - > maybe other people feel the same? Earlier we had discussed whether to use SKIP, but felt SKIP was not appropriate and planned to use except as in [1]. Let's use except unless we find a better alternative. > ~~~ > > 3. General > > The ONLY keyword seems supported by the syntax for tables of the > except-list (more on this in later comments) but: > a) I am not sure if the patch code is accounting for that, and I have kept the behavior similar to FOR TABLE > b) There are no test cases using ONLY. Added tests for the same > ~~~ > > 4. Commit message > > A new option "EXCEPT TABLE" in Create/Alter Publication allows > one or more tables to be excluded, publisher will exclude sending the data > of the excluded tables to the subscriber. > > SUGGESTION > A new "EXCEPT TABLE" clause for CREATE/ALTER PUBLICATION allows one or > more tables to be excluded. The publisher will not send the data of > excluded tables to the subscriber. Modified > ~~ > > 5. Commit message > > The new syntax allows specifying exclude relations while creating a > publication > or exclude relations in alter publication. For example: > > SUGGESTION > The new syntax allows specifying excluded relations when creating or > altering a publication. For example: Modified > ~~~ > > 6. Commit message > > A new column prexcept is added to table "pg_publication_rel", to maintain > the relations that the user wants to exclude publishing through the > publication. > > SUGGESTION > A new column "prexcept" is added to table "pg_publication_rel", to > maintain the relations that the user wants to exclude from the > publications. Modified > ~~~ > > 7. Commit message > > Modified the output plugin (pgoutput) to exclude publishing the changes of the > excluded tables. > > I did not feel it was necessary to say this. It is already said above > that the data is not sent, so that seems enough. Modified > ~~~ > > 8. Commit message > > Updates pg_dump to identify and dump the excluded tables of the publications. > Updates the \d family of commands to display excluded tables of the > publications and \dRp+ variant will now display associated except tables if > any. > > SUGGESTION > pg_dump is updated to identify and dump the excluded tables of the > publications. > > The psql \d family of commands to display excluded tables. e.g. psql > \dRp+ variant will now display associated "except tables" if any. Modified > ~~~ > > 9. doc/src/sgml/catalogs.sgml > > @@ -6426,6 +6426,15 @@ SCRAM-SHA-256$count>:&l >if there is no publication qualifying condition. > > > + > + > + prexcept bool > + > + > + True if the table must be excluded > + > + > > Other descriptions on this page refer to "relation" instead of > "table". Probably this should do the same to be consistent. Modified > ~~~ > > 10. doc/src/sgml/logical-replication.sgml > > @@ -1167,8 +1167,9 @@ CONTEXT: processing remote data for replication > origin "pg_16395" during "INSER > > To add tables to a publication, the user must have ownership rights on the > table. To add all tables in schema to a publication, the user must be a > - superuser. To create a publication that publishes all tables or > all tables in > - schema automatically, the user must be a superuser. > + superuser. To add all tables to a publication, the user must be a > superuser. > + To create a publication that publishes all tables or all tables in schema > + automatically, the user must be a superuser. > > > It seems like a valid change but how is this related to this EXCEPT > patch. Maybe this fix should be patched separately? Earlier we were not allowed to a
Re: Skipping schema changes in publication
On Mon, May 16, 2022 at 2:00 PM osumi.takami...@fujitsu.com
wrote:
>
> On Saturday, May 14, 2022 10:33 PM vignesh C wrote:
> > Thanks for the comments, the attached v5 patch has the changes for the same.
> > Also I have made the changes for SKIP Table based on the new syntax, the
> > changes for the same are available in
> > v5-0002-Skip-publishing-the-tables-specified-in-EXCEPT-TA.patch.
> Hi,
>
>
>
> Several comments on v5-0002.
>
> (1) One unnecessary space before "except_pub_obj_list" syntax definition
>
> + except_pub_obj_list: ExceptPublicationObjSpec
> + { $$ = list_make1($1); }
> + | except_pub_obj_list ',' ExceptPublicationObjSpec
> + { $$ = lappend($1, $3); }
> + | /*EMPTY*/
> { $$ = NULL; }
> + ;
> +
>
> From above part, kindly change
> FROM:
> " except_pub_obj_list: ExceptPublicationObjSpec"
> TO:
> "except_pub_obj_list: ExceptPublicationObjSpec"
>
Modified
> (2) doc/src/sgml/ref/create_publication.sgml
>
> (2-1)
>
> @@ -22,7 +22,7 @@ PostgreSQL documentation
>
>
> CREATE PUBLICATION name
> -[ FOR ALL TABLES
> +[ FOR ALL TABLES [EXCEPT TABLE [ ONLY ] class="parameter">table_name [ * ] [, ... ]]
>| FOR publication_object
> [, ... ] ]
> [ WITH ( class="parameter">publication_parameter [= class="parameter">value] [, ... ] ) ]
>
>
> Here I think we need to add two more whitespaces around square brackets.
> Please change
> FROM:
> "[ FOR ALL TABLES [EXCEPT TABLE [ ONLY ] class="parameter">table_name [ * ] [, ... ]]"
> TO:
> "[ FOR ALL TABLES [ EXCEPT TABLE [ ONLY ] class="parameter">table_name [ * ] [, ... ] ]"
>
> When I check other documentations, I see whitespaces before/after square
> brackets.
>
> (2-2)
> This whitespace alignment applies to alter_publication.sgml as well.
Modified
> (3)
>
>
> @@ -156,6 +156,24 @@ CREATE PUBLICATION class="parameter">name
>
>
>
> +
> +
> +EXCEPT TABLE
> +
> +
> + Marks the publication as one that excludes replicating changes for the
> + specified tables.
> +
> +
> +
> + EXCEPT TABLE can be specified only for
> + FOR ALL TABLES publication. It is not supported for
> + FOR ALL TABLES IN SCHEMA publication and
> + FOR TABLE publication.
> +
> +
> +
> +
>
> This EXCEPT TABLE clause is only for FOR ALL TABLES.
> So, how about extracting the main message from above part and
> moving it to an exising paragraph below, instead of having one independent
> paragraph ?
>
>
> FOR ALL TABLES
>
>
> Marks the publication as one that replicates changes for all tables in
> the database, including tables created in the future.
>
>
>
>
> Something like
> "Marks the publication as one that replicates changes for all tables in
> the database, including tables created in the future. EXCEPT TABLE indicates
> excluded tables for the defined publication.
> "
>
Modified
> (4) One minor confirmation about the syntax
>
> Currently, we allow one way of writing to indicate excluded tables like below.
>
> (example) CREATE PUBLICATION mypub FOR ALL TABLES EXCEPT TABLE tab3, tab4,
> EXCEPT TABLE tab5;
>
> This is because we define ExceptPublicationObjSpec with EXCEPT TABLE.
> Is it OK to have a room to write duplicate "EXCEPT TABLE" clauses ?
> I think there is no harm in having this,
> but I'd like to confirm whether this syntax might be better to be adjusted or
> not.
Changed it to allow except table only once
>
> (5) CheckAlterPublication
>
> +
> + if (excepttable && !stmt->for_all_tables)
> + ereport(ERROR,
> +
> (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> +errmsg("publication \"%s\" is not defined as
> FOR ALL TABLES",
> + NameStr(pubform->pubname)),
> +errdetail("except table cannot be added to,
> dropped from, or set on NON ALL TABLES publications.")));
>
> Could you please add a test for this ?
This code can be removed because of grammar optimization, it will not
allow tables without "ALL TABLES". Removed this code
The v6 patch attached at [1] has the changes for the same.
[1] -
https://www.postgresql.org/message-id/CALDaNm0iZZDB300Dez_97S8G6_RW5QpQ8ef6X3wq8tyK-8wnXQ%40mail.gmail.com
Regards,
Vignesh
Re: Skipping schema changes in publication
On Wed, May 18, 2022 at 8:30 AM shiy.f...@fujitsu.com
wrote:
>
> On Sat, May 14, 2022 9:33 PM vignesh C wrote:
> >
> > Thanks for the comments, the attached v5 patch has the changes for the
> > same. Also I have made the changes for SKIP Table based on the new
> > syntax, the changes for the same are available in
> > v5-0002-Skip-publishing-the-tables-specified-in-EXCEPT-TA.patch.
> >
>
> Thanks for your patch. Here are some comments on v5-0001 patch.
>
> + Oid relid = lfirst_oid(lc);
> +
> + prid = GetSysCacheOid2(PUBLICATIONRELMAP,
> Anum_pg_publication_rel_oid,
> +
> ObjectIdGetDatum(relid),
> +
> ObjectIdGetDatum(pubid));
> + if (!OidIsValid(prid))
> + ereport(ERROR,
> + (errcode(ERRCODE_UNDEFINED_OBJECT),
> +errmsg("relation \"%s\" is not part
> of the publication",
> +
> RelationGetRelationName(rel;
>
> I think the relation in the error message should be the one whose oid is
> "relid", instead of relation "rel".
Modified it
> Besides, I think it might be better not to report an error in this case. If
> "prid" is invalid, just ignore this relation. Because in RESET cases, we want
> to
> drop all tables in the publications, and there is no specific table.
> (If you agree with that, similarly missing_ok should be set to true when
> calling
> PublicationDropSchemas().)
Ideally this scenario should not happen, but if it happens I felt we
should throw an error in this case.
The v6 patch attached at [1] has the changes for the same.
[1] -
https://www.postgresql.org/message-id/CALDaNm0iZZDB300Dez_97S8G6_RW5QpQ8ef6X3wq8tyK-8wnXQ%40mail.gmail.com
Regards,
Vignesh
Re: Skipping schema changes in publication
On Mon, May 16, 2022 at 2:53 PM Peter Smith wrote: > > Below are my review comments for v5-0001. > > There is some overlap with comments recently posted by Osumi-san [1]. > > (I also have review comments for v5-0002; will post them tomorrow) > > == > > 1. Commit message > > This patch adds a new RESET clause to ALTER PUBLICATION which will reset > the publication to default state which includes resetting the publication > options, setting ALL TABLES option to false and dropping the relations and > schemas that are associated with the publication. > > SUGGEST > "to default state" -> "to the default state" > "ALL TABLES option" -> "ALL TABLES flag" Modified > ~~~ > > 2. doc/src/sgml/ref/alter_publication.sgml > > + > + The RESET clause will reset the publication to the > + default state which includes resetting the publication options, setting > + ALL TABLES option to false and > + dropping all relations and schemas that are associated with the > publication. > > > "ALL TABLES option" -> "ALL TABLES flag" Modified > ~~~ > > 3. doc/src/sgml/ref/alter_publication.sgml > > + invoking user to be a superuser. RESET of publication > + requires the invoking user to be a superuser. To alter the owner, you must > > SUGGESTION > To RESET a publication requires the invoking user > to be a superuser. I have combined it with the earlier sentence. > ~~~ > > 4. src/backend/commands/publicationcmds.c > > @@ -53,6 +53,13 @@ > #include "utils/syscache.h" > #include "utils/varlena.h" > > +#define PUB_ATION_INSERT_DEFAULT true > +#define PUB_ACTION_UPDATE_DEFAULT true > +#define PUB_ACTION_DELETE_DEFAULT true > +#define PUB_ACTION_TRUNCATE_DEFAULT true > +#define PUB_VIA_ROOT_DEFAULT false > +#define PUB_ALL_TABLES_DEFAULT false > > 4a. > Typo: "ATION" -> "ACTION" Modified > 4b. > I think these #defines deserve a 1 line comment. > e.g. > /* CREATE PUBLICATION default values for flags and options */ Added comment > 4c. > Since the "_DEFAULT" is a common part of all the names, maybe it is > tidier if it comes first. > e.g. > #define PUB_DEFAULT_ACTION_INSERT true > #define PUB_DEFAULT_ACTION_UPDATE true > #define PUB_DEFAULT_ACTION_DELETE true > #define PUB_DEFAULT_ACTION_TRUNCATE true > #define PUB_DEFAULT_VIA_ROOT false > #define PUB_DEFAULT_ALL_TABLES false Modified The v6 patch attached at [1] has the changes for the same. [1] - https://www.postgresql.org/message-id/CALDaNm0iZZDB300Dez_97S8G6_RW5QpQ8ef6X3wq8tyK-8wnXQ%40mail.gmail.com Regards, Vignesh
Re: Skipping schema changes in publication
On Mon, May 16, 2022 at 8:32 AM osumi.takami...@fujitsu.com
wrote:
>
> On Saturday, May 14, 2022 10:33 PM vignesh C wrote:
> > Thanks for the comments, the attached v5 patch has the changes for the same.
> > Also I have made the changes for SKIP Table based on the new syntax, the
> > changes for the same are available in
> > v5-0002-Skip-publishing-the-tables-specified-in-EXCEPT-TA.patch.
> Hi,
>
>
> Thank you for updating the patch.
> I'll share few minor review comments on v5-0001.
>
>
> (1) doc/src/sgml/ref/alter_publication.sgml
>
> @@ -73,12 +85,13 @@ ALTER PUBLICATION class="parameter">name RENAME TO Adding a table to a publication additionally requires owning that table.
> The ADD ALL TABLES IN SCHEMA and
> SET ALL TABLES IN SCHEMA to a publication requires the
> - invoking user to be a superuser. To alter the owner, you must also be a
> - direct or indirect member of the new owning role. The new owner must have
> - CREATE privilege on the database. Also, the new owner
> - of a FOR ALL TABLES or FOR ALL TABLES IN
> - SCHEMA publication must be a superuser. However, a superuser can
> - change the ownership of a publication regardless of these restrictions.
> + invoking user to be a superuser. RESET of publication
> + requires the invoking user to be a superuser. To alter the owner, you must
> ...
>
>
> I suggest to combine the first part of your change with one existing sentence
> before your change, to make our description concise.
>
> FROM:
> "The ADD ALL TABLES IN SCHEMA and
> SET ALL TABLES IN SCHEMA to a publication requires the
> invoking user to be a superuser. RESET of publication
> requires the invoking user to be a superuser."
>
> TO:
> "The ADD ALL TABLES IN SCHEMA,
> SET ALL TABLES IN SCHEMA to a publication and
> RESET of publication requires the invoking user to be a
> superuser."
Modified
>
> (2) typo
>
> +++ b/src/backend/commands/publicationcmds.c
> @@ -53,6 +53,13 @@
> #include "utils/syscache.h"
> #include "utils/varlena.h"
>
> +#define PUB_ATION_INSERT_DEFAULT true
> +#define PUB_ACTION_UPDATE_DEFAULT true
>
>
> Kindly change
> FROM:
> "PUB_ATION_INSERT_DEFAULT"
> TO:
> "PUB_ACTION_INSERT_DEFAULT"
Modified
>
> (3) src/test/regress/expected/publication.out
>
> +-- Verify that only superuser can reset a publication
> +ALTER PUBLICATION testpub_reset OWNER TO regress_publication_user2;
> +SET ROLE regress_publication_user2;
> +ALTER PUBLICATION testpub_reset RESET; -- fail
>
>
> We have "-- fail" for one case in this patch.
> On the other hand, isn't better to add "-- ok" (or "-- success") for
> other successful statements,
> when we consider the entire tests description consistency ?
We generally do not mention success comments for all the success cases
as that might be an overkill. I felt it is better to keep it as it is.
Thoughts?
The attached v6 patch has the changes for the same.
Regards,
Vignesh
From 72f81a7c56ef764f13949721e227541acf6b719c Mon Sep 17 00:00:00 2001
From: Vigneshwaran C
Date: Sat, 14 May 2022 13:13:46 +0530
Subject: [PATCH v6 1/2] Add RESET clause to Alter Publication which will reset
the publication with default values.
This patch adds a new RESET clause to ALTER PUBLICATION which will reset
the publication to the default state which includes resetting the publication
options, setting ALL TABLES flag to false and dropping the relations and
schemas that are associated with the publication.
Usage:
ALTER PUBLICATION pub1 RESET;
---
doc/src/sgml/ref/alter_publication.sgml | 38 ++---
src/backend/commands/publicationcmds.c| 99 +--
src/backend/parser/gram.y | 9 +++
src/bin/psql/tab-complete.c | 2 +-
src/include/nodes/parsenodes.h| 3 +-
src/test/regress/expected/publication.out | 69
src/test/regress/sql/publication.sql | 37 +
7 files changed, 241 insertions(+), 16 deletions(-)
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index e2cce49471..47bd15f1fa 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -27,6 +27,7 @@ ALTER PUBLICATION name DROP name SET ( publication_parameter [= value] [, ... ] )
ALTER PUBLICATION name OWNER TO { new_owner | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION name RENAME TO new_name
+ALTER PUBLICATION name RESET
where publication_object is one of:
@@ -65,20 +66,33 @@ ALTER PUBLICATION name RENAME TO
- The remaining variants change the owner and the name of the publication.
+ The OWNER clause will change
Re: Skipping schema changes in publication
On Fri, May 13, 2022 at 9:37 AM Peter Smith wrote:
>
> On Thu, May 12, 2022 at 2:24 PM vignesh C wrote:
> >
> ...
> > The attached patch has the implementation for "ALTER PUBLICATION
> > pubname RESET". This command will reset the publication to default
> > state which includes resetting the publication options, setting ALL
> > TABLES option to false and dropping the relations and schemas that are
> > associated with the publication.
> >
>
> Please see below my review comments for the v1-0001 (RESET) patch
>
> ==
>
> 1. Commit message
>
> This patch adds a new RESET option to ALTER PUBLICATION which
>
> Wording: "RESET option" -> "RESET clause"
Modified
> ~~~
>
> 2. doc/src/sgml/ref/alter_publication.sgml
>
> +
> + The RESET clause will reset the publication to default
> + state which includes resetting the publication options, setting
> + ALL TABLES option to false
> and drop the
> + relations and schemas that are associated with the publication.
>
>
> 2a. Wording: "to default state" -> "to the default state"
Modified
> 2b. Wording: "and drop the relations..." -> "and dropping all relations..."
Modified
> ~~~
>
> 3. doc/src/sgml/ref/alter_publication.sgml
>
> + invoking user to be a superuser. RESET of publication
> + requires invoking user to be a superuser. To alter the owner, you must
> also
>
> Wording: "requires invoking user" -> "requires the invoking user"
Modified
> ~~~
>
> 4. doc/src/sgml/ref/alter_publication.sgml - Example
>
> @@ -207,6 +220,12 @@ ALTER PUBLICATION sales_publication ADD ALL
> TABLES IN SCHEMA marketing, sales;
> production_publication:
>
> ALTER PUBLICATION production_publication ADD TABLE users,
> departments, ALL TABLES IN SCHEMA production;
> +
> +
> +
> + Resetting the publication production_publication:
> +
> +ALTER PUBLICATION production_publication RESET;
>
> Wording: "Resetting the publication" -> "Reset the publication"
Modified
> ~~~
>
> 5. src/backend/commands/publicationcmds.c
>
> + /* Check and reset the options */
>
> IMO the code can just reset all these options unconditionally. I did
> not see the point to check for existing option values first. I feel
> the simpler code outweighs any negligible performance difference in
> this case.
Modified
> ~~~
>
> 6. src/backend/commands/publicationcmds.c
>
> + /* Check and reset the options */
>
> Somehow it seemed a pity having to hardcode all these default values
> true/false in multiple places; e.g. the same is already hardcoded in
> the parse_publication_options function.
>
> To avoid multiple hard coded bools you could just call the
> parse_publication_options with an empty options list. That would set
> the defaults which you can then use:
> values[Anum_pg_publication_pubinsert - 1] =
> BoolGetDatum(pubactiondefs->insert);
>
> Alternatively, maybe there should be #defines to use instead of having
> the scattered hardcoded bool defaults:
> #define PUBACTION_DEFAULT_INSERT true
> #define PUBACTION_DEFAULT_UPDATE true
> etc
I have used #define for default value and used it in both the functions.
> ~~~
>
> 7. src/include/nodes/parsenodes.h
>
> @@ -4033,7 +4033,8 @@ typedef enum AlterPublicationAction
> {
> AP_AddObjects, /* add objects to publication */
> AP_DropObjects, /* remove objects from publication */
> - AP_SetObjects /* set list of objects */
> + AP_SetObjects, /* set list of objects */
> + AP_ReSetPublication /* reset the publication */
> } AlterPublicationAction;
>
> Unusual case: "AP_ReSetPublication" -> "AP_ResetPublication"
Modified
> ~~~
>
> 8. src/test/regress/sql/publication.sql
>
> 8a.
> +-- Test for RESET PUBLICATION
> SUGGESTED
> +-- Tests for ALTER PUBLICATION ... RESET
Modified
> 8b.
> +-- Verify that 'ALL TABLES' option is reset
> SUGGESTED:
> +-- Verify that 'ALL TABLES' flag is reset
Modified
> 8c.
> +-- Verify that publish option and publish via root option is reset
> SUGGESTED:
> +-- Verify that publish options and publish_via_partition_root option are
> reset
Modified
> 8d.
> +-- Verify that only superuser can execute RESET publication
> SUGGESTED
> +-- Verify that only superuser can reset a publication
Modified
Thanks for the comments, the attached v5 patch has the changes for the
same. Also I have made the changes for SKIP Table based on the new
syntax, the changes for the same are available in
v5-0002-Skip-publishing-the-tables-specified-in-EXCEPT-TA.patch.
R
Re: First draft of the PG 15 release notes
I wonder if this is worth mentioning: Raise a WARNING for missing publications. commit 8f2e2bbf145384784bad07a96d461c6bbd91f597 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=8f2e2bbf145384784bad07a96d461c6bbd91f597 Regards, Vignesh On Thu, May 12, 2022 at 7:52 PM Bruce Momjian wrote: > > On Thu, May 12, 2022 at 11:12:54AM -0300, Euler Taveira wrote: > OB> On Thu, May 12, 2022, at 11:03 AM, Bruce Momjian wrote: > > > > I looked at that but thought that everyone would already assume we > > skipped replication of empty transactions, and I didn't see much impact > > for the user, so I didn't include it. > > > > It certainly has an impact on heavy workloads that replicate tables with few > > modifications. It receives a high traffic of 'begin' and 'commit' messages > > that > > the previous Postgres versions have to handle (discard). I would classify > > it as > > a performance improvement for logical replication. Don't have a strong > > opinion > > if it should be mentioned or not. > > Oh, so your point is that a transaction that only has SELECT would > previously send an empty transaction? I thought this was only for apps > that create literal empty transactions, which seem rare. > > -- > Bruce Momjian https://momjian.us > EDB https://enterprisedb.com > > Indecision is a decision. Inaction is an action. Mark Batterson > > >
Re: Skipping schema changes in publication
On Tue, May 10, 2022 at 9:08 AM vignesh C wrote:
>
> On Fri, May 6, 2022 at 8:05 AM Peter Smith wrote:
> >
> > On Thu, Apr 28, 2022 at 9:32 PM Amit Kapila wrote:
> > >
> > ...
> > >
> > > Another idea that occurred to me today for tables this is as follows:
> > > 1. Allow to mention except during create publication ... For All Tables.
> > > CREATE PUBLICATION pub1 FOR ALL TABLES EXCEPT TABLE t1,t2;
> > > 2. Allow to Reset it. This new syntax will reset all objects in the
> > > publications.
> > > Alter Publication ... RESET;
> > > 3. Allow to add it to an existing publication
> > > Alter Publication ... Add ALL TABLES [EXCEPT TABLE t1,t2];
> > >
> > > I think it can be extended in a similar way for schema syntax as well.
> > >
> >
> > If the proposed syntax ALTER PUBLICATION ... RESET will reset all the
> > objects in the publication then there still seems simple way to remove
> > only the EXCEPT list but leave everything else intact. IIUC to clear
> > just the EXCEPT list would require a 2 step process - 1) ALTER ...
> > RESET then 2) ALTER ... ADD ALL TABLES again.
> >
> > I was wondering if it might be useful to have a variation that *only*
> > resets the EXCEPT list, but still leaves everything else as-is?
> >
> > So, instead of:
> > ALTER PUBLICATION pubname RESET
>
> +1 for this syntax as this syntax can be extendable to include options
> like (except/all/etc) later.
> Currently we can support this syntax and can be extended later based
> on the requirements.
The attached patch has the implementation for "ALTER PUBLICATION
pubname RESET". This command will reset the publication to default
state which includes resetting the publication options, setting ALL
TABLES option to false and dropping the relations and schemas that are
associated with the publication.
Regards,
Vignesh
From c55befe6f53649babce1dd526b1c123b77731dcd Mon Sep 17 00:00:00 2001
From: Vigneshwaran C
Date: Thu, 12 May 2022 08:29:38 +0530
Subject: [PATCH v1] Add RESET option to Alter Publication which will reset the
publication with default values.
This patch adds a new RESET option to ALTER PUBLICATION which will reset
the publication to default state which includes resetting the publication
options, setting ALL TABLES option to false and dropping the relations and
schemas that are associated with the publication.
Usage:
ALTER PUBLICATION pub1 RESET;
---
doc/src/sgml/ref/alter_publication.sgml | 29 +--
src/backend/commands/publicationcmds.c| 99 +++
src/backend/parser/gram.y | 9 +++
src/bin/psql/tab-complete.c | 2 +-
src/include/nodes/parsenodes.h| 3 +-
src/test/regress/expected/publication.out | 69
src/test/regress/sql/publication.sql | 37 +
7 files changed, 241 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index e2cce49471..29f3858de1 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -27,6 +27,7 @@ ALTER PUBLICATION name DROP name SET ( publication_parameter [= value] [, ... ] )
ALTER PUBLICATION name OWNER TO { new_owner | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION name RENAME TO new_name
+ALTER PUBLICATION name RESET
where publication_object is one of:
@@ -65,7 +66,18 @@ ALTER PUBLICATION name RENAME TO
- The remaining variants change the owner and the name of the publication.
+ The OWNER clause will change the owner of the publication.
+
+
+
+ The RENAME clause will change the name of the publication.
+
+
+
+ The RESET clause will reset the publication to default
+ state which includes resetting the publication options, setting
+ ALL TABLES option to false and drop the
+ relations and schemas that are associated with the publication.
@@ -73,10 +85,11 @@ ALTER PUBLICATION name RENAME TO ADD ALL TABLES IN SCHEMA and
SET ALL TABLES IN SCHEMA to a publication requires the
- invoking user to be a superuser. To alter the owner, you must also be a
- direct or indirect member of the new owning role. The new owner must have
- CREATE privilege on the database. Also, the new owner
- of a FOR ALL TABLES or FOR ALL TABLES IN
+ invoking user to be a superuser. RESET of publication
+ requires invoking user to be a superuser. To alter the owner, you must also
+ be a direct or indirect member of the new owning role. The new owner must
+ have CREATE privilege on the database. Also, the new
+ owner of a FOR ALL TABLES or FOR ALL TABLES IN
SCHEMA publication must be a superuser. However, a superuser can
change the ownership of a publication regardless of these re
Re: Multi-Master Logical Replication
On Thu, Apr 28, 2022 at 5:20 AM Peter Smith wrote: > > MULTI-MASTER LOGICAL REPLICATION > > 1.0 BACKGROUND > > Let’s assume that a user wishes to set up a multi-master environment > so that a set of PostgreSQL instances (nodes) use logical replication > to share tables with every other node in the set. > > We define this as a multi-master logical replication (MMLR) node-set. > > > > 1.1 ADVANTAGES OF MMLR > > - Increases write scalability (e.g., all nodes can write arbitrary data). > - Allows load balancing > - Allows rolling updates of nodes (e.g., logical replication works > between different major versions of PostgreSQL). > - Improves the availability of the system (e.g., no single point of failure) > - Improves performance (e.g., lower latencies for geographically local nodes) > > 2.0 MMLR AND POSTGRESQL > > It is already possible to configure a kind of MMLR set in PostgreSQL > 15 using PUB/SUB, but it is very restrictive because it can only work > when no two nodes operate on the same table. This is because when two > nodes try to share the same table then there becomes a circular > recursive problem where Node1 replicates data to Node2 which is then > replicated back to Node1 and so on. > > To prevent the circular recursive problem Vignesh is developing a > patch [1] that introduces new SUBSCRIPTION options "local_only" (for > publishing only data originating at the publisher node) and > "copy_data=force". Using this patch, we have created a script [2] > demonstrating how to set up all the above multi-node examples. An > overview of the necessary steps is given in the next section. > > 2.1 STEPS – Adding a new node N to an existing node-set > > step 1. Prerequisites – Apply Vignesh’s patch [1]. All nodes in the > set must be visible to each other by a known CONNECTION. All shared > tables must already be defined on all nodes. > > step 2. On node N do CREATE PUBLICATION pub_N FOR ALL TABLES > > step 3. All other nodes then CREATE SUBSCRIPTION to PUBLICATION pub_N > with "local_only=on, copy_data=on" (this will replicate initial data > from the node N tables to every other node). > > step 4. On node N, temporarily ALTER PUBLICATION pub_N to prevent > replication of 'truncate', then TRUNCATE all tables of node N, then > re-allow replication of 'truncate'. > > step 5. On node N do CREATE SUBSCRIPTION to the publications of all > other nodes in the set > 5a. Specify "local_only=on, copy_data=force" for exactly one of the > subscriptions (this will make the node N tables now have the same > data as the other nodes) > 5b. Specify "local_only=on, copy_data=off" for all other subscriptions. > > step 6. Result - Now changes to any table on any node should be > replicated to every other node in the set. > > Note: Steps 4 and 5 need to be done within the same transaction to > avoid loss of data in case of some command failure. (Because we can't > perform create subscription in a transaction, we need to create the > subscription in a disabled mode first and then enable it in the > transaction). > > 2.2 DIFFICULTIES > > Notice that it becomes increasingly complex to configure MMLR manually > as the number of nodes in the set increases. There are also some > difficulties such as > - dealing with initial table data > - coordinating the timing to avoid concurrent updates > - getting the SUBSCRIPTION options for copy_data exactly right. > > 3.0 PROPOSAL > > To make the MMLR setup simpler, we propose to create a new API that > will hide all the step details and remove the burden on the user to > get it right without mistakes. > > 3.1 MOTIVATION > - MMLR (sharing the same tables) is not currently possible > - Vignesh's patch [1] makes MMLR possible, but the manual setup is > still quite difficult > - An MMLR implementation can solve the timing problems (e.g., using > Database Locking) > > 3.2 API > > Preferably the API would be implemented as new SQL functions in > PostgreSQL core, however, implementation using a contrib module or > some new SQL syntax may also be possible. > > SQL functions will be like below: > - pg_mmlr_set_create = create a new set, and give it a name > - pg_mmlr_node_attach = attach the current node to a specified set > - pg_mmlr_node_detach = detach a specified node from a specified set > - pg_mmlr_set_delete = delete a specified set > > For example, internally the pg_mmlr_node_attach API function would > execute the equivalent of all the CREATE PUBLICATION, CREATE > SUBSCRIPTION, and TRUNCATE steps described above. > > Notice this proposal has some external API similarities with the BDR >
Re: Skipping schema changes in publication
On Fri, May 6, 2022 at 8:05 AM Peter Smith wrote: > > On Thu, Apr 28, 2022 at 9:32 PM Amit Kapila wrote: > > > ... > > > > Another idea that occurred to me today for tables this is as follows: > > 1. Allow to mention except during create publication ... For All Tables. > > CREATE PUBLICATION pub1 FOR ALL TABLES EXCEPT TABLE t1,t2; > > 2. Allow to Reset it. This new syntax will reset all objects in the > > publications. > > Alter Publication ... RESET; > > 3. Allow to add it to an existing publication > > Alter Publication ... Add ALL TABLES [EXCEPT TABLE t1,t2]; > > > > I think it can be extended in a similar way for schema syntax as well. > > > > If the proposed syntax ALTER PUBLICATION ... RESET will reset all the > objects in the publication then there still seems simple way to remove > only the EXCEPT list but leave everything else intact. IIUC to clear > just the EXCEPT list would require a 2 step process - 1) ALTER ... > RESET then 2) ALTER ... ADD ALL TABLES again. > > I was wondering if it might be useful to have a variation that *only* > resets the EXCEPT list, but still leaves everything else as-is? > > So, instead of: > ALTER PUBLICATION pubname RESET +1 for this syntax as this syntax can be extendable to include options like (except/all/etc) later. Currently we can support this syntax and can be extended later based on the requirements. The new feature will handle the various use cases based on the behavior given below: -- CREATE Publication with EXCEPT TABLE syntax CREATE PUBLICATION pub1 FOR ALL TABLES EXCEPT TABLE t1,t2; -- ok Alter Publication pub1 RESET; -- All Tables and options are reset similar to creating publication without any publication object and publication option (create publication pub1) \dRp+ pub1 Publication pub2 Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root -++-+-+-+---+-- vignesh | f | t | t | t | t | f (1 row) -- Can add except table after reset of publication ALTER PUBLICATION pub1 Add ALL TABLES EXCEPT TABLE t1,t2; -- ok -- Cannot add except table without reset of publication ALTER PUBLICATION pub1 Add EXCEPT TABLE t3,t4; -- not ok, need to be reset Alter Publication pub1 RESET; -- Cannot add table to ALL TABLES Publication ALTER PUBLICATION pub1 Add ALL TABLES EXCEPT TABLE t1,t2, t3, t4, TABLE t5; -- not ok, ALL TABLES Publications does not support including of TABLES Alter Publication pub1 RESET; -- Cannot add table to ALL TABLES Publication ALTER PUBLICATION pub1 Add ALL TABLES TABLE t1,t2; -- not ok, ALL TABLES Publications does not support including of TABLES -- Cannot add ALL TABLES IN SCHEMA to ALL TABLES Publication ALTER PUBLICATION pub1 Add ALL TABLES ALL TABLES IN SCHEMA sch1, sch2; -- not ok, ALL TABLES Publications does not support including of ALL TABLES IN SCHEMA -- Existing syntax should work as it is CREATE PUBLICATION pub1 FOR TABLE t1; ALTER PUBLICATION pub1 ADD TABLE t1; -- ok, existing ALTER should work as it is (ok without reset) ALTER PUBLICATION pub1 ADD ALL TABLES IN SCHEMA sch1; -- ok, existing ALTER should work as it is (ok without reset) ALTER PUBLICATION pub1 DROP TABLE t1; -- ok, existing ALTER should work as it is (ok without reset) ALTER PUBLICATION pub1 DROP ALL TABLES IN SCHEMA sch1; -- ok, existing ALTER should work as it is (ok without reset) ALTER PUBLICATION pub1 SET TABLE t1; -- ok, existing ALTER should work as it is (ok without reset) ALTER PUBLICATION pub1 SET ALL TABLES IN SCHEMA sch1; -- ok, existing ALTER should work as it is (ok without reset) I will modify the patch to handle this. Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Thu, May 5, 2022 at 2:41 PM Alvaro Herrera wrote: > > On 2022-May-04, vignesh C wrote: > > > On Mon, May 2, 2022 at 5:49 AM Peter Smith wrote: > > > > 1. Commit message > > > > > > In another thread using the terminology "multi-master" seems to be > > > causing some problems. Maybe you should choose different wording in > > > this commit message to avoid having the same problems? > > > > I felt, we can leave this to committer > > Please don't. See > https://postgr.es/m/20200615182235.x7lch5n6kcjq4...@alap3.anarazel.de In this thread changing multimaster to multi-source did not get agreed and the changes did not get committed. The code still uses multimaster as in [1]. I think we should keep it as "multimaster" as that is already being used in [1]. [1] - https://www.postgresql.org/docs/current/different-replication-solutions.html Regards, Vignesh
Re: Handle infinite recursion in logical replication setup
On Mon, May 2, 2022 at 5:49 AM Peter Smith wrote: > > Thanks for updating the patches for all my prior feedback. > > For v12* I have only minor feedback for the docs. > > == > V12-0001 > > no comments > > == > V12-0002 > > 1. Commit message > > In another thread using the terminology "multi-master" seems to be > causing some problems. Maybe you should choose different wording in > this commit message to avoid having the same problems? I felt, we can leave this to committer > ~~~ > > 2. doc/src/sgml/logical-replication.sgml > > These are all similar minor wording suggestions to split the sentences. > Wording: ", here copy_data is specified as XXX ..." -> ". Use > copy_data specified as XXX ..." > > Also: > Wording: "... to subscribe the changes from nodeXXX..." > -> "... to subscribe to the changes from nodeXXX... " (add "to") > -> "... to subscribe to nodeXXX." (or I preferred just remove the > whole "changes" part) > > 2a. > Create a subscription in node3 to subscribe the changes from node1, > here copy_data is specified as force so that the existing table data > is copied during initial sync: > > SUGGESTION > Create a subscription in node3 to subscribe to node1. Use copy_data > specified as force so that the existing table data is copied during > initial sync: Modified > 2b. > Create a subscription in node1 to subscribe the changes fromnode3, > here copy_data is specified as force so that the existing table data > is copied during initial sync: > > SUGGESTION > Create a subscription in node1 to subscribe to node3. Use copy_data > specified as force so that the existing table data is copied during > initial sync: Modified > 2c. > Create a subscription in node2 to subscribe the changes from node3, > here copy_data is specified as force so that the existing table data > is copied during initial sync: > > SUGGESTION > Create a subscription in node2 to subscribe to node3. Use copy_data > specified as force so that the existing table data is copied during > initial sync: Modified > 2d. > Create a subscription in node3 to subscribe the changes from node1, > here copy_data is specified as force when creating a subscription to > node1 so that the existing table data is copied during initial sync: > > SUGGESTION > Create a subscription in node3 to subscribe to node1. Use copy_data > specified as force when creating a subscription to node1 so that the > existing table data is copied during initial sync: Modified > 2e. > Create a subscription in node3 to subscribe the changes from node2, > here copy_data is specified as off as the initial table data would > have been copied in the earlier step: > > SUGGESTION (this one has a bit more re-wording than the others) > Create a subscription in node3 to subscribe to node2. Use copy_data > specified as off because the initial table data would have been > already copied in the previous step: Modified > ~~~ > > 3. doc/src/sgml/logical-replication.sgml > > 31.11.2. Adding new node when there is no data in any of the nodes > 31.11.3. Adding new node when data is present in the existing nodes > 31.11.4. Adding new node when data is present in the new node > > Minor change to the above heading? > > Wording: "Adding new node ..." -> "Adding a new node ..." Modified Thanks for the comments, the attached v13 patch has the changes for the same. Regards, Vignesh From 657265a56dfc2eee8f4b6cd2137e6c9b8c26d6a6 Mon Sep 17 00:00:00 2001 From: Vigneshwaran C Date: Fri, 8 Apr 2022 11:10:05 +0530 Subject: [PATCH v13 1/2] Skip replication of non local data. This patch adds a new SUBSCRIPTION boolean option "local_only". The default is false. When a SUBSCRIPTION is created with this option enabled, the publisher will only publish data that originated at the publisher node. Usage: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=postgres port=' PUBLICATION pub1 with (local_only = true); --- doc/src/sgml/ref/alter_subscription.sgml | 5 +- doc/src/sgml/ref/create_subscription.sgml | 12 ++ src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 4 +- src/backend/commands/subscriptioncmds.c | 26 ++- .../libpqwalreceiver/libpqwalreceiver.c | 5 + src/backend/replication/logical/worker.c | 2 + src/backend/replication/pgoutput/pgoutput.c | 23 +++ src/bin/pg_dump/pg_dump.c | 17 +- src/bin/pg_dump/pg_dump.h | 1 + src/bin/psql/describe.c | 8 +- src/bin/psql/tab-complete.c | 4 +- src/include/catalog/pg_subscription.h | 3 + src/include/replication/logicalproto.h| 10 +- src/include/replication/pgoutput.h| 1 + src/include/replication/walreceiver.h | 1 + src/test/regress/expected/subscription.out| 142 --- src/test/regress/sql/subscription.sql | 10 ++ src/test/subscription/t/032_localonly.pl | 162 ++ 19 files ch
Re: Skipping schema changes in publication
On Thu, Apr 28, 2022 at 4:50 PM osumi.takami...@fujitsu.com
wrote:
>
> On Wednesday, April 27, 2022 9:50 PM vignesh C wrote:
> > Thanks for the comments, the attached v3 patch has the changes for the same.
> Hi
>
> Thank you for updating the patch. Several minor comments on v3.
>
> (1) commit message
>
> The new syntax allows specifying schemas. For example:
> CREATE PUBLICATION pub1 FOR ALL TABLES EXCEPT TABLE t1,t2;
> OR
> ALTER PUBLICATION pub1 ADD EXCEPT TABLE t1,t2;
>
> We have above sentence, but it looks better
> to make the description a bit more accurate.
>
> Kindly change
> From :
> "The new syntax allows specifying schemas"
> To :
> "The new syntax allows specifying excluded relations"
>
> Also, kindly change "OR" to "or",
> because this description is not syntax.
Slightly reworded and modified
> (2) publication_add_relation
>
> @@ -396,6 +400,9 @@ publication_add_relation(Oid pubid, PublicationRelInfo
> *pri,
> ObjectIdGetDatum(pubid);
> values[Anum_pg_publication_rel_prrelid - 1] =
> ObjectIdGetDatum(relid);
> + values[Anum_pg_publication_rel_prexcept - 1] =
> + BoolGetDatum(pri->except);
> +
>
> /* Add qualifications, if available */
>
> It would be better to remove the blank line,
> because with this change, we'll have two blank
> lines in a row.
Modified
> (3) pg_dump.h & pg_dump_sort.c
>
> @@ -80,6 +80,7 @@ typedef enum
> DO_REFRESH_MATVIEW,
> DO_POLICY,
> DO_PUBLICATION,
> + DO_PUBLICATION_EXCEPT_REL,
> DO_PUBLICATION_REL,
> DO_PUBLICATION_TABLE_IN_SCHEMA,
> DO_SUBSCRIPTION
>
> @@ -90,6 +90,7 @@ enum dbObjectTypePriorities
> PRIO_FK_CONSTRAINT,
> PRIO_POLICY,
> PRIO_PUBLICATION,
> + PRIO_PUBLICATION_EXCEPT_REL,
> PRIO_PUBLICATION_REL,
> PRIO_PUBLICATION_TABLE_IN_SCHEMA,
> PRIO_SUBSCRIPTION,
> @@ -144,6 +145,7 @@ static const int dbObjectTypePriority[] =
> PRIO_REFRESH_MATVIEW, /* DO_REFRESH_MATVIEW */
> PRIO_POLICY,/* DO_POLICY */
> PRIO_PUBLICATION, /* DO_PUBLICATION */
> + PRIO_PUBLICATION_EXCEPT_REL,/* DO_PUBLICATION_EXCEPT_REL */
> PRIO_PUBLICATION_REL, /* DO_PUBLICATION_REL */
> PRIO_PUBLICATION_TABLE_IN_SCHEMA, /*
> DO_PUBLICATION_TABLE_IN_SCHEMA */
> PRIO_SUBSCRIPTION /* DO_SUBSCRIPTION */
>
> How about having similar order between
> pg_dump.h and pg_dump_sort.c, like
> we'll add DO_PUBLICATION_EXCEPT_REL
> after DO_PUBLICATION_REL in pg_dump.h ?
>
Modified
> (4) GetAllTablesPublicationRelations
>
> + /*
> +* pg_publication_rel and pg_publication_namespace will only have
> except
> +* tables in case of all tables publication, no need to pass except
> flag
> +* to get the relations.
> +*/
> + List *exceptpubtablelist = GetPublicationRelations(pubid,
> PUBLICATION_PART_ALL);
> +
>
> There is one unnecessary space in a comment
> "...pg_publication_namespace will only have...". Kindly remove it.
>
> Then, how about diving the variable declaration and
> the insertion of the return value of GetPublicationRelations ?
> That might be aligned with other places in this file.
Modified
> (5) GetTopMostAncestorInPublication
>
>
> @@ -302,8 +303,9 @@ GetTopMostAncestorInPublication(Oid puboid, List
> *ancestors, int *ancestor_level
> foreach(lc, ancestors)
> {
> Oid ancestor = lfirst_oid(lc);
> - List *apubids = GetRelationPublications(ancestor);
> + List *apubids = GetRelationPublications(ancestor,
> false);
> List *aschemaPubids = NIL;
> + List *aexceptpubids;
>
> level++;
>
> @@ -317,7 +319,9 @@ GetTopMostAncestorInPublication(Oid puboid, List
> *ancestors, int *ancestor_level
> else
> {
> aschemaPubids =
> GetSchemaPublications(get_rel_namespace(ancestor));
> - if (list_member_oid(aschemaPubids, puboid))
> + aexceptpubids = GetRelationPublications(ancestor,
> true);
> + if (list_member_oid(aschemaPubids, puboid) ||
> + (puballtables &&
> !list_member_oid(aexceptpubids, puboid)))
> {
> topmost_reli
