On Tue, Jun 27, 2023 at 6:16 AM vignesh C wrote:
> While development, below are some of the challenges we faced:
> 1. Almost all the members of the AlterTableType enum will have to be
> annotated.
> 2. Complex functionalities which require access to catalog tables
> cannot be auto generated,
On Wed, Mar 29, 2023 at 5:13 AM Amit Kapila wrote:
>
> On Wed, Mar 29, 2023 at 2:49 AM Zheng Li wrote:
> >
> >
> > I agree that a full fledged DDL deparser and DDL replication is too
> > big of a task for one patch. I think we may consider approaching this
>
On Sun, Mar 26, 2023 at 5:22 PM Tom Lane wrote:
> I spent some time looking through this thread to try to get a sense
> of the state of things, and I came away quite depressed. The patchset
> has ballooned to over 2MB, which is a couple orders of magnitude
> larger than anyone could hope to
> I think that there are some (possibly) tricky challenges that haven't
> been discussed yet to support replicating global objects.
>
> First, as for publications having global objects (roles, databases,
> and tablespaces), but storing them in database specific tables like
> pg_publication doesn't
> > Yes. Do we have any concrete use case where the subscriber is an older
> > version, in the first place?
> >
>
> As per my understanding, it is mostly due to the reason that it can
> work today. Today, during an off-list discussion with Jonathan on this
> point, he pointed me to a similar
On Mon, Mar 6, 2023 at 5:17 AM wangw.f...@fujitsu.com
wrote:
>
> For v-75-0003* patch.
> 2. In the function deparse_CreateSeqStmt.
> It seems that we are not deparsing the "AS data_type" clause (CREATE SEQUENCE
> ... AS data_type). I think this causes all data_type to be default (bigint)
> after
On Fri, Feb 17, 2023 at 4:48 AM Amit Kapila wrote:
>
> On Fri, Feb 17, 2023 at 10:58 AM Zheng Li wrote:
> >
> > > > > Actually, I intend something for global objects. But the main thing
> > > > > that is worrying me about this is that we don't have a
On Mon, Feb 20, 2023 at 3:23 AM Masahiko Sawada wrote:
>
> On Fri, Feb 17, 2023 at 1:13 PM Zheng Li wrote:
> >
> > > > I've implemented a prototype to allow replicated objects to have the
> > > > same owner from the publisher in
> > > > v69-0008
> > > Actually, I intend something for global objects. But the main thing
> > > that is worrying me about this is that we don't have a clean way to
> > > untie global object replication from database-specific object
> > > replication.
> >
> > I think ultimately we need a clean and efficient way to
> > I've implemented a prototype to allow replicated objects to have the
> > same owner from the publisher in
> > v69-0008-Allow-replicated-objects-to-have-the-same-owner-from.patch.
> >
>
> I also think it would be a helpful addition for users.A few points
Thanks for supporting this addition.
>
On Fri, Feb 10, 2023 at 11:31 AM vignesh C wrote:
>
> On Fri, 10 Feb 2023 at 21:50, vignesh C wrote:
> > The attached v68 version patch has the changes for the same.
>
> I was not sure if we should support ddl replication of
> create/alter/drop subscription commands as there might be some data
>
> > Yes, CREATE/ALTER SERVER commands are also supported by the current
> > DDL replication patch.
> >
> > >But what about data inserted by the publisher on the
> > > foreign server?
> >
> > I thought the data inserted to a foreign table will always be stored
> > on the foreign server unless I'm
On Thu, Jan 19, 2023 at 2:05 AM Amit Kapila wrote:
>
> On Thu, Jan 19, 2023 at 8:39 AM Zheng Li wrote:
> >
> > On Wed, Jan 18, 2023 at 6:27 AM Amit Kapila wrote:
> > >
> > > On Sat, Jan 7, 2023 at 8:58 PM Zheng Li wrote:
> > > >
> >
> &g
On Wed, Jan 18, 2023 at 6:27 AM Amit Kapila wrote:
>
> On Sat, Jan 7, 2023 at 8:58 PM Zheng Li wrote:
> >
> > I added documentation and changed user interface design in the
> > attached v60 patch set.
> > The patch set addressed comments from Peter in [1].
>
Hi,
This documentation change looks good to me. I verified in testing and in code
that the value for boolean parameters in PUB/SUB commands can be omitted. which
is equivalent to specifying TRUE. For example,
CREATE PUBLICATIOIN mypub for ALL TABLES with (publish_via_partition_root);
is
On Wed, Dec 28, 2022 at 5:42 PM Zheng Li wrote:
>
> >- CreatePublication has a long list of command tags; is that good?
> >Maybe it'd be better to annotate the list in cmdtaglist.h somehow.
>
> I've addressed this comment by introducing a new flag ddlreplok in the
> PG_C
Hi,
Thanks for working on this and for the feedback!
I've added the updated deparser testing module to the DDL replication
thread in [1].
We'll add more test cases to the testing module and continue the
discussion there.
[1]
Hello,
Thanks for the feedback.
> I have been following this patch for a long time.
> Recently, I started to try to test it. I found several bugs
> here and want to give you feedback.
>
> 1. CREATE TABLE LIKE
> I found that this case may be repication incorrectly.
>You can run the
> Hi, authors on this thread.
>
> The patch v32-0001 is very large, so it will take some time to review
> the code in detail.
Thanks for reviewing!
> Meanwhile, here are some general comments about the patch:
>
> ==
>
> 1. It might be useful to add this thread to the commitfest, if only so
>
> > I think a publication of ALL OBJECTS sounds intuitive. Does it mean we'll
> > publish all DDL commands, all commit and abort operations in every
> > database if there is such publication of ALL OBJECTS?
> >
>
> Actually, I intend something for global objects. But the main thing
> that is
> Can we think of relying to send WAL of such DDLs just based on whether
> there is a corresponding publication (ex. publication of ALL OBJECTS)?
> I mean avoid database-specific filtering in decoding for such DDL
> commands but not sure how much better it is than the current proposal?
I think a
> In general, I agree with your comment below that we can work on this
> after we have some more concrete plans/discussions. I think we can
> probably consider this when we have more discussion around the
> publication commands for the DDL objects. However, it would be good if
> you can add some
bZOnUXyguJ-uBNejwD75Kyo%3DOjdQGJ9g%40mail.gmail.com
[2]https://www.postgresql.org/message-id/OS0PR01MB5716009FDCCC0B50BCB14A99949D9%40OS0PR01MB5716.jpnprd01.prod.outlook.com
With Regards,
Zheng Li
Amazon RDS/Aurora for PostgreSQL
0005-Support-replication-of-global-object-commands-these-.patch
D
On Sat, Jul 23, 2022 at 11:33 AM Joe Conway wrote:
>
> On 7/22/22 17:18, Zheng Li wrote:
> > Here is a patch that supports replication of global object commands,
> > these include ROLE statements, database statements and tablespace
> > statements.
> > The
Hello,
Here is a patch that supports replication of global object commands,
these include ROLE statements, database statements and tablespace statements.
The patch should be applied on top of the v13 DDL replication patch set that
ZJ Hou sent in the previous email.
Global objects commands are
> Here are some points in my mind about the two approaches discussed here.
>
> 1) search_patch vs schema qualify
>
> Again, I still think it will bring more flexibility and security by schema
> qualify the
> objects in DDL command as mentioned before[1].
I wonder what security concerns you have?
> Thanks for providing this idea.
>
> I looked at the string that is used for replication:
>
> """
> {ALTERTABLESTMT :relation {RANGEVAR :schemaname public :relname foo
> :inh true :relpersistence p :alias <> :location 12} :cmds ({ALTERTABLECMD
> :subtype 0 :name <> :num 0 :newowner <> :def
Hi Masahiko,
> Thank you for updating the patches!
>
> I've not looked at these patches in-depth yet but with this approach,
> what do you think we can handle the DDL syntax differences between
> major versions? DDL syntax or behavior could be changed by future
> changes and I think we need to
> But of course, if we expect/require to have a perfect snapshot for that
> exact position in the transaction, this won't work. IMO the whole idea
> that we can have non-transactional bits in naturally transactional
> decoding seems a bit suspicious (at least in hindsight).
>
> No matter what we
> Now, say, the user has added a bar column with "ALTER TABLE foo ADD
> COLUMN bar double precision NOT NULL DEFAULT random();" If we compare
> with replication of DMLs like (UPDATE ddl_test SET bar = random();),
> the replication won't update rows with values (3 and 4) on subscriber
> as they
> > > I agree that it adds to our maintainability effort, like every time we
> > > enhance any DDL or add a new DDL that needs to be replicated, we
> > > probably need to change the deparsing code. But OTOH this approach
> > > seems to provide better flexibility. So, in the long run, maybe the
> >
> Attached is a set of two patches as an attempt to evaluate this approach.
>
> The first patch provides functions to deparse DDL commands. Currently,
> it is restricted to just a simple CREATE TABLE statement, the required
> code is extracted from one of the patches posted in the thread [1].
>
>
> > >Another somewhat unrelated problem I see with this work is how to save
> > >recursion of the same command between nodes (when the involved nodes
> > >replicate DDLs). For DMLs, we can avoid that via replication origins
> > >as is being done in the patch proposed [1] but not sure how will we
>
ar(20)). In
> this
> case, the DDL command can de decomposed removing the typmod information (ALTER
> TABLE ... ALTER COLUMN ... TYPE varchar) and replacing it with a CHECK
> constraint.
I tested ALTER TABLE ... ALTER COLUMN ... TYPE. It seems to be working
fine. Is there a particular case
> > But then this could be true for DML as well right? Like after
> > replicating the function to the subscriber if we are sending the DML
> > done by function then what's the problem in DDL. I mean if there is
> > no design issue in implementing this then I don't think there is much
> > point
> You should forbid it. Unless you can decompose the command into multiple SQL
> commands to make it a safe operation for logical replication.
>
> Let's say you want to add a column with a volatile default.
>
> ALTER TABLE foo ADD COLUMN bar double precision DEFAULT random();
>
> If you replicate
Hi,
Here is the rebased new branch
https://github.com/zli236/postgres/commits/ddl_replication
Regards,
Zheng
> I'm planning to work on the initial DDL replication. I'll open a new thread as
> soon as I write a design for it. Just as an example, the pglogical approach is
> to use pg_dump behind the scenes to provide the schema [1]. It is a reasonable
> approach but an optimal solution should be an API to
Hi Amit,
> Some initial comments:
> ===
> 1.
> +/*
> + * Write logical decoding DDL message into XLog.
> + */
> +XLogRecPtr
> +LogLogicalDDLMessage(const char *prefix, Oid roleoid, const char *message,
> + size_t size, bool transactional)
>
> I don't see anywhere the patch using a
Hi,
> > Good catch. The reason for having isTopLevel in the condition is
> > because I haven't decided if a DDL statement inside a PL should
> > be replicated from the user point of view. For example, if I execute a
> > plpgsql function or a stored procedure which creates a table under the hood,
Hi Dilip,
Thanks for the feedback.
> > > > The table creation WAL and table insert WAL are available. The tricky
> > > > part is how do we break down this command into two parts (a normal
> > > > CREATE TABLE followed by insertions) either from the parsetree or the
> > > > WALs. I’ll have to dig
Hi Japin,
> You should use a different user that has different length from your current
> one.
> For example:
>
> px@localhost$ make check-world
This is fixed in the latest commit:
https://github.com/zli236/postgres/commits/ddl_replication
Thanks,
Zheng
x/Codes/postgresql/Debug/src/bin/pg_dump/tmp_check/tmp_test_jgRI
..
> Failed 84/7709 subtests
> t/003_pg_dump_with_server.pl .. ok
> t/010_dump_connstr.pl . ok
>
> Test Summary Report
> ---
> t/002_pg_dump.pl (Wstat: 21504 Tests: 7709 Failed: 84)
This is fixed in the latest version. I need to remind myself to run
make check-world in the future.
Regards,
Zheng Li
Hello Alvaro,
> I think this is a pretty interesting and useful feature.
>
> Did you see some old code I wrote towards this goal?
> https://www.postgresql.org/message-id/20150215044814.gl3...@alvh.no-ip.org
> The intention was that DDL would produce some JSON blob that accurately
> describes the
o INT_MAX (the max timeout allowed by
WaitLatch()) in which case a worker in hibernation only relies on
wakeup? I think it would be nice to run experiments to verify that the
patch reduces power consumption while varying the value of
HIBERNATE_DELAY_SEC.
Regards,
Zheng Li
Amazon RDS/Aurora for Po
Hi Aleksander,
>That's great!
Thanks!
>Maybe the first implementation shouldn't be perfect as long as known
>limitations are documented and the future improvements are unlikely to
>break anything for the users. Committing an MVP and iterating on this is
>much simpler in terms of development and
the logic
to conditionally log the DDL commands or to conditionally decode/ship
the logical DDL message.
Thoughts? Your feedback is appreciated.
Thanks,
Zheng Li
Amazon RDS/Aurora for PostgreSQL
47 matches
Mail list logo
