Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-30 Thread Amit Kapila
retreating the value of replication_slot_xmin. +*/ + -- With Regards, Amit Kapila.

Re: pub/sub - specifying optional parameters without values.

2023-01-30 Thread Amit Kapila
roducing a new option. [1] - https://www.postgresql.org/message-id/CAA4eK1Kt67RdW0WTR-LTxasj3pyukPCYhfA0arDUNnsz2wh03A%40mail.gmail.com -- With Regards, Amit Kapila.

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-01-30 Thread Amit Kapila
nced hackers on the trade-offs for the > above, and I'd be open to work on that if there is a clear winner. For me (3) > is a decent solution for the problem. > >From the discussion above it is not very clear that adding maintenance costs in this area is worth it even though that can give better results as far as this feature is concerned. -- With Regards, Amit Kapila.

Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-30 Thread Amit Kapila
is a good idea to remove 'already_locked' parameter, especially in back branches as this is an exposed API. -- With Regards, Amit Kapila.

Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-30 Thread Amit Kapila
On Mon, Jan 30, 2023 at 11:34 AM Amit Kapila wrote: > > I have reproduced it manually. For this, I had to manually make the > debugger call ReplicationSlotsComputeRequiredXmin(false) via path > SnapBuildProcessRunningXacts()->LogicalIncreaseXminForSlot()->LogicalConfi

Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-30 Thread Amit Kapila
On Thu, Dec 8, 2022 at 8:17 AM Masahiko Sawada wrote: > > On Mon, Nov 21, 2022 at 4:31 PM Amit Kapila wrote: > > One idea to fix this issue is that in > ReplicationSlotsComputeRequiredXmin(), we compute the minimum xmin > while holding both ProcArrayLock and ReplicationSl

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-30 Thread Amit Kapila
On Mon, Jan 30, 2023 at 12:38 PM Kyotaro Horiguchi wrote: > > At Mon, 30 Jan 2023 11:56:33 +0530, Amit Kapila > wrote in > > > > > > The GUC is not stored in a catalog, but.. oh... it is multiplied by > > > 1000. > > > > Which part of the pat

Re: Logical replication timeout problem

2023-01-29 Thread Amit Kapila
that is not processed by the plugin. So, in such cases, the downstream can timeout. To avoid that we try to send a keepalive message if required. Trying to send a keepalive message after every change has some overhead, but testing showed there is no noticeable overhead if we do it after every ~100 changes." -- With Regards, Amit Kapila.

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-29 Thread Amit Kapila
On Mon, Jan 30, 2023 at 9:20 AM vignesh C wrote: > > On Sat, 28 Jan 2023 at 11:26, Amit Kapila wrote: > > > > One thing that looks a bit odd is that we will anyway have a similar > > check in replorigin_drop_guts() which is a static function and called > > from

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-29 Thread Amit Kapila
On Mon, Jan 30, 2023 at 9:43 AM Kyotaro Horiguchi wrote: > > At Mon, 30 Jan 2023 08:51:05 +0530, Amit Kapila > wrote in > > On Mon, Jan 30, 2023 at 8:32 AM Kyotaro Horiguchi > > wrote: > > > > > > At Sat, 28 Jan 2023 04:28:29 +, "Takamichi

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-29 Thread Amit Kapila
On Mon, Jan 30, 2023 at 5:40 AM Peter Smith wrote: > > Patch v88-0001 LGTM. > Pushed. -- With Regards, Amit Kapila.

Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-29 Thread Amit Kapila
On Mon, Jan 30, 2023 at 10:27 AM Amit Kapila wrote: > > On Sun, Jan 29, 2023 at 9:15 PM Masahiko Sawada wrote: > > > > On Sat, Jan 28, 2023 at 11:54 PM Hayato Kuroda (Fujitsu) > > wrote: > > > > > > Dear Amit, Sawada-san, > > > > > >

Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-29 Thread Amit Kapila
c create a replication slot with > > NOEXPORT_SNAPSHOT option. I think in this case, CreateInitDecodingContext() > > is > > called with need_full_snapshot = false, and slot->effective_xmin is not > > updated. > > Right. This is how we create a slot used by an apply worker. > I was thinking about how that led to this problem because GetOldestSafeDecodingTransactionId() ignores InvalidTransactionId. It seems that is possible when both builder->xmin and replication_slot_catalog_xmin precede replication_slot_catalog_xmin. Do you see any different reason for it? -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-29 Thread Amit Kapila
On Mon, Jan 30, 2023 at 8:32 AM Kyotaro Horiguchi wrote: > > At Sat, 28 Jan 2023 04:28:29 +, "Takamichi Osumi (Fujitsu)" > wrote in > > On Friday, January 27, 2023 8:00 PM Amit Kapila > > wrote: > > > So, you have changed min_apply_de

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-27 Thread Amit Kapila
On Sat, Jan 28, 2023 at 9:36 AM houzj.f...@fujitsu.com wrote: > > On Friday, January 27, 2023 8:16 PM Amit Kapila > > > > On Fri, Jan 27, 2023 at 3:45 PM vignesh C wrote: > > > > > > On Mon, 23 Jan 2023 at 10:52, Amit Kapila > > wrote: > > >

Re: suppressing useless wakeups in logical/worker.c

2023-01-27 Thread Amit Kapila
*/ + now = GetCurrentTimestamp(); + if (wakeup[LRW_WAKEUP_SYNC_START] < now) + wakeup[LRW_WAKEUP_SYNC_START] = TimestampTzPlusSeconds(wakeup[LRW_WAKEUP_SYNC_START], 1); Do we see unnecessary wakeups without this, or delay in sync? BTW, do we need to do something about wakeups in wait_for_relation_state_change()? -- With Regards, Amit Kapila.

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-27 Thread Amit Kapila
On Fri, Jan 27, 2023 at 3:45 PM vignesh C wrote: > > On Mon, 23 Jan 2023 at 10:52, Amit Kapila wrote: > > > > IIRC, this is done to prevent concurrent drops of origin drop say by > > exposed API pg_replication_origin_drop(). See the discussion in [1] > > related to

Re: Logical replication timeout problem

2023-01-27 Thread Amit Kapila
On Fri, Jan 27, 2023 at 5:18 PM houzj.f...@fujitsu.com wrote: > > On Wednesday, January 25, 2023 7:26 PM Amit Kapila > > > > On Tue, Jan 24, 2023 at 8:15 AM wangw.f...@fujitsu.com > > wrote: > > > > > > Attach the new patch. > > > > &g

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-27 Thread Amit Kapila
r subowner? 2. + + diffms = TimestampDifferenceMilliseconds(GetCurrentTimestamp(), + TimestampTzPlusMilliseconds(finish_ts, MySubscription->minapplydelay)); The above code appears a bit unreadable. Can we store the result of TimestampTzPlusMilliseconds() in a separate variable say "TimestampTz delayUntil;"? -- With Regards, Amit Kapila.

Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-26 Thread Amit Kapila
hich is processing running_xacts via SnapBuildProcessRunningXacts()? Isn't that walsender slot's effective_xmin have a non-zero value? If not, then why? -- With Regards, Amit Kapila.

Re: Logical replication timeout problem

2023-01-25 Thread Amit Kapila
e comments for the callback. Additionally, I think we should have a test case to show we don't time out because of not processing non-transactional messages. See pgoutput_message for cases where it doesn't process the message. -- With Regards, Amit Kapila. v7-0001-Fix-the-logical-re

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-24 Thread Amit Kapila
ations between servers." [1] - https://www.postgresql.org/docs/devel/runtime-config-replication.html -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-24 Thread Amit Kapila
ly of > > send_feedback to "has_unprocessed_change". > > > > At Tue, 24 Jan 2023 12:27:58 +0530, Amit Kapila > > wrote in > > > > send_feedback(): > > > > +* If the subscriber side apply is delayed (because of > > time-delayed &

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-24 Thread Amit Kapila
On Wed, Jan 25, 2023 at 10:05 AM Amit Kapila wrote: > > On Wed, Jan 25, 2023 at 3:15 AM Peter Smith wrote: > > > > 1. > > @@ -210,7 +210,7 @@ int logical_decoding_work_mem; > > static const Size max_changes_in_memory = 4096; /* XXX for restore only */ > &

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-24 Thread Amit Kapila
ansaction. > == > src/include/replication/reorderbuffer.h > > 3. > @@ -18,14 +18,14 @@ > #include "utils/timestamp.h" > > extern PGDLLIMPORT int logical_decoding_work_mem; > -extern PGDLLIMPORT int logical_decoding_mode; > +extern PGDLLIMPORT int logical_replication_mode; > > Probably here should also be a comment to say "/* GUC variables */" > Okay, we can do this. -- With Regards, Amit Kapila.

Re: Test failures of 100_bugs.pl

2023-01-24 Thread Amit Kapila
_syncing_table_states() to ensure that invalidation has been processed; (c) print rel states and relids from table_states_not_ready in process_syncing_tables_for_apply() to see if t2 has ever appeared in that list. -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-24 Thread Amit Kapila
On Tue, Jan 24, 2023 at 12:44 PM Peter Smith wrote: > > On Tue, Jan 24, 2023 at 5:58 PM Amit Kapila wrote: > > > > On Tue, Jan 24, 2023 at 8:15 AM Kyotaro Horiguchi > > wrote: > > > > > > > Attached the updated patch v19. > > > > > &g

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-23 Thread Amit Kapila
ok at the both two varaibles instead. > I think this is true without this patch, so why that has not been followed in the first place? One comment, I see in this regard is as below: /* It's legal to not pass a recvpos */ if (recvpos < last_recvpos) recvpos = last_recvpos; -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-23 Thread Amit Kapila
LOGs to be in reverse order as it doesn't make much sense to me to first say that we are skipping changes and then say the transaction is delayed. What do you think? -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-23 Thread Amit Kapila
On Tue, Jan 24, 2023 at 6:17 AM Kyotaro Horiguchi wrote: > > At Mon, 23 Jan 2023 17:36:13 +0530, Amit Kapila > wrote in > > On Sun, Jan 22, 2023 at 6:12 PM Takamichi Osumi (Fujitsu) > > wrote: > > > > > > > > > Attached the updated pat

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-23 Thread Amit Kapila
help us to make it move forward. [1] - https://www.postgresql.org/message-id/TYAPR01MB586668E50FC2447AD7F92491F5E89%40TYAPR01MB5866.jpnprd01.prod.outlook.com -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-23 Thread Amit Kapila
On Tue, Jan 24, 2023 at 3:46 AM Peter Smith wrote: > > On Mon, Jan 23, 2023 at 9:44 PM Amit Kapila wrote: > > > > > > > 6. defGetMinApplyDelay > > > > ... > > > > > > 6b. > > > I thought this function should be implemented as st

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-23 Thread Amit Kapila
publisher and subscriber but not really sure whether that will simplify the usage. [1] - https://www.postgresql.org/message-id/CAJpy0uDzddK_ZUsB2qBJUbW_ZODYGoUHTaS5pVcYE2tzATCVXQ%40mail.gmail.com -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-23 Thread Amit Kapila
feature? Currently, it doesn't care whether the changes of a particular xact are skipped or not. I think that might be okay because anyway the purpose of this feature is to make subscriber lag from publishers. What do you think? I feel we can add some comments to indicate the same. -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-23 Thread Amit Kapila
sql > > 9. Add new test? > > Should there be an additional test to check redundant parameter > setting -- eg. "... WITH (min_apply_delay=123, min_apply_delay=456)" > I don't think that will be of much help. We don't seem to have other tests for subscription parameters. -- With Regards, Amit Kapila.

Re: Support logical replication of DDLs

2023-01-23 Thread Amit Kapila
On Thu, Jan 19, 2023 at 11:24 PM Zheng Li wrote: > > On Thu, Jan 19, 2023 at 2:05 AM Amit Kapila wrote: > > > > > > > > Foreign Tables can also be considered replicated with DDL replication > > > because we > > > don't even need to replicat

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-22 Thread Amit Kapila
mentioned in the comments in process_syncing_tables_for_sync() before the start of the second transaction which leads to this change. See the report and discussion about that race condition in the email [1]. [1] - https://www.postgresql.org/message-id/CAD21AoAw0Oofi4kiDpJBOwpYyBBBkJj=sluon4gd2gjuakg...@mail.gmail.com -- With Regards, Amit Kapila.

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-22 Thread Amit Kapila
o prevent. > IIRC, this is done to prevent concurrent drops of origin drop say by exposed API pg_replication_origin_drop(). See the discussion in [1] related to it. If we want we can optimize it so that we can acquire the lock on the specific origin as mentioned in comments replorigin_drop_by_n

Re: Logical replication timeout problem

2023-01-22 Thread Amit Kapila
ck multiple times (it is already checked inside OutputPluginUpdateProgress). So, I would prefer a neat code here. -- With Regards, Amit Kapila.

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-22 Thread Amit Kapila
al' is used for different purposes on > > subscriber and standby nodes. > > Using the existing parameter makes sense to me. But if we use > logical_decoding_mode also on the subscriber, as Shveta Malik also > suggested, probably it's better to rename it so as not to conf

Re: Exit walsender before confirming remote flush in logical replication

2023-01-20 Thread Amit Kapila
On Tue, Jan 17, 2023 at 2:41 PM Amit Kapila wrote: > > Let me try to summarize the discussion till now. The problem we are > trying to solve here is to allow a shutdown to complete when walsender > is not able to send the entire WAL. Currently, in such cases, the > shutdown fa

Re: Adjust the description of OutputPluginCallbacks in pg-doc

2023-01-19 Thread Amit Kapila
On Fri, Jan 20, 2023 at 8:03 AM wangw.f...@fujitsu.com wrote: > > On Thurs, Jan 19, 2023 at 19:18 PM Amit Kapila > wrote: > > On Wed, Jan 11, 2023 at 4:20 PM wangw.f...@fujitsu.com > > wrote: > > > > > > When I was reading the "Logical Decod

Re: Logical replication timeout problem

2023-01-19 Thread Amit Kapila
> e.g. > BEFORE > ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, false); > AFTER > OutputPluginUpdateProgress(ctx, false); > We already check whether ctx->update_progress is defined or not which is the only extra job done by OutputPluginUpdateProgress but probably we can consolidate the checks and directly invoke OutputPluginUpdateProgress. -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-19 Thread Amit Kapila
cify the reason for not allowing this combination in the comments. -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-19 Thread Amit Kapila
subscription. If you agree this can be 0002 > patch. > Do we have any similar stats for recovery_min_apply_delay? If not, I suggest let's postpone this to see if users really need such a parameter. -- With Regards, Amit Kapila.

Re: Logical replication timeout problem

2023-01-19 Thread Amit Kapila
On Thu, Jan 19, 2023 at 4:13 PM Ashutosh Bapat wrote: > > On Wed, Jan 18, 2023 at 6:00 PM Amit Kapila wrote: > > > + */ > > + ReorderBufferUpdateProgressCB update_progress; > > > > Are you suggesting changing the name of the above variable? If so, how >

Re: Adjust the description of OutputPluginCallbacks in pg-doc

2023-01-19 Thread Amit Kapila
On Thu, Jan 19, 2023 at 4:47 PM Amit Kapila wrote: > > On Wed, Jan 11, 2023 at 4:20 PM wangw.f...@fujitsu.com > wrote: > > > > When I was reading the "Logical Decoding Output Plugins" chapter in pg-doc > > [1], > > I think in the summary s

Re: Adjust the description of OutputPluginCallbacks in pg-doc

2023-01-19 Thread Amit Kapila
hese to be more reader friendly. So I tried to write > a patch for these and attach it. > > Any thoughts? > This looks mostly good to me. I have made minor adjustments in the attached. Do those make sense to you? -- With Regards, Amit Kapila. v2-0001-Improve-the-description-of-Output-Plugin-Callback.patch Description: Binary data

Re: Support logical replication of DDLs

2023-01-18 Thread Amit Kapila
On Thu, Jan 19, 2023 at 8:39 AM Zheng Li wrote: > > On Wed, Jan 18, 2023 at 6:27 AM Amit Kapila wrote: > > > > On Sat, Jan 7, 2023 at 8:58 PM Zheng Li wrote: > > > > > Foreign Tables can also be considered replicated with DDL replication because > we > d

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-18 Thread Amit Kapila
On Wed, Jan 18, 2023 at 12:09 PM Amit Kapila wrote: > > On Fri, Jan 13, 2023 at 11:50 AM Peter Smith wrote: > > > > Here are some review comments for patch v79-0002. > > > > So, this is about the latest v84-0001-Stop-extra-worker-if-GUC-was-changed. > > &g

Re: Logical replication timeout problem

2023-01-18 Thread Amit Kapila
On Wed, Jan 18, 2023 at 5:37 PM Ashutosh Bapat wrote: > > On Wed, Jan 18, 2023 at 1:49 PM wangw.f...@fujitsu.com > wrote: > > > > On Wed, Jan 18, 2023 at 13:29 PM Amit Kapila > > wrote: > > > On Tue, Jan 17, 2023 at 6:41 PM Ashutosh Bapat > > > wro

Re: Support logical replication of DDLs

2023-01-18 Thread Amit Kapila
ght. When you say 'all', how do you think it will help to support DDL replication for foreign tables, materialized views, views, etc where changes to such relations are currently not supported by logical replication? We should also think about initial sync for all those objects as well. -- With Regards, Amit Kapila.

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-17 Thread Amit Kapila
ng-running transactions that can block parallel apply for a bit longer time). I know with this as well it may not be straightforward to test the functionality because we can't be sure how many changes would be required for a timeout to occur. This is just for brainstorming other options to test the partial serialization functionality. Thoughts? -- With Regards, Amit Kapila.

Re: Logical replication timeout problem

2023-01-17 Thread Amit Kapila
On Tue, Jan 17, 2023 at 6:41 PM Ashutosh Bapat wrote: > > On Tue, Jan 17, 2023 at 3:34 PM Amit Kapila wrote: > > > > > > > I am a bit worried about the indirections that the wrappers and hooks > > > create. Output plugins call OutputPluginUpdateProgress() in

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-17 Thread Amit Kapila
o get completed after the subscription is disabled then we can anyway do it later as well. -- With Regards, Amit Kapila.

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-17 Thread Amit Kapila
be contended. > > > > Thanks for the comment and I think your suggestion makes sense. > > I have added the check before getting the leader pid. Here is the new > > version patch. > > Thank you for updating the patch. Looks good to me. > Pushed. -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-17 Thread Amit Kapila
moting in the middle of > delay. > What causes such a transaction to be visible after promotion? Ideally, if the commit doesn't succeed, the transaction shouldn't be visible. Do, we allow the transaction waiting due to delay to get committed on promotion? > I'm not sure if I should make the time-delayed LR aligned with this behavior. > Does someone has an opinion for this ? > Can you please explain a bit more as asked above to understand the difference? -- With Regards, Amit Kapila.

Re: Logical replication timeout problem

2023-01-17 Thread Amit Kapila
On Mon, Jan 16, 2023 at 10:06 PM Ashutosh Bapat wrote: > > On Wed, Jan 11, 2023 at 4:11 PM wangw.f...@fujitsu.com > wrote: > > > > On Mon, Jan 9, 2023 at 13:04 PM Amit Kapila wrote: > > > > > > > Thanks for your comments. > > > > > O

Re: Exit walsender before confirming remote flush in logical replication

2023-01-17 Thread Amit Kapila
the patch and document the risks related to (a); (2) Fix both (a) and (b); (3) Do nothing and document that users need to unblock the subscribers to complete the shutdown. Thoughts? -- With Regards, Amit Kapila.

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-16 Thread Amit Kapila
On Tue, Jan 17, 2023 at 8:59 AM Amit Kapila wrote: > > On Tue, Jan 17, 2023 at 8:35 AM Masahiko Sawada wrote: > > > > On Mon, Jan 16, 2023 at 3:19 PM Amit Kapila wrote: > > > > > > Okay, I have added the comments in get_transaction_apply_action() and > &

Re: typo in the subscription command tests

2023-01-16 Thread Amit Kapila
On Tue, Jan 17, 2023 at 8:30 AM Takamichi Osumi (Fujitsu) wrote: > > While working on some logical replication patch, > I've find a typo on HEAD. > Attached the modification patch for this. > LGTM. -- With Regards, Amit Kapila.

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-16 Thread Amit Kapila
On Tue, Jan 17, 2023 at 8:35 AM Masahiko Sawada wrote: > > On Mon, Jan 16, 2023 at 3:19 PM Amit Kapila wrote: > > > > Okay, I have added the comments in get_transaction_apply_action() and > > updated the comments to refer to the enum TransApplyAction where all >

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-15 Thread Amit Kapila
procpid) > ~ > > 3b. > It may be unrelated to this patch, but it seems strange to me that the > nulls[28]/values[28] assignments are done where they are. Every other > nulls/values assignment of this function here is pretty much in the > correct numerical order except this one, so IMO this code ought to be > relocated to later in this same function. > This is not related to the current patch but I see there is merit in the current coding as it is better to retrieve all the fields of proc together. -- With Regards, Amit Kapila.

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-15 Thread Amit Kapila
nother type of protocol message with the transactional vs. > non-transactional behavior, similar to "logical messages" except that in > this case the worker does not ignore that. > > Also, I think get_transaction_apply_action() would deserve better > comments explaining how/why it makes the decision

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-14 Thread Amit Kapila
anges in the comments, docs, and commit message. I am planning to push this next week by Tuesday unless you or others have any major comments. -- With Regards, Amit Kapila. v81-0001-Display-the-leader-apply-worker-s-PID-for-parall.patch Description: Binary data

Re: backup.sgml typo

2023-01-13 Thread Amit Kapila
On Sat, Jan 14, 2023 at 7:32 AM Tatsuo Ishii wrote: > > There seem to be a small typo in backup.sgml > (archive_command is unnecessarily > repeated). Attached is a patch to fix that. > LGTM. -- With Regards, Amit Kapila.

Re: Add BufFileRead variants with short read and EOF detection

2023-01-13 Thread Amit Kapila
On Thu, Jan 12, 2023 at 2:44 PM Peter Eisentraut wrote: > > On 10.01.23 07:20, Amit Kapila wrote: > > Yeah, we can do that but not sure if it is worth doing any of those > > because there are already other places that don't use the exact > > context. > > Ok, up

Re: Fix pg_publication_tables to exclude generated columns

2023-01-13 Thread Amit Kapila
On Thu, Jan 12, 2023 at 12:33 PM shiy.f...@fujitsu.com wrote: > > On Wed, Jan 11, 2023 2:40 PM Amit Kapila wrote: > > > > On Wed, Jan 11, 2023 at 10:07 AM Tom Lane wrote: > > > > > > Amit Kapila writes: > > > >> On Mon, Jan 9, 2023 11:06 PM

Re: Exit walsender before confirming remote flush in logical replication

2023-01-13 Thread Amit Kapila
it = remote_apply? In that case, won't it ensure that apply has also happened? If so, then shouldn't the time delay feature also cause a similar problem for physical replication as well? -- With Regards, Amit Kapila.

Re: releasing ParallelApplyTxnHash when pa_launch_parallel_worker returns NULL

2023-01-13 Thread Amit Kapila
On Thu, Jan 12, 2023 at 8:25 AM Ted Yu wrote: > On Wed, Jan 11, 2023 at 6:54 PM Amit Kapila wrote: >> >> On Wed, Jan 11, 2023 at 9:31 AM Ted Yu wrote: >> > >> > On Tue, Jan 10, 2023 at 7:55 PM houzj.f...@fujitsu.com >> > wrote: >> >>

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-12 Thread Amit Kapila
On Fri, Jan 13, 2023 at 9:06 AM Amit Kapila wrote: > > On Fri, Jan 13, 2023 at 7:56 AM Peter Smith wrote: > > > > > > > 3. > > > > > > + leader_pid integer > > + > > + > > + Process ID of the lead

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-12 Thread Amit Kapila
st.received_lsn, > st.last_msg_send_time, > > IMO it would be very useful to have an additional "kind" attribute for > this view. This will save the user from needing to do mental > gymnastics every time just to recognise what kind of process they are > looking at. > This could be a separate enhancement as the same should be true for sync workers. -- With Regards, Amit Kapila.

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-12 Thread Amit Kapila
On Thu, Jan 12, 2023 at 4:21 PM shveta malik wrote: > > On Thu, Jan 12, 2023 at 10:34 AM Amit Kapila wrote: > > > > On Thu, Jan 12, 2023 at 9:54 AM Peter Smith wrote: > > > > > > > > > doc/src/sgml/monitoring.sgml > > > > > > 5.

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-11 Thread Amit Kapila
t #1 about terminology) > > "apply_leader_pid" --> "leader_apply_pid" > How about naming this as just leader_pid? I think it could be helpful in the future if we decide to parallelize initial sync (aka parallel copy) because then we could use this for the leader PID of parallel sync workers as well. -- With Regards, Amit Kapila.

Re: How to generate the new expected out file.

2023-01-11 Thread Amit Kapila
On Thu, Jan 12, 2023 at 4:39 AM Andres Freund wrote: > > On 2023-01-05 16:22:01 +0530, Amit Kapila wrote: > > On Thu, Jan 5, 2023 at 4:12 PM jian he wrote: > > > Hi. > > > > > > I changed the src/test/regress/sql/interval.sql, How can I generate t

Re: releasing ParallelApplyTxnHash when pa_launch_parallel_worker returns NULL

2023-01-11 Thread Amit Kapila
allel mode. >> > > > I think even though the chance is rare, we shouldn't leak resource. > But that is true iff we are never able to start the worker. Anyway, I think it is probably fine either way but we can change it as per your suggestion to make it more robust and probably for the code clarity sake. I'll push this tomorrow unless someone thinks otherwise. -- With Regards, Amit Kapila.

Re: Fix pg_publication_tables to exclude generated columns

2023-01-10 Thread Amit Kapila
On Wed, Jan 11, 2023 at 10:07 AM Tom Lane wrote: > > Amit Kapila writes: > >> On Mon, Jan 9, 2023 11:06 PM Tom Lane wrote: > >>> We could just not fix it in the back branches. I'd argue that this is > >>> as much a definition change as a bug fix, so

Re: Fix pg_publication_tables to exclude generated columns

2023-01-10 Thread Amit Kapila
On Tue, Jan 10, 2023 at 8:38 AM shiy.f...@fujitsu.com wrote: > > On Mon, Jan 9, 2023 11:06 PM Tom Lane wrote: > > > > Amit Kapila writes: > > > On Mon, Jan 9, 2023 at 5:29 PM shiy.f...@fujitsu.com > > > wrote: > > >> I think one way to fi

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-10 Thread Amit Kapila
eader apply worker, if this process is a parallel apply worker. NULL if this process is a leader apply worker or does not participate in parallel apply, or a synchronization worker."? -- With Regards, Amit Kapila.

Re: Handle infinite recursion in logical replication setup

2023-01-10 Thread Amit Kapila
On Tue, Jan 10, 2023 at 11:24 PM Jonathan S. Katz wrote: > > On 1/10/23 10:17 AM, Amit Kapila wrote: > > On Tue, Jan 10, 2023 at 8:13 AM Jonathan S. Katz > > wrote: > > > One can use local or higher > > for reducing the latency for COMMIT when synchrono

Re: Handle infinite recursion in logical replication setup

2023-01-10 Thread Amit Kapila
On Tue, Jan 10, 2023 at 8:13 AM Jonathan S. Katz wrote: > > On 9/12/22 1:23 AM, vignesh C wrote: > > On Fri, 9 Sept 2022 at 11:12, Amit Kapila wrote: > > > > Thanks for pushing the patch. I have closed this entry in commitfest. > > I will wait for some more time

Re: typos

2023-01-10 Thread Amit Kapila
On Tue, Jan 10, 2023 at 1:18 PM Michael Paquier wrote: > > On Tue, Jan 10, 2023 at 12:24:40PM +0530, Amit Kapila wrote: > > Thanks for noticing this. I'll take care of this and some other typo > > patches together. > > Does this include 0010? I was just looking at

Re: typos

2023-01-09 Thread Amit Kapila
; > > > It seems to matter because otherwise the translators sometimes re-type > > the view name, which (not surprisingly) can get messed up, which is how > > I mentioned having noticed this. > > > > On Tue, Jan 03, 2023 at 05:41:58PM +0900, Michael Paquier wrote: &g

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-09 Thread Amit Kapila
On Tue, Jan 10, 2023 at 11:16 AM Kyotaro Horiguchi wrote: > > At Mon, 9 Jan 2023 14:21:03 +0530, Amit Kapila > wrote in > > Pushed the first (0001) patch. > > It added the following error message. > > + seg = dsm_attach(handle); > + if (!seg) &

Re: Add BufFileRead variants with short read and EOF detection

2023-01-09 Thread Amit Kapila
On Fri, Jan 6, 2023 at 6:18 PM Peter Eisentraut wrote: > > On 02.01.23 13:13, Amit Kapila wrote: > > On Wed, Dec 28, 2022 at 4:17 PM Peter Eisentraut > > wrote: > >> > >> Most callers of BufFileRead() want to check whether they read the full > >> s

Re: wake up logical workers after ALTER SUBSCRIPTION

2023-01-09 Thread Amit Kapila
? Another minor point: Don't we need to set the launcher's latch after removing the entry from the hash table to avoid the launcher waiting on the latch for a bit longer? -- With Regards, Amit Kapila.

Re: Fix pg_publication_tables to exclude generated columns

2023-01-09 Thread Amit Kapila
ack-branch. Another way is to modify function > pg_get_publication_tables()'s return value to contain all supported columns if > no column list is specified, and we don't need to change system view. > That sounds like a reasonable approach to fix the issue. -- With Regards, Amit Kapila.

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-09 Thread Amit Kapila
gain with commit message. > Pushed the first (0001) patch. -- With Regards, Amit Kapila.

Re: Unwanted file mode modification?

2023-01-09 Thread Amit Kapila
On Mon, Jan 9, 2023 at 1:21 PM Japin Li wrote: > > Commit 216a784829 change the src/backend/replication/logical/worker.c file > mode > from 0644 to 0755, which is unwanted, right? > Right, it is by mistake. I'll fix it. -- With Regards, Amit Kapila.

Re: Logical replication timeout problem

2023-01-08 Thread Amit Kapila
API via reorder buffer as suggested previously [2] similar to other reorder buffer APIs instead of directly using reorderbuffer API to invoke plugin API. [1] - https://www.postgresql.org/message-id/OS3PR01MB6275DFFDAC7A59FA148931529E209%40OS3PR01MB6275.jpnprd01.prod.outlook.com [2] - https://www.postgresql.org/message-id/CAA4eK1%2BfQjndoBOFUn9Wy0hhm3MLyUWEpcT9O7iuCELktfdBiQ%40mail.gmail.com -- With Regards, Amit Kapila.

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-07 Thread Amit Kapila
But otherwise, it should be okay w.r.t DDLs because this patch allows the leader worker to restart logical replication for subscription parameter change which will in turn stop/restart parallel workers if required. -- With Regards, Amit Kapila.

Re: Notify downstream to discard the streamed transaction which was aborted due to crash.

2023-01-07 Thread Amit Kapila
eem easy to write a stable test for this. > > > > Thoughts ? > > Fix looks good to me. Thanks for working on this. > Pushed! -- With Regards, Amit Kapila.

Re: Fix showing XID of a spectoken lock in an incorrect field of pg_locks view.

2023-01-06 Thread Amit Kapila
action id and spec token number. As this is not a very critical issue and is not reported till now, so it may be better to leave backpatching it. What do you think? -- With Regards, Amit Kapila.

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-05 Thread Amit Kapila
IZE_IN_PROGRESS it will frequently call > pa_get_fileset_state() consecutively 2 times, and I think we can > easily achieve the same behavior with just one call. > This is just to keep the code easy to follow. As this would be a rare case, so thought of giving preference to code clarity. -- With Regards, Amit Kapila.

Re: Notify downstream to discard the streamed transaction which was aborted due to crash.

2023-01-05 Thread Amit Kapila
looks good to me. Have you tried this in PG-14 where it was introduced? -- With Regards, Amit Kapila.

Re: wake up logical workers after ALTER SUBSCRIPTION

2023-01-05 Thread Amit Kapila
On Thu, Jan 5, 2023 at 10:49 PM Nathan Bossart wrote: > > On Thu, Jan 05, 2023 at 11:34:37AM +0530, Amit Kapila wrote: > > On Thu, Jan 5, 2023 at 10:16 AM Nathan Bossart > > wrote: > >> In v12, I moved the restart for two_phase mode to the end of > >> proces

Re: Ignore dropped columns when checking the column list in logical replication

2023-01-05 Thread Amit Kapila
On Wed, Jan 4, 2023 at 12:28 PM shiy.f...@fujitsu.com wrote: > > I tried to fix them in the attached patch. > Don't we need a similar handling for generated columns? We don't send those to the subscriber side, see checks in proto.c. -- With Regards, Amit Kapila.

Re: How to generate the new expected out file.

2023-01-05 Thread Amit Kapila
rc/test/regress/expected/interval.out -- With Regards, Amit Kapila.

Re: pgsql: Delay commit status checks until freezing executes.

2023-01-04 Thread Amit Kapila
n miss something important even after detailed reviews by others but I think the chances will be much lower. You are an extremely valuable person for this project and I wish that you continue working with the same enthusiasm. -- With Regards, Amit Kapila.

Re: wake up logical workers after ALTER SUBSCRIPTION

2023-01-04 Thread Amit Kapila
On Thu, Jan 5, 2023 at 10:16 AM Nathan Bossart wrote: > > On Wed, Jan 04, 2023 at 08:12:37PM -0800, Nathan Bossart wrote: > > On Thu, Jan 05, 2023 at 09:09:12AM +0530, Amit Kapila wrote: > >> But there doesn't appear to be any guarantee that the result for > >&g

<    12   13   14   15   16   17   18   19   20   21   >