Re: Parallel Aggregates for string_agg and array_agg

2022-12-26 Thread Pavel Stehule
Hi út 27. 12. 2022 v 6:26 odesílatel David Rowley napsal: > On Wed, 20 Jun 2018 at 19:08, David Rowley > wrote: > > I'll submit it again when there more consensus that we want this. > > Waking up this old thread again. If you don't have a copy, the entire > thread is in [1]. > > The remaining i

RE: New strategies for freezing, advancing relfrozenxid early

2022-12-26 Thread Hayato Kuroda (Fujitsu)
Dear Peter, > Anyway, this assertion is wrong, and simply needs to be removed. > Thanks for the report Thanks for modifying for quickly! I found your commit in the remote repository. I will watch and report again if there are another issue. Best Regards, Hayato Kuroda FUJITSU LIMITED

Re: Exit walsender before confirming remote flush in logical replication

2022-12-26 Thread Dilip Kumar
On Thu, Dec 22, 2022 at 11:16 AM Hayato Kuroda (Fujitsu) wrote: > In case of logical replication, however, we cannot support the use-case that > switches the role publisher <-> subscriber. Suppose same case as above, > additional > transactions are committed while doing step2. To catch up such c

Re: New strategies for freezing, advancing relfrozenxid early

2022-12-26 Thread Peter Geoghegan
On Mon, Dec 26, 2022 at 10:57 PM Hayato Kuroda (Fujitsu) wrote: > I guessed that this assertion failure seemed to be caused by the commit > 4ce3af[2], > because the Assert() seemed to be added by the commit. I agree that the problem is with this assertion, which is on the master branch (not in r

Re: An oversight in ExecInitAgg for grouping sets

2022-12-26 Thread Richard Guo
On Thu, Dec 22, 2022 at 2:02 PM Richard Guo wrote: > I happened to notice $subject. It happens when we build eqfunctions for > each grouping set. > > /* for each grouping set */ > for (int k = 0; k < phasedata->numsets; k++) > { > int length = phasedata->gset_leng

RE: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-26 Thread Hayato Kuroda (Fujitsu)
Hi hackers, > On Thursday, December 22, 2022 3:02 PM Takamichi Osumi (Fujitsu) > wrote: > > Attached the updated patch. > > Again, I used one basic patch in another thread to wake up logical > > replication > > worker shared in [2] for TAP tests. > The v11 caused a cfbot failure in [1]. But, fai

Re: Allow placeholders in ALTER ROLE w/o superuser

2022-12-26 Thread Tom Lane
Justin Pryzby writes: > FYI: this causes meson test running ("installcheck") fail when run > twice. I guess that's expected to work, per: We do indeed expect that to work ... but I don't see any problem with repeat "make installcheck" on HEAD. Can you provide more detail about what you're seein

RE: New strategies for freezing, advancing relfrozenxid early

2022-12-26 Thread Hayato Kuroda (Fujitsu)
Dear Peter, Jeff, While reviewing other patches, I found that cfbot raised ERROR during the VACUUM FREEZE [1] on FreeBSD instance. It seemed that same error has been occurred in other threads. ``` 2022-12-23 08:50:20.175 UTC [34653][postmaster] LOG: server process (PID 37171) was terminated by

Re: build gcc warning

2022-12-26 Thread Tom Lane
Pavel Stehule writes: > I got new warning > analyze.c: In function ‘transformStmt’: > analyze.c:550:21: warning: ‘sub_rteperminfos’ may be used uninitialized > [-Wmaybe-uninitialized] A couple of buildfarm animals are warning about that too ... but only a couple. regards,

Re: Allow placeholders in ALTER ROLE w/o superuser

2022-12-26 Thread Justin Pryzby
On Fri, Dec 09, 2022 at 01:23:56PM +0300, Alexander Korotkov wrote: > Pushed, thanks to everyone! FYI: this causes meson test running ("installcheck") fail when run twice. I guess that's expected to work, per: b62303794efd97f2afb55f1e1b82fffae2cf8a2d f3bbe81db0e84fb486c6423a234c47091b30 6928

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-26 Thread Amit Kapila
On Tue, Dec 27, 2022 at 11:42 AM Dilip Kumar wrote: > > On Tue, Dec 27, 2022 at 9:33 AM Amit Kapila wrote: > > > > > > > BTW, the blocking problem with this patch is to deal with shutdown as > > discussed in the thread [1]. > > I will have a look. > Thanks! > In short, the problem is that at >

Re: Perform streaming logical transactions by background workers and parallel apply

2022-12-26 Thread Amit Kapila
On Tue, Dec 27, 2022 at 11:28 AM Masahiko Sawada wrote: > > On Mon, Dec 26, 2022 at 10:29 PM Amit Kapila wrote: > > > > On Mon, Dec 26, 2022 at 6:33 PM Masahiko Sawada > > wrote: > > > > > > --- > > > +if (!pa_can_start(xid)) > > > +return; > > > + > > > +/* Firs

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-12-26 Thread Masahiko Sawada
On Tue, Dec 27, 2022 at 2:24 PM John Naylor wrote: > > On Tue, Dec 27, 2022 at 12:14 AM Masahiko Sawada > wrote: > > > > On Fri, Dec 23, 2022 at 8:47 PM John Naylor > > wrote: > > > These 4 patches make sense to me.We can merge them into 0002 patch > > Okay, then I'll squash them when I post my

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-26 Thread Dilip Kumar
On Tue, Dec 27, 2022 at 9:33 AM Amit Kapila wrote: > > > BTW, the blocking problem with this patch is to deal with shutdown as > discussed in the thread [1]. I will have a look. In short, the problem is that at > shutdown, we wait for walsender to send all pending data and ensure > all data is

Re: Perform streaming logical transactions by background workers and parallel apply

2022-12-26 Thread Masahiko Sawada
On Mon, Dec 26, 2022 at 10:29 PM Amit Kapila wrote: > > On Mon, Dec 26, 2022 at 6:33 PM Masahiko Sawada wrote: > > > > --- > > +if (!pa_can_start(xid)) > > +return; > > + > > +/* First time through, initialize parallel apply worker state > > hashtable. */ > > +

build gcc warning

2022-12-26 Thread Pavel Stehule
Hi I got new warning -o session.bc session.c analyze.c: In function ‘transformStmt’: analyze.c:550:21: warning: ‘sub_rteperminfos’ may be used uninitialized [-Wmaybe-uninitialized] 550 | List *sub_rteperminfos; | ^~~~ <-->if (isGeneralSelect

Re: [BUG] pg_upgrade test fails from older versions.

2022-12-26 Thread Michael Paquier
On Mon, Dec 26, 2022 at 09:22:08AM +0300, Anton A. Melnikov wrote: > Made a separate patch for it: v3-0001-Fix-dumps-filtering.patch Well, the thing about this part is that is it is not needed: the same can be achieved with 0002 in place. > Yes, indeed. It will be really simpler. > Made it in the

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-12-26 Thread John Naylor
On Tue, Dec 27, 2022 at 12:14 AM Masahiko Sawada wrote: > > On Fri, Dec 23, 2022 at 8:47 PM John Naylor > wrote: > These 4 patches make sense to me.We can merge them into 0002 patch Okay, then I'll squash them when I post my next patch. > and I'll do similar changes for functions for leaf node

Re: Perform streaming logical transactions by background workers and parallel apply

2022-12-26 Thread Amit Kapila
On Tue, Dec 27, 2022 at 10:36 AM Dilip Kumar wrote: > > On Tue, Dec 27, 2022 at 9:15 AM Amit Kapila wrote: > > > > On Mon, Dec 26, 2022 at 7:35 PM Dilip Kumar wrote: > > > > > > In the commit message, there is a statement like this > > > > > > "However, if the leader apply worker times out while

Re: Perform streaming logical transactions by background workers and parallel apply

2022-12-26 Thread Dilip Kumar
On Tue, Dec 27, 2022 at 9:15 AM Amit Kapila wrote: > > On Mon, Dec 26, 2022 at 7:35 PM Dilip Kumar wrote: > > > > In the commit message, there is a statement like this > > > > "However, if the leader apply worker times out while attempting to > > send a message to the > > parallel apply worker, i

Re: Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2022-12-26 Thread Amit Kapila
On Mon, Dec 26, 2022 at 8:50 PM Michail Nikolaev wrote: > > Hello again. > > Just small a fix for: > > > 2022-12-14 09:21:25.705 to > > 2022-12-14 09:49:20.664 (after synchronization start, but before finish). > > Correct values are: > > 2022-12-14 09:49:31.340 > 2022-12-14 09:49:41.683 > > So, it

RE: Perform streaming logical transactions by background workers and parallel apply

2022-12-26 Thread wangw.f...@fujitsu.com
On Mon, Dec 26, 2022 21:02 PM Masahiko Sawada wrote: > Thank you for updating the patches. Here are some comments for 0001 > and 0002 patches: Thanks for your comments. > I think it'd be better to write logs when the leader enters the > serialization mode. It would be helpful for investigating i

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-26 Thread Amit Kapila
On Mon, Dec 26, 2022 at 7:37 PM Dilip Kumar wrote: > > On Mon, Dec 26, 2022 at 2:44 PM Amit Kapila wrote: > > > > On Mon, Dec 26, 2022 at 2:12 PM Dilip Kumar wrote: > > > > > > On Fri, Dec 23, 2022 at 9:16 PM Takamichi Osumi (Fujitsu) > > > wrote: > > > > > > > > > > 4. > > > > > > + * Alth

Re: Perform streaming logical transactions by background workers and parallel apply

2022-12-26 Thread Amit Kapila
On Mon, Dec 26, 2022 at 7:35 PM Dilip Kumar wrote: > > In the commit message, there is a statement like this > > "However, if the leader apply worker times out while attempting to > send a message to the > parallel apply worker, it will switch to "partial serialize" mode - in this > mode the lead

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-26 Thread Michael Paquier
On Mon, Dec 26, 2022 at 04:18:18PM +0530, Bharath Rupireddy wrote: > +1. I think this feature will also be useful in pg_walinspect. > However, I'm a bit concerned that it can flood the running database > disk - if someone generates a lot of FPI files. pg_read_file() and pg_waldump can be fed absol

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-26 Thread Michael Paquier
On Mon, Dec 26, 2022 at 02:39:03PM -0600, Justin Pryzby wrote: > fclose() should be tested, too: Sure. Done that too, and applied the change after a last lookup. -- Michael signature.asc Description: PGP signature

Re: Error-safe user functions

2022-12-26 Thread Tom Lane
I wrote: > (Perhaps we should go further than this, and convert all these > functions to just be DirectInputFunctionCallSafe wrappers > around the corresponding input functions? That would save > some duplicative code, but I've not done it here.) I looked closer at that idea, and realized that it

Re: Passing relation metadata to Exec routine

2022-12-26 Thread Tom Lane
Nikita Malakhov writes: > While working on Pluggable TOAST [1] we found out that creation > of new relation with CREATE TABLE AS... or CREATE TABLE LIKE - > method > static ObjectAddress create_ctas_internal(List *attrList, IntoClause *into) > does not receive any metadata from columns or tables u

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-26 Thread Justin Pryzby
On Mon, Dec 26, 2022 at 04:28:52PM +0900, Michael Paquier wrote: > Comments? > + file = fopen(filename, PG_BINARY_W); > + if (!file) > + pg_fatal("could not open file \"%s\": %m", filename); > + > + if (fwrite(page, BLCKSZ, 1, file) != 1) > +

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-26 Thread David Christensen
> On Dec 26, 2022, at 1:29 AM, Michael Paquier wrote: > > On Sat, Dec 24, 2022 at 06:23:29PM +0530, Bharath Rupireddy wrote: >> Thanks for the patch. I've made the above change as well as renamed >> the test file name to be save_fpi.pl, everything else remains the same >> as v11. Here's the v12

Passing relation metadata to Exec routine

2022-12-26 Thread Nikita Malakhov
Hi hackers! While working on Pluggable TOAST [1] we found out that creation of new relation with CREATE TABLE AS... or CREATE TABLE LIKE - method static ObjectAddress create_ctas_internal(List *attrList, IntoClause *into) does not receive any metadata from columns or tables used in query (if any).

Re: Error-safe user functions

2022-12-26 Thread Andrew Dunstan
On 2022-12-26 Mo 12:47, Tom Lane wrote: > Here's a proposed patch for making tsvectorin and tsqueryin > report errors softly. We have to take the changes down a > couple of levels of subroutines, but it's not hugely difficult. Great! > > With the other patches I've posted recently, this cove

Re: ARRNELEMS Out-of-bounds possible errors

2022-12-26 Thread Ranier Vilela
Em seg., 26 de dez. de 2022 às 15:45, Nikita Malakhov escreveu: > Hi, > > My bad, I was misleaded by unconditional return in ereturn_domain. > Sorry for the noise. > By no means, the mistake was entirely mine, I apologize to you. regards, Ranier Vilela

Re: ARRNELEMS Out-of-bounds possible errors

2022-12-26 Thread Nikita Malakhov
Hi, My bad, I was misleaded by unconditional return in ereturn_domain. Sorry for the noise. On Sat, Dec 24, 2022 at 7:05 PM Tom Lane wrote: > Nikita Malakhov writes: > > Even with null context it does not turn to ereport, and returns dummy > value > > Read the code. ArrayGetNItems passes NUL

Re: Error-safe user functions

2022-12-26 Thread Tom Lane
Here's a proposed patch for making tsvectorin and tsqueryin report errors softly. We have to take the changes down a couple of levels of subroutines, but it's not hugely difficult. A couple of points worthy of comment: * To reduce API changes, I made the functions in tsvector_parser.c and tsquer

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-12-26 Thread Masahiko Sawada
On Fri, Dec 23, 2022 at 8:47 PM John Naylor wrote: > > I wrote: > > > - Try templating out the differences between local and shared memory. > > Here is a brief progress report before Christmas vacation. Thanks! > > I thought the best way to approach this was to go "inside out", that is, > start

Re: Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2022-12-26 Thread Michail Nikolaev
Hello again. Just small a fix for: > 2022-12-14 09:21:25.705 to > 2022-12-14 09:49:20.664 (after synchronization start, but before finish). Correct values are: 2022-12-14 09:49:31.340 2022-12-14 09:49:41.683 So, it looks like we lost about 10s of one of the tables WAL.

Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2022-12-26 Thread Michail Nikolaev
Hello. Just a small story about small data-loss on logical replication. We were logically replicating a 4 TB database from > PostgreSQL 12.12 (Ubuntu 12.12-201-yandex.49163.d86383ed5b) on > x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, > 64-bit to > PostgreSQL 14.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-26 Thread Dilip Kumar
On Mon, Dec 26, 2022 at 2:44 PM Amit Kapila wrote: > > On Mon, Dec 26, 2022 at 2:12 PM Dilip Kumar wrote: > > > > On Fri, Dec 23, 2022 at 9:16 PM Takamichi Osumi (Fujitsu) > > wrote: > > > > > > > 4. > > > > + * Although the delay is applied in BEGIN messages, streamed > > transactions > >

Re: Perform streaming logical transactions by background workers and parallel apply

2022-12-26 Thread Dilip Kumar
On Mon, Dec 26, 2022 at 6:59 PM Amit Kapila wrote: > In the commit message, there is a statement like this "However, if the leader apply worker times out while attempting to send a message to the parallel apply worker, it will switch to "partial serialize" mode - in this mode the leader serial

Re: Error-safe user functions

2022-12-26 Thread Andrew Dunstan
On 2022-12-25 Su 12:13, Tom Lane wrote: > Robert Haas writes: >> On Fri, Dec 16, 2022 at 1:31 PM Tom Lane wrote: >>> The reg* functions probably need a unified plan as to how far >>> down we want to push non-error behavior. >> I would be in favor of an aggressive approach. > Here's a proposed p

Re: Perform streaming logical transactions by background workers and parallel apply

2022-12-26 Thread Amit Kapila
On Mon, Dec 26, 2022 at 6:33 PM Masahiko Sawada wrote: > > --- > +if (!pa_can_start(xid)) > +return; > + > +/* First time through, initialize parallel apply worker state > hashtable. */ > +if (!ParallelApplyTxnHash) > +{ > +HASHCTL

Re: Perform streaming logical transactions by background workers and parallel apply

2022-12-26 Thread Masahiko Sawada
On Mon, Dec 26, 2022 at 1:22 PM houzj.f...@fujitsu.com wrote: > > On Friday, December 23, 2022 5:20 PM houzj.f...@fujitsu.com > wrote: > > > > I noticed a CFbot failure in one of the new testcases in 015_stream.pl which > > comes from old 032_xx.pl. It's because I slightly adjusted the change si

Make IsInstallXLogFileSegmentActive() an assert-only function

2022-12-26 Thread Bharath Rupireddy
Hi, IsInstallXLogFileSegmentActive() is currently being used for assert checks. How about making it an assert-only function to disable it on production builds? This can shave an unused function on production builds. We can easily switch back when there comes a real caller outside of xlog.c. I'm at

Make use of assign_checkpoint_completion_target() to calculate CheckPointSegments correctly

2022-12-26 Thread Bharath Rupireddy
Hi, It looks like assign_checkpoint_completion_target() is defined [1], but never used, because of which CheckPointSegments may miss to account for changed checkpoint_completion_target. I'm attaching a tiny patch to fix this. Thoughts? [1] commit 88e982302684246e8af785e78a467ac37c76dee9 Author:

RE: Exit walsender before confirming remote flush in logical replication

2022-12-26 Thread Hayato Kuroda (Fujitsu)
Dear Horiguchi-san, > > Thus how about before entering an apply_delay, logrep worker sending a > > kind of crafted feedback, which reports commit_data.end_lsn as > > flushpos? A little tweak is needed in send_feedback() but seems to > > work.. > > Thanks for replying! I tested your saying but it

Re: Perform streaming logical transactions by background workers and parallel apply

2022-12-26 Thread Amit Kapila
On Mon, Dec 26, 2022 at 9:52 AM houzj.f...@fujitsu.com wrote: > > On Friday, December 23, 2022 5:20 PM houzj.f...@fujitsu.com > wrote: > > Since the GUC used to force stream changes has been committed, I removed that > patch from the patch set here and rebased the testcases based on that commit.

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-26 Thread Bharath Rupireddy
On Mon, Dec 26, 2022 at 12:59 PM Michael Paquier wrote: > > I have done a review of that, and here are my notes: > - The variable names were a bit inconsistent, so I have switched most > of the new code to use "fullpage". > > - The new test has been renamed. > > - RestoreBlockImage() would report

Re: Apply worker fails if a relation is missing on subscriber even if refresh publication has not been refreshed yet

2022-12-26 Thread Amit Kapila
On Mon, Dec 26, 2022 at 3:41 PM Melih Mutlu wrote: > >> Do you have any use case in mind where the user has added a table to >> the publication even though she doesn't want it to be replicated? One >> thing that came to my mind is that due to some reason after adding a >> table to the publication,

Re: Apply worker fails if a relation is missing on subscriber even if refresh publication has not been refreshed yet

2022-12-26 Thread Melih Mutlu
Hi Amit, Amit Kapila , 23 Ara 2022 Cum, 09:39 tarihinde şunu yazdı: > I also have the same understanding but I think if we skip replicating > some table due to the reason that the corresponding publication has > not been refreshed then it is better to LOG that information instead > of silently sk

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-26 Thread Amit Kapila
On Mon, Dec 26, 2022 at 2:12 PM Dilip Kumar wrote: > > On Fri, Dec 23, 2022 at 9:16 PM Takamichi Osumi (Fujitsu) > wrote: > > > > 4. > > + * Although the delay is applied in BEGIN messages, streamed transactions > + * apply the delay in a STREAM COMMIT message. That's ok because no > +

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2022-12-26 Thread Bharath Rupireddy
On Sun, Dec 25, 2022 at 4:55 PM Dilip Kumar wrote: > > > > This adds copying of the whole page (at least) at every WAL *record* > > > read, > > > > In the worst case yes, but that may not always be true. On a typical > > production server with decent write traffic, it happens that the > > callers

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-26 Thread Dilip Kumar
On Fri, Dec 23, 2022 at 9:16 PM Takamichi Osumi (Fujitsu) wrote: > > On Thursday, December 22, 2022 3:02 PM Takamichi Osumi (Fujitsu) > wrote: > > Attached the updated patch. > > Again, I used one basic patch in another thread to wake up logical > > replication > > worker shared in [2] for TAP

Re: Force streaming every change in logical decoding

2022-12-26 Thread Amit Kapila
On Sat, Dec 24, 2022 at 3:28 PM Dilip Kumar wrote: > > On Sat, Dec 24, 2022 at 8:56 AM Amit Kapila wrote: > > > > > > > > > > OK, I removed the modification in 022_twophase_cascade.pl and combine > > > > the two patches. > > > > > > Thank you for updating the patch. The v6 patch looks good to me

Re: Avoid lost result of recursion (src/backend/optimizer/util/inherit.c)

2022-12-26 Thread Amit Langote
On Fri, Dec 23, 2022 at 9:10 PM David Rowley wrote > On Fri, 23 Dec 2022 at 17:04, Richard Guo wrote: > > Thanks for the test! I looked at this and found that with WCO > > constraints we can also hit the buggy code. Based on David's test case, > > I came up with the following in the morning. >