Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Kyotaro Horiguchi
At Wed, 8 Dec 2021 11:47:30 +0530, Bharath Rupireddy wrote in > On Wed, Dec 8, 2021 at 10:59 AM Bossart, Nathan wrote: > > >> Another option we might want to consider is to just skip updating the > > >> state entirely for end-of-recovery checkpoints. The state would > > >> instead go straight

Re: Make pg_waldump report replication origin ID, LSN, and timestamp.

2021-12-07 Thread Michael Paquier
On Mon, Dec 06, 2021 at 11:24:09PM +0900, Masahiko Sawada wrote: > On Mon, Dec 6, 2021 at 5:09 PM Michael Paquier wrote: >> Shouldn't you check for parsed.origin_lsn instead? The replication >> origin is stored there as far as I read EndPrepare(). > > Also, looking at PrepareRedoAdd(), we check

Re: add recovery, backup, archive, streaming etc. activity messages to server logs along with ps display

2021-12-07 Thread Michael Paquier
On Wed, Dec 08, 2021 at 10:47:57AM +0530, Bharath Rupireddy wrote: > Thanks for taking a look at the patch. How about the attached v4? > > I added a CF entry - https://commitfest.postgresql.org/36/3443/ + else if (source == XLOG_FROM_STREAM) + ereport(LOG, +

Re: Skipping logical replication transactions on subscriber side

2021-12-07 Thread Masahiko Sawada
On Wed, Dec 8, 2021 at 3:50 PM Amit Kapila wrote: > > On Wed, Dec 8, 2021 at 11:48 AM Masahiko Sawada wrote: > > > > On Wed, Dec 8, 2021 at 2:16 PM Amit Kapila wrote: > > > > > > On Tue, Dec 7, 2021 at 5:06 PM Masahiko Sawada > > > wrote: > > > > > > > > On Mon, Dec 6, 2021 at 2:17 PM Amit

Re: Make mesage at end-of-recovery less scary.

2021-12-07 Thread Kyotaro Horiguchi
At Tue, 09 Nov 2021 16:27:51 +0900 (JST), Kyotaro Horiguchi wrote in > This is the updated version. > > - emode_for_currupt_record() now uses currentSource instead of > readSource. > > - If zero record length is faced, make sure the whole header is zeroed > before deciding it as the

Re: Skipping logical replication transactions on subscriber side

2021-12-07 Thread Amit Kapila
On Wed, Dec 8, 2021 at 11:48 AM Masahiko Sawada wrote: > > On Wed, Dec 8, 2021 at 2:16 PM Amit Kapila wrote: > > > > On Tue, Dec 7, 2021 at 5:06 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Dec 6, 2021 at 2:17 PM Amit Kapila > > > wrote: > > > > > > I'll submit the patch tomorrow. > > >

Re: row filtering for logical replication

2021-12-07 Thread Amit Kapila
On Mon, Dec 6, 2021 at 6:04 PM Euler Taveira wrote: > > On Mon, Dec 6, 2021, at 3:35 AM, Dilip Kumar wrote: > > On Mon, Dec 6, 2021 at 6:49 AM Euler Taveira wrote: > > > > On Fri, Dec 3, 2021, at 8:12 PM, Euler Taveira wrote: > > > > PS> I will update the commit message in the next version. I

Re: GUC flags

2021-12-07 Thread Michael Paquier
On Mon, Dec 06, 2021 at 07:36:55AM -0600, Justin Pryzby wrote: > The script checks that guc.c and sample config are consistent. > > I think your undertanding of INTENTIONALLY_NOT_INCLUDED is not right. > That's a list of stuff it "avoids reporting" as an suspected error, not an > additional list

Re: Skipping logical replication transactions on subscriber side

2021-12-07 Thread Masahiko Sawada
On Wed, Dec 8, 2021 at 2:16 PM Amit Kapila wrote: > > On Tue, Dec 7, 2021 at 5:06 PM Masahiko Sawada wrote: > > > > On Mon, Dec 6, 2021 at 2:17 PM Amit Kapila wrote: > > > > I'll submit the patch tomorrow. > > > > While updating the patch, I realized that skipping a transaction that > > is

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Bharath Rupireddy
On Wed, Dec 8, 2021 at 10:59 AM Bossart, Nathan wrote: > >> Another option we might want to consider is to just skip updating the > >> state entirely for end-of-recovery checkpoints. The state would > >> instead go straight from DB_IN_CRASH_RECOVERY to DB_IN_PRODUCTION. I > >> don't know if

Re: Data is copied twice when specifying both child and parent table in publication

2021-12-07 Thread Amit Kapila
On Wed, Dec 8, 2021 at 11:30 AM vignesh C wrote: > > On Wed, Dec 8, 2021 at 11:11 AM Amit Kapila wrote: > > > > On Tue, Dec 7, 2021 at 5:53 PM vignesh C wrote: > > > > > > On Fri, Dec 3, 2021 at 11:24 AM houzj.f...@fujitsu.com > > > wrote: > > > > > > > > > > 2) Any particular reason why the

Re: row filtering for logical replication

2021-12-07 Thread Peter Smith
On Thu, Dec 2, 2021 at 2:59 PM houzj.f...@fujitsu.com wrote: > ... > Attach the v44-0005 top-up patch. > This version addressed all the comments received so far, > mainly including the following changes: > 1) rename rfcol_valid_for_replica to rfcol_valid > 2) Remove the struct PublicationInfo and

Re: Data is copied twice when specifying both child and parent table in publication

2021-12-07 Thread vignesh C
On Wed, Dec 8, 2021 at 11:11 AM Amit Kapila wrote: > > On Tue, Dec 7, 2021 at 5:53 PM vignesh C wrote: > > > > On Fri, Dec 3, 2021 at 11:24 AM houzj.f...@fujitsu.com > > wrote: > > > > > > > 2) Any particular reason why the code and tests are backbranched but > > not the document changes? > > >

Re: Data is copied twice when specifying both child and parent table in publication

2021-12-07 Thread Amit Kapila
On Tue, Dec 7, 2021 at 5:53 PM vignesh C wrote: > > On Fri, Dec 3, 2021 at 11:24 AM houzj.f...@fujitsu.com > wrote: > > > > 2) Any particular reason why the code and tests are backbranched but > not the document changes? > I am not sure whether we need the doc change or not as this is not a new

RE: [PATCH]Comment improvement in publication.sql

2021-12-07 Thread tanghy.f...@fujitsu.com
On Wednesday, December 8, 2021 1:49 PM, vignesh C wrote: > The patch no longer applies, could you post a rebased patch. Thanks for your kindly reminder. Attached a rebased patch. Some changes in v4 patch has been fixed by 5a28324, so I deleted those changes. Regards, Tang

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 8:42 PM, "Bharath Rupireddy" wrote: > On Wed, Dec 8, 2021 at 9:49 AM Bossart, Nathan wrote: >> I think that's alright. The only other small suggestion I have would >> be to say "during end-of-recovery checkpoint" instead of "while in >> end-of-recovery checkpoint." > > "while in"

Re: row filtering for logical replication

2021-12-07 Thread Amit Kapila
On Tue, Dec 7, 2021 at 6:31 PM Ashutosh Bapat wrote: > > On Tue, Dec 7, 2021 at 12:18 PM tanghy.f...@fujitsu.com > wrote: > > > > I have another problem with your patch. The document says: > > > > ... If the subscription has several publications in > > + which the same table has been published

Re: add recovery, backup, archive, streaming etc. activity messages to server logs along with ps display

2021-12-07 Thread Bharath Rupireddy
On Wed, Dec 8, 2021 at 10:05 AM Michael Paquier wrote: > > On Tue, Dec 07, 2021 at 06:28:24PM +0530, Bharath Rupireddy wrote: > > Upon thinking further, having at least the messages at LOG level [1] > > would be helpful to know what's happening with the system while in > > recovery. Although

Re: Skipping logical replication transactions on subscriber side

2021-12-07 Thread Amit Kapila
On Tue, Dec 7, 2021 at 5:06 PM Masahiko Sawada wrote: > > On Mon, Dec 6, 2021 at 2:17 PM Amit Kapila wrote: > > I'll submit the patch tomorrow. > > While updating the patch, I realized that skipping a transaction that > is prepared on the publisher will be tricky a bit; > > First of all, since

Re: [PATCH]Comment improvement in publication.sql

2021-12-07 Thread vignesh C
On Sun, Aug 8, 2021 at 4:26 PM tanghy.f...@fujitsu.com wrote: > > On Sunday, August 8, 2021 6:34 PM, vignesh C wrote > >Thanks for the updated patch, your changes look good to me. You might > >want to include the commit message in the patch, that will be useful. > > Thanks for your kindly

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Bharath Rupireddy
On Wed, Dec 8, 2021 at 9:49 AM Bossart, Nathan wrote: > > On 12/7/21, 5:21 PM, "Bharath Rupireddy" > wrote: > > On Wed, Dec 8, 2021 at 2:50 AM Bossart, Nathan wrote: > >> I noticed that some (but not all) of the surrounding messages say > >> "last known up at" the control file time. I'm

Re: parallel vacuum comments

2021-12-07 Thread Masahiko Sawada
On Wed, Dec 8, 2021 at 12:22 PM houzj.f...@fujitsu.com wrote: > > On Tuesday, December 7, 2021 1:42 PM Masahiko Sawada > wrote: > > I've attached an updated patch. I've removed 0003 patch that added > > regression tests as per discussion. Regarding the terminology like "bulkdel" > > and

Re: add recovery, backup, archive, streaming etc. activity messages to server logs along with ps display

2021-12-07 Thread Michael Paquier
On Tue, Dec 07, 2021 at 06:28:24PM +0530, Bharath Rupireddy wrote: > Upon thinking further, having at least the messages at LOG level [1] > would be helpful to know what's happening with the system while in > recovery. Although these messages at LOG level seem to be filling up > the server logs,

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 5:21 PM, "Bharath Rupireddy" wrote: > On Wed, Dec 8, 2021 at 2:50 AM Bossart, Nathan wrote: >> I noticed that some (but not all) of the surrounding messages say >> "last known up at" the control file time. I'm curious why you chose >> not to use that phrasing in this case. > > If

Re: parse_subscription_options - suggested improvements

2021-12-07 Thread Peter Smith
On Wed, Dec 8, 2021 at 2:51 PM Michael Paquier wrote: > > On Tue, Dec 07, 2021 at 08:12:59AM +1100, Peter Smith wrote: > > On Tue, Dec 7, 2021 at 6:07 AM Bossart, Nathan wrote: > >> Attached a v14 with the initializations added back. > > > > LGTM. > > All the code paths previously covered still

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 5:21 PM, "Bharath Rupireddy" wrote: > On Wed, Dec 8, 2021 at 4:17 AM Bossart, Nathan wrote: >> I agree with Tom. I would just s/server/backend/ (as per the >> attached) and call it a day. > > Thanks. v5 patch looks good to me. I've marked the commitfest entry as

Re: parse_subscription_options - suggested improvements

2021-12-07 Thread Michael Paquier
On Tue, Dec 07, 2021 at 08:12:59AM +1100, Peter Smith wrote: > On Tue, Dec 7, 2021 at 6:07 AM Bossart, Nathan wrote: >> Attached a v14 with the initializations added back. > > LGTM. All the code paths previously covered still are, so applied this one. Thanks! -- Michael signature.asc

Re: 回复:Re: Is it worth pushing conditions to sublink/subplan?

2021-12-07 Thread Tomas Vondra
Hi, On 12/7/21 10:44, 曾文旌(义从) wrote: > Hi Hackers > > For my previous proposal, I developed a prototype and passed > regression testing. It works similarly to subquery's qual pushdown. > We know that sublink expands at the beginning of each level of > query. At this stage, The query's conditions

RE: parallel vacuum comments

2021-12-07 Thread houzj.f...@fujitsu.com
On Tuesday, December 7, 2021 1:42 PM Masahiko Sawada wrote: > I've attached an updated patch. I've removed 0003 patch that added > regression tests as per discussion. Regarding the terminology like "bulkdel" > and "cleanup" you pointed out, I've done that in 0002 patch while moving the > code to

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2021-12-07 Thread Tomas Vondra
On 12/8/21 02:54, Bharath Rupireddy wrote: > On Wed, Dec 8, 2021 at 4:07 AM Justin Pryzby wrote: >>> I'd bet squashing all of this into a single string (not really a flag) >>> will just mean people will have to parse it, etc. Keeping individual >>> boolean flags (or even separate string

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2021-12-07 Thread Bharath Rupireddy
On Wed, Dec 8, 2021 at 7:34 AM Tomas Vondra wrote: > >> I agree it might be useful to provide information about the nature of > >> the checkpoint, and perhaps even PID of the backend that triggered it > >> (although that may be tricky, if the backend terminates). > > > > Thanks. I agree to have

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2021-12-07 Thread Tomas Vondra
On 12/8/21 02:54, Bharath Rupireddy wrote: > On Wed, Dec 8, 2021 at 3:48 AM Tomas Vondra > wrote: >> >> On 12/7/21 15:36, Bharath Rupireddy wrote: >>> Hi, >>> >>> Currently one can know the kind of on-going/last checkpoint (shutdown, >>> end-of-recovery, immediate, force etc.) only via server

Re: Transparent column encryption

2021-12-07 Thread Tomas Vondra
On 12/8/21 00:26, Jacob Champion wrote: > On Tue, 2021-12-07 at 22:21 +0100, Tomas Vondra wrote: >> IMO it's impossible to solve this attack within TCE, because it requires >> ensuring consistency at the row level, but TCE obviously works at column >> level only. > > I was under the

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2021-12-07 Thread Bharath Rupireddy
On Wed, Dec 8, 2021 at 4:07 AM Justin Pryzby wrote: > > I'd bet squashing all of this into a single string (not really a flag) > > will just mean people will have to parse it, etc. Keeping individual > > boolean flags (or even separate string fields) would be better, I guess. > > Since the size

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2021-12-07 Thread Bharath Rupireddy
On Wed, Dec 8, 2021 at 3:48 AM Tomas Vondra wrote: > > On 12/7/21 15:36, Bharath Rupireddy wrote: > > Hi, > > > > Currently one can know the kind of on-going/last checkpoint (shutdown, > > end-of-recovery, immediate, force etc.) only via server logs that to > > when log_checkpoints GUC is on. At

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2021-12-07 Thread Bharath Rupireddy
On Wed, Dec 8, 2021 at 4:17 AM Bossart, Nathan wrote: > > On 11/18/21, 8:27 PM, "Bharath Rupireddy" > wrote: > > Here's the v4 patch with the above changes, the output looks like [1]. > > Please review it further. > > I agree with Tom. I would just s/server/backend/ (as per the > attached) and

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Bharath Rupireddy
On Wed, Dec 8, 2021 at 2:50 AM Bossart, Nathan wrote: > > On 12/7/21, 12:10 AM, "Bharath Rupireddy" > wrote: > > Here's a patch that I've come up with. Please see if this looks okay > > and let me know if we want to take it forward so that I can add a CF > > entry. > > Overall, the patch seems

Re: logical decoding and replication of sequences

2021-12-07 Thread Tomas Vondra
On 12/7/21 15:38, Peter Eisentraut wrote: > I have checked the 0001 and 0003 patches.  (I think we have dismissed > the approach in 0002 for now.  And let's talk about 0004 later.) > Right, I think that's correct. > I have attached a few fixup patches, described further below. > > # 0001 > >

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-07 Thread Imseih (AWS), Sami
I also want to +1 this this effort. Exposing subtransaction usage is very useful. It would also be extremely beneficial to add both subtransaction usage and overflow counters to pg_stat_database. Monitoring tools that capture deltas on pg_stat_database will be able to generate historical

Re: Transparent column encryption

2021-12-07 Thread Jacob Champion
On Tue, 2021-12-07 at 22:21 +0100, Tomas Vondra wrote: > IMO it's impossible to solve this attack within TCE, because it requires > ensuring consistency at the row level, but TCE obviously works at column > level only. I was under the impression that clients already had to be modified to figure

Re: Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-07 Thread Peter Geoghegan
On Tue, Dec 7, 2021 at 1:59 PM Robert Haas wrote: > If we're only trying to decide whether or not to vacuum a table, we > don't need units: the output is a Boolean. I was imagining a world in which we preserve the autovacuum_vacuum_scale_factor design, but interpret it creatively (but never too

Re: RecoveryInProgress() has critical side effects

2021-12-07 Thread Robert Haas
On Sat, Dec 4, 2021 at 7:44 PM Michael Paquier wrote: > My main worry here is that this changes slightly the definition of > doPageWrites across stable branches at the end of recovery as there > could be records generated there. Note that GetFullPageWriteInfo() > gets called in XLogInsert(),

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2021-12-07 Thread Justin Pryzby
On Tue, Dec 07, 2021 at 11:18:37PM +0100, Tomas Vondra wrote: > On 12/7/21 15:36, Bharath Rupireddy wrote: > > Currently one can know the kind of on-going/last checkpoint (shutdown, > > end-of-recovery, immediate, force etc.) only via server logs that to > > when log_checkpoints GUC is on. > >

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2021-12-07 Thread Tomas Vondra
On 12/7/21 15:36, Bharath Rupireddy wrote: > Hi, > > Currently one can know the kind of on-going/last checkpoint (shutdown, > end-of-recovery, immediate, force etc.) only via server logs that to > when log_checkpoints GUC is on. At times, the users/service layer > components would want to know

Re: Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-07 Thread Robert Haas
On Tue, Dec 7, 2021 at 3:44 PM Peter Geoghegan wrote: > Fair enough, but even then we still ultimately have to generate a > final number that represents how close we are to a configurable "do an > autovacuum" threshold (such as an autovacuum_vacuum_scale_factor-based > threshold) -- the

Re: Transparent column encryption

2021-12-07 Thread Tomas Vondra
On 12/7/21 19:02, Jacob Champion wrote: On Tue, 2021-12-07 at 16:39 +0100, Peter Eisentraut wrote: On 06.12.21 21:44, Jacob Champion wrote: I think reusing a zero IV will potentially leak more information than just equality, depending on the cipher in use. You may be interested in synthetic

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 12:10 AM, "Bharath Rupireddy" wrote: > Here's a patch that I've come up with. Please see if this looks okay > and let me know if we want to take it forward so that I can add a CF > entry. Overall, the patch seems reasonable to me. + case

Re: Pre-allocating WAL files

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 9:35 AM, "Bossart, Nathan" wrote: > On 12/7/21, 12:29 AM, "Bharath Rupireddy" > wrote: >> Why can't the walwriter pre-allocate some of the WAL segments instead >> of a new background process? Of course, it might delay the main >> functionality of the walwriter i.e. flush and sync

Re: Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-07 Thread Peter Geoghegan
On Tue, Dec 7, 2021 at 12:27 PM Robert Haas wrote: > Well... I mean, I think we're almost saying the same thing, then, but > I think you're saying it more confusingly. I have no objection to > counting the number of dead HOT chains rather than the number of dead > tules, because that's what

Re: enable certain TAP tests for MSVC builds

2021-12-07 Thread Andrew Dunstan
On 12/5/21 18:32, Noah Misch wrote: > On Sun, Dec 05, 2021 at 06:00:08PM -0500, Andrew Dunstan wrote: >> On 12/5/21 14:47, Noah Misch wrote: >>> On Sun, Dec 05, 2021 at 11:57:31AM -0500, Andrew Dunstan wrote: Certain TAP tests rely on settings that the Make files provide for them.

Re: Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-07 Thread Robert Haas
On Tue, Dec 7, 2021 at 2:13 PM Peter Geoghegan wrote: > For example, why should we count dead heap-only tuples from earlier in > a HOT chain, even when we see no evidence that opportunistic HOT > pruning can't keep up on that page? Since we actually care about the > direction of things, not just

Re: Dubious usage of TYPCATEGORY_STRING

2021-12-07 Thread Tom Lane
I wrote: > Peter Eisentraut writes: >> Could we add explicit casts (like polcmd::text) here? Or would it break >> too much? > I assumed it'd break too much to consider doing that. But I suppose > that since a typcategory change would be initdb-forcing anyway, maybe > it's not out of the

Re: MSVC SSL test failure

2021-12-07 Thread Andrew Dunstan
On 12/7/21 13:22, Tom Lane wrote: > Alexander Lakhin writes: >> 07.12.2021 19:25, Tom Lane wrote: >>> Hmm. I wonder whether using SD_BOTH behaves any differently. >> With shutdown(MyProcPort->sock, SD_BOTH) the test failed for me on >> iterations 1, 2, 3, 16 (just as without shutdown() at

Re: pg_dump versus ancient server versions

2021-12-07 Thread Andrew Dunstan
On 12/7/21 13:59, Mark Dilger wrote: >> On Dec 7, 2021, at 10:52 AM, Tom Lane wrote: >> >> I'm not entirely following ... are you suggesting that each released minor >> version needs to be kept buildable separately? > No. I'm just wondering if we want to share the product of such efforts if >

Re: Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-07 Thread Peter Geoghegan
On Tue, Dec 7, 2021 at 7:58 AM Robert Haas wrote: > I think that's a good observation. I think the current autovacuum > algorithm works pretty well when things are normal. But in extreme > scenarios it does not degrade gracefully. +1 to all of the specific examples you go on to describe. > > I

Re: pg_dump versus ancient server versions

2021-12-07 Thread Mark Dilger
> On Dec 7, 2021, at 10:52 AM, Tom Lane wrote: > > I'm not entirely following ... are you suggesting that each released minor > version needs to be kept buildable separately? No. I'm just wondering if we want to share the product of such efforts if anybody (me, for instance) volunteers to

Re: pg_dump versus ancient server versions

2021-12-07 Thread Tom Lane
Mark Dilger writes: > Wouldn't you be able to see what changed by comparing the last released tag > for version X.Y against the RELX_Y_STABLE branch? Something like `git diff > REL8_4_22 origin/REL8_4_STABLE > buildability.patch`? > Having such a patch should make reproducing old corruption

Re: Post-CVE Wishlist

2021-12-07 Thread Jacob Champion
On Tue, 2021-11-23 at 18:27 +, Jacob Champion wrote: > Now that the MITM CVEs are published [1], I wanted to share my wishlist > of things that would have made those attacks difficult/impossible to > pull off. Now that we're post-commitfest, here's my summary of the responses so far: > =

Re: pg_dump versus ancient server versions

2021-12-07 Thread Mark Dilger
> On Dec 7, 2021, at 8:33 AM, Robert Haas wrote: > > However, it wouldn't be a great idea to back-patch a > completely arbitrary subset of our fixes into those branches, because > then it sort of gets confusing to understand what the status of that > branch is. I don't know that I'm terribly

Re: MSVC SSL test failure

2021-12-07 Thread Tom Lane
Alexander Lakhin writes: > 07.12.2021 19:25, Tom Lane wrote: >> Hmm. I wonder whether using SD_BOTH behaves any differently. > With shutdown(MyProcPort->sock, SD_BOTH) the test failed for me on > iterations 1, 2, 3, 16 (just as without shutdown() at all). > So shutdown with the SD_SEND flag

Re: ExecTypeSetColNames is fundamentally broken

2021-12-07 Thread Tom Lane
Robert Haas writes: > On Tue, Dec 7, 2021 at 12:30 PM Tom Lane wrote: >> If we consider that the alias renames the columns "for all purposes", >> how is it okay for f() to select the "a" column? > I'd say it isn't. In a green field I'd probably agree with you, but IMO that will break far too

Re: Dubious usage of TYPCATEGORY_STRING

2021-12-07 Thread Robert Haas
On Tue, Dec 7, 2021 at 12:19 PM Tom Lane wrote: > > What's wrong with that? > > Well, we don't allow things like > > regression=# select '(1,2)'::point::float8; > ERROR: cannot cast type point to double precision > LINE 1: select '(1,2)'::point::float8; > ^ > > It's

Re: Transparent column encryption

2021-12-07 Thread Jacob Champion
On Tue, 2021-12-07 at 16:39 +0100, Peter Eisentraut wrote: > On 06.12.21 21:44, Jacob Champion wrote: > > I think reusing a zero IV will potentially leak more information than > > just equality, depending on the cipher in use. You may be interested in > > synthetic IVs and nonce-misuse resistance

Re: MSVC SSL test failure

2021-12-07 Thread Alexander Lakhin
Hello Tom, 07.12.2021 19:25, Tom Lane wrote: > Hmm. I wonder whether using SD_BOTH behaves any differently. With shutdown(MyProcPort->sock, SD_BOTH) the test failed for me on iterations 1, 2, 3, 16 (just as without shutdown() at all). So shutdown with the SD_SEND flag definitely behaves much

Re: ExecTypeSetColNames is fundamentally broken

2021-12-07 Thread Robert Haas
On Tue, Dec 7, 2021 at 12:30 PM Tom Lane wrote: > If we consider that the alias renames the columns "for all purposes", > how is it okay for f() to select the "a" column? I'd say it isn't. > Another way to phrase the issue is that the column names seen > by f() are currently different from

Re: Alter all tables in schema owner fix

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 2:47 AM, "Greg Nancarrow" wrote: > On Tue, Dec 7, 2021 at 9:01 PM Amit Kapila wrote: >> >> Thanks, the patch looks mostly good to me. I have slightly modified it >> to incorporate one of Michael's suggestions, ran pgindent, and >> modified the commit message. >> > > LGTM, except in

Re: pg_dump versus ancient server versions

2021-12-07 Thread Tom Lane
Robert Haas writes: > I guess the point about user-visible bug fixes is that, as soon as we > start doing that, we don't really want it to be hit-or-miss. We could > make a decision to back-patch all bug fixes or those of a certain > severity or whatever we like back to older branches, and then

Re: Pre-allocating WAL files

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 12:29 AM, "Bharath Rupireddy" wrote: > Why can't the walwriter pre-allocate some of the WAL segments instead > of a new background process? Of course, it might delay the main > functionality of the walwriter i.e. flush and sync the WAL files, but > having checkpointer do the

Re: ExecTypeSetColNames is fundamentally broken

2021-12-07 Thread Tom Lane
Robert Haas writes: > On Mon, Dec 6, 2021 at 4:05 PM Tom Lane wrote: >> select f(t) from t(x,y); >> >> If we adopt the "rename for all purposes" interpretation, then >> the second SELECT must fail, because what f() is being passed is >> no longer of type t. > For me, the second SELECT does

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-07 Thread Bossart, Nathan
On 12/6/21, 8:19 PM, "Dilip Kumar" wrote: > If the subtransaction cache is overflowed in some of the transactions > then it will affect all the concurrent queries as they need to access > the SLRU for checking the visibility of each tuple. But currently > there is no way to identify whether in

Re: Dubious usage of TYPCATEGORY_STRING

2021-12-07 Thread Tom Lane
Robert Haas writes: > On Thu, Dec 2, 2021 at 4:22 PM Tom Lane wrote: >> An example of the reasons not to treat these types as being >> general-purpose strings can be seen at [1], where the "char" >> type has acquired some never-intended cast behaviors. Taking >> that to an extreme, we currently

Re: pg_dump versus ancient server versions

2021-12-07 Thread Robert Haas
On Mon, Dec 6, 2021 at 4:19 PM Tom Lane wrote: > Right. The question that's on the table is how much is the right > amount of maintenance. I think that back-patching user-visible bug > fixes, for example, is taking things too far. What we want is to > be able to replicate the behavior of the

Re: MSVC SSL test failure

2021-12-07 Thread Tom Lane
Alexander Lakhin writes: > It seems that the test failure rate may depend on the specs/environment. No surprise there, since the issue is almost surely timing-dependent. > shutdown(MyProcPort->sock, SD_SEND) apparently fixes the issue, I've got > 83 successful runs, but then iteration 84

Re: ExecTypeSetColNames is fundamentally broken

2021-12-07 Thread Robert Haas
On Mon, Dec 6, 2021 at 4:05 PM Tom Lane wrote: > Well, that was what I thought when I wrote bf7ca1587, but it leads > to logical contradictions. Consider > > create table t (a int, b int); > > create function f(t) returns ... ; > > select f(t) from t; > > select f(t) from t(x,y); > > If we adopt

Re: AW: Assorted improvements in pg_dump

2021-12-07 Thread Tom Lane
Hans Buschmann writes: > I noticed that you patched master with all the improvements in pg_dump. > Did you change your mind about backpatching patch 0005 to fix the toast size > matter? I looked briefly at that and found that the patch would have to be largely rewritten, because getTables()

Re: Dubious usage of TYPCATEGORY_STRING

2021-12-07 Thread Robert Haas
On Thu, Dec 2, 2021 at 4:22 PM Tom Lane wrote: > An example of the reasons not to treat these types as being > general-purpose strings can be seen at [1], where the "char" > type has acquired some never-intended cast behaviors. Taking > that to an extreme, we currently accept > > regression=#

Re: Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-07 Thread Robert Haas
On Mon, Dec 6, 2021 at 9:42 PM Peter Geoghegan wrote: > On Mon, Dec 6, 2021 at 6:11 PM Robert Haas wrote: > > This doesn't seem convincing. Launching autovacuum too soon surely has > > costs that someone might not want to pay. Clearly in the degenerate > > case where we always autovacuum every

Re: Triage for unimplemented geometric operators

2021-12-07 Thread Anton Voloshin
On 07/12/2021 06:18, Tom Lane wrote: So my inclination for HEAD is to implement poly_distance and nuke the others. I'm a bit less sure about the back branches, but maybe just de-document all of them there. I agree, seems to be a reasonable compromise. Removing 20+-years old unused and

Re: types reliant on encodings [was Re: Dubious usage of TYPCATEGORY_STRING]

2021-12-07 Thread Peter Eisentraut
On 03.12.21 19:42, Chapman Flack wrote: Is there any way to find out, from the catalogs or in any automatable way, which types are implemented with a dependence on the database encoding (or on some encoding)? What is this needed for? C code can internally do whatever it wants, and the

Re: Dubious usage of TYPCATEGORY_STRING

2021-12-07 Thread Tom Lane
Peter Eisentraut writes: > On 02.12.21 22:22, Tom Lane wrote: >> My first thought about fixing point 1 was to put "char" into some >> other typcategory, but that turns out to break some of psql's >> catalog queries, with results like: >> >> regression=# \dp >> ERROR: operator is not unique:

Re: Dubious usage of TYPCATEGORY_STRING

2021-12-07 Thread Peter Eisentraut
On 02.12.21 22:22, Tom Lane wrote: My first thought about fixing point 1 was to put "char" into some other typcategory, but that turns out to break some of psql's catalog queries, with results like: regression=# \dp ERROR: operator is not unique: unknown || "char" LINE 16:E' (' ||

Re: Is there any documentation on how to correctly create extensions in HA(primary-standby) setup?

2021-12-07 Thread Peter Eisentraut
On 03.12.21 15:28, Bharath Rupireddy wrote: I'm thinking of adding the above steps into the "Additional Supplied Modules" section documentation. Any thoughts please? [1] - https://www.postgresql.org/docs/devel/contrib.html The chapter about extensions is probably better:

Re: Transparent column encryption

2021-12-07 Thread Peter Eisentraut
On 06.12.21 21:44, Jacob Champion wrote: I think reusing a zero IV will potentially leak more information than just equality, depending on the cipher in use. You may be interested in synthetic IVs and nonce-misuse resistance (e.g. [1]), since they seem like they would match this use case

pg_dump/restore --no-tableam

2021-12-07 Thread Justin Pryzby
I first suggested this a couple years ago. Is it desirable to implement in pg_dump and pg_restore ? It'd be just like --tablespace. On Tue, Jan 28, 2020 at 07:33:17AM -0600, Justin Pryzby wrote: > I made these casual comments. If there's any agreement on their merit, it'd > be > nice to

Re: Transparent column encryption

2021-12-07 Thread Peter Eisentraut
On 06.12.21 19:28, Robert Haas wrote: Speaking of parameter descriptions, the trickiest part of this whole thing appears to be how to get transparently encrypted data into the database (as opposed to reading it out). It is required to use protocol-level prepared statements (i.e., extended

Re: Non-superuser subscription owners

2021-12-07 Thread Mark Dilger
> On Dec 7, 2021, at 2:29 AM, Amit Kapila wrote: > > Okay, let me try to explain again. Following is the text from docs > [1]: " (a) To create a subscription, the user must be a superuser. (b) > The subscription apply process will run in the local database with the > privileges of a

Re: Skipping logical replication transactions on subscriber side

2021-12-07 Thread Peter Eisentraut
On 03.12.21 03:53, Amit Kapila wrote: I don't know how difficult it would be, but allowing multiple xids might be desirable. Are there many cases where there could be multiple xid failures that the user can skip? Apply worker always keeps looping at the same error failure so the user wouldn't

Re: logical decoding and replication of sequences

2021-12-07 Thread Peter Eisentraut
I have checked the 0001 and 0003 patches. (I think we have dismissed the approach in 0002 for now. And let's talk about 0004 later.) I have attached a few fixup patches, described further below. # 0001 The argument "create" for fill_seq_with_data() is always true (and patch 0002 removes it).

Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2021-12-07 Thread Bharath Rupireddy
Hi, Currently one can know the kind of on-going/last checkpoint (shutdown, end-of-recovery, immediate, force etc.) only via server logs that to when log_checkpoints GUC is on. At times, the users/service layer components would want to know the kind of checkpoint (along with other checkpoint

Re: should we document an example to set multiple libraries in shared_preload_libraries?

2021-12-07 Thread Bharath Rupireddy
On Mon, Dec 6, 2021 at 9:20 PM Bharath Rupireddy wrote: > > On Fri, Dec 3, 2021 at 11:25 PM Bossart, Nathan wrote: > > > > On 12/3/21, 6:21 AM, "Bharath Rupireddy" > > wrote: > > > +1 to add here in the "Parameter Names and Values section", but do we > > > want to backlink every string

Re: row filtering for logical replication

2021-12-07 Thread Ashutosh Bapat
On Tue, Dec 7, 2021 at 12:18 PM tanghy.f...@fujitsu.com wrote: > > On Friday, December 3, 2021 10:09 AM Peter Smith > wrote: > > > > On Thu, Dec 2, 2021 at 2:32 PM tanghy.f...@fujitsu.com > > wrote: > > > > > > On Thursday, December 2, 2021 5:21 AM Peter Smith > > wrote: > > > > > > > > PSA

Re: add recovery, backup, archive, streaming etc. activity messages to server logs along with ps display

2021-12-07 Thread Bharath Rupireddy
On Wed, Nov 17, 2021 at 8:01 AM Michael Paquier wrote: > > On Tue, Nov 16, 2021 at 01:26:49PM -0300, Alvaro Herrera wrote: > > My opinion is that adding these things willy-nilly is not a solution to > > any actual problem. Adding a few additional log lines that are > > low-volume at DEBUG1 might

Re: pg_get_publication_tables() output duplicate relid

2021-12-07 Thread Amit Langote
On Tue, Dec 7, 2021 at 3:20 PM Amit Kapila wrote: > On Tue, Dec 7, 2021 at 10:52 AM Amit Langote wrote: > > So if a partition is > > explicitly present in a publication with a filter defined on it, I > > suppose following are the possible scenarios: > > > > * publish_via_partition_root=true > >

Re: Data is copied twice when specifying both child and parent table in publication

2021-12-07 Thread vignesh C
On Fri, Dec 3, 2021 at 11:24 AM houzj.f...@fujitsu.com wrote: > > On Thursday, December 2, 2021 4:54 PM houzj.f...@fujitsu.com > wrote: > > On Thursday, December 2, 2021 12:50 PM Amit Kapila > > wrote: > > > On Thu, Dec 2, 2021 at 9:41 AM Greg Nancarrow > > > wrote: > > > > > > > > On Thu,

Re: Skipping logical replication transactions on subscriber side

2021-12-07 Thread Masahiko Sawada
On Mon, Dec 6, 2021 at 2:17 PM Amit Kapila wrote: > > On Fri, Dec 3, 2021 at 12:12 PM Masahiko Sawada wrote: > > > > On Fri, Dec 3, 2021 at 11:53 AM Amit Kapila wrote: > > > > > > > But this syntax gives you flexibility, so we can also > > > > start with a simple implementation. > > > > > > >

Re: Alter all tables in schema owner fix

2021-12-07 Thread Greg Nancarrow
On Tue, Dec 7, 2021 at 9:01 PM Amit Kapila wrote: > > Thanks, the patch looks mostly good to me. I have slightly modified it > to incorporate one of Michael's suggestions, ran pgindent, and > modified the commit message. > LGTM, except in the patch commit message I'd change "Create Publication"

Re: Non-superuser subscription owners

2021-12-07 Thread Amit Kapila
On Mon, Dec 6, 2021 at 9:26 PM Mark Dilger wrote: > > > On Dec 6, 2021, at 2:19 AM, Amit Kapila wrote: > > > >>> If we want to maintain the property that subscriptions can only be > >>> owned by superuser > > We don't want to maintain such a property, or at least, that's not what I > want. I

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-07 Thread Dilip Kumar
On Tue, Dec 7, 2021 at 10:29 AM Nikolay Samokhvalov wrote: > > On Mon, Dec 6, 2021 at 8:16 PM Dilip Kumar wrote: >> >> If the subtransaction cache is overflowed in some of the transactions >> then it will affect all the concurrent queries as they need to access >> the SLRU for checking the

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-07 Thread Dilip Kumar
On Tue, Dec 7, 2021 at 11:11 AM Justin Pryzby wrote: Thanks for the review I will work on these comments. > > + > > + subxact_count xid > > + > > + > > + The current backend's active subtransactions count. > > subtransaction (no s) > > > + Set to true if

Re: Alter all tables in schema owner fix

2021-12-07 Thread Amit Kapila
On Tue, Dec 7, 2021 at 8:21 AM vignesh C wrote: > > Thanks for your comments, I have made the changes. Additionally I have > renamed IsSchemaPublication to is_schema_publication for keeping the > naming similar around the code. The attached v3 patch has the changes > for the same. > Thanks, the

  1   2   >