Re: [PATCH] Make jsonapi usable from libpq

2021-06-23 Thread Michael Paquier
On Tue, Jun 22, 2021 at 10:59:37PM +, Jacob Champion wrote: > Hmm. I'm honestly hesitant to start splitting files apart, mostly > because json_log_and_abort() is only called from two places, and both > those places are triggered by programmer error as opposed to user > error. > > Would it

Re: Decoding speculative insert with toast leaks memory

2021-06-23 Thread Michael Paquier
On Thu, Jun 24, 2021 at 12:25:15AM -0400, Tom Lane wrote: > Amit Kapila writes: >> Okay, I'll change this in back branches and HEAD to keep the code >> consistent, or do you think it is better to retain the order in HEAD >> as it is and just change it for back-branches? > > As I said, I'd keep

Re: Decoding speculative insert with toast leaks memory

2021-06-23 Thread Tom Lane
Amit Kapila writes: >> I think it's OK in HEAD. I agree we shouldn't do it like that >> in the back branches. > Okay, I'll change this in back branches and HEAD to keep the code > consistent, or do you think it is better to retain the order in HEAD > as it is and just change it for

Re: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-06-23 Thread Greg Nancarrow
On Thu, Jun 24, 2021 at 1:38 PM Zhihong Yu wrote: > > How about walking the partition hierarchy bottom up, recording the parents > but not taking the locks. > Once top-most parent is found, take the locks in reverse order (top down) ? > Is it safe to walk up the partition hierarchy (to record

Re: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-06-23 Thread Zhihong Yu
On Wed, Jun 23, 2021 at 8:10 PM Amit Kapila wrote: > On Wed, Jun 16, 2021 at 6:10 PM houzj.f...@fujitsu.com > wrote: > > > > On Tuesday, June 15, 2021 10:01 PM Robert Haas > wrote: > > > > > > Now, maybe it could be done, and I think that's worth a little more > thought. For > > > example,

Re: Decoding speculative insert with toast leaks memory

2021-06-23 Thread Amit Kapila
On Wed, Jun 23, 2021 at 8:21 PM Tom Lane wrote: > > Tomas Vondra writes: > > While rebasing a patch broken by 4daa140a2f5, I noticed that the patch > > does this: > > > @@ -63,6 +63,7 @@ enum ReorderBufferChangeType > > REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID, > >

Re: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-06-23 Thread Amit Kapila
On Wed, Jun 16, 2021 at 6:10 PM houzj.f...@fujitsu.com wrote: > > On Tuesday, June 15, 2021 10:01 PM Robert Haas wrote: > > > > Now, maybe it could be done, and I think that's worth a little more > > thought. For > > example, perhaps whenever we invalidate a relation, we could also somehow > >

Re: pgbench logging broken by time logic changes

2021-06-23 Thread Michael Paquier
On Wed, Jun 23, 2021 at 10:01:28PM +0200, Fabien COELHO wrote: > Thanks for the hint! Why not, having the ability to collect data is a good > thing, so attached v10 does that. If something go wrong, the TODO section > could be extended around all calls. +check_pgbench_logs($bdir,

Re: Deadlock risk while inserting directly into partition?

2021-06-23 Thread David Rowley
On Thu, 24 Jun 2021 at 12:45, Tom Lane wrote: > I don't think that the > ability to access partitions directly is a material problem here; > I doubt that we need to lock every partition in the plan when run-time > routing is working (surely we only need to lock the partition mapping); > and most

Re: Assertion failure in HEAD and 13 after calling COMMIT in a stored proc

2021-06-23 Thread Tom Lane
Greg Nancarrow writes: > On Wed, Jun 23, 2021 at 11:01 PM Tom Lane wrote: >> The comment is written in terms of "when can we >> skip taking a snapshot", while the test in the code is written for >> the inverse condition "when do we need a snapshot". > Perhaps that code could have been written

Re: Deadlock risk while inserting directly into partition?

2021-06-23 Thread Tom Lane
David Rowley writes: > ... What I mean is that as we add > more and more fixes to improve performance of partitioning, that there > comes a point where the ability to directly reference partitions is a > hindrance rather than something that's useful. Right now that > hindrance is the fact that we

Re: Assertion failure in HEAD and 13 after calling COMMIT in a stored proc

2021-06-23 Thread Greg Nancarrow
On Wed, Jun 23, 2021 at 11:01 PM Tom Lane wrote: > > > The code is correct as-is; the proposed change would result in taking > more snapshots than needed. Perhaps the comment needs revision, since > you both misread it. The comment is written in terms of "when can we > skip taking a snapshot",

Re: Deadlock risk while inserting directly into partition?

2021-06-23 Thread Tom Lane
David Rowley writes: > On Thu, 24 Jun 2021 at 10:38, Tom Lane wrote: >> I strongly disagree. That's essentially betting the whole farm on >> our always being able to optimize parent-level operations fully, >> which I do not think we are anywhere close to. > Did you have anything in particular

Re: Deadlock risk while inserting directly into partition?

2021-06-23 Thread David Rowley
On Thu, 24 Jun 2021 at 10:38, Tom Lane wrote: > > David Rowley writes: > > I've recently been thinking it would be good if you were unable to > > access partitions directly by name at all. > > I strongly disagree. That's essentially betting the whole farm on > our always being able to optimize

Re: Deadlock risk while inserting directly into partition?

2021-06-23 Thread Tom Lane
David Rowley writes: > I've recently been thinking it would be good if you were unable to > access partitions directly by name at all. I strongly disagree. That's essentially betting the whole farm on our always being able to optimize parent-level operations fully, which I do not think we are

Re: Deadlock risk while inserting directly into partition?

2021-06-23 Thread David Rowley
On Wed, 23 Jun 2021 at 21:07, Amit Kapila wrote: > I noticed that while inserting directly into a partition table we > compute the PartitionCheckExpr by traversing all the parent partitions > via > ExecPartitionCheck()->RelationGetPartitionQual()->generate_partition_qual(). > We take

Re: Emit namespace in post-copy output

2021-06-23 Thread Corey Huinker
On Tue, Jun 22, 2021 at 6:08 PM Mike wrote: > When running a VACUUM or CLUSTER command, the namespace name is not part > of the emitted message. > > Using `vacuumdb` CLI tool recently with multiple jobs, I found that > reading the output messages harder to match the relations with their >

Re: pgbench logging broken by time logic changes

2021-06-23 Thread Fabien COELHO
Ola Álvaro, ... or, actually, even better would be to use a TODO block, so that the test is run and reports its status, but if it happens not to succeed it will not cause the whole test to fail. That way you'll accumulate some evidence that may serve to improve the test in the future until it

Re: Pipeline mode and PQpipelineSync()

2021-06-23 Thread Alvaro Herrera
On 2021-Jun-23, Boris Kolpackov wrote: > I think one change that is definitely needed is to make it clear that > the PQpipelineSync() call is not optional. > > I would also add a note saying that while the server starts processing > the pipeline immediately, it may buffer the results and the

Re: Using indexUnchanged with nbtree

2021-06-23 Thread Peter Geoghegan
On Wed, Jun 23, 2021 at 9:31 AM Simon Riggs wrote: > You're right that skipping the check might also skip killing a prior > row version, but it doesn't prevent later scans from killing them, so > there is no correctness aspect to that. Probably not, no. I'll assume for now that there is no

Re: Using indexUnchanged with nbtree

2021-06-23 Thread Simon Riggs
On Wed, Jun 23, 2021 at 5:17 PM Peter Geoghegan wrote: > > On Mon, Jun 21, 2021 at 5:31 AM Simon Riggs > wrote: > > Seems like we can skip the uniqueness check if indexUnchanged, which > > will speed up non-HOT UPDATEs on tables with B-Trees. > > I thought about doing this myself. Never got as

Re: Using indexUnchanged with nbtree

2021-06-23 Thread Peter Geoghegan
On Mon, Jun 21, 2021 at 5:31 AM Simon Riggs wrote: > Seems like we can skip the uniqueness check if indexUnchanged, which > will speed up non-HOT UPDATEs on tables with B-Trees. I thought about doing this myself. Never got as far as thinking about the correctness implications in detail. One

Re: PG 14 release notes, first draft

2021-06-23 Thread Peter Geoghegan
On Wed, Jun 23, 2021 at 5:50 AM Simon Riggs wrote: > I just noticed that these commits are missing, yet are very important > new features: > d9d076222f5b94a8 > f9900df5f9 > c98763bf51bf > > These are important enough to be major features of PG14. I certainly think that they're important enough

Re: SSL Tests for sslinfo extension

2021-06-23 Thread Daniel Gustafsson
> On 17 Jun 2021, at 09:29, Michael Paquier wrote: > > On Thu, May 20, 2021 at 08:40:48PM +0200, Daniel Gustafsson wrote: >>> On 19 May 2021, at 21:05, Andrew Dunstan wrote: >>> >>> On 5/19/21 1:01 PM, Dagfinn Ilmari Mannsåker wrote: Daniel Gustafsson writes: > In order to be

Re: logical decoding and replication of sequences

2021-06-23 Thread Tomas Vondra
On 6/23/21 4:14 PM, Tomas Vondra wrote: > A rebased patch, addressing a minor bitrot due to 4daa140a2f5. > Meh, forgot to attach the patch as usual, of course ... regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company >From

Postgres picks suboptimal index after building of an extended statistics

2021-06-23 Thread Andrey V. Lepikhov
Hi, Ivan Frolkov reported a problem with choosing a non-optimal index during a query optimization. This problem appeared after building of an extended statistics. I prepared the test case (see t.sql in attachment). For reproduction of this case we need to have a composite primary key index

Re: logical decoding and replication of sequences

2021-06-23 Thread Tomas Vondra
A rebased patch, addressing a minor bitrot due to 4daa140a2f5. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Race condition in InvalidateObsoleteReplicationSlots()

2021-06-23 Thread Álvaro Herrera
On 2021-Jun-20, Tom Lane wrote: > Actually ... isn't there a second race, in the opposite direction? > IIUC, the point of this is that once we force some WAL to be sent > to the frozen sender/receiver, they'll be killed for failure to > respond. But the advance_wal call is not the only possible

Re: Reducing the cycle time for CLOBBER_CACHE_ALWAYS buildfarm members

2021-06-23 Thread Tom Lane
Andrew Dunstan writes: > Looks OK for the buildfarm patch. I wonder if we just want to run initdb > once with --clobber-cache instead of once per locale? I thought about that, but I'm not sure it's appropriate for the buildfarm client to be making that decision. I do not think any of the CCA

Re: genbki stricter error handling

2021-06-23 Thread Tom Lane
Peter Eisentraut writes: > I propose the attached patch to make genbki.pl error out if it > encounters any errors in this routine, while preserving the property > that all errors in one run are reported. +1, looks sane in a quick read-through. regards, tom lane

Re: Reducing the cycle time for CLOBBER_CACHE_ALWAYS buildfarm members

2021-06-23 Thread Andrew Dunstan
On 6/22/21 5:11 PM, Tom Lane wrote: > Andrew Dunstan writes: >> On 6/20/21 6:10 PM, Tom Lane wrote: >>> (3) Since this only works in v14 and up, older branches would >>> have to fall back to -DCLOBBER_CACHE_ALWAYS. Perhaps we could >>> improve the buildfarm client script so that buildfarm

Re: pgbench logging broken by time logic changes

2021-06-23 Thread Alvaro Herrera
On 2021-Jun-23, Fabien COELHO wrote: > +# cool check that we are around 2 seconds > +# The rate may results in an unlucky schedule which triggers > +# an early exit, hence the loose bound. > +# > +# THIS TEST IS COMMENTED OUT BUT PLEASE LET IT THERE SO THAT > +# IT CAN BE ENABLED EASILY. > +# >

Re: Assertion failure in HEAD and 13 after calling COMMIT in a stored proc

2021-06-23 Thread Tom Lane
Greg Nancarrow writes: > On Tue, Jun 22, 2021 at 10:56 PM Ranier Vilela wrote: >> The comments in the function are clear: >> If expression is mutable OR is a non-read-only function, so need a snapshot. > I have to agree with you. > Looks like the "&&" should really be an "||". The explanation

Re: PG 14 release notes, first draft

2021-06-23 Thread Simon Riggs
On Mon, May 10, 2021 at 7:03 AM Bruce Momjian wrote: > > I have committed the first draft of the PG 14 release notes. You can > see the most current build of them here: > > https://momjian.us/pgsql_docs/release-14.html > > I need clarification on many items, and the document still needs

RE: Support tab completion for upper character inputs in psql

2021-06-23 Thread tanghy.f...@fujitsu.com
Hi I've updated the patch to V8 since Tom, Kyotaro and Laurenz discussed the lower case issue of German/Turkish language at [1]. Differences from V7 are: * Add a function valid_input_text which checks the input text to see if it only contains alphabet letters, numbers etc. * Delete the flag

Re: intermittent failures in Cygwin from select_parallel tests

2021-06-23 Thread Andrew Dunstan
On 6/22/21 11:02 PM, Noah Misch wrote: > On Tue, Jun 22, 2021 at 08:18:42AM -0400, Andrew Dunstan wrote: >> On 6/22/21 2:42 AM, Noah Misch wrote: >>> I was wondering about suggesting some kind of official warning, but I guess the manual already covers it with this 10 year old notice.

Re: Doc chapter for Hash Indexes

2021-06-23 Thread Simon Riggs
On Wed, Jun 23, 2021 at 5:12 AM Amit Kapila wrote: > > On Tue, Jun 22, 2021 at 2:31 PM Simon Riggs > wrote: > > > > I attach both clean and compare versions. > > > > Do we want to hold this work for PG15 or commit in PG14 and backpatch > it till v10 where we have made hash indexes crash-safe? I

Re: Assertion failure in HEAD and 13 after calling COMMIT in a stored proc

2021-06-23 Thread Ranier Vilela
Em qua., 23 de jun. de 2021 às 03:04, Greg Nancarrow escreveu: > On Tue, Jun 22, 2021 at 10:56 PM Ranier Vilela > wrote: > > > > On Mon, Jun 21, 2021 at 04:19:27PM -0700, Jim Nasby wrote: > > > The following generates an assertion failure. Quick testing with start > and > > > stop as well as

Re: Pipeline mode and PQpipelineSync()

2021-06-23 Thread Boris Kolpackov
Alvaro Herrera writes: > On 2021-Jun-22, Alvaro Herrera wrote: > > > > So I think it would be useful to clarify the server behavior and > > > specify it in the documentation. > > > > I'll see about improving the docs on these points. > > So I started to modify the second paragraph to indicate

Re: [PATCH] Hooks at XactCommand level

2021-06-23 Thread Nicolas CHAHWEKILIAN
Hello, As far as I am concerned, I am totally awaiting for this kind of feature exposed here, for one single reason at this time : the extension pg_statement_rollback will be much more valuable with the ability of processing "rollback to savepoint" without the need for explicit instruction

Re: pgbench logging broken by time logic changes

2021-06-23 Thread Fabien COELHO
Bonjour Michaël, Could it be possible to document the intention of the test and its coverage then? With the current patch, one has to guess what's the intention behind this case. Ok, see attached. +check_pgbench_logs($bdir, '001_pgbench_log_1', $nthreads, 1, 3, + qr{^\d{10,} \d{1,2}

genbki stricter error handling

2021-06-23 Thread Peter Eisentraut
genbki.pl says: # Perform OID lookups on an array of OID names. # If we don't have a unique value to substitute, warn and # leave the entry unchanged. # (A warning seems sufficient because the bootstrap backend will reject # non-numeric values anyway. So we might as well detect multiple

RE: Deadlock risk while inserting directly into partition?

2021-06-23 Thread houzj.f...@fujitsu.com
On Wednesday, June 23, 2021 5:07 PM Amit Kapila wrote: > I noticed that while inserting directly into a partition table we compute the > PartitionCheckExpr by traversing all the parent partitions via > ExecPartitionCheck()->RelationGetPartitionQual()->generate_partition_qual(). > We take

Re: decoupling table and index vacuum

2021-06-23 Thread Antonin Houska
Andres Freund wrote: > On 2021-04-21 11:21:31 -0400, Robert Haas wrote: > > This scheme adds a lot of complexity, which is a concern, but it seems > > to me that it might have several benefits. One is concurrency. You > > could have one process gathering dead TIDs and adding them to the > >

Deadlock risk while inserting directly into partition?

2021-06-23 Thread Amit Kapila
I noticed that while inserting directly into a partition table we compute the PartitionCheckExpr by traversing all the parent partitions via ExecPartitionCheck()->RelationGetPartitionQual()->generate_partition_qual(). We take AccessShareLock on parent tables while generating qual. Now, on the

Re: improvements in Unicode tables generation code

2021-06-23 Thread Peter Eisentraut
On 22.06.21 10:20, Heikki Linnakangas wrote: On 22/06/2021 10:20, Peter Eisentraut wrote: I have accumulated a few patches to improve the output of the scripts in src/backend/utils/mb/Unicode/ to be less non-standard-looking and fix a few other minor things in that area.

Re: Added schema level support for publication.

2021-06-23 Thread Greg Nancarrow
On Tue, Jun 22, 2021 at 2:15 PM vignesh C wrote: > > > Updated patch has the fix for this, this also includes the fixes for > the other comments you had given. > I have removed the skip table patches to keep the focus on the main > patch, once this patch gets into committable shape, I will focus

Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors

2021-06-23 Thread Fabien COELHO
Hello Yugo-san: # About v12.1 This is a refactoring patch, which creates a separate structure for holding variables. This will become handy in the next patch. There is also a benefit from a software engineering point of view, so it has merit on its own. ## Compilation Patch applies

Re: Pipeline mode and PQpipelineSync()

2021-06-23 Thread Boris Kolpackov
Alvaro Herrera writes: > > I think always requiring PQpipelineSync() is fine since it also serves > > as an error recovery boundary. But the fact that the server waits until > > the sync message to start executing the pipeline is surprising. To me > > this seems to go contrary to the idea of a

Re: pgbench logging broken by time logic changes

2021-06-23 Thread Michael Paquier
On Wed, Jun 23, 2021 at 08:26:32AM +0200, Fabien COELHO wrote: >> Could we make that shorter at 1s? That will shorten the duration of >> the test run. It is easy to miss that this test is for >> --aggregate-interval (aka the logAgg() path) without a comment. > > It is for -T, -P and

Re: [PATCH] Pull general SASL framework out of SCRAM

2021-06-23 Thread Michael Paquier
On Tue, Jun 22, 2021 at 10:37:29PM +, Jacob Champion wrote: > Currently, the SASL logic is tightly coupled to the SCRAM > implementation. This patch untangles the two, by introducing callback > structs for both the frontend and backend. The approach to define and have a set callbacks feels

RE: Partition Check not updated when insert into a partition

2021-06-23 Thread houzj.f...@fujitsu.com
On Wednesday, June 23, 2021 12:16 PM I wrote: > When directly INSERT INTO partition, postgres will invoke ExecPartitionCheck > which will execute its parent's and grandparent's partition constraint check. > From the code, the whole constraint check is saved in relcache::rd_partcheck. > > For a

Re: pgbench logging broken by time logic changes

2021-06-23 Thread Fabien COELHO
Hello, +# note: this test is time sensitive, and may fail on a very +# loaded host. +# note: --progress-timestamp is not tested +my $delay = pgbench( + '-T 2 -P 1 -l --aggregate-interval=1 -S -b se@2' + . ' --rate=20 --latency-limit=1000 -j ' . $nthreads + . ' -c 3 -r',

Re: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-06-23 Thread Amit Kapila
On Wed, Jun 23, 2021 at 6:35 AM houzj.f...@fujitsu.com wrote: > > On Tuesday, June 22, 2021 8:25 PM Amit Kapila wrote: > > On Wed, Jun 16, 2021 at 8:57 AM houzj.f...@fujitsu.com > > wrote: > > > > > > I think the check of partition could be even more complicated if we > > > need to check the

Re: [PoC] Federated Authn/z with OAUTHBEARER

2021-06-23 Thread Michael Paquier
On Tue, Jun 22, 2021 at 11:26:03PM +, Jacob Champion wrote: > Done [1, 2]. I've copied your comments into those threads with my > responses, and I'll have them registered in commitfest shortly. Thanks! -- Michael signature.asc Description: PGP signature

Re: Assertion failure in HEAD and 13 after calling COMMIT in a stored proc

2021-06-23 Thread Greg Nancarrow
On Tue, Jun 22, 2021 at 10:56 PM Ranier Vilela wrote: > > On Mon, Jun 21, 2021 at 04:19:27PM -0700, Jim Nasby wrote: > > The following generates an assertion failure. Quick testing with start and > > stop as well as the core dump shows it’s failing on the execution of > > `schema_name :=