Re: Track statistics for streaming of in-progress transactions

2020-10-19 Thread Amit Kapila
On Mon, Oct 19, 2020 at 1:52 PM Ajin Cherian wrote: > > On Wed, Oct 14, 2020 at 2:39 PM Amit Kapila wrote: > > > > Commit 464824323e has added the support of the streaming of > > in-progress transactions into the built-in logical replication. The > > attached patch adds the statistics about

RE: Transactions involving multiple postgres foreign servers, take 2

2020-10-19 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > I don't think the inability to cancel all session at once cannot be a > reason not to not to allow operators to cancel a stuck session. Yeah, I didn't mean to discount the ability to cancel queries. I just want to confirm how the user can use the cancellation in

Re: [HACKERS] logical decoding of two-phase transactions

2020-10-19 Thread Peter Smith
Hello Ajin. I have gone through the v10 patches to verify if and how my previous v6 review comments got addressed. Some issues remain, and there are a few newly introduced ones. Mostly it is all very minor stuff. Please find my revised review comments below. Kind Regards. Peter Smith Fujitsu

Re: Add statistics to pg_stat_wal view for wal related parameter tuning

2020-10-19 Thread Amit Kapila
On Tue, Oct 20, 2020 at 8:01 AM Masahiro Ikeda wrote: > > Hi, > > I think we need to add some statistics to pg_stat_wal view. > > Although there are some parameter related WAL, > there are few statistics for tuning them. > > I think it's better to provide the following statistics. > Please let me

Re: Transactions involving multiple postgres foreign servers, take 2

2020-10-19 Thread Kyotaro Horiguchi
At Tue, 20 Oct 2020 02:44:09 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Ashutosh Bapat > > Using pg_cancel_backend() and pg_terminate_backend() a DBA can cancel > > running query from any backend or terminate a backend. For either to > > work the backend needs to be interruptible.

RE: Transactions involving multiple postgres foreign servers, take 2

2020-10-19 Thread tsunakawa.ta...@fujitsu.com
From: Ashutosh Bapat > Using pg_cancel_backend() and pg_terminate_backend() a DBA can cancel > running query from any backend or terminate a backend. For either to > work the backend needs to be interruptible. IIRC, Robert had made an > effort to make postgres_fdw interruptible few years back.

Re: Deleting older versions in unique indexes to avoid page splits

2020-10-19 Thread Peter Geoghegan
On Fri, Oct 16, 2020 at 1:58 PM Peter Geoghegan wrote: > Attached is v3, which is rebased against the master branch as of > today. No real changes, though. And now here's v4. This version adds handling of posting list tuples, which I was skipping over before. Highly contended leaf pages with

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2020-10-19 Thread tsunakawa.ta...@fujitsu.com
Hi Andrey-san, Thanks for the revision. The patch looks good except for the following two items. (18) + if (target_resultRelInfo->ri_FdwRoutine != NULL) + { + if (target_resultRelInfo->ri_usesMultiInsert) + { +

Add statistics to pg_stat_wal view for wal related parameter tuning

2020-10-19 Thread Masahiro Ikeda
Hi, I think we need to add some statistics to pg_stat_wal view. Although there are some parameter related WAL, there are few statistics for tuning them. I think it's better to provide the following statistics. Please let me know your comments. ``` postgres=# SELECT * from pg_stat_wal; -[

Re: partition routing layering in nodeModifyTable.c

2020-10-19 Thread Amit Langote
On Mon, Oct 19, 2020 at 8:48 PM Heikki Linnakangas wrote: > On 19/10/2020 07:54, Amit Langote wrote: > > On Sun, Oct 18, 2020 at 12:54 AM Alvaro Herrera > > wrote: > >> Well, I was thinking on making the ri_PartitionInfo be about > >> partitioning in general, not just specifically for partition

Re: Reduce the dependence on access/xlog_internal.h

2020-10-19 Thread Andres Freund
Hi, On 2020-10-19 18:29:27 -0700, Mark Dilger wrote: > Please find access/xlog_internal.h refactored in the attached patch > series. This header is included from many places, including external > tools. It is aesthetically displeasing to have something called > "internal" used from so many

Re: speed up unicode normalization quick check

2020-10-19 Thread Michael Paquier
On Mon, Oct 19, 2020 at 12:12:00PM -0400, John Naylor wrote: > I see, I should have looked for that when Michael mentioned it. We could > left-justify instead, as in the attached. If it were up to me, though, I'd > just format it like pgindent expects, even if not nice looking. It's just a > bunch

Reduce the dependence on access/xlog_internal.h

2020-10-19 Thread Mark Dilger
Hackers, Please find access/xlog_internal.h refactored in the attached patch series. This header is included from many places, including external tools. It is aesthetically displeasing to have something called "internal" used from so many places, especially when many of those places do not

Re: Make procedure OUT parameters work with JDBC

2020-10-19 Thread Craig Ringer
On Mon, 19 Oct 2020, 19:16 Andrew Dunstan, wrote: > > On 10/19/20 5:19 AM, Peter Eisentraut wrote: > > A follow-up to the recently added support for OUT parameters for > > procedures. The JDBC driver sends OUT parameters with type void. > > This makes sense when calling a function, so that the

Re: Hash support for row types

2020-10-19 Thread Andres Freund
Hi, On 2020-10-19 10:01:14 +0200, Peter Eisentraut wrote: > In [0] it was discussed that hash support for row types/record would be > handy. So I implemented that. > The implementation hashes each field and combines the hash values. Most of > the code structure can be borrowed from the record

Re: Enumize logical replication message actions

2020-10-19 Thread Andres Freund
Hi, On 2020-10-16 12:55:26 +0530, Ashutosh Bapat wrote: > Here's a patch simplifying that for top level logical replication > messages. I think that's a good plan. One big benefit for me is that it's much easier to search for an enum than for a single letter constant. Including searching for all

Re: [PATCH] Add extra statistics to explain for Nested Loop

2020-10-19 Thread Andres Freund
Hi, On 2020-10-16 10:42:43 +0300, e.sokol...@postgrespro.ru wrote: > For some distributions of data in tables, different loops in nested loop > joins can take different time and process different amounts of entries. It > makes average statistics returned by explain analyze not very useful for >

Re: Non-configure build of thread_test has been broken for awhile

2020-10-19 Thread Alvaro Herrera
On 2020-Oct-18, Tom Lane wrote: > It doesn't really seem sane to me to support two different build > environments for thread_test, especially when one of them is so > little-used that it can be broken for years before we notice. > So I'd be inclined to rip out the Makefile and just consider >

Re: Probable documentation errors or improvements

2020-10-19 Thread Tom Lane
Heikki Linnakangas writes: > I left out these changes in sepgsql docs: >> --- a/doc/src/sgml/sepgsql.sgml >> +++ b/doc/src/sgml/sepgsql.sgml >> @@ -88,7 +88,7 @@ Policy from config file:targeted >> >> To build this module, include the option --with-selinux in >> your PostgreSQL

Re: Parallel Inserts in CREATE TABLE AS

2020-10-19 Thread Bharath Rupireddy
On Thu, Oct 15, 2020 at 3:18 PM Amit Kapila wrote: > > > > > 1. How to represent the parallel insert for CTAS in explain plans? The > > > > explain CTAS shows the plan for only the SELECT part. How about having > > > > some textual info along with the Gather node? I'm not quite sure on > > > >

Re: Probable documentation errors or improvements

2020-10-19 Thread Justin Pryzby
On Mon, Oct 19, 2020 at 07:36:29PM +0300, Heikki Linnakangas wrote: > On 05/10/2020 22:19, Justin Pryzby wrote: > > On Fri, Sep 25, 2020 at 09:30:00AM -0500, Justin Pryzby wrote: > > > Split one patch about text search, added another one (sequences), added > > > some > > > info to commit

Re: Probable documentation errors or improvements

2020-10-19 Thread Heikki Linnakangas
On 05/10/2020 22:19, Justin Pryzby wrote: On Fri, Sep 25, 2020 at 09:30:00AM -0500, Justin Pryzby wrote: Split one patch about text search, added another one (sequences), added some info to commit messages, and added here. https://commitfest.postgresql.org/30/2744/ Added an additional patch

Re: Reduce the time required for a database recovery from archive.

2020-10-19 Thread Stephen Frost
Greetings, * Dmitry Shulga (d.shu...@postgrespro.ru) wrote: > > On 9 Sep 2020, at 21:26, Stephen Frost wrote: > > * Dmitry Shulga (d.shu...@postgrespro.ru) wrote: > >> Overall archive file processing is done one by one, and this might > >> create a performance bottleneck if archived WAL files

Re: speed up unicode normalization quick check

2020-10-19 Thread John Naylor
On Mon, Oct 19, 2020 at 10:38 AM Tom Lane wrote: > John Naylor writes: > > On Mon, Oct 19, 2020 at 2:16 AM Peter Eisentraut < > > peter.eisentr...@2ndquadrant.com> wrote: > >> Could you adjust the generation script so that the resulting header file > >> passes the git whitespace check? Check

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-10-19 Thread Andrey Lepikhov
19.10.2020 09:12, tsunakawa.ta...@fujitsu.com пишет: Hello Andrey-san, Thank you for challenging an interesting feature. Below are my review comments. (1) - /* for use by copy.c when performing multi-inserts */ + /* +* The following fields are currently only relevant

parallel distinct union and aggregate support patch

2020-10-19 Thread bu...@sohu.com
Hi hackers, I write a path for soupport parallel distinct, union and aggregate using batch sort. steps: 1. generate hash value for group clauses values, and using mod hash value save to batch 2. end of outer plan, wait all other workers finish write to batch 3. echo worker get a unique batch

Re: speed up unicode normalization quick check

2020-10-19 Thread Tom Lane
John Naylor writes: > On Mon, Oct 19, 2020 at 2:16 AM Peter Eisentraut < > peter.eisentr...@2ndquadrant.com> wrote: >> Could you adjust the generation script so that the resulting header file >> passes the git whitespace check? Check the output of >> git show --check

Re: speed up unicode decomposition and recomposition

2020-10-19 Thread John Naylor
On Fri, Oct 16, 2020 at 2:08 PM Daniel Verite wrote: > John Naylor wrote: > > > I'd be curious how it compares to ICU now > > I've made another run of the test in [1] with your v2 patches > from this thread against icu_ext built with ICU-67.1. > The results show the times in milliseconds

Re: speed up unicode decomposition and recomposition

2020-10-19 Thread John Naylor
On Thu, Oct 15, 2020 at 11:32 PM Michael Paquier wrote: > > The binary sizes of libpgcommon_shlib.a and libpgcommon.a change > because Decomp_hash_func() gets included, impacting libpq. > I don't see any difference on gcc/Linux in those two files, nor in unicode_norm_shlib.o -- I do see a

Re: [PATCH] Add extra statistics to explain for Nested Loop

2020-10-19 Thread Pierre Giraud
Le 17/10/2020 à 06:26, Julien Rouhaud a écrit : > On Sat, Oct 17, 2020 at 12:15 PM Pavel Stehule > wrote: >> >> so 17. 10. 2020 v 0:11 odesílatel Anastasia Lubennikova >> napsal: >>> >>> On 16.10.2020 12:07, Julien Rouhaud wrote: >>> >>> Le ven. 16 oct. 2020 à 16:12, Pavel Stehule a >>>

Re: speed up unicode normalization quick check

2020-10-19 Thread John Naylor
On Mon, Oct 19, 2020 at 2:16 AM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2020-10-12 13:36, Michael Paquier wrote: > > On Mon, Oct 12, 2020 at 03:39:51PM +0900, Masahiko Sawada wrote: > >> Yes, this patch resolves the problem. > > > > Okay, applied then. > > Could you

Is Recovery actually paused?

2020-10-19 Thread Dilip Kumar
Hello, We have an interface to pause the WAL replay (pg_wal_replay_pause) and to know whether the WAL replay pause is requested (pg_is_wal_replay_paused). But there is no way to know whether the recovery is actually paused or not. Actually, the recovery process might process an extra WAL before

Re: [doc] improve tableoid description

2020-10-19 Thread Ian Lawrence Barwick
2020年10月19日(月) 20:22 Ashutosh Bapat : > > On Sat, Oct 17, 2020 at 6:35 PM Ian Lawrence Barwick > wrote: > > > > [doc] improve tableoid description > > > > Hi > > > > Attached patch aims to improve the description of the tableoid system > > column [1] > > by: > > > > - mentioning it's useful for

Re: Transactions involving multiple postgres foreign servers, take 2

2020-10-19 Thread Masahiko Sawada
On Mon, 19 Oct 2020 at 20:37, Ashutosh Bapat wrote: > > On Mon, Oct 19, 2020 at 2:37 PM tsunakawa.ta...@fujitsu.com > wrote: > > > > Really. I thought we were talking about canceling commits with Ctrl + C as > > you referred, right? I couldn't imagine, in production environments where > >

Re: Use PointerGetDatum(cstring_to_text_with_len()) instead of CStringGetTextDatum() to avoid duplicate strlen

2020-10-19 Thread Heikki Linnakangas
On 19/10/2020 09:32, Hou, Zhijie wrote: Hi I found some code like the following: StringInfoData s; ... values[6] = CStringGetTextDatum(s.data); The length of string can be found in ' StringInfoData.len', but the macro CStringGetTextDatum will use strlen to count the length again. I think we

Re: partition routing layering in nodeModifyTable.c

2020-10-19 Thread Heikki Linnakangas
On 17/10/2020 18:54, Alvaro Herrera wrote: On 2020-Oct-17, Amit Langote wrote: As I said in my previous email, I don't see how we can make initializing the map any lazier than it already is. If a partition has a different tuple descriptor than the root table, then we know for sure that any

Re: partition routing layering in nodeModifyTable.c

2020-10-19 Thread Heikki Linnakangas
On 19/10/2020 07:54, Amit Langote wrote: On Sun, Oct 18, 2020 at 12:54 AM Alvaro Herrera wrote: On 2020-Oct-17, Amit Langote wrote: Hmm, I don't see ri_PartitionCheckExpr as being a piece of routing information, because it's primarily meant to be used when inserting *directly* into a

Re: Transactions involving multiple postgres foreign servers, take 2

2020-10-19 Thread Ashutosh Bapat
On Mon, Oct 19, 2020 at 2:37 PM tsunakawa.ta...@fujitsu.com wrote: > > Really. I thought we were talking about canceling commits with Ctrl + C as > you referred, right? I couldn't imagine, in production environments where > many sessions are running transactions concurrently, how the user

Assertion failure when ATTACH partition followed by CREATE PARTITION.

2020-10-19 Thread Amul Sul
Hi, Assertion added in commits 6b2c4e59d016 is failing with following test: CREATE TABLE sales ( prod_id int, prod_quantity int, sold_monthdate ) PARTITION BY RANGE(sold_month); CREATE TABLE public.sales_p1 PARTITION OF public.sales FOR VALUES FROM (MINVALUE) TO

Re: [doc] improve tableoid description

2020-10-19 Thread Ashutosh Bapat
On Sat, Oct 17, 2020 at 6:35 PM Ian Lawrence Barwick wrote: > > [doc] improve tableoid description > > Hi > > Attached patch aims to improve the description of the tableoid system column > [1] > by: > > - mentioning it's useful for determining table names for partitioned tables as > well as

Re: Make procedure OUT parameters work with JDBC

2020-10-19 Thread Andrew Dunstan
On 10/19/20 5:19 AM, Peter Eisentraut wrote: > A follow-up to the recently added support for OUT parameters for > procedures.  The JDBC driver sends OUT parameters with type void.  > This makes sense when calling a function, so that the parameters are > ignored in ParseFuncOrColumn().  For a

Re: Assertion failure with LEFT JOINs among >500 relations

2020-10-19 Thread David Rowley
On Mon, 19 Oct 2020 at 13:06, Tom Lane wrote: > > David Rowley writes: > > I guess we could resolve that concern by just changing the failing > > assert to become: Assert(outer_skip_rows <= outer_rows || > > isinf(outer_rows)); > > I can't really object to just weakening the Assert a tad. > My

Make procedure OUT parameters work with JDBC

2020-10-19 Thread Peter Eisentraut
A follow-up to the recently added support for OUT parameters for procedures. The JDBC driver sends OUT parameters with type void. This makes sense when calling a function, so that the parameters are ignored in ParseFuncOrColumn(). For a procedure call we want to treat them as unknown. This

Re: Parallel copy

2020-10-19 Thread Amit Kapila
On Sun, Oct 18, 2020 at 7:47 AM Hou, Zhijie wrote: > > Hi Vignesh, > > After having a look over the patch, > I have some suggestions for > 0003-Allow-copy-from-command-to-process-data-from-file.patch. > > 1. > > +static uint32 > +EstimateCstateSize(ParallelContext *pcxt, CopyState cstate, List >

RE: Transactions involving multiple postgres foreign servers, take 2

2020-10-19 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > On Mon, 19 Oct 2020 at 14:39, tsunakawa.ta...@fujitsu.com > wrote: > > I have to say that's nitpicking. I believe almost nobody does, or cares > > about, > canceling commits, > > Really? I don’t think so. I think It’s terrible that the query gets > stuck for a long

Re: Track statistics for streaming of in-progress transactions

2020-10-19 Thread Ajin Cherian
On Wed, Oct 14, 2020 at 2:39 PM Amit Kapila wrote: > > Commit 464824323e has added the support of the streaming of > in-progress transactions into the built-in logical replication. The > attached patch adds the statistics about transactions streamed to the > decoding output plugin from

Hash support for row types

2020-10-19 Thread Peter Eisentraut
In [0] it was discussed that hash support for row types/record would be handy. So I implemented that. The implementation hashes each field and combines the hash values. Most of the code structure can be borrowed from the record comparison functions/btree support. To combine the hash

Re: Reduce the time required for a database recovery from archive.

2020-10-19 Thread Dmitry Shulga
Hello Stephen > On 9 Sep 2020, at 21:26, Stephen Frost wrote: > > Greetings, > > * Dmitry Shulga (d.shu...@postgrespro.ru) wrote: >> Overall archive file processing is done one by one, and this might >> create a performance bottleneck if archived WAL files are delivered slowly, >> because the

Re: Online checksums verification in the backend

2020-10-19 Thread Michael Paquier
On Mon, Oct 19, 2020 at 11:16:38AM +0800, Julien Rouhaud wrote: > On Mon, Oct 19, 2020 at 10:39 AM Michael Paquier wrote: >> Somewhat related to the first point, NoComputedChecksum exists >> because, as the current patch is shaped, we need to report an existing >> checksum to the user even for

Re: Resetting spilled txn statistics in pg_stat_replication

2020-10-19 Thread Masahiko Sawada
On Mon, 19 Oct 2020 at 14:24, Amit Kapila wrote: > > On Mon, Oct 19, 2020 at 9:04 AM Masahiko Sawada > wrote: > > > > On Thu, 15 Oct 2020 at 17:51, Amit Kapila wrote: > > > > > > > > > AFAICS, we use name data-type in many other similar stats views like > > > pg_stat_subscription,

Re: Transactions involving multiple postgres foreign servers, take 2

2020-10-19 Thread Masahiko Sawada
On Mon, 19 Oct 2020 at 14:39, tsunakawa.ta...@fujitsu.com wrote: > > From: Masahiko Sawada > > > Unfortunately, I'm afraid we can do nothing about it. If the DBMS's > > > client > > library doesn't support cancellation (e.g. doesn't respond to Ctrl+C or > > provide a > > function that cancel

Re: speed up unicode normalization quick check

2020-10-19 Thread Michael Paquier
On Mon, Oct 19, 2020 at 08:15:56AM +0200, Peter Eisentraut wrote: > On 2020-10-12 13:36, Michael Paquier wrote: > > On Mon, Oct 12, 2020 at 03:39:51PM +0900, Masahiko Sawada wrote: > > > Yes, this patch resolves the problem. > > > > Okay, applied then. > > Could you adjust the generation script

Use PointerGetDatum(cstring_to_text_with_len()) instead of CStringGetTextDatum() to avoid duplicate strlen

2020-10-19 Thread Hou, Zhijie
Hi I found some code like the following: > StringInfoData s; > ... > values[6] = CStringGetTextDatum(s.data); The length of string can be found in ' StringInfoData.len', but the macro CStringGetTextDatum will use strlen to count the length again. I think we can use

Re: speed up unicode normalization quick check

2020-10-19 Thread Peter Eisentraut
On 2020-10-12 13:36, Michael Paquier wrote: On Mon, Oct 12, 2020 at 03:39:51PM +0900, Masahiko Sawada wrote: Yes, this patch resolves the problem. Okay, applied then. Could you adjust the generation script so that the resulting header file passes the git whitespace check? Check the output