Re: [HACKERS] logical decoding of two-phase transactions

2020-09-21 Thread Ajin Cherian
> Why can't we call ReorderBufferCleanupTXN() from > ReorderBufferFinishPrepared after your changes? > Since the truncate already removed the changes, it would fail on the below Assert in ReorderBufferCleanupTXN() /* Check we're not mixing changes from different transactions. */ As

Re: VACUUM PARALLEL option vs. max_parallel_maintenance_workers

2020-09-21 Thread Peter Eisentraut
On 2020-09-21 05:48, Amit Kapila wrote: What according to you should be the behavior here and how will it be better than current? I think if I write VACUUM (PARALLEL 5), it should use up to 5 workers (up to the number of indexes), even if max_parallel_maintenance_workers is 2. -- Peter Eise

Re: Yet another fast GiST build

2020-09-21 Thread Heikki Linnakangas
On 21/09/2020 02:06, Tom Lane wrote: Justin Pryzby writes: This also appears to break checksums. Thanks, I'll go fix it. I was wondering about that, because the typical pattern for use of smgrextend for indexes seems to be RelationOpenSmgr(rel); PageSetChecksumInplace(page,

Re: Yet another fast GiST build

2020-09-21 Thread Heikki Linnakangas
On 21/09/2020 11:08, Heikki Linnakangas wrote: I think they need to, so that they can stamp the page with the LSN of the WAL record. But GiST build is special in that regard, because it stamps all pages with GistBuildLSN. Actually, don't we have a problem with that, even before this patch? Eve

Re: Binaries on s390x arch

2020-09-21 Thread Christoph Berg
Re: Namrata Bhave > As seen from downloads page, the Apt repo/rpms are not yet available for > s390x for latest versions. Hi, are you asking about apt (.deb) or yum (.rpm) packages? > Wanted to know if there is any work going on/planned to provide Postgres in > ready-to-use package or installer

Re: Yet another fast GiST build

2020-09-21 Thread Andrey M. Borodin
> 21 сент. 2020 г., в 13:45, Heikki Linnakangas написал(а): > > On 21/09/2020 11:08, Heikki Linnakangas wrote: >> I think they need to, so that they can stamp the page with the LSN of >> the WAL record. But GiST build is special in that regard, because it >> stamps all pages with GistBuildLSN.

Re: Is deduplicate_items ever used in nbtree?

2020-09-21 Thread Julien Rouhaud
Hi, On Mon, Sep 21, 2020 at 2:21 PM Nikolay Shaplov wrote: > > Hi! > > I am still working with reloptions patches, and noticed that in current master > deduplicate_items option exists, but never actually used. > You can set, it, you can dump it, it has proper representation in BTOptions, > it is

Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Ashutosh Sharma
Hi All, Today, while exploring logical replication in PostgreSQL, I noticed that logical replication from PG version 13 and below to PG v14 (development version) is not working. It has stopped working from the following git commit onwards: commit 464824323e57dc4b397e8b05854d779908b55304 Author: A

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-21 Thread Amit Kapila
On Mon, Sep 21, 2020 at 12:36 PM Ajin Cherian wrote: > > > Why can't we call ReorderBufferCleanupTXN() from > > ReorderBufferFinishPrepared after your changes? > > > > Since the truncate already removed the changes, it would fail on the > below Assert in ReorderBufferCleanupTXN() > /* Chec

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-21 Thread Dilip Kumar
On Mon, Sep 21, 2020 at 10:20 AM Amit Kapila wrote: > > On Sun, Sep 20, 2020 at 11:01 AM Dilip Kumar wrote: > > > > On Fri, Sep 18, 2020 at 6:02 PM Ajin Cherian wrote: > > > > > > > 3. > > > > + /* > > + * If it's ROLLBACK PREPARED then handle it via callbacks. > > + */ > > + if (TransactionIdIs

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-21 Thread Ajin Cherian
On Sun, Sep 20, 2020 at 3:31 PM Dilip Kumar wrote: > + /* > + * If it's ROLLBACK PREPARED then handle it via callbacks. > + */ > + if (TransactionIdIsValid(xid) && > + !SnapBuildXactNeedsSkip(ctx->snapshot_builder, buf->origptr) && > + parsed->dbId == ctx->slot->data.database && > + !FilterByOrig

Re: VACUUM PARALLEL option vs. max_parallel_maintenance_workers

2020-09-21 Thread Amit Kapila
On Mon, Sep 21, 2020 at 12:45 PM Peter Eisentraut wrote: > > On 2020-09-21 05:48, Amit Kapila wrote: > > What according to you should be the behavior here and how will it be > > better than current? > > I think if I write VACUUM (PARALLEL 5), it should use up to 5 workers > (up to the number of in

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Bharath Rupireddy
On Mon, Sep 21, 2020 at 3:26 PM Ashutosh Sharma wrote: > > commit 464824323e57dc4b397e8b05854d779908b55304 > Author: Amit Kapila > Date: Thu Sep 3 07:54:07 2020 +0530 > > Above select query produces no result. When this experiment is > performed below the mentioned git commit, it works fine. >

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Dilip Kumar
On Mon, Sep 21, 2020 at 3:26 PM Ashutosh Sharma wrote: > > Hi All, > > Today, while exploring logical replication in PostgreSQL, I noticed > that logical replication from PG version 13 and below to PG v14 > (development version) is not working. It has stopped working from the > following git commi

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-21 Thread Andrey Lepikhov
This patch currently looks very ready for use. And I'm taking a close look at the error reporting. Here we have difference in behavior of local and foreign table: regression test in postgres_fdw.sql: copy rem2 from stdin; -1 xyzzy \. reports error (1): = ERROR: new row fo

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-21 Thread Amit Kapila
On Mon, Sep 21, 2020 at 3:45 PM Ajin Cherian wrote: > > On Sun, Sep 20, 2020 at 3:31 PM Dilip Kumar wrote: > > > + /* > > + * If it's ROLLBACK PREPARED then handle it via callbacks. > > + */ > > + if (TransactionIdIsValid(xid) && > > + !SnapBuildXactNeedsSkip(ctx->snapshot_builder, buf->origptr)

Re: pg_service.conf file with unknown parameters

2020-09-21 Thread Daniel Gustafsson
> On 11 Sep 2020, at 14:39, Magnus Hagander wrote: > For example, if I have a service file with gssencmode=disable set, that > service file cannot be used by a psql client linked against libpq from > version 10. Even if the behavior would be identical (since v10 doesn't have > gssencmode). >

Re: Planner, check if can use consider HASH for groupings (src/backend/optimizer/plan/planner.c)

2020-09-21 Thread Ranier Vilela
Em dom., 20 de set. de 2020 às 21:10, Tomas Vondra < tomas.von...@2ndquadrant.com> escreveu: > On Sun, Sep 20, 2020 at 08:09:56PM -0300, Ranier Vilela wrote: > >Em sex., 18 de set. de 2020 às 10:37, Tomas Vondra < > >tomas.von...@2ndquadrant.com> escreveu: > > > >> On Thu, Sep 17, 2020 at 06:31:12

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-21 Thread Ajin Cherian
On Mon, Sep 21, 2020 at 9:24 PM Amit Kapila wrote: > I think we need to call ReorderBufferAbort at least to clean up the > TXN. Also, if what you are saying is correct then that should be true > without this patch as well, no? If so, we don't need to worry about it > as far as this patch is conce

Re: make MaxBackends available in _PG_init

2020-09-21 Thread Bharath Rupireddy
On Mon, Sep 21, 2020 at 9:14 AM Wang, Shenhao wrote: > > In source, I find that the postmaster will first load library, and then > calculate the value of MaxBackends. > > In the old version, the MaxBackends was calculated by: > MaxBackends = MaxConnections + autovacuum_max_workers + 1 +

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-21 Thread Amit Kapila
On Mon, Sep 21, 2020 at 5:23 PM Ajin Cherian wrote: > > On Mon, Sep 21, 2020 at 9:24 PM Amit Kapila wrote: > > > I think we need to call ReorderBufferAbort at least to clean up the > > TXN. Also, if what you are saying is correct then that should be true > > without this patch as well, no? If so,

Lift line-length limit for pg_service.conf

2020-09-21 Thread Daniel Gustafsson
The pg_service.conf parsing thread [0] made me realize that we have a hardwired line length of max 256 bytes. Lifting this would be in line with recent work for ecpg, pg_regress and pg_hba (784b1ba1a2 and 8f8154a50). The attached moves pg_service.conf to use the new pg_get_line_append API and a S

Re: Yet another fast GiST build

2020-09-21 Thread Heikki Linnakangas
On 21/09/2020 02:06, Tom Lane wrote: Justin Pryzby writes: This also appears to break checksums. Fixed, thanks for the report! I was wondering about that, because the typical pattern for use of smgrextend for indexes seems to be RelationOpenSmgr(rel); PageSetChecksumInplace

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Dilip Kumar
On Mon, Sep 21, 2020 at 4:15 PM Dilip Kumar wrote: > > On Mon, Sep 21, 2020 at 3:26 PM Ashutosh Sharma wrote: > > > > Hi All, > > > > Today, while exploring logical replication in PostgreSQL, I noticed > > that logical replication from PG version 13 and below to PG v14 > > (development version) i

Re: Yet another fast GiST build

2020-09-21 Thread Heikki Linnakangas
On 21/09/2020 12:06, Andrey M. Borodin wrote: 21 сент. 2020 г., в 13:45, Heikki Linnakangas написал(а): Actually, don't we have a problem with that, even before this patch? Even though we set the LSN to the magic GistBuildLSN value when we build the index, WAL replay will write the LSN of the r

Re: PATCH: Batch/pipelining support for libpq

2020-09-21 Thread Dave Cramer
Alvaro, On Fri, 4 Sep 2020 at 17:26, Alvaro Herrera wrote: > On 2020-Aug-31, Matthieu Garrigues wrote: > > > It seems like this patch is nearly finished. I fixed all the remaining > > issues. I'm also asking a confirmation of the test scenarios you want > > to see in the next version of the pat

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Ashutosh Sharma
Thanks Dilip for the patch. AFAIU, the fix looks good. One small comment: In the error message we are still referring to the native protocol version number. Shouldn't it be replaced with the greatest protocol version number we support now (i.e. LOGICALREP_PROTO_MAX_VERSION_NUM)? - if (data-

Re: Dynamic gathering the values for seq_page_cost/xxx_cost

2020-09-21 Thread Ashutosh Bapat
On Mon, Sep 21, 2020 at 9:11 AM Andy Fan wrote: > > Here are some changes for my detection program. > > | | seq_read_lat (us) | > random_read_lat (us) | > | FIO |12 | >14

Re: PATCH: Batch/pipelining support for libpq

2020-09-21 Thread Dave Cramer
Dave Cramer www.postgres.rocks On Mon, 31 Aug 2020 at 11:46, Matthieu Garrigues < matthieu.garrig...@gmail.com> wrote: > Hi, > > It seems like this patch is nearly finished. I fixed all the remaining > issues. I'm also asking > a confirmation of the test scenarios you want to see in the next > v

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Dilip Kumar
On Mon, Sep 21, 2020 at 6:27 PM Ashutosh Sharma wrote: > > Thanks Dilip for the patch. AFAIU, the fix looks good. One small comment: > > In the error message we are still referring to the native protocol > version number. Shouldn't it be replaced with the greatest protocol > version number we supp

Re: PATCH: Batch/pipelining support for libpq

2020-09-21 Thread Alvaro Herrera
On 2020-Sep-21, Dave Cramer wrote: Hello Dave, > I am looking for this in the commitfest and can't find it. However there is > an old commitfest entry > > https://commitfest.postgresql.org/13/1024/ > > Do you have the link for the new one ? Here you go: https://commitfest.postgresql.org/29/27

Re: PATCH: Batch/pipelining support for libpq

2020-09-21 Thread Matthieu Garrigues
Matthieu Garrigues On Mon, Sep 21, 2020 at 3:09 PM Dave Cramer wrote: >> > There was a comment upthread a while back that people should look at the > comments made in > https://www.postgresql.org/message-id/20180322.211148.187821341.horiguchi.kyotaro%40lab.ntt.co.jp > by Horiguchi-San. > > Fro

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Amit Kapila
On Mon, Sep 21, 2020 at 6:27 PM Ashutosh Sharma wrote: > > Thanks Dilip for the patch. AFAIU, the fix looks good. One small comment: > Thanks Ashutosh and Dilip for working on this. I'll look into it in a day or two. -- With Regards, Amit Kapila.

Re: Yet another fast GiST build

2020-09-21 Thread Andrey M. Borodin
> 21 сент. 2020 г., в 17:15, Heikki Linnakangas написал(а): > > On 21/09/2020 12:06, Andrey M. Borodin wrote >> >> I think we don't set rightlinks during index build. > > The new GiST sorting code does not, but the regular insert-based code does. > > That's a bit questionable in the new cod

Re: Inconsistent Japanese name order in v13 contributors list

2020-09-21 Thread Alvaro Herrera
On 2020-Sep-18, Bruce Momjian wrote: > This thread from 2015 is the most comprehensive discussion I remember of > Japanese name ordering, including a suggestion to use small caps: > > > https://www.postgresql.org/message-id/flat/20150613231826.GY133018%40postgresql.org#88d245a5cdd2b32e1e3e

Re: PATCH: Batch/pipelining support for libpq

2020-09-21 Thread Dave Cramer
On Mon, 21 Sep 2020 at 09:21, Matthieu Garrigues < matthieu.garrig...@gmail.com> wrote: > Matthieu Garrigues > > On Mon, Sep 21, 2020 at 3:09 PM Dave Cramer > wrote: > >> > > There was a comment upthread a while back that people should look at the > comments made in > https://www.postgresql.org/m

Re: Command statistics system (cmdstats)

2020-09-21 Thread Alvaro Herrera
On 2020-Sep-18, Michael Paquier wrote: > Based on the low level of activity and the fact that the patch was > marked as waiting on author for a couple of weeks, it looks like > little could be achieved by the end of the CF, and the attention was > elsewhere, so it looked better (and it still does

Re: PATCH: Batch/pipelining support for libpq

2020-09-21 Thread Matthieu Garrigues
On Mon, Sep 21, 2020 at 3:39 PM Dave Cramer wrote: > > > > On Mon, 21 Sep 2020 at 09:21, Matthieu Garrigues > wrote: >> >> Matthieu Garrigues >> >> On Mon, Sep 21, 2020 at 3:09 PM Dave Cramer >> wrote: >> >> >> > There was a comment upthread a while back that people should look at the >> > co

Re: Yet another fast GiST build

2020-09-21 Thread Andrey M. Borodin
> 21 сент. 2020 г., в 18:29, Andrey M. Borodin > написал(а): > > It was a conscious decision with incorrect motivation. I was thinking that it > will help to reduce number of "false positive" inspecting right pages. But > now I see that: > 1. There should be no such "false positives" that we

RE: Binaries on s390x arch

2020-09-21 Thread Namrata Bhave
Hi Christoph, Thank you for your response. We will be glad to obtain binaries for s390x on RHEL, SLES and Ubuntu distros. We are ready to provide the necessary infra. To enable this, we would need more information about VM configuration(No of VMs, OS, vCPUs, memory, Storage). Secondly T &C's wo

Re: Yet another fast GiST build

2020-09-21 Thread Tom Lane
Heikki Linnakangas writes: > On 21/09/2020 02:06, Tom Lane wrote: >> Another interesting point is that all the other index AMs seem to WAL-log >> the new page before the smgrextend call, whereas this code is doing it >> in the other order. I strongly doubt that both patterns are equally >> correc

Re: Global snapshots

2020-09-21 Thread Alexey Kondratov
On 2020-09-18 00:54, Bruce Momjian wrote: On Tue, Sep 8, 2020 at 01:36:16PM +0300, Alexey Kondratov wrote: Thank you for the link! After a quick look on the Sawada-san's patch set I think that there are two major differences: 1. There is a built-in foreign xacts resolver in the [1], which s

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-21 Thread Tom Lane
Amit Kapila writes: > On Sun, Sep 20, 2020 at 10:43 PM Tom Lane wrote: >> AFAICS, there is no chance of the existing pg_surgery regression test >> being fully stable if we don't fix both things. > What if ensure that it runs with autovacuum = off and there is no > parallel test running? I am not

PGXS testing upgrade paths

2020-09-21 Thread James Coleman
If there's a better list than this, please let me know, but I figured hackers is appropriate since the extension building infrastructure is documented in core. While working on an in-house extension I realized that while PGXS provides the standard regression test infrastructure, I'm not aware of a

Re: Lift line-length limit for pg_service.conf

2020-09-21 Thread Tom Lane
Daniel Gustafsson writes: > The pg_service.conf parsing thread [0] made me realize that we have a > hardwired > line length of max 256 bytes. Lifting this would be in line with recent work > for ecpg, pg_regress and pg_hba (784b1ba1a2 and 8f8154a50). The attached > moves > pg_service.conf to u

Re: PGXS testing upgrade paths

2020-09-21 Thread Tom Lane
James Coleman writes: > If there's a better list than this, please let me know, but I figured > hackers is appropriate since the extension building infrastructure is > documented in core. > While working on an in-house extension I realized that while PGXS > provides the standard regression test i

Re: PGXS testing upgrade paths

2020-09-21 Thread James Coleman
On Mon, Sep 21, 2020 at 11:36 AM Tom Lane wrote: > > James Coleman writes: > > If there's a better list than this, please let me know, but I figured > > hackers is appropriate since the extension building infrastructure is > > documented in core. > > > While working on an in-house extension I rea

Re: PGXS testing upgrade paths

2020-09-21 Thread Tom Lane
James Coleman writes: > On Mon, Sep 21, 2020 at 11:36 AM Tom Lane wrote: >> The recommended way to deal with updates these days is to leave the >> original extension script as-is and just write update scripts >> (1.0--1.1, 1.1--1.2, etc). That way, application of the updates >> is tested automat

Re: Planner, check if can use consider HASH for groupings (src/backend/optimizer/plan/planner.c)

2020-09-21 Thread Tomas Vondra
On Mon, Sep 21, 2020 at 08:32:35AM -0300, Ranier Vilela wrote: Em dom., 20 de set. de 2020 às 21:10, Tomas Vondra < tomas.von...@2ndquadrant.com> escreveu: On Sun, Sep 20, 2020 at 08:09:56PM -0300, Ranier Vilela wrote: >Em sex., 18 de set. de 2020 às 10:37, Tomas Vondra < >tomas.von...@2ndquadr

Re: doc review for v13

2020-09-21 Thread Tom Lane
Justin Pryzby writes: > Thanks. Here's the remainder, with some new ones. LGTM. I tweaked one or two places a bit more, and pushed it. regards, tom lane

Re: Planner, check if can use consider HASH for groupings (src/backend/optimizer/plan/planner.c)

2020-09-21 Thread Ranier Vilela
Em seg., 21 de set. de 2020 às 13:37, Tomas Vondra < tomas.von...@2ndquadrant.com> escreveu: > On Mon, Sep 21, 2020 at 08:32:35AM -0300, Ranier Vilela wrote: > >Em dom., 20 de set. de 2020 às 21:10, Tomas Vondra < > >tomas.von...@2ndquadrant.com> escreveu: > > > >> On Sun, Sep 20, 2020 at 08:09:56

Re: Planner, check if can use consider HASH for groupings (src/backend/optimizer/plan/planner.c)

2020-09-21 Thread Tomas Vondra
On Sun, Sep 20, 2020 at 01:50:34PM -0300, Ranier Vilela wrote: Em seg., 21 de set. de 2020 às 13:37, Tomas Vondra < tomas.von...@2ndquadrant.com> escreveu: On Mon, Sep 21, 2020 at 08:32:35AM -0300, Ranier Vilela wrote: >Em dom., 20 de set. de 2020 às 21:10, Tomas Vondra < >tomas.von...@2ndquadr

Re: PATCH: Batch/pipelining support for libpq

2020-09-21 Thread Matthieu Garrigues
Hi Dave, I merged PQbatchProcessQueue into PQgetResult. One first init call to PQbatchProcessQueue was also required in PQsendQueue to have PQgetResult ready to read the first batch query. Tests and documentation are updated accordingly. Matthieu Garrigues On Mon, Sep 21, 2020 at 3:39 PM Dave Cr

Re: Planner, check if can use consider HASH for groupings (src/backend/optimizer/plan/planner.c)

2020-09-21 Thread Ranier Vilela
Em seg., 21 de set. de 2020 às 14:24, Tomas Vondra < tomas.von...@2ndquadrant.com> escreveu: > On Sun, Sep 20, 2020 at 01:50:34PM -0300, Ranier Vilela wrote: > >Em seg., 21 de set. de 2020 às 13:37, Tomas Vondra < > >tomas.von...@2ndquadrant.com> escreveu: > > > >> On Mon, Sep 21, 2020 at 08:32:35

Re: WIP: BRIN multi-range indexes

2020-09-21 Thread John Naylor
On Fri, Sep 18, 2020 at 6:27 PM Tomas Vondra wrote: > But maybe we could still use this scheme by actually computing > > h1 = hash_uint32_extended(value, seed1); > h2 = hash_uint32_extended(value, seed2); > > and then use this as the independent hash functions. I think that would > meet t

Re: pgindent vs dtrace on macos

2020-09-21 Thread Tom Lane
I wrote: > We still have to deal with > src/backend/utils/sort/qsort_tuple.c > src/pl/plpgsql/src/plerrcodes.h > src/pl/plpython/spiexceptions.h > src/pl/tcl/pltclerrcodes.h > if we want to be entirely clean about this. I took care of those, so I think we're done here. reg

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-21 Thread Robert Haas
On Sun, Sep 20, 2020 at 1:13 PM Tom Lane wrote: > 1. My patch a7212be8b does indeed have a problem. It will allow > vacuum_set_xid_limits to compute freezeLimit = nextXid for a temp > table if freeze_min_age is zero (ie VACUUM FREEZE). If there's > any concurrent transactions, this falls foul of

Re: Improper use about DatumGetInt32

2020-09-21 Thread Robert Haas
On Sun, Sep 20, 2020 at 9:17 PM Hou, Zhijie wrote: > In (/contrib/bloom/blutils.c:277), I found it use DatumGetInt32 to get UInt32 > type. > Is it more appropriate to use DatumGetUInt32 here? Typically, the DatumGetBlah() function that you pick should match the SQL data type that the function is

Re: pgindent vs dtrace on macos

2020-09-21 Thread Daniel Gustafsson
> On 21 Sep 2020, at 19:59, Tom Lane wrote: > > I wrote: >> We still have to deal with >> src/backend/utils/sort/qsort_tuple.c >> src/pl/plpgsql/src/plerrcodes.h >> src/pl/plpython/spiexceptions.h >> src/pl/tcl/pltclerrcodes.h >> if we want to be entirely clean about this. > > I took care of tho

Re: OpenSSL 3.0.0 compatibility

2020-09-21 Thread Daniel Gustafsson
> On 19 Sep 2020, at 04:11, Michael Paquier wrote: > On Fri, Sep 18, 2020 at 04:11:13PM +0200, Daniel Gustafsson wrote: >> The other problem was that the cipher context >> padding must be explicitly set, whereas in previous versions relying on the >> default worked fine. EVP_CIPHER_CTX_set_paddi

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-21 Thread Tom Lane
Robert Haas writes: > On Sun, Sep 20, 2020 at 1:13 PM Tom Lane wrote: >> 2. As Amit suspected, there's an inconsistency between pruneheap.c's >> rules for which tuples are removable and vacuum.c's rules for that. >> This seems like a massive bug in its own right: what's the point of >> pruneheap.

Re: Improper use about DatumGetInt32

2020-09-21 Thread Tom Lane
Robert Haas writes: > Typically, the DatumGetBlah() function that you pick should match the > SQL data type that the function is returning. So if the function > returns pg_catalog.int4, which corresponds to the C data type int32, > you would use DatumGetInt32. There is no SQL type corresponding to

Re: pgindent vs dtrace on macos

2020-09-21 Thread Tom Lane
Oh wait, I forgot about the fmgrprotos.h discrepancy. I wrote: > It strikes me that a low-cost workaround would be to rename these > C functions. There's no law that their C names must match the > SQL names. Here's a proposed patch to fix it that way. regards, tom lane

Re: pgindent vs dtrace on macos

2020-09-21 Thread Alvaro Herrera
On 2020-Sep-21, Tom Lane wrote: > Oh wait, I forgot about the fmgrprotos.h discrepancy. > > I wrote: > > It strikes me that a low-cost workaround would be to rename these > > C functions. There's no law that their C names must match the > > SQL names. > > Here's a proposed patch to fix it that

Re: pgindent vs dtrace on macos

2020-09-21 Thread Tom Lane
Alvaro Herrera writes: > On 2020-Sep-21, Tom Lane wrote: >> Here's a proposed patch to fix it that way. > pgtypes_numeric.h still contains > typedef struct > { > } numeric; > ... isn't this more likely to create a typedef entry than merely a > function name? Well, yeah, it *is* a typedef. My p

Re: pgindent vs dtrace on macos

2020-09-21 Thread Alvaro Herrera
On 2020-Sep-21, Tom Lane wrote: > > ... isn't this more likely to create a typedef entry than merely a > > function name? > > Well, yeah, it *is* a typedef. My proposal is to rename the C function > to avoid the conflict, rather than renaming the typedef. Given the > small number of direct call

Re: Improper use about DatumGetInt32

2020-09-21 Thread Andres Freund
Hi, On 2020-09-21 14:08:22 -0400, Robert Haas wrote: > There is no SQL type corresponding to the C data type uint32, so I'm > not sure why we even have DatumGetUInt32. I'm sort of suspicious that > there's some fuzzy thinking going on there. I think we mostly use it for the few places where we c

Re: WIP: BRIN multi-range indexes

2020-09-21 Thread Tomas Vondra
On Mon, Sep 21, 2020 at 01:42:34PM -0400, John Naylor wrote: On Fri, Sep 18, 2020 at 6:27 PM Tomas Vondra wrote: But maybe we could still use this scheme by actually computing h1 = hash_uint32_extended(value, seed1); h2 = hash_uint32_extended(value, seed2); and then use this as the i

Re: Feature improvement for pg_stat_statements

2020-09-21 Thread Andres Freund
Hi, On 2020-09-18 17:55:12 +0900, btnakamichin wrote: > I’m thinking of adding adding a function called > pg_stat_statements_reset_time() that returns the last timestamp when > executed pg_stat_statements_reset(). pg_stat_statements can reset each SQL > statement. We can record each sql reset timi

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-21 Thread Robert Haas
On Mon, Sep 21, 2020 at 2:21 PM Tom Lane wrote: > Right, but what we end up with is that the very same tuple xmin and > xmax might result in pruning/deletion, or not, depending on whether > it's part of a HOT chain or not. That's at best pretty weird, and > at worst it means that corner-case bugs

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-21 Thread Andres Freund
Hi, On 2020-09-21 16:02:29 -0400, Robert Haas wrote: > On Mon, Sep 21, 2020 at 2:21 PM Tom Lane wrote: > > Right, but what we end up with is that the very same tuple xmin and > > xmax might result in pruning/deletion, or not, depending on whether > > it's part of a HOT chain or not. That's at be

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-21 Thread Tom Lane
Robert Haas writes: > On Mon, Sep 21, 2020 at 2:21 PM Tom Lane wrote: >> Right, but what we end up with is that the very same tuple xmin and >> xmax might result in pruning/deletion, or not, depending on whether >> it's part of a HOT chain or not. That's at best pretty weird, and >> at worst it

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-21 Thread Andres Freund
Hi, On 2020-09-20 13:13:16 -0400, Tom Lane wrote: > 2. As Amit suspected, there's an inconsistency between pruneheap.c's > rules for which tuples are removable and vacuum.c's rules for that. > This seems like a massive bug in its own right: what's the point of > pruneheap.c going to huge effort to

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-21 Thread Tom Lane
Andres Freund writes: > The reason for that is that the GlobalVisState stuff is computed > heuristically (and then re-checked if that's not sufficient to prune a > tuple, unless already done so). That's done so GetSnapshotData() doesn't > have to look at each backends ->xmin, which is quite a mass

Re: Parallel Full Hash Join

2020-09-21 Thread Melanie Plageman
On Wed, Sep 11, 2019 at 11:23 PM Thomas Munro wrote: > > While thinking about looping hash joins (an alternative strategy for > limiting hash join memory usage currently being investigated by > Melanie Plageman in a nearby thread[1]), the topic of parallel query > deadlock hazards came back to ha

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-21 Thread Andres Freund
Hi, On 2020-09-21 16:40:40 -0400, Tom Lane wrote: > Andres Freund writes: > >> I think to move forward, we need to figure out what the freezing > >> behavior ought to be for temp tables. We could make it the same > >> as it was before a7212be8b, which'd just require some more complexity > >> in

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-21 Thread Tom Lane
Andres Freund writes: > On 2020-09-21 16:40:40 -0400, Tom Lane wrote: >> Andres Freund writes: >>> I think that's an argument for what I suggested elsewhere, which is that >>> we should move the logic for a different horizon for temp tables out of >>> vacuum_set_xid_limits, and into procarray. >

Re: Handing off SLRU fsyncs to the checkpointer

2020-09-21 Thread Thomas Munro
On Mon, Sep 21, 2020 at 2:19 PM Thomas Munro wrote: > While scanning for comments and identifier names that needed updating, > I realised that this patch changed the behaviour of the ShutdownXXX() > functions, since they currently flush the SLRUs but are not followed > by a checkpoint. I'm not en

Re: new heapcheck contrib module

2020-09-21 Thread Robert Haas
On Tue, Aug 25, 2020 at 10:36 AM Mark Dilger wrote: > Thanks for the review! + msg OUT text + ) Looks like atypical formatting. +REVOKE ALL ON FUNCTION +verify_heapam(regclass, boolea

Re: Command statistics system (cmdstats)

2020-09-21 Thread Robert Haas
On Mon, Sep 21, 2020 at 9:41 AM Alvaro Herrera wrote: > "A couple of weeks" of inactivity is not sufficient, in my view, to boot > a patch out of the commitfest process. Whenever the patch is > resurrected, it will be a new entry which won't have the history that it > had accumulated in the long

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-21 Thread Andres Freund
On 2020-09-21 17:03:53 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2020-09-21 16:40:40 -0400, Tom Lane wrote: > >> Andres Freund writes: > >>> I think that's an argument for what I suggested elsewhere, which is that > >>> we should move the logic for a different horizon for temp tables

Re: Compatible defaults for LEAD/LAG

2020-09-21 Thread Tom Lane
Pavel Stehule writes: > I see few possibilities how to finish and close this issue: > 1. use anycompatible type and add note to documentation so expression of > optional argument can change a result type, and so this is Postgres > specific - other databases and ANSI SQL disallow this. > It is the

RE: Global snapshots

2020-09-21 Thread tsunakawa.ta...@fujitsu.com
Hi Andrey-san, all, From: Andrey V. Lepikhov > On 7/27/20 11:22 AM, tsunakawa.ta...@fujitsu.com wrote: > > Could you take a look at this patent? I'm afraid this is the Clock-SI for > > MVCC. > Microsoft holds this until 2031. I couldn't find this with the keyword > "Clock-SI."" > > > > > > US8

Re: Index Skip Scan (new UniqueKeys)

2020-09-21 Thread Peter Geoghegan
On Sat, Aug 15, 2020 at 7:09 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > Here is a new version that hopefully address most of the concerns > mentioned in this thread so far. As before, first two patches are taken > from UniqueKeys thread and attached only for the reference. List of > changes

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-21 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > Yes, but it still seems hard to me that we require for all FDW > implementations to commit/rollback prepared transactions without the > possibility of ERROR. Of course we can't eliminate the possibility of error, because remote servers require network communication. What

Load TIME fields - proposed performance improvement

2020-09-21 Thread Peter Smith
Hi Hackers. I have a test table with multiple (10) columns defined as TIME WITHOUT TIME ZONE. When loading this table with a lot of data (e.g. "COPY tbl FROM /my/path/2GB.csv WITH (FORMAT CSV)") I observed it was spending an excessive amount of time within the function GetCurrentDateTime. IIUC t

Re: Load TIME fields - proposed performance improvement

2020-09-21 Thread Tom Lane
Peter Smith writes: > IIUC the code is calling GetCurrentDateTime only to acquire the > current TX timestamp as a struct pg_tm in order to derive some > timezone information. > ... > I have attached a patch which caches this struct, so now those 225 > million calls are reduced to just 1 call. Int

Re: Load TIME fields - proposed performance improvement

2020-09-21 Thread Tom Lane
I wrote: > Interesting idea, but this implementation is leaving a *lot* > on the table. If we want to cache the result of > timestamp2tm applied to GetCurrentTransactionStartTimestamp(), > there are half a dozen different call sites that could make > use of such a cache, eg, GetSQLCurrentDate and

Re: shared-memory based stats collector

2020-09-21 Thread Andres Freund
Hi, On 2020-09-08 17:55:57 +0900, Kyotaro Horiguchi wrote: > Locks on the shared statistics is acquired by the units of such like > tables, functions so the expected chance of collision are not so high. I can't really parse that... > Furthermore, until 1 second has elapsed since the last flushi

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Ashutosh Sharma
On Mon, Sep 21, 2020 at 6:58 PM Amit Kapila wrote: > > On Mon, Sep 21, 2020 at 6:27 PM Ashutosh Sharma wrote: > > > > Thanks Dilip for the patch. AFAIU, the fix looks good. One small comment: > > > > Thanks Ashutosh and Dilip for working on this. I'll look into it in a > day or two. > Just a tho

Re: Load TIME fields - proposed performance improvement

2020-09-21 Thread Peter Smith
Hi Tom. Thanks for your feedback. On Tue, Sep 22, 2020 at 12:44 PM Tom Lane wrote: > Still, for the size of the patch I'm envisioning, it'd be well > worth the trouble. The OP patch I gave was just a POC to test the effect and to see if the idea was judged as worthwhile... I will rewrite/fix

Re: Index Skip Scan (new UniqueKeys)

2020-09-21 Thread Peter Geoghegan
On Mon, Sep 21, 2020 at 5:59 PM Peter Geoghegan wrote: > That's all I have for now. One more thing. I don't think that this should be a bitwise AND: if ((offnum > maxoff) & (so->currPos.nextPage == P_NONE)) {

Re: Parallel Full Hash Join

2020-09-21 Thread Thomas Munro
On Tue, Sep 22, 2020 at 8:49 AM Melanie Plageman wrote: > On Wed, Sep 11, 2019 at 11:23 PM Thomas Munro wrote: >> 1. You could probably make it so that the PHJ_BATCH_SCAN_INNER phase >> in this patch (the scan for unmatched tuples) is executed by only one >> process, using the "detach-and-see-if

Parallel INSERT (INTO ... SELECT ...)

2020-09-21 Thread Greg Nancarrow
Hi Hackers, Following on from Dilip Kumar's POC patch for allowing parallelism of the SELECT part of "INSERT INTO ... SELECT ...", I have attached a POC patch for allowing parallelism of both the INSERT and SELECT parts, where it can be allowed. For cases where it can't be allowed (e.g. INSERT int

Use appendStringInfoString and appendPQExpBufferStr where possible

2020-09-21 Thread Hou, Zhijie
Hi hackers In(/src/bin/scripts/reindexdb.c; /src/backend/access/rmgrdesc/dbasedesc.c; /src/pl/plpython/plpy_elog.c) I found some more places that should use appendPQExrBufferStr instead of appendPQExpBuffer. Here is the patch. Previous Discussion: https://www.postgresql.org/message-id/CAKJS1f

Re: Dynamic gathering the values for seq_page_cost/xxx_cost

2020-09-21 Thread Andy Fan
Thanks Ashutosh for coming:) On Mon, Sep 21, 2020 at 9:03 PM Ashutosh Bapat wrote: > On Mon, Sep 21, 2020 at 9:11 AM Andy Fan wrote: > > > > Here are some changes for my detection program. > > > > | | seq_read_lat (us) | > random_read_lat (us) | >

Re: Dynamic gathering the values for seq_page_cost/xxx_cost

2020-09-21 Thread Andy Fan
> > > It's probably worth testing on various other storage systems to see >> how that applies to those. >> >> Yes, I can test more on new hardware once I get it. Now it is still in > progress. > However I can only get a physical machine with SSD or Virtual machine with > SSD, other types are hard

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Amit Kapila
On Tue, Sep 22, 2020 at 8:34 AM Ashutosh Sharma wrote: > > On Mon, Sep 21, 2020 at 6:58 PM Amit Kapila wrote: > > > > On Mon, Sep 21, 2020 at 6:27 PM Ashutosh Sharma > > wrote: > > > > > > Thanks Dilip for the patch. AFAIU, the fix looks good. One small comment: > > > > > > > Thanks Ashutosh an

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Ashutosh Sharma
On Tue, Sep 22, 2020 at 12:02 PM Amit Kapila wrote: > > On Tue, Sep 22, 2020 at 8:34 AM Ashutosh Sharma wrote: > > > > On Mon, Sep 21, 2020 at 6:58 PM Amit Kapila wrote: > > > > > > On Mon, Sep 21, 2020 at 6:27 PM Ashutosh Sharma > > > wrote: > > > > > > > > Thanks Dilip for the patch. AFAIU,

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Ashutosh Sharma
On Tue, Sep 22, 2020 at 12:22 PM Ashutosh Sharma wrote: > > On Tue, Sep 22, 2020 at 12:02 PM Amit Kapila wrote: > > > > On Tue, Sep 22, 2020 at 8:34 AM Ashutosh Sharma > > wrote: > > > > > > On Mon, Sep 21, 2020 at 6:58 PM Amit Kapila > > > wrote: > > > > > > > > On Mon, Sep 21, 2020 at 6:27

  1   2   >