Re: doc review for v14

2021-02-25 Thread Michael Paquier
On Wed, Feb 24, 2021 at 01:39:55AM -0600, Justin Pryzby wrote: > On Wed, Feb 24, 2021 at 04:18:51PM +0900, Michael Paquier wrote: >> +The status of each kind of extended statistics is shown in a column >> +named after the "kind" (e.g. Ndistinct). >> +NULL means that it doesn

Re: Improvements and additions to COPY progress reporting

2021-02-25 Thread Michael Paquier
On Wed, Feb 24, 2021 at 01:53:03AM -0600, Justin Pryzby wrote: > On Sun, Feb 21, 2021 at 08:10:09PM +0100, Matthias van de Meent wrote: > I think these should say that they report their progress *in* the view (not > "to"): > > > +Each backend running ANALYZE will report its > > progress to >

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-02-25 Thread Dmitry Dolgov
> On Wed, Feb 24, 2021 at 01:45:10PM -0800, Andres Freund wrote: > > > I'm curious if it makes sense > > to explore possibility to have these sort of "backpressure", e.g. if > > number of inflight requests is too large calculate inflight_limit a bit > > lower than possible (to avoid hard performanc

Re: [PATCH] Feature improvement for TRUNCATE tab completion.

2021-02-25 Thread Fujii Masao
On 2021/02/24 12:59, Fujii Masao wrote: On 2021/02/23 1:44, Muhammad Usama wrote: The following review has been posted through the commitfest application: make installcheck-world:  not tested Implements feature:   tested, passed Spec compliant:   tested, passed Documentation:   

Re: WIP: System Versioned Temporal Table

2021-02-25 Thread Li Japin
On Jan 27, 2021, at 12:39 AM, Surafel Temesgen mailto:surafel3...@gmail.com>> wrote: On Tue, Jan 26, 2021 at 2:33 PM Vik Fearing mailto:v...@postgresfriends.org>> wrote: I'm still in the weeds of reviewing this patch, but why should this fail? It should not fail. Attached is rebased patch t

Re: archive_command / pg_stat_archiver & documentation

2021-02-25 Thread Benoit Lobréau
Le mer. 24 févr. 2021 à 14:52, Julien Rouhaud a écrit : > Hi, > > On Wed, Feb 24, 2021 at 8:21 PM talk to ben wrote: > > > > The documentation describes how a return code > 125 on the > restore_command would prevent the server from starting [1] : > > > > " > > It is important that the command re

Re: repeated decoding of prepared transactions

2021-02-25 Thread vignesh C
On Wed, Feb 24, 2021 at 5:06 PM Ajin Cherian wrote: > > On Wed, Feb 24, 2021 at 4:48 PM Ajin Cherian wrote: > > > I plan to split this into two patches next. But do review and let me > > know if you have any comments. > > Attaching an updated patch-set with the changes for > snapshot_was_exported

Re: repeated decoding of prepared transactions

2021-02-25 Thread Amit Kapila
On Wed, Feb 24, 2021 at 5:06 PM Ajin Cherian wrote: > > On Wed, Feb 24, 2021 at 4:48 PM Ajin Cherian wrote: > > > I plan to split this into two patches next. But do review and let me > > know if you have any comments. > > Attaching an updated patch-set with the changes for > snapshot_was_exported

Re: [PATCH] Note effect of max_replication_slots on subscriber side in documentation.

2021-02-25 Thread Amit Kapila
On Thu, Feb 25, 2021 at 2:19 AM Paul Martinez wrote: > > Hey, all, > > I went ahead and made a patch for introducing a new GUC variable, > max_replication_origins, to replace the awkward re-use of > max_replication_slots. > > I'm mostly indifferent whether a new GUC variable is necessary, or > sim

Re: 64-bit XIDs in deleted nbtree pages

2021-02-25 Thread Masahiko Sawada
On Thu, Feb 25, 2021 at 1:42 PM Peter Geoghegan wrote: > > On Mon, Feb 22, 2021 at 2:54 PM Peter Geoghegan wrote: > > Good point. I think that the structure should make the page deletion > > triggering condition have only secondary importance -- it is only > > described at all to be complete and

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-25 Thread Amit Langote
On Wed, Feb 24, 2021 at 6:03 PM Amit Kapila wrote: > On Wed, Feb 24, 2021 at 2:14 PM Greg Nancarrow wrote: > > On Wed, Feb 24, 2021 at 3:12 PM Amit Kapila wrote: > > > On Wed, Feb 24, 2021 at 8:41 AM Greg Nancarrow > > > wrote: > > > > On Tue, Feb 23, 2021 at 10:53 PM Amit Kapila > > > > wro

Re: archive_command / pg_stat_archiver & documentation

2021-02-25 Thread Julien Rouhaud
On Thu, Feb 25, 2021 at 7:25 PM Benoit Lobréau wrote: > > Le mer. 24 févr. 2021 à 14:52, Julien Rouhaud a écrit : >> >> I thought that this behavior was documented, especially for the lack >> of update of pg_stat_archiver. If it's not the case then we should >> definitely fix that! > > I tried t

Re: cursor sensitivity misunderstanding

2021-02-25 Thread Peter Eisentraut
On 18.02.21 19:14, Peter Eisentraut wrote: On 18.02.21 17:11, David G. Johnston wrote: The OP was doing a course based on Oracle and was confused regarding our behavior.  The documentation failed to help me provide a useful response, so I'd agree there is something here that needs reworking if

Re: SSL SNI

2021-02-25 Thread Peter Eisentraut
On 15.02.21 15:28, Matthias van de Meent wrote: +/* + * Set Server Name Indication (SNI), but not if it's a literal IP address. + * (RFC 6066) + */ +if (!((conn->pghost[0] >= '0' && conn->pghost[0] <= '9') || strchr(conn->pghost, ':'))) '1one.example.com' is a valid hostname,

Re: SSL SNI

2021-02-25 Thread Peter Eisentraut
On 17.02.21 00:01, Jacob Champion wrote: On Mon, 2021-02-15 at 15:09 +0100, Peter Eisentraut wrote: The question I had was whether this should be an optional behavior, or conversely a behavior that can be turned off, or whether it should just be turned on all the time. Personally I think there

Re: logical decoding of two-phase transactions

2021-02-25 Thread onlinebusinessindia
> That's where you've misunderstood - it isn't committed yet. The point or > this change is to allow us to do logical decoding at the PREPARE > TRANSACTION > point. The xact is not yet committed or rolled back. Yes, I got that. I was looking for a why or an actual use-case. > Stas wants this for

Re: Removing support for COPY FROM STDIN in protocol version 2

2021-02-25 Thread John Naylor
On Thu, Feb 4, 2021 at 11:47 AM Heikki Linnakangas wrote: > > On 04/02/2021 17:35, Tom Lane wrote: > > Alvaro Herrera writes: > >> Yeah, the changes I was thinking about are all in libpq-int.h so that's > >> not really a problem. But one enum in libpq-fe.h renumbers values, and > >> I think it's

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-02-25 Thread Alexey Lesovsky
Hi, On 25.02.2021 02:03, Andres Freund wrote: pg_stat_aio_backends. This stat is based on COUNTERs, which is great, but the issue here is that its lifespan is limited by the lifespan of the backend processes - once the backend exits the stat will no longer be available - which could be inappropr

Re: libpq debug log

2021-02-25 Thread Álvaro Herrera
I tweaked this code a little bit more. I didn't like that libpq-trace.h was exposing all the internal details of the API to the world; and I liked it even less that libpq-int.h had to include the file, exposing everything everywhere. I added some forward struct declarations in libpq-int.h to avo

Re: SSL SNI

2021-02-25 Thread Jacob Champion
On Thu, 2021-02-25 at 17:00 +0100, Peter Eisentraut wrote: > Just as additional data points, it has come to my attention that both > the Go driver ("lib/pq") and the JDBC environment already send SNI > automatically. (In the case of JDBC this is done by the Java system > libraries, not the JDBC

Re: libpq compression

2021-02-25 Thread Daniil Zakhlystov
Hi, thanks for your review, > On Feb 22, 2021, at 10:38 AM, Craig Ringer > wrote: > > > On Thu, 11 Feb 2021, 21:09 Daniil Zakhlystov, > wrote:: > > 3. Chunked compression allows to compress only well compressible messages and > save the CPU cycles by not compressing the others > 4. Chunked

Re: libpq debug log

2021-02-25 Thread Álvaro Herrera
It appears that something is still wrong. I applied lipq pipeline v27 from [1] and ran src/test/modules/test_libpq/pipeline singlerow, after patching it to do PQtrace() after PQconn(). Below is the output I get from that. The noteworthy point is that "ParseComplete" messages appear multiple tim

Re: libpq debug log

2021-02-25 Thread Álvaro Herrera
This assert I added? > +static bool > +pqTraceMaybeBreakLine(int size, PGconn *conn) > +{ > + Assert(conn->be_msg->length > 0); It breaks. Maybe there's a good reason, maybe there ain't, but in the meantime I just removed it. -- Álvaro Herrera Valdivia, Chile "Para tener más hay que

Re: Some regular-expression performance hacking

2021-02-25 Thread Tom Lane
I wrote: > However, in a different line of thought, I realized that the > memory allocation logic could use some polishing. It gives out > ten arcs per NFA state initially, and then adds ten more at a time. > However, that's not very bright when you look at the actual usage > patterns, because mos

Re: Single transaction in the tablesync worker?

2021-02-25 Thread Masahiko Sawada
On Thu, Feb 25, 2021 at 1:52 PM Amit Kapila wrote: > > On Wed, Feb 24, 2021 at 5:55 PM Amit Kapila wrote: > > > > On Wed, Feb 24, 2021 at 12:47 PM Masahiko Sawada > > wrote: > > > > > > On Fri, Feb 12, 2021 at 2:49 PM Amit Kapila > > > wrote: > > > > > > > > > > > > Thanks, I have pushed the

Re: 64-bit XIDs in deleted nbtree pages

2021-02-25 Thread Peter Geoghegan
On Thu, Feb 25, 2021 at 5:42 AM Masahiko Sawada wrote: > btvacuumcleanup() has been playing two roles: recycling deleted pages > and collecting index statistics. Right. I pushed the VACUUM VERBOSE "index pages newly deleted" instrumentation patch earlier - it really isn't complicated or controv

Re: Disallow SSL compression?

2021-02-25 Thread Michael Paquier
On Mon, Feb 22, 2021 at 12:27:38PM +0100, Daniel Gustafsson wrote: > On 22 Feb 2021, at 11:52, Magnus Hagander wrote: >> I think for libpq we want to keep the option for a while but making it >> a no-op, to not unnecessarily break systems where people just upgrade >> libpq, though. And document it

Re: SSL SNI

2021-02-25 Thread Greg Stark
Hate to be that guy but This still doesn't seem like it is IPv6-ready. Is there any harm in having SNI with an IPv6 address there if it gets through?

RE: libpq debug log

2021-02-25 Thread iwata....@fujitsu.com
Alvaro san, Thank you very much for your updating and organizing this patch. It appears that something is still wrong. I applied lipq pipeline v27 from [1] and ran src/test/modules/test_libpq/pipeline singlerow, after patching it to do PQtrace() after PQconn(). Below is the output I get from

Update docs of logical replication for commit ce0fdbfe97.

2021-02-25 Thread Amit Kapila
We forgot to update the logical replication configuration settings page in commit ce0fdbfe97. After commit ce0fdbfe97, table synchronization workers also started using replication origins to track the progress and the same should be reflected in docs. Attached patch for the same. v1-0001-Update-

Re: Some regular-expression performance hacking

2021-02-25 Thread Tom Lane
I wrote: > So I rearranged things to allocate arcs out of a common pool, and for > good measure made the state allocation code do the same thing. I was > pretty much blown away by the results: not only is the average-case > space usage about half what it is on HEAD, but the worst-case drops > by w

[PATCH] pgbench: Bug fix for the -d option

2021-02-25 Thread miyake_kouta
Hi. I found a bug in pgbench's -d option and created a patch. The bug is the following: pgbench's option -d can display debug log about connection, which is like "pghost: foo pgport: 5432 nclients: 1 nxacts: 10 dbName: bar". This configuration is supplied by other options or environment variabl

Re: [HACKERS] logical decoding of two-phase transactions

2021-02-25 Thread Amit Kapila
On Thu, Feb 25, 2021 at 12:32 PM Peter Smith wrote: > > Please find attached the latest patch set v43* > > Differences from v42* > > - Rebased to HEAD as @ today > > - Added new patch 0006 "Tablesync early exit" as discussed here [1] > I feel we can start a separate thread for this as it can be d

Re: libpq debug log

2021-02-25 Thread Kyotaro Horiguchi
At Thu, 25 Feb 2021 21:04:00 -0300, Álvaro Herrera wrote in > This assert I added? > > > +static bool > > +pqTraceMaybeBreakLine(int size, PGconn *conn) > > +{ > > + Assert(conn->be_msg->length > 0); > > It breaks. Maybe there's a good reason, maybe there ain't, but in the > meantime I just

Re: libpq debug log

2021-02-25 Thread Kyotaro Horiguchi
At Fri, 26 Feb 2021 13:35:26 +0900 (JST), Kyotaro Horiguchi wrote in > At Thu, 25 Feb 2021 21:04:00 -0300, Álvaro Herrera > wrote in > > This assert I added? > > > > > +static bool > > > +pqTraceMaybeBreakLine(int size, PGconn *conn) > > > +{ > > > + Assert(conn->be_msg->length > 0); > > >

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-25 Thread Amit Langote
Hi Greg, Replying to an earlier email in the thread because I think I found a problem relevant to the topic that was brought up. On Wed, Feb 17, 2021 at 10:34 PM Amit Langote wrote: > On Wed, Feb 17, 2021 at 10:44 AM Greg Nancarrow wrote: > > Is the use of "table_close(rel, NoLock)'' intentiona

Re: libpq debug log

2021-02-25 Thread Kyotaro Horiguchi
(we had better avoid top-posting:p) At Fri, 26 Feb 2021 02:52:49 +, "iwata@fujitsu.com" wrote in > Alvaro san, > > Thank you very much for your updating and organizing this patch. > > It appears that something is still wrong. I applied lipq pipeline v27 from > [1] and ran src/test/m

Re: libpq debug log

2021-02-25 Thread Kyotaro Horiguchi
(Is what I'm drinking now decafe?) At Fri, 26 Feb 2021 14:33:26 +0900 (JST), Kyotaro Horiguchi wrote in > - Tweaked psql to enable tracing. (attached) > > - Connect to a server using SSL. > > - Restart the server. -> This is a mistake. Just restaring the server doesn't cause this. > - Re

Replace buffer I/O locks with condition variables (reviving an old patch)

2021-02-25 Thread Thomas Munro
Hello hackers, Back in 2016, Robert Haas proposed to replace I/O locks with condition variables[1]. Condition variables went in and have found lots of uses, but this patch to replace a bunch of LWLocks and some busy looping did not. Since then, it has been tested quite a lot as part of the AIO p

Re: Global snapshots

2021-02-25 Thread Andrey V. Lepikhov
On 1/1/21 8:14 AM, tsunakawa.ta...@fujitsu.com wrote: -- 11. A method comprising: receiving information relating to a distributed database transaction operating on data in data stores associated with respective participating nodes associated with

RE: libpq debug log

2021-02-25 Thread tsunakawa.ta...@fujitsu.com
From: Álvaro Herrera > It appears that something is still wrong. I applied lipq pipeline v27 from > [1] and ran src/test/modules/test_libpq/pipeline singlerow, after patching it > to do PQtrace() after PQconn(). Below is the output I get from that. The > noteworthy point is that "ParseComplete

RE: Parallel INSERT (INTO ... SELECT ...)

2021-02-25 Thread houzj.f...@fujitsu.com
> I add some code to track the time spent in index operation. > From the results[1], we can see more workers will bring more cost in > _bt_search_insert() in each worker. > After debugged, the most cost part is the following: > - > /* drop the read lock on the page, then acquire o

RE: Global snapshots

2021-02-25 Thread tsunakawa.ta...@fujitsu.com
From: Andrey V. Lepikhov > After researching in depth, I think this is the real problem. > My idea was that we are not using real clocks, we only use clock ticks to > measure time intervals. It can also be interpreted as a kind of clock. Yes, patent claims tend to be written to cover broad interp

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-25 Thread Greg Nancarrow
On Fri, Feb 26, 2021 at 4:07 PM Amit Langote wrote: > > Hi Greg, > > Replying to an earlier email in the thread because I think I found a > problem relevant to the topic that was brought up. > > On Wed, Feb 17, 2021 at 10:34 PM Amit Langote wrote: > > On Wed, Feb 17, 2021 at 10:44 AM Greg Nancarr

Re: Postgresql network transmission overhead

2021-02-25 Thread Michael J. Baars
On Wed, 2021-02-24 at 19:18 -0600, Justin Pryzby wrote: > On Wed, Feb 24, 2021 at 09:14:19AM +0100, Michael J. Baars wrote: > > I've written this function to insert several rows at once, and noticed a > > certain postgresql overhead as you can see from the log file. A lot more > > data than > > t

Add --tablespace option to reindexdb

2021-02-25 Thread Michael Paquier
Hi all, Since c5b2860, it is possible to specify a tablespace for a REINDEX, but the equivalent option has not been added to reindexdb. Attached is a patch to take care of that. This includes documentation and tests. While on it, I have added tests for toast tables and indexes with a tablespace

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-25 Thread Amit Langote
On Fri, Feb 26, 2021 at 3:35 PM Greg Nancarrow wrote: > On Fri, Feb 26, 2021 at 4:07 PM Amit Langote wrote: > > The attached patch fixes this, although I am starting to have second > > thoughts about how we're tracking partitions in this patch. Wondering > > if we should bite the bullet and add

Re: SSL SNI

2021-02-25 Thread Peter Eisentraut
On 26.02.21 03:40, Greg Stark wrote: This still doesn't seem like it is IPv6-ready. Do you mean the IPv6 detection code is not correct? What is the problem? > Is there any harm in> having SNI with an IPv6 address there if it gets through? I doubt it.

Re: libpq debug log

2021-02-25 Thread Kyotaro Horiguchi
At Fri, 26 Feb 2021 06:21:15 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Álvaro Herrera > > It appears that something is still wrong. I applied lipq pipeline v27 from > > [1] and ran src/test/modules/test_libpq/pipeline singlerow, after patching > > it to do PQtrace() after PQconn(

Re: pg_attribute.attname inconsistency when renaming primary key columns

2021-02-25 Thread Michael Paquier
On Wed, Feb 24, 2021 at 04:55:11PM +0900, Kyotaro Horiguchi wrote: > At Mon, 22 Feb 2021 21:42:44 +0100, "Joel Jacobson" wrote > in > > I solved my problem by using attnum::text instead of attname for > > pg_class.relkind = ‘i’ as a work-around to avoid a diff. > > For your information, note t

Re: a misbehavior of partition row movement (?)

2021-02-25 Thread Masahiko Sawada
On Mon, Feb 22, 2021 at 3:04 PM Amit Langote wrote: > > On Fri, Feb 19, 2021 at 5:04 PM Masahiko Sawada wrote: > > On Mon, Feb 15, 2021 at 10:37 PM Amit Langote > > wrote: > > > Regarding the patch, I would have liked if it only prevented the > > > update when the foreign key that would be viol

Re: a misbehavior of partition row movement (?)

2021-02-25 Thread Amit Langote
Hi Rahila, On Wed, Feb 24, 2021 at 3:07 PM Rahila Syed wrote: >> > I think the documentation update is missing from the patches. >> >> Hmm, I don't think we document the behavior that is improved by the v3 >> patches as a limitation of any existing feature, neither of foreign >> keys referencing

Re: REINDEX backend filtering

2021-02-25 Thread Julien Rouhaud
Hi, On Wed, Feb 24, 2021 at 09:34:59PM +0300, Darafei "Komяpa" Praliaskouski wrote: > Hello, > > The PostGIS project needed this from time to time. Would be great if > reindex by opclass can be made possible. > > We changed the semantics of btree at least twice (in 2.4 and 3.0), fixed > some ND

Re: REINDEX backend filtering

2021-02-25 Thread Julien Rouhaud
On Thu, Feb 25, 2021 at 07:36:02AM +1300, Thomas Munro wrote: > On Thu, Feb 25, 2021 at 1:22 AM Julien Rouhaud wrote: > > #define reindexHasFilter(x)((x & REINDEXOPT_COLL_NOT_CURRENT) != 0) > > It's better to use "(x) & ..." in macros to avoid weird operator > precedence problems in futur