Re: Inconsistent behavior of smart shutdown handling for queries with and without parallel workers

2020-08-13 Thread Bharath Rupireddy
I think the inconsistent behaviour reported in this thread gets resolved with the approach and patch being discussed in [1]. > > 1. In general, do we need to allow postmaster to send different > signals to bgworkers for fast and smart shutdowns and let them > differentiate the two modes(if

Re: doc examples for pghandler

2020-08-13 Thread Michael Paquier
On Tue, Aug 11, 2020 at 01:01:10PM -0700, Mark Wong wrote: > Ah, right. For the moment I've added some empty conditionals for > trigger and event trigger handling. > > I've created a new entry in the commitfest app. [1] I'll keep at it. :) Thanks for the patch. I have reviewed and reworked it

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-13 Thread Thomas Munro
On Thu, Aug 13, 2020 at 6:38 PM Amit Kapila wrote: > I have pushed that patch last week and attached are the remaining > patches. I have made a few changes in the next patch > 0001-Extend-the-BufFile-interface.patch and have some comments on it > which are as below: Hi Amit, I noticed that

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-13 Thread Masahiko Sawada
On Wed, 12 Aug 2020 at 22:27, Ashutosh Sharma wrote: > > Thanks Robert for the review. Please find my comments inline below: > > On Fri, Aug 7, 2020 at 9:21 PM Robert Haas wrote: > > > > On Thu, Aug 6, 2020 at 9:23 AM Ashutosh Sharma > > wrote: > > > Attached v4 patch fixes the latest comments

Re: massive FPI_FOR_HINT load after promote

2020-08-13 Thread Masahiko Sawada
On Wed, 12 Aug 2020 at 02:42, Alvaro Herrera wrote: > > On 2020-Aug-11, Masahiko Sawada wrote: > > > On Tue, 11 Aug 2020 at 07:56, Alvaro Herrera > > wrote: > > > > So if you have some table where tuples gain hint bits in bulk, and > > > rarely modify the pages afterwards, and promote before

Re: Collation versioning

2020-08-13 Thread Thomas Munro
On Thu, Aug 13, 2020 at 9:52 PM Julien Rouhaud wrote: > Here's a rebased v27 that removes the current approach to ignore indexes that > don't rely on a stable ordering. I'll start a new thread on that matter once > the infrastructure pieces will be committed. Thanks Julien. I'm planning to do

Re: Asynchronous Append on postgres_fdw nodes.

2020-08-13 Thread Thomas Munro
On Thu, Jul 2, 2020 at 3:20 PM Etsuro Fujita wrote: > On Thu, Jul 2, 2020 at 11:14 AM Kyotaro Horiguchi > wrote: > > As the result of a discussion with Fujita-san off-list, I'm going to > > hold off development until he decides whether mine or Thomas' is > > better. > > I'd like to join the

Re: fixing old_snapshot_threshold's time->xid mapping

2020-08-13 Thread Thomas Munro
On Fri, Aug 14, 2020 at 12:52 PM Thomas Munro wrote: > Here's a rebase. And another, since I was too slow and v6 is already in conflict... sorry for the high frequency patches. From 1ced7b8c881676f21623c048f5e9e012ca8416ec Mon Sep 17 00:00:00 2001 From: Robert Haas Date: Thu, 16 Apr 2020

Re: security_context_t marked as deprecated in libselinux 3.1

2020-08-13 Thread Michael Paquier
On Thu, Aug 13, 2020 at 08:47:28PM -0400, Tom Lane wrote: > Should we back-patch that? Usually I figure that people might want > to build back PG branches on newer platforms at some point, so that > it's useful to apply portability fixes across-the-board. On the > other hand, since it's only a

Re: pg_dump from v13 is slow

2020-08-13 Thread Tom Lane
Justin Pryzby writes: > I can reproduce the issue with generated data: > pryzbyj=# SELECT format('create table t%s(i int)', i) FROM > generate_series(1,)i; Hm, I tried this case and didn't really detect much runtime difference between v12 and HEAD. > I don't know if it's any issue, but I

Re: pg_dump from v13 is slow

2020-08-13 Thread Justin Pryzby
On Thu, Aug 13, 2020 at 08:53:46PM -0400, Alvaro Herrera wrote: > Hmm, I wonder if you're comparing an assert-enabled pg13 build to a > non-assert-enabled pg12 build, or something like that. Great question - I thought of it myself but then forgot to look.. $ rpm -q postgresql1{2,3}-server

Re: pg_dump from v13 is slow

2020-08-13 Thread Alvaro Herrera
Hmm, I wonder if you're comparing an assert-enabled pg13 build to a non-assert-enabled pg12 build, or something like that. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: fixing old_snapshot_threshold's time->xid mapping

2020-08-13 Thread Thomas Munro
On Wed, Apr 22, 2020 at 5:39 PM Dilip Kumar wrote: > - if (ts == update_ts) > + if (ts >= update_ts) Hi Dilip, I didn't follow this bit -- could you explain? Here's a rebase. In the 0004 patch I chose to leave behind some unnecessary braces to avoid

Re: security_context_t marked as deprecated in libselinux 3.1

2020-08-13 Thread Tom Lane
Michael Paquier writes: > Applied on HEAD then. Thanks for the discussion! Should we back-patch that? Usually I figure that people might want to build back PG branches on newer platforms at some point, so that it's useful to apply portability fixes across-the-board. On the other hand, since

Re: pg_dump from v13 is slow

2020-08-13 Thread Justin Pryzby
I can reproduce the issue with generated data: pryzbyj=# SELECT format('create table t%s(i int)', i) FROM generate_series(1,)i; \set ECHO errors \set QUIET \gexec $ time pg_dump --section=pre-data -d pryzbyj -h /tmp -p 5613 |wc 110015 240049 1577087 real0m50.445s $ time pg_dump

Re: security_context_t marked as deprecated in libselinux 3.1

2020-08-13 Thread Michael Paquier
On Thu, Aug 13, 2020 at 02:35:28PM +0900, Michael Paquier wrote: > Okay, thanks for confirming. Let's do so then, I'll just wait a bit > to see if there are more comments from others. Applied on HEAD then. Thanks for the discussion! -- Michael signature.asc Description: PGP signature

Re: 回复:how to create index concurrently on partitioned table

2020-08-13 Thread Michael Paquier
On Wed, Aug 12, 2020 at 10:37:08PM +0900, Michael Paquier wrote: > I have been able to work more on this patch today, and finally added > an error context for the transaction block check, as that's cleaner. > In my manual tests, I have also bumped into a case that failed with > the original patch

RE: Libpq support to connect to standby server as priority

2020-08-13 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > I think it would be better to have read-write and read-only check > trnasaction_read_only, and primary and standby can check the new > thing. There can never be any real advantage in having synonyms for > the same thing, but there can be an advantage to letting users choose >

Re: security_context_t marked as deprecated in libselinux 3.1

2020-08-13 Thread Michael Paquier
On Thu, Aug 13, 2020 at 06:54:41AM -0400, Joe Conway wrote: > On 8/13/20 1:22 AM, Michael Paquier wrote: >> Joe, what's the version of libselinux used in rhinoceros? 2.5? > > rpm -q libselinux > libselinux-2.5-15.el7.x86_64 Thanks! -- Michael signature.asc Description: PGP signature

Re: pg_dump from v13 is slow

2020-08-13 Thread Andres Freund
Hi, On 2020-08-13 17:48:23 -0500, Justin Pryzby wrote: > I'm trying to narrow this down, but I'd be very happy for suggestions. Would be worth knowing how much of the time pgbench is 100% CPU utilized, and how much of the time it is basically waiting for server side queries and largely idle. If

Re: pg_dump from v13 is slow

2020-08-13 Thread Alvaro Herrera
On 2020-Aug-13, Justin Pryzby wrote: > I'm trying to narrow this down, but I'd be very happy for suggestions. Maybe you can time "pg_dump --binary-upgrade" to see if the slowness comes from there. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7

pg_dump from v13 is slow

2020-08-13 Thread Justin Pryzby
I ran into this while running pg_upgrade from beta2 to beta3: $ time sudo -u postgres sh -c 'cd /var/lib/pgsql; /usr/pgsql-13/bin/pg_upgrade -b /usr/pgsql-13b2/bin/ -d ./13b2/data -D ./13/data --link' real94m18.335s This instances has many table partitions, and the production

Re: [BUG] Error in BRIN summarization

2020-08-13 Thread Alvaro Herrera
On 2020-Aug-13, Anastasia Lubennikova wrote: > Cool. > This version looks much simpler than mine and passes the tests fine. Thanks, pushed it to all branches. Thanks for diagnosing this problem! -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7

Re: proposal: possibility to read dumped table's name from file

2020-08-13 Thread Justin Pryzby
On Mon, Jul 27, 2020 at 07:25:54AM +0200, Pavel Stehule wrote: > so 25. 7. 2020 v 15:26 odesílatel vignesh C napsal: > > > On Tue, Jul 14, 2020 at 12:03 PM Pavel Stehule > > wrote: > > >> I meant can this: > > >> printf(_(" --filter=FILENAMEread object name filter > > >>

Re: run pgindent on a regular basis / scripted manner

2020-08-13 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Thu, Aug 13, 2020 at 12:30 PM Stephen Frost wrote: > > So, in our world, wouldn't this translate to 'make cfbot complain'? > > This seems like it would be useful, but we'd have to figure out what > to do about typedefs.list. If the

Re: run pgindent on a regular basis / scripted manner

2020-08-13 Thread Andres Freund
Hi, On 2020-08-13 12:50:16 -0400, Tom Lane wrote: > Stephen Frost writes: > > So, in our world, wouldn't this translate to 'make cfbot complain'? > > > I'm definitely a fan of the idea of having cfbot flag these and then we > > maybe get to a point where it's not the committers dealing with

Re: run pgindent on a regular basis / scripted manner

2020-08-13 Thread Robert Haas
On Thu, Aug 13, 2020 at 3:47 PM Tom Lane wrote: > The traditional reason for not doing pgindent too often has been that > it'd cause more work for people who have to rebase their patches over > pgindent's results. If we want to do it more often, then in order to > respond to that concern, I

Re: Libpq support to connect to standby server as priority

2020-08-13 Thread Robert Haas
On Fri, Dec 27, 2019 at 8:08 AM Alvaro Herrera wrote: > So, we can know whether server is primary/standby by checking > in_recovery, as opposed to knowing whether read-write which is done by > checking transaction_read_only. So we can keep read-write as a synonym > for "primary", and check

Re: run pgindent on a regular basis / scripted manner

2020-08-13 Thread Tom Lane
Robert Haas writes: > I am not sure whether weekly or after-every-commit pgindent runs is a > good idea, but I think we should try to do it once a month or so. It's > too annoying otherwise. I could go either way on the question of > automation. The traditional reason for not doing pgindent too

Re: run pgindent on a regular basis / scripted manner

2020-08-13 Thread Robert Haas
On Thu, Aug 13, 2020 at 12:30 PM Stephen Frost wrote: > So, in our world, wouldn't this translate to 'make cfbot complain'? This seems like it would be useful, but we'd have to figure out what to do about typedefs.list. If the patch is indented with the current one (which is auto-generated by

Re: run pgindent on a regular basis / scripted manner

2020-08-13 Thread Robert Haas
On Wed, Aug 12, 2020 at 7:47 PM Tom Lane wrote: > I'm not in favor of unsupervised pgindent runs, really. It can do a lot > of damage to code that was written without thinking about it --- in > particular, it'll make a hash of comment blocks that were manually > formatted and not protected with

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-13 Thread Robert Haas
On Thu, Aug 13, 2020 at 3:52 AM Ashutosh Sharma wrote: > This looks like a very good suggestion to me. I will do this change in > the next version. Just wondering if we should be doing similar changes > in other contrib modules (like pgrowlocks, pageinspect and > pgstattuple) as well? It seems

Re: jsonb, collection & postgres_fdw

2020-08-13 Thread Tom Lane
Konstantin Knizhnik writes: > Or funccollid=0 doesn't mean that collations of function arguments do > not affect function behavior? No, it does not. As I said already, there is no way to tell from outside a function whether it pays attention to collation or not. funccollid is the collation to

Re: jsonb, collection & postgres_fdw

2020-08-13 Thread Konstantin Knizhnik
On 13.08.2020 20:00, Tom Lane wrote: Konstantin Knizhnik writes: Right now jsonb functions are treated as non-shippable by postgres_fdw and so predicates with them are not pushed down to foreign server: Yeah, that's kind of annoying, but breaking the collation check is not an acceptable

Re: jsonb, collection & postgres_fdw

2020-08-13 Thread Tom Lane
Konstantin Knizhnik writes: > Right now jsonb functions are treated as non-shippable by postgres_fdw > and so predicates with them are not pushed down to foreign server: Yeah, that's kind of annoying, but breaking the collation check is not an acceptable fix. And what you're proposing *does*

Re: run pgindent on a regular basis / scripted manner

2020-08-13 Thread Magnus Hagander
On Thu, Aug 13, 2020 at 6:30 PM Stephen Frost wrote: > > * Magnus Hagander (mag...@hagander.net) wrote: > > > There are many solutions that do such things but that do it in the > "github > > workflow" way, which is they do change -> commit -> create pull request, > > and then somebody eyeballs

Re: run pgindent on a regular basis / scripted manner

2020-08-13 Thread Tom Lane
Stephen Frost writes: > So, in our world, wouldn't this translate to 'make cfbot complain'? > I'm definitely a fan of the idea of having cfbot flag these and then we > maybe get to a point where it's not the committers dealing with fixing > patches that weren't pgindent'd properly, it's the

Re: Parallel query hangs after a smart shutdown is issued

2020-08-13 Thread Tom Lane
I wrote: > Hmmm ... maybe that should be more like > if (smartShutState != SMART_NORMAL_USAGE && > backend_type == BACKEND_TYPE_NORMAL) After some more rethinking and testing, here's a v5 that feels fairly final to me. I realized that the logic in canAcceptConnections was

Re: run pgindent on a regular basis / scripted manner

2020-08-13 Thread Stephen Frost
Greetings, * Magnus Hagander (mag...@hagander.net) wrote: > There's another option here as well, that is a bit "softer", is to use a > pre-commit hook. Yeah, +1 on a pre-commit idea to help address this. I certainly agree with Andres that it's quite annoying to deal with commits coming in that

Re: Switch to multi-inserts for pg_depend

2020-08-13 Thread Alvaro Herrera
On 2020-Aug-13, Michael Paquier wrote: > On Thu, Aug 13, 2020 at 05:35:14AM -0400, Alvaro Herrera wrote: > > Next to the API definition I guess, is that dependency.h? > > We need something more central, see also MAX_PGATTRIBUTE_INSERT_BYTES > and MAX_PGSHDEPEND_INSERT_BYTES. And the definition

jsonb, collection & postgres_fdw

2020-08-13 Thread Konstantin Knizhnik
Hi hackers, Right now jsonb functions are treated as non-shippable by postgres_fdw and so predicates with them are not pushed down to foreign server: create table jt(content jsonb); create extension postgres_fdw; create server pg_fdw  FOREIGN DATA WRAPPER postgres_fdw options(host

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-13 Thread Amit Kapila
On Thu, Aug 13, 2020 at 12:08 PM Amit Kapila wrote: > > On Fri, Aug 7, 2020 at 2:04 PM Dilip Kumar wrote: > > > > On Thu, Aug 6, 2020 at 2:43 PM Amit Kapila wrote: > > > > .. > > > This patch's functionality can be independently verified by SQL APIs > > > > Your changes look fine to me. > > > >

Re: security_context_t marked as deprecated in libselinux 3.1

2020-08-13 Thread Joe Conway
On 8/13/20 1:22 AM, Michael Paquier wrote: > On Wed, Aug 12, 2020 at 10:50:21PM -0400, Tom Lane wrote: >> Ummm ... aren't you going to get some cast-away-const warnings now? >> Or are all of the called functions declared as taking "const char *" >> not just "char *"? > > Let me see.. The

Re: [BUG] Error in BRIN summarization

2020-08-13 Thread Anastasia Lubennikova
On 12.08.2020 22:58, Alvaro Herrera wrote: On 2020-Aug-12, Alvaro Herrera wrote: 'anyvisible' mode is not required AFAICS; reading the code, I think this could also hit REINDEX CONCURRENTLY and CREATE INDEX CONCURRENTLY, which do not use that flag. I didn't try to reproduce it there, though.

Re: Switch to multi-inserts for pg_depend

2020-08-13 Thread Michael Paquier
On Thu, Aug 13, 2020 at 05:35:14AM -0400, Alvaro Herrera wrote: > Next to the API definition I guess, is that dependency.h? We need something more central, see also MAX_PGATTRIBUTE_INSERT_BYTES and MAX_PGSHDEPEND_INSERT_BYTES. And the definition should be named something like

Missing unknown status in PGresult

2020-08-13 Thread Hao Wu
Hi hackers, PGRES_FATAL_ERROR is the result status in most cases in the client-side when the backend process raises an error. When the query is failed to execute, PGRES_FATAL_ERROR is returned. But in another case, PGRES_FATAL_ERROR is also returned. The situation is that the network is broken

Re: Switch to multi-inserts for pg_depend

2020-08-13 Thread Alvaro Herrera
On 2020-Aug-13, Michael Paquier wrote: > Okay. Would src/include/catalog/catalog.h be a suited location for > this flag or somebody has a better idea? Next to the API definition I guess, is that dependency.h? -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-13 Thread Ashutosh Sharma
Hi Asim, Thanks for having a look into the patch and for sharing your feedback. Please find my comments inline below: On Thu, Aug 13, 2020 at 12:36 PM Asim Praveen wrote: > > Hi Ashutosh > > I stumbled upon this thread today, went through your patch and it looks good. > A minor suggestion in

Re: run pgindent on a regular basis / scripted manner

2020-08-13 Thread Magnus Hagander
On Thu, Aug 13, 2020 at 6:08 AM Tom Lane wrote: > Noah Misch writes: > > On Wed, Aug 12, 2020 at 07:47:01PM -0400, Tom Lane wrote: > >> If the workflow is commit first and re-indent later, then we can always > >> revert the pgindent commit and clean things up manually; but an auto > >>

Re: [PG13] Planning (time + buffers) data structure in explain plan (format text)

2020-08-13 Thread Julien Rouhaud
On Fri, Aug 7, 2020 at 3:51 PM Pierre Giraud wrote: > > Le 07/08/2020 à 14:52, Julien Rouhaud a écrit : > > Hi, > > > > On Fri, Aug 7, 2020 at 2:30 PM Pierre Giraud > > wrote: > >> > >> Hi all, > >> > >> As far as I understand, in the upcoming version 13, information about > >> buffers used

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-13 Thread Asim Praveen
Hi Ashutosh I stumbled upon this thread today, went through your patch and it looks good. A minor suggestion in sanity_check_relation(): if (rel->rd_rel->relam != HEAP_TABLE_AM_OID) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),

Re: WIP: WAL prefetch (another approach)

2020-08-13 Thread Thomas Munro
On Thu, Aug 6, 2020 at 10:47 PM Tomas Vondra wrote: > On Thu, Aug 06, 2020 at 02:58:44PM +1200, Thomas Munro wrote: > >On Tue, Aug 4, 2020 at 3:47 AM Tomas Vondra > >> Any luck trying to reproduce thigs? Should I try again and collect some > >> additional debug info? > > > >No luck. I'm working