Re: Creating a function for exposing memory usage of backend process

2020-06-25 Thread Kasahara Tatsuhito
Hi ! On Thu, Jun 18, 2020 at 12:56 PM Fujii Masao wrote: > Agreed. The feature to view how local memory contexts are used in > each process is very useful! +1 > >=# SELECT * FROM pg_stat_get_backend_memory_context(PID); > > I'm afraid that this interface is not convenient when we want to

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-06-25 Thread Amit Kapila
On Fri, Jun 26, 2020 at 10:39 AM Dilip Kumar wrote: > > On Thu, Jun 25, 2020 at 7:10 PM Dilip Kumar wrote: > > > > On Wed, Jun 24, 2020 at 4:04 PM Amit Kapila wrote: > > > > > > Comments on other patches: > > > = > > > 5. > > > > 3. On concurrent abort we are truncating

Re: EXPLAIN: Non-parallel ancestor plan nodes exclude parallel worker instrumentation

2020-06-25 Thread Maciek Sakrejda
On Wed, Jun 24, 2020 at 2:44 AM Amit Kapila wrote: > So, that leads to loops as 2 on "Parallel Seq Scan" and "Nested Loop" nodes. > Does this make sense now? Yes, I think we're on the same page. Thanks for the additional details. It turns out that the plan I sent at the top of the thread is

Re: Transactions involving multiple postgres foreign servers, take 2

2020-06-25 Thread Masahiko Sawada
On Tue, 23 Jun 2020 at 13:26, Amit Kapila wrote: > > On Tue, Jun 23, 2020 at 9:03 AM Masahiko Sawada > wrote: > > > > > > I've attached the latest version patches. I've incorporated the review > > comments I got so far and improved locking strategy. > > > > Thanks for updating the patch. > > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-06-25 Thread Dilip Kumar
On Thu, Jun 25, 2020 at 7:10 PM Dilip Kumar wrote: > > On Wed, Jun 24, 2020 at 4:04 PM Amit Kapila wrote: > > > > On Tue, Jun 23, 2020 at 7:00 PM Dilip Kumar wrote: > > > > > > Here is the POC patch to discuss the idea of a cleanup of shared > > > fileset on proc exit. As discussed offlist,

Re: min_safe_lsn column in pg_replication_slots view

2020-06-25 Thread Amit Kapila
On Fri, Jun 26, 2020 at 4:54 AM Alvaro Herrera wrote: > > On 2020-Jun-26, Michael Paquier wrote: > > > On Thu, Jun 25, 2020 at 11:24:27AM -0400, Alvaro Herrera wrote: > > > I don't understand the proposal. Michael posted a patch that adds > > > pg_wal_oldest_lsn(), and you say we should apply

Re: [Patch] ALTER SYSTEM READ ONLY

2020-06-25 Thread Amul Sul
On Wed, Jun 24, 2020 at 1:54 PM tushar wrote: > > On 6/22/20 11:59 AM, Amul Sul wrote: > > 2. Now skipping the startup checkpoint if the system is read-only mode, as > > discussed [2]. > > I am not able to perform pg_checksums o/p after shutting down my server > in read only mode . > > Steps - >

Re: Removal of currtid()/currtid2() and some table AM cleanup

2020-06-25 Thread Michael Paquier
On Thu, Jun 25, 2020 at 10:14:00PM +0900, Inoue, Hiroshi wrote: > I seem to have invalidated KEYSET-DRIVEN cursors used in positioned-update > test. It was introduced by the commit 4a272fd but was invalidated by the > commit 2be35a6. > > I don't object to the removal of currtid(2) because

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Bruce Momjian
On Fri, Jun 26, 2020 at 01:53:57AM +0200, Tomas Vondra wrote: > I'm not saying it's not beneficial to use different limits for different > nodes. Some nodes are less sensitive to the size (e.g. sorting often > gets faster with smaller work_mem). But I think we should instead have a > per-session

Re: Backpatch b61d161c14 (Introduce vacuum errcontext ...)

2020-06-25 Thread Amit Kapila
On Fri, Jun 26, 2020 at 7:25 AM Justin Pryzby wrote: > > > > I have done some testing with both the patches and would like to do > > more unless there are objections with these. > > Comments: > > > * The index name is saved only during this phase and restored > > immediately > > => I

Re: Backpatch b61d161c14 (Introduce vacuum errcontext ...)

2020-06-25 Thread Justin Pryzby
On Thu, Jun 25, 2020 at 02:31:51PM +0530, Amit Kapila wrote: > I have looked at both the patches (using separate variables (by > Justin) and using a struct (by Andres)) and found the second one bit > better. Thanks for looking. > I have improved some comments in the code and for now, kept as two

Re: Keep elog(ERROR) and ereport(ERROR) calls in the cold path

2020-06-25 Thread David Rowley
On Fri, 26 Jun 2020 at 04:35, Andres Freund wrote: > On 2020-06-25 13:50:37 +1200, David Rowley wrote: > > In the attached, I've come up with a way that works. Basically I've > > just added a function named errstart_cold() that is attributed with > > __attribute__((cold)), which will hint to the

Re: Failures with installcheck and low work_mem value in 13~

2020-06-25 Thread Michael Paquier
On Thu, Jun 25, 2020 at 12:15:54PM -0700, Peter Geoghegan wrote: > The RMT met today. We determined that it wasn't worth adjusting this > test to pass with non-standard work_mem values. Okay, thanks for the feedback. We'll see how it works out. -- Michael signature.asc Description: PGP

Re: xid wraparound danger due to INDEX_CLEANUP false

2020-06-25 Thread Peter Geoghegan
On Thu, Jun 25, 2020 at 8:28 AM Robert Haas wrote: > I'm not really convinced. I agree that from a theoretical point of > view an index can have arbitrary needs and is the arbiter of its own > needs, but when I pull the emergency break, I want the vehicle to > stop, not think about stopping.

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2020-06-25 Thread Tomas Vondra
On Thu, Jun 25, 2020 at 03:09:44PM -0700, Melanie Plageman wrote: On Tue, Jun 23, 2020 at 3:24 PM Tomas Vondra wrote: I started looking at the patch to refresh my knowledge both of this patch and parallel hash join, but I think it needs a rebase. The changes in 7897e3bb90 apparently touched

Re: xid wraparound danger due to INDEX_CLEANUP false

2020-06-25 Thread Peter Geoghegan
On Thu, Jun 25, 2020 at 6:59 AM Masahiko Sawada wrote: > I think that with the approach implemented in my patch, it could be a > problem for the user that the user cannot easily know in advance > whether vacuum with INDEX_CLEANUP false will perform index cleanup, > even if page deletion doesn’t

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Tomas Vondra
On Thu, Jun 25, 2020 at 01:17:56PM -0400, Bruce Momjian wrote: On Thu, Jun 25, 2020 at 11:46:54AM -0400, Robert Haas wrote: On Wed, Jun 24, 2020 at 7:38 PM Bruce Momjian wrote: > I think my main point is that work_mem was not being honored for > hash-agg before, but now that PG 13 can do it,

Re: Enabling B-Tree deduplication by default

2020-06-25 Thread Peter Geoghegan
On Thu, Jan 30, 2020 at 11:40 AM Peter Geoghegan wrote: > I think that I should commit the patch without the GUC tentatively. > Just have the storage parameter, so that everyone gets the > optimization without asking for it. We can then review the decision to > enable deduplication generally

Re: min_safe_lsn column in pg_replication_slots view

2020-06-25 Thread Alvaro Herrera
On 2020-Jun-26, Michael Paquier wrote: > On Thu, Jun 25, 2020 at 11:24:27AM -0400, Alvaro Herrera wrote: > > I don't understand the proposal. Michael posted a patch that adds > > pg_wal_oldest_lsn(), and you say we should apply the patch except the > > part that adds that function -- so what

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Tomas Vondra
On Thu, Jun 25, 2020 at 09:42:33AM -0700, Jeff Davis wrote: On Wed, 2020-06-24 at 12:31 -0700, Andres Freund wrote: nodeAgg.c already treats those separately: void hash_agg_set_limits(double hashentrysize, uint64 input_groups, int used_bits, Size

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Tomas Vondra
On Thu, Jun 25, 2020 at 11:16:23AM -0700, Andres Freund wrote: Hi, On 2020-06-25 10:44:42 -0700, Jeff Davis wrote: There are only two possible paths: HashAgg and Sort+Group, and we need to pick one. If the planner expects one to spill, it is likely to expect the other to spill. If one spills

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Tomas Vondra
On Thu, Jun 25, 2020 at 02:28:02PM -0700, Jeff Davis wrote: On Thu, 2020-06-25 at 15:56 -0400, Bruce Momjian wrote: It is my understanding that spill of sorts is mostly read sequentially, while hash reads are random. Is that right? Is that not being costed properly? I don't think there's a

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Alvaro Herrera
On 2020-Jun-25, Andres Freund wrote: > >My point here is that maybe we don't need to offer a GUC to explicitly > >turn spilling off; it seems sufficient to let users change work_mem so > >that spilling will naturally not occur. Why do we need more? > > That's not really a useful escape hatch,

Re: should libpq also require TLSv1.2 by default?

2020-06-25 Thread Bruce Momjian
On Thu, Jun 25, 2020 at 06:44:05PM -0400, Tom Lane wrote: > BTW, the server-side report of the problem looks like > > LOG: could not accept SSL connection: wrong version number I like that one. ;-) -- Bruce Momjian https://momjian.us EnterpriseDB

Re: min_safe_lsn column in pg_replication_slots view

2020-06-25 Thread Michael Paquier
On Thu, Jun 25, 2020 at 11:24:27AM -0400, Alvaro Herrera wrote: > I don't understand the proposal. Michael posted a patch that adds > pg_wal_oldest_lsn(), and you say we should apply the patch except the > part that adds that function -- so what part would be applying? I have sent last week a

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Andres Freund
Hi, On June 25, 2020 3:44:22 PM PDT, Alvaro Herrera wrote: >On 2020-Jun-25, Andres Freund wrote: > >> > What are people doing for those cases already? Do we have an >> > real-world queries that are a problem in PG 13 for this? >> >> I don't know about real world, but it's pretty easy to come

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Alvaro Herrera
On 2020-Jun-25, Andres Freund wrote: > > What are people doing for those cases already? Do we have an > > real-world queries that are a problem in PG 13 for this? > > I don't know about real world, but it's pretty easy to come up with > examples. > > query: > SELECT a, array_agg(b) FROM

Re: should libpq also require TLSv1.2 by default?

2020-06-25 Thread Tom Lane
Daniel Gustafsson writes: >> On 24 Jun 2020, at 10:46, Magnus Hagander wrote: >> It might also be worth noting that it's not really "any protocol version", >> it means it will be "whatever the openssl configuration says", I think? For >> example, debian buster sets: >> >>

Re: create database with template doesn't copy database ACL

2020-06-25 Thread Bruce Momjian
On Tue, Jun 16, 2020 at 06:10:54AM -0400, Bruce Momjian wrote: > On Mon, Jun 15, 2020 at 10:10:32AM -0400, Bruce Momjian wrote: > > On Mon, Jun 15, 2020 at 12:14:55AM -0400, Tom Lane wrote: > > > Bruce Momjian writes: > > > > Well, I thought we copied everything except things tha can be specified

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Jeff Davis
On Thu, 2020-06-25 at 15:56 -0400, Bruce Momjian wrote: > It is my understanding that spill of sorts is mostly read > sequentially, > while hash reads are random. Is that right? Is that not being > costed > properly? I don't think there's a major problem with the cost model, but it could

Re: CUBE_MAX_DIM

2020-06-25 Thread Alastair McKinley
> From: Tom Lane > Sent: 25 June 2020 17:43 > > Alastair McKinley writes: > > I know that Cube in it's current form isn't suitable for nearest-neighbour > > searching these vectors in their raw form (I have tried recompilation with > > higher CUBE_MAX_DIM myself), but conceptually kNN GiST

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Andres Freund
Hi, On 2020-06-25 14:25:12 -0400, Bruce Momjian wrote: > I am still trying to get my head around why the spill is going to be so > much work to adjust for hash agg than our other spillable nodes. Aggregates are the classical case used to process large amounts of data. For larger amounts of data

Re: should libpq also require TLSv1.2 by default?

2020-06-25 Thread Tom Lane
Robert Haas writes: > I wonder how much of a problem this really is. Yeah. I find Robert's points about that pretty persuasive: by now needing to connect to a server without TLSv1.2 support, *and* needing to do so with SSL on, ought to be a tiny niche use case (much smaller than the number of

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Bruce Momjian
On Thu, Jun 25, 2020 at 03:24:42PM -0400, Bruce Momjian wrote: > On Thu, Jun 25, 2020 at 11:10:58AM -0700, Jeff Davis wrote: > > On Wed, 2020-06-24 at 13:29 -0400, Tom Lane wrote: > > > If we feel we need something to let people have the v12 behavior > > > back, let's have > > > (1) enable_hashagg

Re: Open Item: Should non-text EXPLAIN always show properties?

2020-06-25 Thread Tom Lane
James Coleman writes: > On Thu, Jun 25, 2020 at 12:33 PM Tom Lane wrote: >> I think the right way to think about this is that we are building >> an output data structure according to a schema that should be fixed >> for any particular plan shape. If event X happened zero times in >> a given

Re: should libpq also require TLSv1.2 by default?

2020-06-25 Thread Robert Haas
On Wed, Jun 24, 2020 at 9:50 PM Michael Paquier wrote: > Yeah, and I would not be surprised to see cases where people complain > to us about that when attempting to reach one of their old boxes, > breaking some stuff they have been relying on for years by forcing the > addition of a

Re: Open Item: Should non-text EXPLAIN always show properties?

2020-06-25 Thread James Coleman
On Thu, Jun 25, 2020 at 12:33 PM Tom Lane wrote: > > Robert Haas writes: > > On Thu, Jun 25, 2020 at 8:42 AM James Coleman wrote: > >> Yesterday I'd replied [1] to Justin's proposal for this WRT > >> incremental sort and expressed my opinion that including both > >> unnecessarily (i.e.,

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Bruce Momjian
On Thu, Jun 25, 2020 at 11:10:58AM -0700, Jeff Davis wrote: > On Wed, 2020-06-24 at 13:29 -0400, Tom Lane wrote: > > If we feel we need something to let people have the v12 behavior > > back, let's have > > (1) enable_hashagg on/off --- controls planner, same as it ever was > > (2)

Re: Failures with installcheck and low work_mem value in 13~

2020-06-25 Thread Peter Geoghegan
On Fri, Jun 19, 2020 at 7:48 PM Michael Paquier wrote: > We cared about such plan stability that in the past FWIW, see for > example c588df9 as work_mem is a setting that people like to change. > Why should this be different? work_mem is a popular configuration > setting. The RMT met today. We

Re: Why forbid "INSERT INTO t () VALUES ();"

2020-06-25 Thread Robert Haas
On Thu, Jun 25, 2020 at 12:07 PM Tom Lane wrote: > Yeah, the multi-insert case is a plausible reason that hadn't been > mentioned before. On the other hand, you can already do that pretty > painlessly: Sure, but it means if you're writing code to generate queries programmatically, then you have

Re: Weird failures on lorikeet

2020-06-25 Thread Andrew Dunstan
On 6/25/20 12:52 PM, Alvaro Herrera wrote: > Is there something going on with lorikeet again? I see this: > > 2020-06-25 01:55:13.380 EDT [5ef43c40.21e0:85] pg_regress/typed_table LOG: > statement: SELECT c.oid::pg_catalog.regclass > FROM pg_catalog.pg_class c, pg_catalog.pg_inherits i

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Bruce Momjian
On Thu, Jun 25, 2020 at 11:02:30AM -0700, Jeff Davis wrote: > If we have the GUCs there, then at least if someone comes to the > mailing list with a problem, we can offer them a temporary solution, > and have time to try to avoid the problem in a future release (tweaking > estimates, cost model,

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Andres Freund
Hi, On 2020-06-25 10:44:42 -0700, Jeff Davis wrote: > There are only two possible paths: HashAgg and Sort+Group, and we need > to pick one. If the planner expects one to spill, it is likely to > expect the other to spill. If one spills in the executor, then the > other is likely to spill, too.

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Jeff Davis
On Wed, 2020-06-24 at 13:29 -0400, Tom Lane wrote: > If we feel we need something to let people have the v12 behavior > back, let's have > (1) enable_hashagg on/off --- controls planner, same as it ever was > (2) enable_hashagg_spill on/off --- controls executor by disabling > spill > > But I'm

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Jeff Davis
On Thu, 2020-06-25 at 13:17 -0400, Bruce Momjian wrote: > Frankly, if it took me this long to get my head around this, I am > unclear how many people will understand this tuning feature enough to > actually use it. The way I think about it is that v13 HashAgg is much more consistent with the way

Remove a redundant condition check

2020-06-25 Thread Ádám Balogh
Hello, A one line change to remove a duplicate check. This duplicate check was detected during testing my contribution to a static code analysis tool. There is no functional change, no new tests needed. Regards, Ádám Balogh CodeChecker Team Ericsson Hungary

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Robert Haas
On Thu, Jun 25, 2020 at 1:15 PM Jeff Davis wrote: > Unexpected things (meaning underestimates) are not independent. All the > queries are based on the same stats, so if you have a lot of similar > queries, they will all get the same underestimate at once, and all be > surprised when they need to

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Jeff Davis
On Thu, 2020-06-25 at 09:37 -0700, Andres Freund wrote: > > Let's say you have work_mem=32MB and a query that's expected to use > > 16MB of memory. In reality, it uses 64MB of memory. So you are > > saying > > this query would get to use all 64MB of memory, right? > > > > But then you run

Re: [patch] demote

2020-06-25 Thread Jehan-Guillaume de Rorthais
Hi, Here is a summary of my work during the last few days on this demote approach. Please, find in attachment v2-0001-Demote-PoC.patch and the comments in the commit message and as FIXME in code. The patch is not finished or bug-free yet, I'm still not very happy with the coding style, it

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Bruce Momjian
On Thu, Jun 25, 2020 at 11:46:54AM -0400, Robert Haas wrote: > On Wed, Jun 24, 2020 at 7:38 PM Bruce Momjian wrote: > > I think my main point is that work_mem was not being honored for > > hash-agg before, but now that PG 13 can do it, we are again allowing > > work_mem not to apply in certain

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Jeff Davis
On Thu, 2020-06-25 at 11:46 -0400, Robert Haas wrote: > Because the first problem is so bad, most people > set the value relatively conservatively and just live with the > performance consequences. But this also means that they have memory > left over most of the time, so the idea of letting a

Re: Weird failures on lorikeet

2020-06-25 Thread Alvaro Herrera
Is there something going on with lorikeet again? I see this: 2020-06-25 01:55:13.380 EDT [5ef43c40.21e0:85] pg_regress/typed_table LOG: statement: SELECT c.oid::pg_catalog.regclass FROM pg_catalog.pg_class c, pg_catalog.pg_inherits i WHERE c.oid = i.inhparent AND i.inhrelid =

Re: CUBE_MAX_DIM

2020-06-25 Thread Tom Lane
Alastair McKinley writes: > I know that Cube in it's current form isn't suitable for nearest-neighbour > searching these vectors in their raw form (I have tried recompilation with > higher CUBE_MAX_DIM myself), but conceptually kNN GiST searches using Cubes > can be useful for these

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Jeff Davis
On Wed, 2020-06-24 at 12:31 -0700, Andres Freund wrote: > nodeAgg.c already treats those separately: > > void > hash_agg_set_limits(double hashentrysize, uint64 input_groups, int > used_bits, > Size *mem_limit, uint64 > *ngroups_limit, >

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Andres Freund
On 2020-06-25 09:24:52 -0700, Jeff Davis wrote: > On Wed, 2020-06-24 at 12:14 -0700, Andres Freund wrote: > > E.g. if the plan isn't expected to spill, > > only spill at 10 x work_mem or something like that. > > Let's say you have work_mem=32MB and a query that's expected to use > 16MB of memory.

Re: Keep elog(ERROR) and ereport(ERROR) calls in the cold path

2020-06-25 Thread Andres Freund
Hi, Thanks for picking this up again! On 2020-06-25 13:50:37 +1200, David Rowley wrote: > In the attached, I've come up with a way that works. Basically I've > just added a function named errstart_cold() that is attributed with > __attribute__((cold)), which will hint to the compiler to keep >

Re: Why forbid "INSERT INTO t () VALUES ();"

2020-06-25 Thread Isaac Morland
On Wed, 24 Jun 2020 at 08:18, Fabien COELHO wrote: > I would like to create an "all defaults" row, i.e. a row composed of the > default values for all attributes, so I wrote: > >INSERT INTO t() VALUES (); > > This is forbidden by postgres, and also sqlite. > This is not the only area where

Re: Open Item: Should non-text EXPLAIN always show properties?

2020-06-25 Thread Tom Lane
Robert Haas writes: > On Thu, Jun 25, 2020 at 8:42 AM James Coleman wrote: >> Yesterday I'd replied [1] to Justin's proposal for this WRT >> incremental sort and expressed my opinion that including both >> unnecessarily (i.e., including disk when an in-memory sort was used) >> is undesirable and

Re: CUBE_MAX_DIM

2020-06-25 Thread Alastair McKinley
> > Devrim =?ISO-8859-1?Q?G=FCnd=FCz?= writes: > > Someone contacted me about increasing CUBE_MAX_DIM > > in contrib/cube/cubedata.h (in the community RPMs). The current value > > is 100 with the following comment: > > > * This limit is pretty arbitrary, but don't make it so large that you > >

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Jeff Davis
On Wed, 2020-06-24 at 12:14 -0700, Andres Freund wrote: > E.g. if the plan isn't expected to spill, > only spill at 10 x work_mem or something like that. Let's say you have work_mem=32MB and a query that's expected to use 16MB of memory. In reality, it uses 64MB of memory. So you are saying this

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Jeff Davis
On Wed, 2020-06-24 at 15:28 -0400, Robert Haas wrote: > On Wed, Jun 24, 2020 at 3:14 PM Andres Freund > wrote: > > FWIW, my gut feeling is that we'll end up have to separate the > > "execution time" spilling from using plain work mem, because it'll > > trigger spilling too often. E.g. if the plan

Re: Why forbid "INSERT INTO t () VALUES ();"

2020-06-25 Thread Tom Lane
Robert Haas writes: > On Thu, Jun 25, 2020 at 12:56 AM Fabien COELHO wrote: >> It also means that if for some reason someone wants to insert several such >> all-default rows, they have to repeat the insert, as "VALUES (), ();" >> would not work, so it is also losing a corner-corner case

Re: Keep elog(ERROR) and ereport(ERROR) calls in the cold path

2020-06-25 Thread Robert Haas
On Wed, Jun 24, 2020 at 9:51 PM David Rowley wrote: > $ pgbench -n -f pg-tpch/queries/q01.sql -T 120 tpch > > Which is about a 1.42% increase in performance. That's not exactly > groundbreaking, but pretty useful to have if that happens to apply > across the board for execution performance. > >

Re: Default setting for enable_hashagg_disk

2020-06-25 Thread Robert Haas
On Wed, Jun 24, 2020 at 7:38 PM Bruce Momjian wrote: > I think my main point is that work_mem was not being honored for > hash-agg before, but now that PG 13 can do it, we are again allowing > work_mem not to apply in certain cases. I am wondering if our hard > limit for work_mem is the issue,

Re: PostgreSQL and big data - FDW

2020-06-25 Thread Bruce Momjian
On Thu, Jun 25, 2020 at 07:02:37AM +, ROS Didier wrote: > Hi Stephen > > My EDF company is very interested in this feature (KERBEROS authentication > method and hdfs_fdw ). > Is it possible to know how many days of development does this represent ? who > can develop this implementation ?

Re: Open Item: Should non-text EXPLAIN always show properties?

2020-06-25 Thread Robert Haas
On Thu, Jun 25, 2020 at 8:42 AM James Coleman wrote: > Yesterday I'd replied [1] to Justin's proposal for this WRT > incremental sort and expressed my opinion that including both > unnecessarily (i.e., including disk when an in-memory sort was used) > is undesirable and confusing and leads to

Re: Parallel Seq Scan vs kernel read ahead

2020-06-25 Thread Robert Haas
On Tue, Jun 23, 2020 at 11:53 PM David Rowley wrote: > In summary, based on these tests, I don't think we're making anything > worse in regards to synchronize_seqscans if we cap the maximum number > of blocks to allocate to each worker at once to 8192. Perhaps there's > some argument for using

Re: xid wraparound danger due to INDEX_CLEANUP false

2020-06-25 Thread Robert Haas
On Wed, Jun 24, 2020 at 4:02 PM Peter Geoghegan wrote: > > Apparently, what we're really doing here is that even when > > INDEX_CLEANUP is OFF, we're still going to keep all the dead tuples. > > That seems sad, but if it's what we have to do then it at least needs > > comments explaining it. > >

Re: min_safe_lsn column in pg_replication_slots view

2020-06-25 Thread Alvaro Herrera
On 2020-Jun-25, Amit Kapila wrote: > +1. I also think let's drop the idea of exposing a function for this > value and revert the min_safe_lsn part of the work as proposed by > Michael above [1] excluding the function pg_wal_oldest_lsn() in that > patch. Then, we can expose this as a new stat

Re: CUBE_MAX_DIM

2020-06-25 Thread Tom Lane
Devrim =?ISO-8859-1?Q?G=FCnd=FCz?= writes: > Someone contacted me about increasing CUBE_MAX_DIM > in contrib/cube/cubedata.h (in the community RPMs). The current value > is 100 with the following comment: > * This limit is pretty arbitrary, but don't make it so large that you > * risk overflow

Re: improving basebackup.c's file-reading code

2020-06-25 Thread Robert Haas
On Thu, Jun 25, 2020 at 10:29 AM Daniel Gustafsson wrote: > As this went in, can we close the 2020-07 CF entry or is there anything left > in > the patchseries? Done. Thanks for the reminder. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Why forbid "INSERT INTO t () VALUES ();"

2020-06-25 Thread Robert Haas
On Thu, Jun 25, 2020 at 12:56 AM Fabien COELHO wrote: > It also means that if for some reason someone wants to insert several such > all-default rows, they have to repeat the insert, as "VALUES (), ();" > would not work, so it is also losing a corner-corner case capability > without obvious

Re: improving basebackup.c's file-reading code

2020-06-25 Thread Daniel Gustafsson
> On 17 Jun 2020, at 17:45, Robert Haas wrote: > > On Wed, Apr 29, 2020 at 5:52 AM Hamid Akhtar wrote: >> The idea and the patch looks good to me. >> >> It makes sense to change fread to basebackup_read_file which internally >> calls pg_pread which is a portable function as opposed to read.

Re: More tzdb fun: POSIXRULES is being deprecated upstream

2020-06-25 Thread Tom Lane
Peter Eisentraut writes: > What you are saying is, instead of the OS dropping POSIXRULES support, > it would be better if we dropped it first and release-noted that. > However, I don't agree with the premise of that. OSes with long-term > support aren't going to drop it. You might be right,

Re: xid wraparound danger due to INDEX_CLEANUP false

2020-06-25 Thread Masahiko Sawada
On Thu, 25 Jun 2020 at 05:02, Peter Geoghegan wrote: > > On Wed, Jun 24, 2020 at 10:21 AM Robert Haas wrote: > > Sorry, I'm so far behind on my email. Argh. > > That's okay. > > > I think, especially on the blog post you linked, that we should aim to > > have INDEX_CLEANUP OFF mode do the

Re: Why forbid "INSERT INTO t () VALUES ();"

2020-06-25 Thread Fabien COELHO
Bonjour Vik, It's forbidden because the SQL standard forbids it. Ok, that is definitely a reason. I'm not sure it is a good reason, though. It's a very good reason. It might not be good *enough*, but it is a good reason. Ok for good, although paradoxically not "good enough":-) Why

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-06-25 Thread Dilip Kumar
On Wed, Jun 24, 2020 at 4:04 PM Amit Kapila wrote: > > On Tue, Jun 23, 2020 at 7:00 PM Dilip Kumar wrote: > > > > Here is the POC patch to discuss the idea of a cleanup of shared > > fileset on proc exit. As discussed offlist, here I am maintaining > > the list of shared fileset. First time

Re: CUBE_MAX_DIM

2020-06-25 Thread Komяpa
Hello, The problem with higher dimension cubes is that starting with dimensionality of ~52 the "distance" metrics in 64-bit float have less than a single bit per dimension in mantissa, making cubes indistinguishable. Developers for facial recognition software had a chat about that on russian

Re: Removal of currtid()/currtid2() and some table AM cleanup

2020-06-25 Thread Inoue, Hiroshi
Hi, I seem to have invalidated KEYSET-DRIVEN cursors used in positioned-update test . It was introduced by the commit 4a272fd but was invalidated by the commit 2be35a6. I don't object to the removal of currtid(2) because keyset-driven cursors in psqlodbc are changed into static cursors in

Re: More tzdb fun: POSIXRULES is being deprecated upstream

2020-06-25 Thread Peter Eisentraut
On 2020-06-19 21:55, Tom Lane wrote: Yeah, we can do nothing in the back branches and hope that that doesn't happen for the remaining lifespan of v12. But I wonder whether that doesn't amount to sticking our heads in the sand. I suppose it'd be possible to have a release-note entry in the back

Re: Open Item: Should non-text EXPLAIN always show properties?

2020-06-25 Thread James Coleman
On Thu, Jun 25, 2020 at 5:15 AM David Rowley wrote: > > Over on [1] Justin mentions that the non-text EXPLAIN ANALYZE should > always show the "Disk Usage" and "HashAgg Batches" properties. I > agree with this. show_wal_usage() is a good example of how we normally > do things. We try to keep

Re: SIGSEGV from START_REPLICATION 0/XXXXXXX in XLogSendPhysical () at walsender.c:2762

2020-06-25 Thread Robert Haas
On Wed, Jun 24, 2020 at 3:41 PM Alvaro Herrera wrote: > People (specifically the jdbc driver) *are* using this feature in this > way, but they didn't realize they were doing it. It was an accident and > they didn't notice. But you don't know that that's true of everyone using this feature, and

Re: Resetting spilled txn statistics in pg_stat_replication

2020-06-25 Thread Amit Kapila
On Tue, Jun 23, 2020 at 6:39 PM Amit Kapila wrote: > > On Tue, Jun 23, 2020 at 3:48 PM Tomas Vondra > wrote: > > > > On Tue, Jun 23, 2020 at 10:58:18AM +0530, Amit Kapila wrote: > > >On Tue, Jun 23, 2020 at 9:32 AM Masahiko Sawada > > > wrote: > > >> > > >> On Sun, 21 Jun 2020 at 06:57, Tomas

Re: min_safe_lsn column in pg_replication_slots view

2020-06-25 Thread Amit Kapila
On Wed, Jun 24, 2020 at 8:45 PM Alvaro Herrera wrote: > > On 2020-Jun-24, Fujii Masao wrote: > > > On 2020/06/24 8:39, Alvaro Herrera wrote: > > > > I think we should publish the value from wal_keep_segments separately > > > from max_slot_wal_keep_size. ISTM that the user might decide to change

CUBE_MAX_DIM

2020-06-25 Thread Devrim Gündüz
Hi, Someone contacted me about increasing CUBE_MAX_DIM in contrib/cube/cubedata.h (in the community RPMs). The current value is 100 with the following comment: * This limit is pretty arbitrary, but don't make it so large that you * risk overflow in sizing calculations. They said they use 500,

Re: Online checksums patch - once again

2020-06-25 Thread Daniel Gustafsson
> On 22 Jun 2020, at 18:29, Robert Haas wrote: > > On Mon, Jun 22, 2020 at 8:27 AM Daniel Gustafsson wrote: >> Restartability is implemented by keeping state in pg_class. I opted for a >> bool >> which is cleared as the first step of checksum enable, since it offers fewer >> synchronization

Re: Why forbid "INSERT INTO t () VALUES ();"

2020-06-25 Thread David Rowley
On Thu, 25 Jun 2020 at 16:56, Fabien COELHO wrote: > It also means that if for some reason someone wants to insert several such > all-default rows, they have to repeat the insert, as "VALUES (), ();" > would not work, so it is also losing a corner-corner case capability > without obvious reason.

Open Item: Should non-text EXPLAIN always show properties?

2020-06-25 Thread David Rowley
Over on [1] Justin mentions that the non-text EXPLAIN ANALYZE should always show the "Disk Usage" and "HashAgg Batches" properties. I agree with this. show_wal_usage() is a good example of how we normally do things. We try to keep the text format as humanly readable as possible but don't really

Re: Missing some ifndef FRONTEND at the top of logging.c and file_utils.c

2020-06-25 Thread Daniel Gustafsson
> On 25 Jun 2020, at 10:07, Michael Paquier wrote: > So, shouldn't that stuff be added as per the attached? That makes sense, logging.c and file_utils.c are indeed only part of libpgcommon.a and should only be compiled for frontend. cheers ./daniel

Re: hashagg slowdown due to spill changes

2020-06-25 Thread Tomas Vondra
On Wed, Jun 24, 2020 at 05:26:07PM -0700, Melanie Plageman wrote: On Tue, Jun 23, 2020 at 10:06 AM Andres Freund wrote: Hi, On 2020-06-23 09:23:57 -0700, Melanie Plageman wrote: > On Mon, Jun 22, 2020 at 9:02 PM Andres Freund wrote: > > It's not this patch's fault, but none, really none, of

Re: Backpatch b61d161c14 (Introduce vacuum errcontext ...)

2020-06-25 Thread Amit Kapila
On Tue, Jun 23, 2020 at 11:49 PM Andres Freund wrote: > > Hi, > > On 2020-06-23 00:14:40 -0400, Tom Lane wrote: > > Andres Freund writes: > > > I am only suggesting that where you save the old location, as currently > > > done with LVRelStats olderrinfo, you instead use a more specific > > >

Re: Why forbid "INSERT INTO t () VALUES ();"

2020-06-25 Thread Vik Fearing
On 6/25/20 6:56 AM, Fabien COELHO wrote: > > Hello Tom, > >   INSERT INTO t() VALUES (); >> >>> I'm still unclear why it would be forbidden though, it seems logical to >>> try that, whereas the working one is quite away from the usual syntax. >> >> It's forbidden because the SQL standard

Re: Review for GetWALAvailability()

2020-06-25 Thread Kyotaro Horiguchi
At Thu, 25 Jun 2020 14:35:34 +0900, Fujii Masao wrote in > > > On 2020/06/25 12:57, Alvaro Herrera wrote: > > On 2020-Jun-25, Fujii Masao wrote: > > > >>/* > >> * Find the oldest extant segment file. We get 1 until checkpoint > >> removes > >> * the first WAL segment file since

Missing some ifndef FRONTEND at the top of logging.c and file_utils.c

2020-06-25 Thread Michael Paquier
Hi all, As subject tells, we have in src/common/ four files that are only compiled as part of the frontend: fe_memutils.c, file_utils.c, logging.c and restricted_token.c. Two of them are missing the following, to make sure that we never try to compile them with the backend: +#ifndef FRONTEND

Re: Ought to use heap_multi_insert() for pg_attribute/depend insertions?

2020-06-25 Thread Daniel Gustafsson
> On 4 Mar 2020, at 23:16, Daniel Gustafsson wrote: > The v9 patch attached addresses, I believe, all comments made to date. I > tried > to read through earlier reviews too to ensure I hadn't missed anything so I > hope I've covered all findings so far. Attached is a rebased version which was

RE: PostgreSQL and big data - FDW

2020-06-25 Thread ROS Didier
Hi Stephen My EDF company is very interested in this feature (KERBEROS authentication method and hdfs_fdw ). Is it possible to know how many days of development does this represent ? who can develop this implementation ? what cost ? Best Regards Didier ROS EDF -Message d'origine- De :

Re: some more pg_dump refactoring

2020-06-25 Thread Fabien COELHO
Hallo Peter, My 0.02 €: Patch applies cleanly, compiles, make check and pg_dump tap tests ok. The refactoring is a definite improvements. You changed the query strings to use "\n" instead of " ". I would not have changed that, because it departs from the style around, and I do not think

Re: archive status ".ready" files may be created too early

2020-06-25 Thread Kyotaro Horiguchi
Hello. Matsumura-san. I agree that WAL writer is not the place to notify segmnet. And the direction you suggested would work. At Fri, 19 Jun 2020 10:18:34 +, "matsumura@fujitsu.com" wrote in > 1. Description in primary side > > [Basic problem] > A process flushing WAL record