Re: make \d pg_toast.foo show its indices ; and, \d toast show its main table ; and \d relkind=I show its partitions

2019-07-23 Thread Fabien COELHO
Find attached updated patches which also work against old servers. I can't check that for sure. * About toast table addition v7: Patch applies cleanly, compiles, make check ok, no doc. This addition show the main table of a toast table, which is useful. Field relnamespace oid in pg_class

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-23 Thread Michael Paquier
On Mon, Jul 22, 2019 at 02:40:19PM +0200, Julien Rouhaud wrote: > Right, so I kept the long option. Also this comment was outdated, as > the --jobs is now just ignored with a list of indexes, so I fixed that > too. + if (!parallel) + { + if (user_list == NULL) + { + /* +

Re: pgbench - add pseudo-random permutation function

2019-07-23 Thread Fabien COELHO
Hello Thomas, Function nbits(), which was previously discussed, has been simplified by using the function pg_popcount64(). Hi Fabien, Suzuki-san, I am not smart enough to commit this I'm in no hurry:-) or judge its value for benchmarking, I think that it is valuable given that we have

Re: Race conditions with TAP test for syncrep

2019-07-23 Thread Michael Paquier
On Mon, Jul 22, 2019 at 11:45:53PM -0700, Noah Misch wrote: > PostgresNode uses "print" the same way. The patch does close the intended > race conditions, and its implementation choices look fine to me. Thanks Noah for the review. I have reviewed the thread and improved a couple of comments base

Re: [PATCH] minor bugfix for pg_basebackup (9.6 ~ )

2019-07-23 Thread Michael Paquier
On Mon, Jul 22, 2019 at 12:58:40PM -0400, Alvaro Herrera wrote: > Maybe it's just me, but it seems weird to try to forestall a problem > that cannot occur by definition. I would rather remove the escaping, > and add a one-line comment explaining why we don't do it? No objections with doing that e

Re: Berserk Autovacuum (let's save next Mandrill)

2019-07-23 Thread Masahiko Sawada
On Sun, Apr 14, 2019 at 9:59 PM Darafei "Komяpa" Praliaskouski wrote: >> >> >> >I don't think it's helpful to force emergency vacuuming more >> >frequently; >> >quite the contrary, it's likely to cause even more issues. We should >> >tweak autovacuum to perform freezing more preemtively instead.

Re: Runtime pruning problem

2019-07-23 Thread David Rowley
On Sat, 25 May 2019 at 18:55, David Rowley wrote: > and an updated patch, rebased after the pgindent run. > > Hopefully, this will make the CF bot happy again. and rebased again due to a conflict with some List changes that touched ruleutils.c. I also made another couple of passes over this addi

Re: Optimze usage of immutable functions as relation

2019-07-23 Thread Anastasia Lubennikova
08.07.2019 4:18, Thomas Munro: The July Commitfest is here. Could we please have a rebase of this patch? Updated patch is in attachments. I've only resolved one small cosmetic merge conflict. Later this week I'm going to send a more thoughtful review. -- Anastasia Lubennikova Postgres Profess

Re: [bug fix] Produce a crash dump before main() on Windows

2019-07-23 Thread Kyotaro Horiguchi
At Mon, 1 Jul 2019 00:04:47 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FC6515F@G01JPEXMBYT05> > From: Amit Kapila [mailto:amit.kapil...@gmail.com] > > Tsunakawa/Haribabu - By reading this thread briefly, it seems we need > > some more inputs from other developers on

Re: SegFault on 9.6.14

2019-07-23 Thread Amit Kapila
On Tue, Jul 23, 2019 at 9:11 AM Thomas Munro wrote: > > On Fri, Jul 19, 2019 at 3:00 PM Amit Kapila wrote: > > I am thinking that why not we remove the part of destroying the > > parallel context (and shared memory) from ExecShutdownGather (and > > ExecShutdownGatherMerge) and then do it at the t

[GSoC] Second period status report for the de-TOAST iterator

2019-07-23 Thread Binguo Bao
Hi hackers, It's a report about my GSoC project[1] during the second period. What I've done: - Submitted the patch[2] to optimize partial TOAST decompression - Submitted the patch[3] as a prototype of de-TOAST iterator and tested its performance on position() function - Perfected patches according

Re: pg_receivewal documentation

2019-07-23 Thread Jesper Pedersen
Hi, On 7/22/19 8:08 PM, Michael Paquier wrote: +to something other than Looks fine to me. Just a tiny nit. For the second reference to synchronous_commit, I would change the link to a markup. Sure. Best regards, Jesper >From f6c5e9128e0779f928d94bf9bcc8813bf03150f0 Mon Sep 17 00:0

Re: Add CREATE DATABASE LOCALE option

2019-07-23 Thread Peter Eisentraut
On 2019-07-23 00:18, Fabien COELHO wrote: > It applies cleanly, compiles, global & pg_dump make check ok, doc gen ok. > > I'm still unconvinced of the interest of breaking backward compatibility, > but this is no big deal. > > I do not like much calling strlen() to check whether a string is empt

Re: Support for jsonpath .datetime() method

2019-07-23 Thread Peter Eisentraut
I think the best way forward here is to focus first on patch 0002 and get the additional format templates in, independent of any surrounding JSON functionality. In particular, remove parse_datetime() and all the related API changes, then it becomes much simpler. The codes FF1..FF6 that you added

Re: SQL/JSON: JSON_TABLE

2019-07-23 Thread Pavel Stehule
Hi út 16. 7. 2019 v 16:06 odesílatel Nikita Glukhov napsal: > On 29.06.2019 8:40, Pavel Stehule wrote: > > Hi > > so 29. 6. 2019 v 7:26 odesílatel Nikita Glukhov > napsal: > >> Attached 36th version of patches rebased onto jsonpath v36. >> > > I cannot to apply these patches on master. Please,

Re: POC: Cleaning up orphaned files using undo logs

2019-07-23 Thread Amit Khandekar
On Tue, 23 Jul 2019 at 08:48, Amit Kapila wrote: > > On Mon, Jul 22, 2019 at 8:39 PM Amit Khandekar wrote: > > > > On Mon, 22 Jul 2019 at 14:21, Amit Kapila wrote: > > > > - > > > > +UndoWorkerGetSlotInfo(int slot, UndoRequestInfo *urinfo) > > +{ > > + /* Block concurrent access. */

Re: [PATCH] Improve performance of NOTIFY over many databases (issue blocking on AccessExclusiveLock on object 0 of class 1262 of database 0)

2019-07-23 Thread Martijn van Oosterhout
Hoi Tom, Thank you for the detailed response. Sorry the delay, I was on holiday. You are absolutely correct when you point out that the queue pointers were accessed without the lock and this is probably unsafe. For the first two patches this is can be remedied, though I find it makes the logic a

Re: pg_upgrade version checking questions

2019-07-23 Thread Daniel Gustafsson
> On 22 Jul 2019, at 10:46, Peter Eisentraut > wrote: > > On 2019-04-04 15:40, Daniel Gustafsson wrote: >> On Wednesday, March 27, 2019 1:43 PM, Christoph Berg wrote: >> >>> Re: Daniel Gustafsson 2019-03-26 >>> pC-NMmh4vQLQP76YTwY4AuoD4OdNw9egikekyQpXFpgqmTlGjIZXOTd2W5RDZPpRski5N3ADRrLYgLk6QU

Fetching timeline during recovery

2019-07-23 Thread Jehan-Guillaume de Rorthais
Hello, Fetching the timeline from a standby could be useful in various situation. Either for backup tools [1] or failover tools during some kind of election process. Please, find in attachment a first trivial patch to support pg_walfile_name() and pg_walfile_name_offset() on a standby. Previous

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-07-23 Thread Tom Lane
David Rowley writes: > I've attached v7, which really is v6 with some comments adjusted and > the order of the hash_get_num_entries and hash_get_max_bucket function > calls swapped. I think hash_get_num_entries() will return 0 most of > the time where we're calling it, so it makes sense to put th

stress test for parallel workers

2019-07-23 Thread Justin Pryzby
Does anyone have a stress test for parallel workers ? On a customer's new VM, I got this several times while (trying to) migrate their DB: < 2019-07-23 10:33:51.552 CDT postgres >FATAL: postmaster exited during a parallel transaction < 2019-07-23 10:33:51.552 CDT postgres >STATEMENT: CREATE U

Re: Change ereport level for QueuePartitionConstraintValidation

2019-07-23 Thread Alvaro Herrera
On 2019-Jul-23, David Rowley wrote: > Also, if I'm not wrong, the votes so far appear to be: > > NOTICE: Robert, Amit > DEBUG1: Tom, Alvaro (I'm entirely basing this on the fact that they > mentioned possible ways to test with DEBUG1) > > I'll be happy with DEBUG1 if we can get tests to test it

Re: [bug fix] Produce a crash dump before main() on Windows

2019-07-23 Thread Tom Lane
Kyotaro Horiguchi writes: > We are obliged to assume that we won't have the desired behavior > without detecting whether running as a service or not. > My investigation convinced me that there is no way for a process > to detect wheter it is running as a service (except the process > directly cal

Re: [PATCH] Improve performance of NOTIFY over many databases (issue blocking on AccessExclusiveLock on object 0 of class 1262 of database 0)

2019-07-23 Thread Tom Lane
Martijn van Oosterhout writes: > There are a number of possible improvements here: > 1. Do what sinval does and separate the reader and writer locks so > they can't block each other. This is the ultimate solution, but it's a > significant refactor and it's not clear that's actually worthwhile > h

Re: stress test for parallel workers

2019-07-23 Thread Tom Lane
Justin Pryzby writes: > Does anyone have a stress test for parallel workers ? > On a customer's new VM, I got this several times while (trying to) migrate > their DB: > < 2019-07-23 10:33:51.552 CDT postgres >FATAL: postmaster exited during a > parallel transaction We've been seeing this irre

Re: [bug fix] Produce a crash dump before main() on Windows

2019-07-23 Thread Alvaro Herrera
On 2019-Jul-23, Tom Lane wrote: > Kyotaro Horiguchi writes: > > My investigation convinced me that there is no way for a process > > to detect wheter it is running as a service (except the process > > directly called from system (aka entry function)). Maybe we can try calling pgwin32_is_service

Re: stress test for parallel workers

2019-07-23 Thread Justin Pryzby
On Tue, Jul 23, 2019 at 01:28:47PM -0400, Tom Lane wrote: > ... you'd think an OOM kill would show up in the kernel log. > (Not necessarily in dmesg, though. Did you try syslog?) Nothing in /var/log/messages (nor dmesg ring). I enabled abrtd while trying to reproduce it last week. Since you ask

Re: block-level incremental backup

2019-07-23 Thread Jeevan Ladhe
Hi Vignesh, This backup technology is extending the pg_basebackup itself, which means we can still take online backups. This is internally done using pg_start_backup and pg_stop_backup. pg_start_backup performs a checkpoint, and this checkpoint is used in the recovery process while starting the cl

pgbench - allow to create partitioned tables

2019-07-23 Thread Fabien COELHO
Hello devs, While doing some performance tests and reviewing patches, I needed to create partitioned tables. Given the current syntax this is time consumming. The attached patch adds two options to create a partitioned "account" table in pgbench. It allows to answer quickly simple questio

Re: Fetching timeline during recovery

2019-07-23 Thread Andrey Borodin
> 23 июля 2019 г., в 21:05, Jehan-Guillaume de Rorthais > написал(а): > > Fetching the timeline from a standby could be useful in various situation. > Either for backup tools [1] or failover tools during some kind of election > process. That backup tool is reading timeline from pg_control_che

Re: [PATCH] Improve performance of NOTIFY over many databases (issue blocking on AccessExclusiveLock on object 0 of class 1262 of database 0)

2019-07-23 Thread Martijn van Oosterhout
On Tue, 23 Jul 2019 at 19:21, Tom Lane wrote: > > Martijn van Oosterhout writes: > > 2. Add a field to AsyncQueueEntry which points to the next listening > > backend. This would allow the loops over all listening backends to > > complete much faster, especially in the normal case where there are

Re: Fetching timeline during recovery

2019-07-23 Thread David Steele
On 7/23/19 2:59 PM, Andrey Borodin wrote: > >> 23 июля 2019 г., в 21:05, Jehan-Guillaume de Rorthais >> написал(а): >> >> Fetching the timeline from a standby could be useful in various situation. >> Either for backup tools [1] or failover tools during some kind of election >> process. > That ba

Re: Fetching timeline during recovery

2019-07-23 Thread Jehan-Guillaume de Rorthais
On Tue, 23 Jul 2019 16:00:29 -0400 David Steele wrote: > On 7/23/19 2:59 PM, Andrey Borodin wrote: > > > >> 23 июля 2019 г., в 21:05, Jehan-Guillaume de Rorthais > >> написал(а): > >> > >> Fetching the timeline from a standby could be useful in various situation. > >> Either for backup tools

Re: make \d pg_toast.foo show its indices ; and, \d toast show its main table ; and \d relkind=I show its partitions

2019-07-23 Thread Tom Lane
Fabien COELHO writes: > Field relnamespace oid in pg_class appears with pg 7.3, maybe it would be > appropriate to guard agains older versions, with "pset.sversion >= 70300". > It seems that there are other unguarded instances in "describe.c", so > maybe this is considered too old. Per the com

Re: stress test for parallel workers

2019-07-23 Thread Thomas Munro
On Wed, Jul 24, 2019 at 4:27 AM Justin Pryzby wrote: > < 2019-07-23 10:33:51.552 CDT postgres >FATAL: postmaster exited during a > parallel transaction > < 2019-07-23 10:33:51.552 CDT postgres >STATEMENT: CREATE UNIQUE INDEX > unused0_huawei_umts_nodeb_locell_201907_unique_idx ON > child.unus

Re: [PATCH] Improve performance of NOTIFY over many databases (issue blocking on AccessExclusiveLock on object 0 of class 1262 of database 0)

2019-07-23 Thread Tom Lane
Martijn van Oosterhout writes: > On Tue, 23 Jul 2019 at 19:21, Tom Lane wrote: >> Martijn van Oosterhout writes: >>> 2. Add a field to AsyncQueueEntry which points to the next listening >>> backend. This would allow the loops over all listening backends to >>> complete much faster, especially in

Re: getting ERROR "relation 16401 has no triggers" with partition foreign key alter

2019-07-23 Thread Alvaro Herrera
On 2019-Jul-22, Alvaro Herrera wrote: > After looking at the code some more, I think calling the new function in > the Prep phase is correct. The attached patch is pretty much final form > for this bugfix. I decided to unwrap a couple of error messages (I did > get bitten while grepping because

Re: stress test for parallel workers

2019-07-23 Thread Thomas Munro
On Wed, Jul 24, 2019 at 5:42 AM Justin Pryzby wrote: > #2 0x0085ddff in errfinish (dummy=) at > elog.c:555 > edata = > elevel = 22 > oldcontext = 0x27e15d0 > econtext = 0x0 > __func__ = "errfinish" > #3 0x006f7e94 in CheckPointReplication

Re: make \d pg_toast.foo show its indices ; and, \d toast show its main table ; and \d relkind=I show its partitions

2019-07-23 Thread Fabien COELHO
Pushed with minor fiddling with the toast-table code, and rather more significant hacking on the partitioned-index code. Notably, 0003 had broken output of Tablespace: footers for everything except indexes. Argh, sorry for the review miss. It's possibly not Justin's fault that that wasn't

Re: stress test for parallel workers

2019-07-23 Thread Tom Lane
Thomas Munro writes: > *I suspect that the only thing implicating parallelism in this failure > is that parallel leaders happen to print out that message if the > postmaster dies while they are waiting for workers; most other places > (probably every other backend in your cluster) just quietly exi

Re: pgbench - allow to create partitioned tables

2019-07-23 Thread Simon Riggs
On Tue, 23 Jul 2019 at 19:26, Fabien COELHO wrote: > > Hello devs, > > While doing some performance tests and reviewing patches, I needed to > create partitioned tables. Given the current syntax this is time > consumming. > Good idea. I wonder why we didn't have it already. > The attached patc

pgbench tests vs Windows

2019-07-23 Thread Andrew Dunstan
In commit ed8a7c6fcf9 we added some extra tests to pgbench, including this snippet: \setshell two\   expr \     1 + :one Unfortunately, this isn't portable, as I've just discovered at the cost of quite a bit of time. In particular, you can't assume expr is present and in the path o

Re: range_agg

2019-07-23 Thread Alvaro Herrera
Hi Paul, Just checking if you've had a chance to make progress on this. Thanks, -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: stress test for parallel workers

2019-07-23 Thread Thomas Munro
On Wed, Jul 24, 2019 at 10:03 AM Thomas Munro wrote: > > edata = > If you have that core, it might be interesting to go to frame 2 and > print *edata or edata->saved_errno. ... Rats. We already saw that it's optimised out so unless we can find that somewhere else in a variable that's p

Re: stress test for parallel workers

2019-07-23 Thread Justin Pryzby
On Wed, Jul 24, 2019 at 10:03:25AM +1200, Thomas Munro wrote: > On Wed, Jul 24, 2019 at 5:42 AM Justin Pryzby wrote: > > #2 0x0085ddff in errfinish (dummy=) at > > elog.c:555 > > edata = > > If you have that core, it might be interesting to go to frame 2 and > print *edata or e

Re: stress test for parallel workers

2019-07-23 Thread Thomas Munro
On Wed, Jul 24, 2019 at 10:42 AM Justin Pryzby wrote: > On Wed, Jul 24, 2019 at 10:03:25AM +1200, Thomas Munro wrote: > > On Wed, Jul 24, 2019 at 5:42 AM Justin Pryzby wrote: > > > #2 0x0085ddff in errfinish (dummy=) at > > > elog.c:555 > > > edata = > > > > If you have that co

Re: Support for jsonpath .datetime() method

2019-07-23 Thread Nikita Glukhov
On 23.07.2019 16:44, Peter Eisentraut wrote: I think the best way forward here is to focus first on patch 0002 and get the additional format templates in, independent of any surrounding JSON functionality. In particular, remove parse_datetime() and all the related API changes, then it becomes m

Re: stress test for parallel workers

2019-07-23 Thread Justin Pryzby
On Wed, Jul 24, 2019 at 10:46:42AM +1200, Thomas Munro wrote: > On Wed, Jul 24, 2019 at 10:42 AM Justin Pryzby wrote: > > On Wed, Jul 24, 2019 at 10:03:25AM +1200, Thomas Munro wrote: > > > On Wed, Jul 24, 2019 at 5:42 AM Justin Pryzby > > > wrote: > > > > #2 0x0085ddff in errfinish (du

Re: pgbench tests vs Windows

2019-07-23 Thread Tom Lane
Andrew Dunstan writes: > In commit ed8a7c6fcf9 we added some extra tests to pgbench, including > this snippet: > \setshell two\ >   expr \ >     1 + :one > Unfortunately, this isn't portable, as I've just discovered at the cost > of quite a bit of time. In particular, you can't assume

Re: stress test for parallel workers

2019-07-23 Thread Tom Lane
Justin Pryzby writes: > I want to say I'm almost certain it wasn't ENOSPC in other cases, since, > failing to find log output, I ran df right after the failure. The fact that you're not finding log output matching what was reported to the client seems to me to be a mighty strong indication that t

Re: stress test for parallel workers

2019-07-23 Thread Thomas Munro
On Wed, Jul 24, 2019 at 11:04 AM Justin Pryzby wrote: > I ought to have remembered that it *was* in fact out of space this AM when > this > core was dumped (due to having not touched it since scheduling transition to > this VM last week). > > I want to say I'm almost certain it wasn't ENOSPC in o

Re: stress test for parallel workers

2019-07-23 Thread Thomas Munro
On Wed, Jul 24, 2019 at 10:11 AM Tom Lane wrote: > Thomas Munro writes: > > *I suspect that the only thing implicating parallelism in this failure > > is that parallel leaders happen to print out that message if the > > postmaster dies while they are waiting for workers; most other places > > (pr

Re: stress test for parallel workers

2019-07-23 Thread Alvaro Herrera
On 2019-Jul-23, Justin Pryzby wrote: > I want to say I'm almost certain it wasn't ENOSPC in other cases, since, > failing to find log output, I ran df right after the failure. I'm not sure that this proves much, since I expect temporary files to be deleted on failure; by the time you run 'df' the

Re: Parallel Append subplan order instability on aye-aye

2019-07-23 Thread Thomas Munro
On Tue, Jul 16, 2019 at 12:21 PM Tom Lane wrote: > In the meantime, we've had *lots* of buildfarm failures in the > added pg_stat_all_tables query, which indicate that indeed the > stats collector mechanism isn't terribly reliable. But that > doesn't directly prove anything about the original pro

Re: SQL/JSON path issues/questions

2019-07-23 Thread Steven Pousty
Ok I have the toolset. Where do I find the PR for the doc on this work. I only feel qualified to review the doc. Thanks Steve On Sat, Jul 20, 2019 at 11:48 AM Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > Hi Steven, > > On Fri, Jul 19, 2019 at 9:53 PM Steven Pousty > wrote: > > I woul

Re: stress test for parallel workers

2019-07-23 Thread Justin Pryzby
On Wed, Jul 24, 2019 at 11:32:30AM +1200, Thomas Munro wrote: > On Wed, Jul 24, 2019 at 11:04 AM Justin Pryzby wrote: > > I ought to have remembered that it *was* in fact out of space this AM when > > this > > core was dumped (due to having not touched it since scheduling transition to > > this V

Re: Fetching timeline during recovery

2019-07-23 Thread Michael Paquier
On Tue, Jul 23, 2019 at 06:05:18PM +0200, Jehan-Guillaume de Rorthais wrote: > Please, find in attachment a first trivial patch to support pg_walfile_name() > and pg_walfile_name_offset() on a standby. > Previous restriction on this functions seems related to ThisTimeLineID not > being safe on stan

RE: [PATCH] Speedup truncates of relation forks

2019-07-23 Thread Jamison, Kirk
Hi, I repeated the recovery performance test before, and found out that I made a wrong measurement. Using the same steps indicated in the previous email (24GB shared_buffers for my case), the recovery time still significantly improved compared to head from "13 minutes" to "4 minutes 44 seconds"

Re: Memory Accounting

2019-07-23 Thread Melanie Plageman
On Thu, Jul 18, 2019 at 11:24 AM Jeff Davis wrote: > Previous discussion: > https://postgr.es/m/1407012053.15301.53.camel%40jeff-desktop > > This patch introduces a way to ask a memory context how much memory it > currently has allocated. Each time a new block (not an individual > chunk, but a ne

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-07-23 Thread Peter Geoghegan
On Fri, Jul 19, 2019 at 7:24 PM Peter Geoghegan wrote: > Hmm. So, the attached test case fails amcheck verification for me with > the latest version of the patch: Attached is a revised version of your v2 that fixes this issue -- I'll call this v3. In general, my goal for the revision was to make

Re: Race conditions with TAP test for syncrep

2019-07-23 Thread Michael Paquier
On Tue, Jul 23, 2019 at 05:04:32PM +0900, Michael Paquier wrote: > Thanks Noah for the review. I have reviewed the thread and improved a > couple of comments based on Alvaro's previous input. Attached is v2. > If there are no objections, I would be fine to commit it. Applied and back-patched dow

pg_basebackup delays closing of stdout

2019-07-23 Thread Jeff Janes
Ever since pg_basebackup was created, it had a comment like this: * End of chunk. If requested, and this is the base tablespace * write configuration file into the tarfile. When done, close the * file (but not stdout). But, why make the exception for output going to stdout? If we

Re: pg_receivewal documentation

2019-07-23 Thread Michael Paquier
On Tue, Jul 23, 2019 at 08:00:41AM -0400, Jesper Pedersen wrote: > Sure. Thanks. Applied down to 9.6 where remote_apply has been introduced, with tweaks for 9.6 as the tool is named pg_receivexlog there. -- Michael signature.asc Description: PGP signature

Re: pgbench tests vs Windows

2019-07-23 Thread Michael Paquier
On Tue, Jul 23, 2019 at 07:13:51PM -0400, Tom Lane wrote: > Andrew Dunstan writes: >> I propose to prepare a patch along these lines. Alternatively we could >> just drop it - I don't think the test matters all that hugely. > > I'd say try that, but if it doesn't work right away, just skip the > t

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-07-23 Thread David Rowley
Thanks for having a look at this. On Wed, 24 Jul 2019 at 04:13, Tom Lane wrote: > > David Rowley writes: > > I'm pretty happy with v7 now. If anyone has any objections to it, > > please speak up very soon. Otherwise, I plan to push it about this > > time tomorrow. > > I dunno, this seems close

Re: On the stability of TAP tests for LDAP

2019-07-23 Thread Thomas Munro
On Fri, Jul 19, 2019 at 3:30 PM Michael Paquier wrote: > # Running: /usr/sbin/slapd -f > /home/bf/bfr/root/HEAD/pgsql.build/src/test/ldap/tmp_check/slapd.conf > -h ldap://localhost:55306 ldaps://localhost:55307 > # loading LDAP data > # Running: ldapadd -x -y > /home/bf/bfr/root/HEAD/pgsql.build

Re: Change atoi to strtol in same place

2019-07-23 Thread Joe Nelson
> Surafel Temesgen wrote: > > we use atoi for user argument processing in same place which return > > zero for both invalid input and input value zero. [...] in same > > place where we accept zero as valued input it case a problem by > > preceding for invalid input as input value zero. strtol whic

Re: block-level incremental backup

2019-07-23 Thread vignesh C
Thanks Jeevan. 1) If relation file has changed due to truncate or vacuum. During incremental backup the new files will be copied. There are chances that both the old file and new file will be present. I'm not sure if cleaning up of the old file is handled. 2) Just a small thought

Re: [PATCH] minor bugfix for pg_basebackup (9.6 ~ )

2019-07-23 Thread Ian Barwick
On 7/23/19 5:10 PM, Michael Paquier wrote: On Mon, Jul 22, 2019 at 12:58:40PM -0400, Alvaro Herrera wrote: Maybe it's just me, but it seems weird to try to forestall a problem that cannot occur by definition. I would rather remove the escaping, and add a one-line comment explaining why we don't

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-07-23 Thread David Rowley
On Wed, 24 Jul 2019 at 15:05, David Rowley wrote: > To be able to reduce the threshold down again we'd need to make a > hash_get_num_entries(LockMethodLocalHash) call before performing the > guts of LockReleaseAll(). We could then weight that onto some running > average counter with a weight of, s

Re: POC: Cleaning up orphaned files using undo logs

2019-07-23 Thread Dilip Kumar
On Mon, Jul 22, 2019 at 3:51 PM Dilip Kumar wrote: > Please find my review comments for 0013-Allow-foreground-transactions-to-perform-undo-action + /* initialize undo record locations for the transaction */ + for (i = 0; i < UndoLogCategories; i++) + { + s->start_urec_ptr[i] = InvalidUndoRecPtr;

Re: On the stability of TAP tests for LDAP

2019-07-23 Thread Thomas Munro
On Wed, Jul 24, 2019 at 3:52 PM Thomas Munro wrote: > I guess we should do that too. I don't know how to write Perl but I'll try... Does this look about right? -- Thomas Munro https://enterprisedb.com wait-for-slapd.patch Description: Binary data

Re: Change atoi to strtol in same place

2019-07-23 Thread David Rowley
On Wed, 24 Jul 2019 at 16:02, Joe Nelson wrote: > > In general, I think it's a good idea to fix those places, but I wonder > > if we need to change the error messages too. > > I'll leave that decision for the community to debate. I did, however, > remove the newlines for the new error messages bei

Re: range_agg

2019-07-23 Thread Paul A Jungwirth
On Tue, Jul 23, 2019 at 3:32 PM Alvaro Herrera wrote: > Just checking if you've had a chance to make progress on this. Not a lot. :-) But I should have more time for it the next few weeks than I did the last few. I do have some code for creating concrete multirange types (used when you create a c

Re: stress test for parallel workers

2019-07-23 Thread Tom Lane
Thomas Munro writes: > On Wed, Jul 24, 2019 at 10:11 AM Tom Lane wrote: >> In any case, the evidence from the buildfarm is pretty clear that >> there is *some* connection. We've seen a lot of recent failures >> involving "postmaster exited during a parallel transaction", while >> the number of p

Re: Change atoi to strtol in same place

2019-07-23 Thread Michael Paquier
On Wed, Jul 24, 2019 at 04:57:42PM +1200, David Rowley wrote: > I'd like to put my vote not to add this complex code to each option > validation that requires an integer number. I'm not sure there > currently is a home for it, but if there was, wouldn't it be better > writing a function that takes

Re: On the stability of TAP tests for LDAP

2019-07-23 Thread Michael Paquier
On Wed, Jul 24, 2019 at 04:41:05PM +1200, Thomas Munro wrote: > On Wed, Jul 24, 2019 at 3:52 PM Thomas Munro wrote: > > I guess we should do that too. I don't know how to write Perl but I'll > > try... > > Does this look about right? Some comments from here. I have not tested the patch. I wo

Re: On the stability of TAP tests for LDAP

2019-07-23 Thread Thomas Munro
On Wed, Jul 24, 2019 at 5:26 PM Michael Paquier wrote: > > Does this look about right? > > Some comments from here. I have not tested the patch. > > I would recommend using TestLib::system_log instead of plain system(). > The command should be a list of arguments with one element per > argument (

Re: POC: Cleaning up orphaned files using undo logs

2019-07-23 Thread Rushabh Lathia
Hi, I have stated review of 0008-Provide-interfaces-to-store-and-fetch-undo-records.patch, here are few quick comments. 1) README.undointerface should provide more information like API details or the sequence in which API should get called. 2) Information about the API's in the undoaccess.c file

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-07-23 Thread Andres Freund
Hi, On 2019-07-21 21:37:28 +1200, David Rowley wrote: > select.sql: > \set p 1 > select * from ht where a = :p > > Master: > > $ pgbench -n -f select.sql -T 60 -M prepared postgres > tps = 10172.035036 (excluding connections establishing) > tps = 10192.780529 (excluding connections establishing)

Re: Change atoi to strtol in same place

2019-07-23 Thread Andres Freund
On 2019-07-24 16:57:42 +1200, David Rowley wrote: > On Wed, 24 Jul 2019 at 16:02, Joe Nelson wrote: > > > In general, I think it's a good idea to fix those places, but I wonder > > > if we need to change the error messages too. > > > > I'll leave that decision for the community to debate. I did, h

Re: Psql patch to show access methods info

2019-07-23 Thread Andres Freund
Hi, On 2019-07-15 22:03:31 +0300, Nikita Glukhov wrote: > + > + > + \dAc[+] > +[ class="parameter">access-method-pattern > + [ class="parameter">input-type-pattern]] > + > + > + > + > +Shows info index access

Re: Psql patch to show access methods info

2019-07-23 Thread Andres Freund
Hi, On 2019-07-23 01:57:29 +0300, Alexander Korotkov wrote: > It was always scary there is no way in psql to see am/opclass/opfamily > information rather than query catalog directly. What does make that scary? > I'm going to push it. Probably, someone find that commands syntax and > output for

Re: ANALYZE: ERROR: tuple already updated by self

2019-07-23 Thread Andres Freund
Hi, On 2019-06-18 17:08:37 -0700, Andres Freund wrote: > On 2019-06-18 18:48:58 -0500, Justin Pryzby wrote: > > Ah: the table is an inheritence parent. If I uninherit its child, there's > > no > > error during ANALYZE. MV stats on the child are ok: > > It's a "classical" inheritance parent, no

Re: initdb recommendations

2019-07-23 Thread Andres Freund
Hi, On 2019-07-22 13:02:13 -0400, Andrew Dunstan wrote: > There are a few things we could do. We could force trust auth, or we > could add an ident map that allowed $USER to login as buildfarm. Finding > all the places we would need to fix that could be a fun project ... Perhaps we could actually

Re: Psql patch to show access methods info

2019-07-23 Thread Andres Freund
Hi, On 2019-07-15 22:03:31 +0300, Nikita Glukhov wrote: > + > + > + \dAc[+] > +[ class="parameter">access-method-pattern > + [ class="parameter">input-type-pattern]] > + > + > + > + > +Shows info index access

Re: POC: Cleaning up orphaned files using undo logs

2019-07-23 Thread Andres Freund
On 2019-07-22 14:21:36 +0530, Amit Kapila wrote: > Another thing is changing subxids to fxids can increase the size of > two-phase file for a xact having many sub-transactions which again > might be okay, but not completely sure. I can't see that being a problem.

Re: pg_basebackup delays closing of stdout

2019-07-23 Thread Andres Freund
Hi, On 2019-07-23 22:16:26 -0400, Jeff Janes wrote: > Ever since pg_basebackup was created, it had a comment like this: > > * End of chunk. If requested, and this is the base tablespace > * write configuration file into the tarfile. When done, close the > * file (but not stdout). >

Re: Problem during Windows service start

2019-07-23 Thread Kyotaro Horiguchi
Sorry in advance for link-breaking message, but the original mail was too old and gmail doesn't allow me to craft required headers to link to it. https://www.postgresql.org/message-id/CAKm4Xs71Ma8bV1fY6Gfz9Mg3AKmiHuoJNpxeDVF_KTVOKoy1WQ%40mail.gmail.com > Please find the proposed patch for review.

Seek failure at end of FSM file during WAL replay (in 11)

2019-07-23 Thread Michael Paquier
Hi all, Recently, one of the test beds we use has blown up once when doing streaming replication like that: FATAL: could not seek to end of file "base/16386/19817_fsm": No such file or directory CONTEXT: WAL redo at 60/8DA22448 for Heap2/CLEAN: remxid 65751197 LOG: startup process (PID 4488

Re: POC: Cleaning up orphaned files using undo logs

2019-07-23 Thread Dilip Kumar
On Wed, Jul 24, 2019 at 11:28 AM Rushabh Lathia wrote: > > Hi, > > I have stated review of > 0008-Provide-interfaces-to-store-and-fetch-undo-records.patch, here are few > quick comments. Thanks for the review, I will work on them soon and post the updated patch along with other comments. I have

Re: make \d pg_toast.foo show its indices ; and, \d toast show its main table ; and \d relkind=I show its partitions

2019-07-23 Thread Fabien COELHO
Find attached updated patches which also work against old servers. I can't check that for sure. * About toast table addition v7: Patch applies cleanly, compiles, make check ok, no doc. This addition show the main table of a toast table, which is useful. Field relnamespace oid in pg_class

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-23 Thread Michael Paquier
On Mon, Jul 22, 2019 at 02:40:19PM +0200, Julien Rouhaud wrote: > Right, so I kept the long option. Also this comment was outdated, as > the --jobs is now just ignored with a list of indexes, so I fixed that > too. + if (!parallel) + { + if (user_list == NULL) + { + /* +

Re: pgbench - add pseudo-random permutation function

2019-07-23 Thread Fabien COELHO
Hello Thomas, Function nbits(), which was previously discussed, has been simplified by using the function pg_popcount64(). Hi Fabien, Suzuki-san, I am not smart enough to commit this I'm in no hurry:-) or judge its value for benchmarking, I think that it is valuable given that we have

Re: Race conditions with TAP test for syncrep

2019-07-23 Thread Michael Paquier
On Mon, Jul 22, 2019 at 11:45:53PM -0700, Noah Misch wrote: > PostgresNode uses "print" the same way. The patch does close the intended > race conditions, and its implementation choices look fine to me. Thanks Noah for the review. I have reviewed the thread and improved a couple of comments base

Re: [PATCH] minor bugfix for pg_basebackup (9.6 ~ )

2019-07-23 Thread Michael Paquier
On Mon, Jul 22, 2019 at 12:58:40PM -0400, Alvaro Herrera wrote: > Maybe it's just me, but it seems weird to try to forestall a problem > that cannot occur by definition. I would rather remove the escaping, > and add a one-line comment explaining why we don't do it? No objections with doing that e

Re: Berserk Autovacuum (let's save next Mandrill)

2019-07-23 Thread Masahiko Sawada
On Sun, Apr 14, 2019 at 9:59 PM Darafei "Komяpa" Praliaskouski wrote: >> >> >> >I don't think it's helpful to force emergency vacuuming more >> >frequently; >> >quite the contrary, it's likely to cause even more issues. We should >> >tweak autovacuum to perform freezing more preemtively instead.

Re: Runtime pruning problem

2019-07-23 Thread David Rowley
On Sat, 25 May 2019 at 18:55, David Rowley wrote: > and an updated patch, rebased after the pgindent run. > > Hopefully, this will make the CF bot happy again. and rebased again due to a conflict with some List changes that touched ruleutils.c. I also made another couple of passes over this addi

Re: Optimze usage of immutable functions as relation

2019-07-23 Thread Anastasia Lubennikova
08.07.2019 4:18, Thomas Munro: The July Commitfest is here. Could we please have a rebase of this patch? Updated patch is in attachments. I've only resolved one small cosmetic merge conflict. Later this week I'm going to send a more thoughtful review. -- Anastasia Lubennikova Postgres Profess

Re: [bug fix] Produce a crash dump before main() on Windows

2019-07-23 Thread Kyotaro Horiguchi
At Mon, 1 Jul 2019 00:04:47 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FC6515F@G01JPEXMBYT05> > From: Amit Kapila [mailto:amit.kapil...@gmail.com] > > Tsunakawa/Haribabu - By reading this thread briefly, it seems we need > > some more inputs from other developers on

  1   2   >