Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2018-09-30 Thread Michael Paquier
On Wed, Jun 06, 2018 at 07:03:47PM +0530, Amit Kapila wrote: > I think Robert's chash stuff [1] might be helpful to reduce the contention > you are seeing. Latest patch available does not apply, so I moved it to next CF. The thread has died a bit as well... -- Michael signature.asc Description:

Re: libpq compression

2018-09-30 Thread Michael Paquier
On Mon, Aug 20, 2018 at 06:00:39PM +0300, Konstantin Knizhnik wrote: > New version of the patch is attached: I removed -Z options form pgbench and > psql and add checking that server and client are implementing the same > compression algorithm. The patch had no reviews, and does not apply anymore,

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2018-09-30 Thread Michael Paquier
On Wed, Sep 12, 2018 at 06:12:29PM +0300, Marina Polyakova wrote: > The discussion about this has become entangled from the beginning, because > as I wrote in [1] at first I misread your original proposal... The last emails are about the last reviews of Fabien, which has remained unanswered for th

Re: pgbench - add pseudo-random permutation function

2018-09-30 Thread Michael Paquier
On Wed, Sep 26, 2018 at 01:20:49PM +0200, Fabien COELHO wrote: > So, am I right to deducing that you are satisfied with the current status of > the patch, with the nbits implementation either based on popcount (v4) or > clz (v5) compiler intrinsics? I think that the clz option is better. Fabien, p

Re: executor relation handling

2018-09-30 Thread Amit Langote
On 2018/09/30 5:04, Tom Lane wrote: > David Rowley writes: >> I've attached v10 which fixes this and aligns the WARNING text in >> ExecInitRangeTable() and addRangeTableEntryForRelation(). > > I started poking at this. Thanks a lot for looking at this. > I thought that it would be a good cross-

Re: Multiple primary key on partition table?

2018-09-30 Thread Rajkumar Raghuwanshi
On Tue, Sep 18, 2018 at 11:20 AM amul sul wrote: > On Mon, Sep 17, 2018 at 9:06 PM amul sul wrote: > > > > Nice catch Rajkumar. > > > > In index_check_primary_key(), relationHasPrimaryKey() called only for > the an > > alter command but I think we need to call in this case as well, like > this:

Re: Adding a note to protocol.sgml regarding CopyData

2018-09-30 Thread Michael Paquier
On Fri, Sep 28, 2018 at 08:16:46PM +0200, Peter Eisentraut wrote: > Yeah, I don't see why we need to document it three times in the same > chapter. > > Also, that chapter is specifically about version 3.0 of the protocol, so > documenting version 2.0 is out of scope. This has been marked as retur

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2018-09-30 Thread Michael Paquier
On Fri, Mar 23, 2018 at 02:18:09PM +0100, Daniel Verite wrote: > There's a batch mode for pgbench in a patch posted in [1], > with \beginbatch and \endbatch commands, but nothing > for psql AFAICS. > psql is more complicated because currently it uses a > blocking PQexec() call at its core. Craig m

Re: Sample values for pg_stat_statements

2018-09-30 Thread Michael Paquier
On Tue, Apr 17, 2018 at 05:58:44PM +0200, Vik Fearing wrote: > I think I handle that well enough with permission checking, but I'm open > to more debate on it. The recent version bump in pg_stat_statements (Sorry my fault!) is causing this patch to not apply anymore. I have moved it to next CF wi

Re: partition tree inspection functions

2018-09-30 Thread Michael Paquier
On Mon, Oct 01, 2018 at 03:16:32PM +0900, Amit Langote wrote: > I wasn't able to respond to some of issues that Jesper brought up with the > approach taken by the latest patch whereby there is no separate > pg_partition_level function. He said that such a function would be useful > to get the info

Re: libpq should append auth failures, not overwrite

2018-09-30 Thread Michael Paquier
On Tue, Aug 14, 2018 at 02:17:09PM +0200, Fabien COELHO wrote: > I think that people would survive having the ip spelled out on localhost > errors when there are several ips associated to the name. > > You could also create an exception for "localhost" if you see it as a large > problem, but if so

Re: partition tree inspection functions

2018-09-30 Thread Amit Langote
Hi, On 2018/10/01 15:03, Michael Paquier wrote: > On Thu, Aug 09, 2018 at 01:05:56PM +0900, Amit Langote wrote: >> Attached updated patch. > > So, except if I am missing something, what we have here is a patch which > has been debatted quite a bit and has semantics which look nice. Thanks. > A

Re: partition tree inspection functions

2018-09-30 Thread Michael Paquier
On Thu, Aug 09, 2018 at 01:05:56PM +0900, Amit Langote wrote: > Attached updated patch. So, except if I am missing something, what we have here is a patch which has been debatted quite a bit and has semantics which look nice. Any objections if we move forward with this patch? +-- all tables in t

Re: ToDo: show size of partitioned table

2018-09-30 Thread Michael Paquier
On Wed, Jul 25, 2018 at 06:09:05PM +0900, Amit Langote wrote: > This looks nice, although I haven't looked at the patch yet. Also, as you > said, we could later replace the method of directly querying pg_inherits > by something else. Yes, at the end we may be looking at something like that which

Re: doc - improve description of default privileges

2018-09-30 Thread Michael Paquier
On Fri, Sep 28, 2018 at 04:15:13PM -0300, Alvaro Herrera wrote: > There is a legitimate point in doing this, though, since the GRANT page > is already explaining how does psql display privileges. Maybe the right > solution is move that stuff all to the psql documentation, and alter the > GRANT pag

Re: csv format for psql

2018-09-30 Thread Michael Paquier
On Tue, Aug 28, 2018 at 08:54:02PM +0200, Fabien COELHO wrote: > Patch applies cleanly, compiles, "make check" ok. > > I tried "\pset fieldsep_csv '\0'" which could be seen as one character, but > it does not want it. I'm okay with this behavior. > > I'd suggest to add a test about rejected field

Re: TABLE tab completion

2018-09-30 Thread Michael Paquier
On Mon, May 28, 2018 at 09:28:12PM +0200, Vik Fearing wrote: > The tab completion for the TABLE command includes indexes but that's a > bug. Attached is a trivial patch to fix it. As far as I can see, this patch is not needed anymore since e3b7a6d1. -- Michael signature.asc Description: PGP sig

Assert failed in snprintf.c

2018-09-30 Thread Jaime Casanova
Hi, sqlsmith made it again, attached is the query (run against regression database) that makes the assert fail and the backtrace. this happens in head only (or at least 11 is fine). -- Jaime Casanova www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Trai

Re: SerializeParamList vs machines with strict alignment

2018-09-30 Thread Amit Kapila
On Sun, Sep 30, 2018 at 10:44 AM Amit Kapila wrote: > > On Mon, Sep 10, 2018 at 7:05 PM Tom Lane wrote: > > > > Amit Kapila writes: > > > On Mon, Sep 10, 2018 at 8:58 AM Tom Lane wrote: > > >> In particular, SerializeParamList does this: > > >> > > >> /* Write flags. */ > > >> memcpy(*start_add

Re: automatic restore point

2018-09-30 Thread Alvaro Herrera
On 2018-Jul-02, Jaime Casanova wrote: > You can also create a normal trigger BEFORE TRUNCATE to create a > restore point just before running the TRUNCATE command. On every single table? -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DB

Re: automatic restore point

2018-09-30 Thread Alvaro Herrera
On 2018-Sep-30, Michael Paquier wrote: > On Fri, Sep 28, 2018 at 09:13:17PM +0200, Peter Eisentraut wrote: > > I think this should be done using event triggers. Right now, you just > > have it hardcoded to TRUNCATE and DROP TABLE, which seems somewhat > > arbitrary. With event triggers, you have

Re: executor relation handling

2018-09-30 Thread Tom Lane
David Rowley writes: > On 1 October 2018 at 06:18, Tom Lane wrote: >> + for (slockmode = lockmode + 1; >> + slockmode <= AccessExclusiveLock; >> + slockmode++) > So would it not be better to add the following to lockdefs.h? > #define MaxLockLevel 8 > then use that to terminate the loop

Re: Some incorrect comments and out-dated README from run-time pruning

2018-09-30 Thread David Rowley
On 28 September 2018 at 09:20, David Rowley wrote: > I've noticed that the comments above the PartitionedRelPruneInfo > struct incorrectly document how subplan_map and subpart_map are > indexed. This seems to have snuck in on 4e232364033. > > Also, while reading the executor README file, I noticed

Re: pgbench - test whether a variable exists

2018-09-30 Thread Michael Paquier
On Tue, Mar 27, 2018 at 02:08:31PM +0200, Fabien COELHO wrote: >> Patch v2 is a rebase. > > Patch v3 is also a rebase. This has rotten for half a year, so I am marking it as returned with feedback. There have been comments from Alvaro and Andres as well... -- Michael signature.asc Description:

Re: heap_sync seems rather oblivious to partitioned tables (wal_level=minimal)

2018-09-30 Thread David Rowley
On 28 September 2018 at 19:22, David Rowley wrote: > I've edited that in the attached patch. Also reworded a comment that > Amit mentioned and made a small change to the COPY FREEZE docs to > mention no support for partitioned tables. Added to the November 'fest. https://commitfest.postgresql.o

Re: pgbench exit code

2018-09-30 Thread Michael Paquier
On Mon, Oct 01, 2018 at 11:29:19AM +0900, Michael Paquier wrote: > On Fri, Aug 10, 2018 at 01:50:49PM +0200, Fabien COELHO wrote: >> BTW, I have added this patch to the next CF: >> >> https://commitfest.postgresql.org/19/1751/ > > The latest patch does not apply anymore. Peter, could you re

Re: pgbench exit code

2018-09-30 Thread Michael Paquier
On Fri, Aug 10, 2018 at 01:50:49PM +0200, Fabien COELHO wrote: > BTW, I have added this patch to the next CF: > > https://commitfest.postgresql.org/19/1751/ The latest patch does not apply anymore. Peter, could you rebase it? -- Michael signature.asc Description: PGP signature

Re: Psql patch to show access methods info

2018-09-30 Thread Michael Paquier
On Tue, Jul 03, 2018 at 01:25:37PM +0300, s.cherkas...@postgrespro.ru wrote: > diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml > index 3ed9021..b699548 100644 > --- a/doc/src/sgml/catalogs.sgml > +++ b/doc/src/sgml/catalogs.sgml Please note that the latest patch proposed does

Re: file cloning in pg_upgrade and CREATE DATABASE

2018-09-30 Thread Michael Paquier
On Fri, Sep 28, 2018 at 02:19:53PM +0900, Michael Paquier wrote: > Thanks for the rebase. At the end I got my hands on only an APFS using > a mac. I ran a test with an instance holding a database with pgbench at > scaling factor 500, which gives close to 6.5GB. The results are nice on > my lapto

Re: Relations being opened without any lock whatever

2018-09-30 Thread Tom Lane
Michael Paquier writes: > Okay, that's bad. Wouldn't it be sufficient to use what the caller > passes out as lockmode instead of enforcing AEL though? No, because at the bottom of that function we're going to do a DROP CONSTRAINT on the old FK constraint, and that needs AEL anyway. If we tried t

Re: Relations being opened without any lock whatever

2018-09-30 Thread Michael Paquier
On Sun, Sep 30, 2018 at 03:20:44PM -0400, Tom Lane wrote: > 1. ALTER TABLE ... ALTER COLUMN TYPE, on a column that is part of > a foreign key constraint, opens the rel that has the other end of > the constraint before it's acquired a lock on said rel. > > The comment in ATPostAlterTypeCleanup clai

Re: Function for listing archive_status directory

2018-09-30 Thread Michael Paquier
On Sun, Sep 30, 2018 at 10:59:20PM +0200, Christoph Moench-Tegeder wrote: > Without further ado, I present a patch to implement pg_ls_archive_status(), > which fills this gap. I believe the function name is long enough and we > don't need an extra wal in there. The patch is based on a very recent >

Re: [HACKERS] Optional message to user when terminating/cancelling backend

2018-09-30 Thread Michael Paquier
On Sun, Sep 30, 2018 at 10:51:44PM +0900, Michael Paquier wrote: > You could have chosen something less complicated, like "ホゲ", which is > the equivalent of "foo" in English. Anyway, non-ASCII characters should > not be included in the final patch. Looking at the refactoring patch 0001, wouldn't

Re: [HACKERS] kqueue

2018-09-30 Thread Thomas Munro
On Sun, Sep 30, 2018 at 9:49 PM Matteo Beccati wrote: > On 30/09/2018 04:36, Thomas Munro wrote: > > On Sat, Sep 29, 2018 at 7:51 PM Matteo Beccati wrote: > >> Out of curiosity, I've installed FreBSD on an identically specced VM, > >> and the select benchmark was ~75k tps for kqueue vs ~90k tps o

Re: executor relation handling

2018-09-30 Thread David Rowley
On 1 October 2018 at 06:18, Tom Lane wrote: > It occurred to me that it'd be reasonable to insist that the caller > holds a lock *at least as strong* as the one being recorded in the RTE, > and that there's also been discussions about verifying that some lock > is held when something like heap_ope

Re: [RFC] Removing "magic" oids

2018-09-30 Thread David Fetter
On Sat, Sep 29, 2018 at 08:48:10PM -0700, Andres Freund wrote: > Hi, > > In my opinion the current WITH OIDs system has numerous weaknesses: > > 1) The fact that oids are so magic means that if we get pluggable >storage, the design of the potential pluggable systems is constrained >and si

Re: Making all nbtree entries unique by having heap TIDs participate in comparisons

2018-09-30 Thread Peter Geoghegan
On Fri, Sep 28, 2018 at 10:58 PM Andrey Lepikhov wrote: > I am reviewing this patch too. And join to Peter Eisentraut opinion > about splitting the patch into a hierarchy of two or three patches: > "functional" - tid stuff and "optimizational" - suffix truncation & > splitting. My reasons are simp

Function for listing archive_status directory

2018-09-30 Thread Christoph Moench-Tegeder
Hi, while setting up monitoring for a new PostgreSQL instance, I noticed that there's no build-in way for a pg_monitor role to check the contents of the archive_status directory. We got pg_ls_waldir() in 10, but that only lists pg_wal - not it's subdirectory. It seems listing the archive_status di

Relations being opened without any lock whatever

2018-09-30 Thread Tom Lane
Running the regression tests with the patch I showed in https://www.postgresql.org/message-id/16565.1538327...@sss.pgh.pa.us exposes two places where HEAD is opening relations without having any lock at all on them: 1. ALTER TABLE ... ALTER COLUMN TYPE, on a column that is part of a foreign key co

Re: executor relation handling

2018-09-30 Thread Tom Lane
I wrote: > 1. You set up transformRuleStmt to insert AccessExclusiveLock into > the "OLD" and "NEW" RTEs for a view. This is surely wrong; we do > not want to take exclusive lock on a view just to run a query using > the view. It should (usually, anyway) just be AccessShareLock. > However, becaus

Re: Cygwin linking rules

2018-09-30 Thread Andrew Dunstan
On 09/29/2018 02:13 PM, Marco Atzeri wrote: building from git and using the attached patch that is used for all cygwin packages on latest cygwin $ uname -svrm CYGWIN_NT-10.0 2.11.1(0.329/5/3) 2018-09-05 10:24 x86_64 I do not see the problem == creating database "contrib_regre

Re: [HACKERS] Optional message to user when terminating/cancelling backend

2018-09-30 Thread Michael Paquier
On Mon, Aug 27, 2018 at 12:06:18PM +0200, Daniel Gustafsson wrote: > I like that idea, so I’ve updated the patch to see what the cfbot > thinks of it. + when pg_cancel_backend(pg_backend_pid(), 'ソケットをブロッキングモードに設定できませんでした') You could have chosen something less complicated, like "ホゲ", which is th

Re: Sync ECPG scanner with core

2018-09-30 Thread Michael Paquier
On Sun, Sep 30, 2018 at 04:32:53PM +0700, John Naylor wrote: > Commit 21c8ee7946 was one of several that sync'd the backend scanner > with the psql scanner, with the commit message stating "Someday it'd > be nice to make ecpg's pgc.l more easily diff'able too, but today is > not that day." Attached

Re: Speed up the removal of WAL files

2018-09-30 Thread Michael Paquier
On Fri, Mar 09, 2018 at 12:50:03AM +, Tsunakawa, Takayuki wrote: > From: Michael Paquier [mailto:mich...@paquier.xyz] >> Hm. durable_xx should remain a sane operation as an isolated call as you >> still get the same problem if a crash happens before flushing the parent... >> Fujii-san idea als

Re: Recovery performance of DROP DATABASE with many tablespaces

2018-09-30 Thread Michael Paquier
On Tue, Jul 10, 2018 at 03:04:05PM +0900, Michael Paquier wrote: > It would be simple to measure the time it takes to replay this single > DROP DATABASE record by putting two gettimeofday() calls or such things > and then take the time difference. There are many methods that you > could use here,

Re: [PATCH] Find additional connection service files in pg_service.conf.d directory

2018-09-30 Thread Michael Paquier
On Mon, Aug 06, 2018 at 11:20:31AM -0400, Curt Tilmes wrote: > On Thu, Jul 19, 2018 at 5:39 AM Heikki Linnakangas wrote: >> I also think we should process the files in alphabetical order. That's >> how most software with conf.d directories work. >> > > In my experience, this is not a problem with

Re: [Patch] Create a new session in postmaster by calling setsid()

2018-09-30 Thread Michael Paquier
On Thu, Sep 13, 2018 at 12:24:38PM -0700, Andres Freund wrote: > On 2018-09-13 15:27:58 +0800, Paul Guo wrote: >> From the respective of users instead of developers, I think for such >> important >> service, tty should be dissociated, i.e. postmaster should be daemonized by >> default (and even sho

Re: automatic restore point

2018-09-30 Thread Michael Paquier
On Fri, Sep 28, 2018 at 09:13:17PM +0200, Peter Eisentraut wrote: > I think this should be done using event triggers. Right now, you just > have it hardcoded to TRUNCATE and DROP TABLE, which seems somewhat > arbitrary. With event triggers, you have the full flexibility to do > what you want. Yo

Re: Use durable_unlink for .ready and .done files for WAL segment removal

2018-09-30 Thread Michael Paquier
On Sat, Sep 29, 2018 at 04:58:57PM +0900, Michael Paquier wrote: > Actually, what you are proposing here sounds much better to me. That's > in the area of what has been done recently with RemoveTempXlogFiles() in > 5fc1008e. Any objections to doing something like that? Okay. I have hacked a pa

Re: Online enabling of checksums

2018-09-30 Thread Tomas Vondra
On 09/29/2018 06:51 PM, Stephen Frost wrote: > Greetings, > > * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: >> On 09/29/2018 02:19 PM, Stephen Frost wrote: >>> * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: While looking at the online checksum verification patch (which I guess

Re: [HACKERS] kqueue

2018-09-30 Thread Matteo Beccati
Hi Thomas, On 30/09/2018 04:36, Thomas Munro wrote: > On Sat, Sep 29, 2018 at 7:51 PM Matteo Beccati wrote: >> Out of curiosity, I've installed FreBSD on an identically specced VM, >> and the select benchmark was ~75k tps for kqueue vs ~90k tps on >> unpatched master, so maybe there's something w

Re: libpq host/hostaddr/conninfo inconsistencies

2018-09-30 Thread Fabien COELHO
So I'm speculating that the cfbot is using a version of patch(1) that doesn't have strip-trailing-CRs logic. Which bemuses me, because I thought they all did. Huh. Yeah. I have now switched it over to GNU patch. It seems to be happier with Fabien's patches so far, but will take a few minu

Re: libpq host/hostaddr/conninfo inconsistencies

2018-09-30 Thread Fabien COELHO
Hello Arthur, Thanks for the comments. However, the actual capability is slightly different: specifying an ip address to "host" does work, without ensuing any name or reverse name look-ups, even if this is undocumented. Agree it may have more details within the documentation. sh> psql "ho