Re: [HACKERS] More race conditions in logical replication

2017-07-14 Thread Noah Misch
On Wed, Jul 12, 2017 at 06:48:28PM -0400, Alvaro Herrera wrote: > Petr Jelinek wrote: > > > So best idea I could come up with is to make use of the new condition > > variable API. That lets us wait for variable which can be set per slot. > > Here's a cleaned up version of that patch, which I inte

Re: [HACKERS] hash index on unlogged tables doesn't behave as expected

2017-07-14 Thread Ashutosh Sharma
On Sat, Jul 15, 2017 at 8:20 AM, Amit Kapila wrote: > On Fri, Jul 14, 2017 at 6:02 PM, Michael Paquier > wrote: >> (catching up finally with this thread) >> >> On Mon, Jul 10, 2017 at 11:57 AM, Kyotaro HORIGUCHI >> wrote: >>> At Mon, 10 Jul 2017 14:58:13 +0530, Amit Kapila >>> wrote in >>> >

Re: [HACKERS] hash index on unlogged tables doesn't behave as expected

2017-07-14 Thread Amit Kapila
On Fri, Jul 14, 2017 at 6:02 PM, Michael Paquier wrote: > (catching up finally with this thread) > > On Mon, Jul 10, 2017 at 11:57 AM, Kyotaro HORIGUCHI > wrote: >> At Mon, 10 Jul 2017 14:58:13 +0530, Amit Kapila >> wrote in >>> I am also not 100% comfortable with adding flush at two new place

Re: [HACKERS] New partitioning - some feedback

2017-07-14 Thread Robert Haas
On Mon, Jul 10, 2017 at 5:46 PM, David Fetter wrote: > With utmost respect, it's less messy than adding '!' to the already > way too random and mysterious syntax of psql's \ commands. What > should '\det!' mean? What about '\dT!'? Since \det lists foreign tables, \det! would list foreign tables

Re: [HACKERS] Pluggable storage

2017-07-14 Thread Robert Haas
On Fri, Jul 14, 2017 at 8:35 AM, Haribabu Kommi wrote: > To replace tuple with slot, I took trigger and SPI calls as the first step > in modifying > from tuple to slot, Here I attached a WIP patch. The notable changes are, > > 1. Replace most of the HeapTuple with Slot in SPI interface functions.

Re: [HACKERS] Pluggable storage

2017-07-14 Thread Robert Haas
On Thu, Jun 22, 2017 at 4:27 PM, Tomas Vondra wrote: > Can you elaborate a bit more about this TID bit pattern issues? I do > remember that not all TIDs are valid due to safeguards on individual fields, > like for example > > Assert(iptr->ip_posid < (1 << MaxHeapTuplesPerPageBits)) > > But per

Re: [HACKERS] Pluggable storage

2017-07-14 Thread Robert Haas
On Thu, Jun 22, 2017 at 9:30 AM, Alexander Korotkov wrote: > If #1 is only about changing tuple and page formats, then could be much > simpler than the patch upthread? We can implement "page format access > methods" with routines for insertion, update, pruning and deletion of tuples > *in particu

[HACKERS] segfault in HEAD when too many nested functions call

2017-07-14 Thread Julien Rouhaud
Hello, Since b8d7f053c5c2bf2a7e8734fe3327f6a8bc711755 (Andres in Cc), if you write queries which result in infinite recursion (or just too many nested function calls), execution ends with segfault instead of intended exhausted max_stack_depth: Program received signal SIGSEGV, Segmentation fault.

Re: [HACKERS] [WIP] Zipfian distribution in pgbench

2017-07-14 Thread Peter Geoghegan
On Fri, Jul 14, 2017 at 6:37 AM, Alik Khilazhev wrote: > I am attaching results of tests for 32 and 128 clients that were running for > 10 minutes, and TPS remains 305 and 115 ktps respectively. > > Tests was executed with configuration set for YCSB. And there is very > aggressively autovacuum,

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2017-07-14 Thread Alvaro Herrera
Mark Rofail wrote: > On Wed, Jul 12, 2017 at 2:30 PM, Mark Rofail wrote: > > > On Wed, Jul 12, 2017 at 12:53 AM, Alvaro Herrera > > wrote: > >> > >> We have one opclass for each type combination -- int4 to int2, int4 to > >> int4, int4 to int8, etc. You just need to add the new strategy to all

[HACKERS] The case for removing replacement selection sort

2017-07-14 Thread Peter Geoghegan
There was a number of improvements to tuplesort.c external sort merging made for Postgres 10. One in particular, the changes to merge heap maintenance that occurred for commit 24598337c8d, really helped with presorted cases -- cases when there was an (inverse) physical/logical correlation. Replace

Re: [HACKERS] SCRAM auth and Pgpool-II

2017-07-14 Thread Stephen Frost
Greetings, * Vladimir Borodin (r...@simply.name) wrote: > > 14 июля 2017 г., в 1:33, Stephen Frost написал(а): > > What would be really nice for such cases is support for Kerberos and > > delegated Kerberos credentials. Having pgpool support that would remove > > the need to deal with passwords

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2017-07-14 Thread Mark Rofail
On Wed, Jul 12, 2017 at 2:30 PM, Mark Rofail wrote: > On Wed, Jul 12, 2017 at 12:53 AM, Alvaro Herrera > wrote: >> >> We have one opclass for each type combination -- int4 to int2, int4 to >> int4, int4 to int8, etc. You just need to add the new strategy to all >> the opclasses. >> > > Can you

[HACKERS] bug in locking an update tuple chain

2017-07-14 Thread Alvaro Herrera
A customer of ours reported a problem in 9.3.14 while inserting tuples in a table with a foreign key, with many concurrent transactions doing the same: after a few insertions worked sucessfully, a later one would return failure indicating that the primary key value was not present in the referenced

Re: [HACKERS] PG 10 release notes

2017-07-14 Thread Adrien Nayrat
On 07/13/2017 04:36 PM, Adrien Nayrat wrote: > Hello hackers, > > From: Peter Geoghegan >> Date: Wed, 5 Jul 2017 15:19:57 -0700 >> Subject: Re: [BUGS] BUG #14722: Segfault in tuplesort_heap_siftup, 32 bit >> overflow >> On pgsql-b...@postgresql.org > > On 07/06/2017 12:19 AM, Peter Geoghegan wr

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2017-07-14 Thread Marina Polyakova
Well, the short version may be to only do a full transaction retry and to document that for now savepoints are not handled, and to let that for future work if need arises. I agree with you. For progress the output must be short and readable, and probably we do not care about whether retries ca

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2017-07-14 Thread Marina Polyakova
Ok, fine. My point was just to check before proceeding. And I'm very grateful for that :) -- Marina Polyakova Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscripti

Re: [HACKERS] [WIP] Zipfian distribution in pgbench

2017-07-14 Thread Fabien COELHO
Algorithm works with theta less than 1. The only problem here is that theta can not be 1, because of next line of code cell->alpha = 1. / (1 - theta); That’s why I put such restriction. Now I see 2 possible solutions for that: 1) Exclude 1, and allow everything in range (0;+∞). Yep. 2) Or

Re: [HACKERS] SCRAM auth and Pgpool-II

2017-07-14 Thread Vladimir Borodin
> 14 июля 2017 г., в 1:33, Stephen Frost написал(а): > > What would be really nice for such cases is support for Kerberos and > delegated Kerberos credentials. Having pgpool support that would remove > the need to deal with passwords at all. Since nearly all systems with some kind of load nowa

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2017-07-14 Thread Fabien COELHO
And I'm not sure that we should do all the stuff for savepoints rollbacks because: - as I see it now it only makes sense for the deadlock failures; - if there's a failure what savepoint we should rollback to and start the execution again? ISTM that it is the point of having savepoint in the

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2017-07-14 Thread Fabien COELHO
If the answer is no, then implement something in pgbench directly. The structure of variables is different, the container structure of the variables is different, so I think that the answer is no. Ok, fine. My point was just to check before proceeding. -- Fabien. -- Sent via pgsql-hacker

Re: [HACKERS] SCRAM auth and Pgpool-II

2017-07-14 Thread Chapman Flack
On 07/13/2017 10:46 PM, Chapman Flack wrote: > Neither is suitable on an unencrypted channel (as has been repeatedly Please forgive my thinko about md5. I had overlooked the second salted md5 used in the protocol, and that had to be some years ago when I was sure I had looked for one in the code.

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2017-07-14 Thread Marina Polyakova
Given that the number of variables of a pgbench script is expected to be pretty small, I'm not sure that the sorting stuff is worth the effort. I think it is a good insurance if there're many variables.. My suggestion is really to look at both implementations and to answer the question "should

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2017-07-14 Thread Marina Polyakova
Not necessarily? It depends on where the locks triggering the issue are set, if they are all set after the savepoint it could work on a second attempt. Don't you mean the deadlock failures where can really help rollback to Yes, I mean deadlock failures can rollback to a savepoint and work on a

Re: [HACKERS] [WIP] Zipfian distribution in pgbench

2017-07-14 Thread Alik Khilazhev
> On 13 Jul 2017, at 23:13, Peter Geoghegan wrote: > > I just figured out that "result.txt" is only a 60 second pgbench run. > Is the same true for other tests? Yes, other tests ran 60 seconds too. > > It would be much more interesting to see runs that lasted 10 minutes > or more. I am atta

[HACKERS] Re: [BUGS] BUG #14634: On Windows pg_basebackup should write tar to stdout in binary mode

2017-07-14 Thread Heikki Linnakangas
On 07/14/2017 05:27 AM, Haribabu Kommi wrote: On Fri, Jul 14, 2017 at 2:54 AM, Heikki Linnakangas wrote: On 05/03/2017 07:32 AM, Haribabu Kommi wrote: [Adding -hackers mailing list] On Fri, Apr 28, 2017 at 6:28 PM, wrote: Executing command pg_basebackup -D -F t writes its output to stdou

Re: [HACKERS] [WIP] Zipfian distribution in pgbench

2017-07-14 Thread Alik Khilazhev
> On 13 Jul 2017, at 19:14, Fabien COELHO wrote: > > Documentation says that the closer theta is from 0 the flatter the > distribution > but the implementation requires at least 1, including strange error messages: > > zipfian parameter must be greater than 1.00 (not 1.00) > > Could

Re: [HACKERS] hash index on unlogged tables doesn't behave as expected

2017-07-14 Thread Michael Paquier
(catching up finally with this thread) On Mon, Jul 10, 2017 at 11:57 AM, Kyotaro HORIGUCHI wrote: > At Mon, 10 Jul 2017 14:58:13 +0530, Amit Kapila > wrote in >> I am also not 100% comfortable with adding flush at two new places, >> but that makes the code for fix simpler and fundamentally the

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-14 Thread Mithun Cy
On Thu, Jul 6, 2017 at 10:52 AM, Amit Kapila wrote: >>> 3. >> -- I do not think that is true pages of the unlogged table are also >> read into buffers for read-only purpose. So if we miss to dump them >> while we shut down then the previous dump should be used. >> > > I am not able to understand w

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2017-07-14 Thread Fabien COELHO
Note that there is something for psql (src/bin/psql/variable.c) which may or may not be shared. It should be checked before recoding eventually the same thing. Thank you very much for pointing this file! As I checked this is another structure: here there's a simple list, while in pgbench we

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2017-07-14 Thread Fabien COELHO
Hello Marina, Not necessarily? It depends on where the locks triggering the issue are set, if they are all set after the savepoint it could work on a second attempt. Don't you mean the deadlock failures where can really help rollback to Yes, I mean deadlock failures can rollback to a savepo

Re: [HACKERS] More flexible LDAP auth search filters?

2017-07-14 Thread Magnus Hagander
On Thu, Jul 13, 2017 at 9:31 AM, Thomas Munro wrote: > Hi hackers, > > A customer asked how to use pg_hba.conf LDAP search+bind > authentication to restrict logins to users in one of a small number of > groups. ldapsearchattribute only lets you make filters like > "(foo=username)", so it couldn'

[HACKERS] Adding -E switch to pg_dumpall

2017-07-14 Thread Michael Paquier
Hi all, While looking at a user problem, I got surprised that pg_dumpall does not have a -E switch. This has been discussed a bit in the past like here: https://www.postgresql.org/message-id/75e4c42d37e6a74e9fb57c2e9f1300d601070...@tiger.nexperience.com Now it is possible to enforce the encoding

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2017-07-14 Thread Marina Polyakova
I was not convinced by the overall memory management around variables to begin with, and it is even less so with their new copy management. Maybe having a clean "Variables" data structure could help improve the situation. Note that there is something for psql (src/bin/psql/variable.c) which ma

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2017-07-14 Thread Marina Polyakova
On 13-07-2017 19:32, Fabien COELHO wrote: Hello, Hi! [...] I didn't make rollbacks to savepoints after the failure because they cannot help for serialization failures at all: after rollback to savepoint a new attempt will be always unsuccessful. Not necessarily? It depends on where the loc

Re: [HACKERS] Authentication mechanisms categorization

2017-07-14 Thread Michael Paquier
On Fri, Jul 14, 2017 at 12:16 PM, Álvaro Hernández Tortosa wrote: > If the parameter authmethod would rather be "authmethods", i.e., a list, > I think it would be significantly more flexible. Yes, but the handling of a list becomes messier if there are some other connection parameters that ar

Re: [HACKERS] Authentication mechanisms categorization

2017-07-14 Thread Álvaro Hernández Tortosa
On 14/07/17 11:09, Michael Paquier wrote: On Sat, Jul 8, 2017 at 2:19 PM, Álvaro Hernández Tortosa wrote: There has been some prior discussion, that we recently continued at pgday.ru, about what to do if a client wants to use a "strong" authentication mechanism but a rogue server forces

Re: [HACKERS] pg_stop_backup(wait_for_archive := true) on standby server

2017-07-14 Thread Michael Paquier
On Fri, Jul 14, 2017 at 12:34 AM, Stephen Frost wrote: > Michael, all, > > * Michael Paquier (michael.paqu...@gmail.com) wrote: >> On Thu, Jul 13, 2017 at 7:13 AM, Masahiko Sawada >> wrote: >> > Sorry, I missed lots of typo in the last patch. All comments from you >> > are incorporated into the

Re: [HACKERS] Adding support for Default partition in partitioning

2017-07-14 Thread Beena Emerson
Hello, On Thu, Jul 13, 2017 at 1:22 AM, Jeevan Ladhe wrote: > >> - Should probably be merged with the patch to add default partitioning >> for ranges. > > > Beena is already rebasing her patch on my latest patches, so I think getting > them merged here won't be an issue, mostly will be just like

Re: [HACKERS] Default Partition for Range

2017-07-14 Thread Beena Emerson
On Tue, Jul 4, 2017 at 4:21 PM, Rahila Syed wrote: > > Hello Beena, > > Thanks for the updated patch. It passes the basic tests which I performed. > > Few comments: > 1. In check_new_partition_bound(), > >> /* Default case is not handled here */ >>if (spec->is_default) >>

Re: [HACKERS] Default Partition for Range

2017-07-14 Thread Beena Emerson
Thank you for your review Dilip and Rahila. PFA the updated patch which is rebased to Jeevan's latest list partition patch [1] and also handles your comments. https://www.postgresql.org/message-id/CAOgcT0OARciE2X%2BU0rjSKp9VuC279dYcCGkc3nCWKhHQ1_m2rw%40mail.gmail.com -- Beena Emerson Enterpri

Re: [HACKERS] Default Partition for Range

2017-07-14 Thread Beena Emerson
On Mon, Jul 3, 2017 at 8:00 PM, Dilip Kumar wrote: > On Fri, Jun 30, 2017 at 11:56 AM, Beena Emerson > wrote: >> Hello Dilip, >> >> On Wed, Jun 21, 2017 at 6:27 PM, Dilip Kumar wrote: >>> On Tue, Jun 20, 2017 at 6:57 PM, Dilip Kumar wrote: This is basically crashing in RelationBuildPartit

Re: [HACKERS] Authentication mechanisms categorization

2017-07-14 Thread Michael Paquier
On Sat, Jul 8, 2017 at 2:19 PM, Álvaro Hernández Tortosa wrote: > There has been some prior discussion, that we recently continued at > pgday.ru, about what to do if a client wants to use a "strong" > authentication mechanism but a rogue server forces the client to use a > weaker authenticati

Re: [HACKERS] Replacing lfirst() with lfirst_node() appropriately in planner.c

2017-07-14 Thread Ashutosh Bapat
On Thu, Jul 13, 2017 at 9:15 PM, Alvaro Herrera wrote: > Ashutosh Bapat wrote: > >> Happened to stumble across some instances of lfirst() which could use >> lfirst_node() in planner.c. Here's patch which replaces calls to >> lfirst() extracting node pointers by lfirst_node() in planner.c. > > Soun

Re: [HACKERS] PgFDW connection invalidation by ALTER SERVER/ALTER USER MAPPING

2017-07-14 Thread Ashutosh Bapat
On Fri, Jul 14, 2017 at 2:04 PM, Kyotaro HORIGUCHI wrote: > Thank you for the comments. > > At Thu, 13 Jul 2017 16:54:42 +0530, Ashutosh Bapat > wrote in > >> On Thu, Jul 13, 2017 at 2:53 PM, Kyotaro HORIGUCHI >> wrote: >> > Hello, moved to pgsql-hackers. >> > >> > This is the revased and rev

Re: [HACKERS] PgFDW connection invalidation by ALTER SERVER/ALTER USER MAPPING

2017-07-14 Thread Kyotaro HORIGUCHI
Thank you for the comments. At Thu, 13 Jul 2017 16:54:42 +0530, Ashutosh Bapat wrote in > On Thu, Jul 13, 2017 at 2:53 PM, Kyotaro HORIGUCHI > wrote: > > Hello, moved to pgsql-hackers. > > > > This is the revased and revised version of the previous patch. > > > > At Thu, 13 Jul 2017 13:42:49