Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-08-19 Thread Fujii Masao
On Tue, Aug 19, 2014 at 2:08 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-18 13:06:15 -0400, Robert Haas wrote: On Mon, Aug 18, 2014 at 7:19 AM, Rahila Syed rahilasye...@gmail.com wrote: According to the measurement result, the amount of WAL generated in Multiple Blocks in one

Re: [HACKERS] 9.5: Memory-bounded HashAgg

2014-08-19 Thread Jeff Davis
On Fri, 2014-08-15 at 13:53 -0400, Robert Haas wrote: I think that's right, and I rather like your (Jeff's) approach. It's definitely true that we could do better if we have a mechanism for serializing and deserializing group states, but (1) I think an awful lot of cases would get an awful

Re: [HACKERS] WAL format and API changes (9.5)

2014-08-19 Thread Michael Paquier
On Mon, Aug 18, 2014 at 10:55 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: All rightey. Here's the next version of this work. It now comes as two patches. The first one refactors adds the XLogReplayBuffer() function and refactors all the redo functions to use it. It doesn't change

Re: [HACKERS] 9.5: Better memory accounting, towards memory-bounded HashAgg

2014-08-19 Thread Jeff Davis
On Thu, 2014-08-14 at 12:52 -0400, Robert Haas wrote: It appears to me that the performance characteristics for this version are not significantly different from version 1. I have not looked at the code. While trying to reproduce your results, I noticed what might be around a 1% regression

Re: [HACKERS] 9.5: Better memory accounting, towards memory-bounded HashAgg

2014-08-19 Thread Jeff Davis
On Sat, 2014-08-16 at 23:09 +0200, Tomas Vondra wrote: But maybe the inheritance really is not necessary - maybe it would be enough to track this per-context, and then just walk through the contexts and collect this. Because my observation is that most of the time is actually spent in walking

Re: [HACKERS] pg_receivexlog and replication slots

2014-08-19 Thread Fujii Masao
On Mon, Aug 18, 2014 at 4:01 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Mon, Aug 18, 2014 at 3:48 PM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, Aug 18, 2014 at 2:38 PM, Michael Paquier michael.paqu...@gmail.com wrote: - IDENTIFY_SYSTEM checks were incorrect (even in HEAD).

Re: [HACKERS] WAL format and API changes (9.5)

2014-08-19 Thread Heikki Linnakangas
On 08/19/2014 11:07 AM, Michael Paquier wrote: Patch 1 looks good. The main difference with the v1 submitted a couple of days back is that the global variable approach is replaced with additional arguments for the LSN position and record pointer in XLogReplayBuffer. I have as well run a couple

Re: [HACKERS] pg_receivexlog and replication slots

2014-08-19 Thread Andres Freund
On 2014-08-18 14:38:06 +0900, Michael Paquier wrote: - IDENTIFY_SYSTEM checks were incorrect (even in HEAD). The following check was done but in 9.4 this command returns 4 fields: (PQntuples(res) != 1 || PQnfields(res) 3) Which is correct. We don't want to error out in the case where 3

Re: [HACKERS] pg_receivexlog and replication slots

2014-08-19 Thread Fujii Masao
On Tue, Aug 19, 2014 at 5:52 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-18 14:38:06 +0900, Michael Paquier wrote: - IDENTIFY_SYSTEM checks were incorrect (even in HEAD). The following check was done but in 9.4 this command returns 4 fields: (PQntuples(res) != 1 ||

Re: [HACKERS] pg_receivexlog and replication slots

2014-08-19 Thread Andres Freund
On 2014-08-19 18:02:32 +0900, Fujii Masao wrote: On Tue, Aug 19, 2014 at 5:52 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-18 14:38:06 +0900, Michael Paquier wrote: - IDENTIFY_SYSTEM checks were incorrect (even in HEAD). The following check was done but in 9.4 this command

Re: [HACKERS] pg_receivexlog and replication slots

2014-08-19 Thread Fujii Masao
On Tue, Aug 19, 2014 at 6:03 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-19 18:02:32 +0900, Fujii Masao wrote: On Tue, Aug 19, 2014 at 5:52 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-18 14:38:06 +0900, Michael Paquier wrote: - IDENTIFY_SYSTEM checks were

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-08-19 Thread Rahila Syed
Hello, Thank you for comments. Could you tell me where the patch for single block in one run is? Please find attached patch for single block compression in one run. Thank you, On Tue, Aug 19, 2014 at 1:17 PM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Aug 19, 2014 at 2:08 AM, Andres

Re: [HACKERS] PoC: Partial sort

2014-08-19 Thread David Rowley
On Tue, Feb 11, 2014 at 7:59 AM, Alexander Korotkov aekorot...@gmail.com wrote: Done. Patch is splitted. I've started to look at this, and for now I'm still finding my way around the patch, so I'm not quite there yet with understanding everything. Never-the-less it seems best to post my

Re: [HACKERS] Reporting the commit LSN at commit time

2014-08-19 Thread Greg Stark
On Tue, Aug 19, 2014 at 1:21 AM, Craig Ringer cr...@2ndquadrant.com wrote: There's plenty of agreement on not a GUC - but what about alternatives? It could be a new protocol message. Currently there are no transaction oriented protocol messages (other than the transaction status in

Re: [HACKERS] Reporting the commit LSN at commit time

2014-08-19 Thread Andres Freund
On 2014-08-19 08:21:10 +0800, Craig Ringer wrote: On 08/19/2014 01:03 AM, Robert Haas wrote: 2. I agree that it's not good to have this get controlled by a GUC. If the behavior change is big enough that it's going to break clients, adding a GUC isn't a sufficient remedy. If it's not,

Re: [HACKERS] After switching primary server while using replication slot.

2014-08-19 Thread Fujii Masao
On Mon, Aug 18, 2014 at 11:16 PM, Sawada Masahiko sawada.m...@gmail.com wrote: Hi all, After switching primary serer while using repliaction slot, the standby server will not able to connect new primary server. Imagine this situation, if primary server has two ASYNC standby servers, also use

Re: [HACKERS] 9.5: Memory-bounded HashAgg

2014-08-19 Thread Tomas Vondra
On 19 Srpen 2014, 9:52, Jeff Davis wrote: On Fri, 2014-08-15 at 13:53 -0400, Robert Haas wrote: I think that's right, and I rather like your (Jeff's) approach. It's definitely true that we could do better if we have a mechanism for serializing and deserializing group states, but (1) I think

Re: [HACKERS] 9.5: Better memory accounting, towards memory-bounded HashAgg

2014-08-19 Thread Tomas Vondra
On 19 Srpen 2014, 10:26, Jeff Davis wrote: On Sat, 2014-08-16 at 23:09 +0200, Tomas Vondra wrote: But maybe the inheritance really is not necessary - maybe it would be enough to track this per-context, and then just walk through the contexts and collect this. Because my observation is that

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2014-08-19 Thread Amit Kapila
On Wed, Aug 13, 2014 at 4:01 PM, Dilip kumar dilip.ku...@huawei.com wrote: On 11 August 2014 10:29, Amit kapila wrote, 5. res = executeQuery(conn, select relname, nspname from pg_class c, pg_namespace ns where (relkind = \'r\' or relkind = \'m\') and

Re: [HACKERS] Reporting the commit LSN at commit time

2014-08-19 Thread Fujii Masao
On Tue, Aug 19, 2014 at 7:21 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-19 08:21:10 +0800, Craig Ringer wrote: On 08/19/2014 01:03 AM, Robert Haas wrote: 2. I agree that it's not good to have this get controlled by a GUC. If the behavior change is big enough that it's going

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2014-08-19 Thread Amit Kapila
On Fri, Aug 15, 2014 at 12:55 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Aug 11, 2014 at 12:59 AM, Amit Kapila amit.kapil...@gmail.com wrote: 1. +Number of parallel connections to perform the operation. This option will enable the vacuum +operation to run on

[HACKERS] Trove with PostgreSQL-XC

2014-08-19 Thread Vivek Singh Raghuwanshi
Hi All, Please let me know is that possible to use Openstack Trove with Postgres-XC. With instances and Baremetal (after Juno Release). I Know it is possible to use other medium like MySQL or PostgreSQL, but i am not sure about XC. -- ViVek Raghuwanshi Mobile -+91-09595950504 Skype -

Re: [HACKERS] Reporting the commit LSN at commit time

2014-08-19 Thread Andres Freund
On 2014-08-19 19:59:51 +0900, Fujii Masao wrote: On Tue, Aug 19, 2014 at 7:21 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-19 08:21:10 +0800, Craig Ringer wrote: On 08/19/2014 01:03 AM, Robert Haas wrote: 2. I agree that it's not good to have this get controlled by a GUC.

Re: [HACKERS] [Postgres-xc-developers] Trove with PostgreSQL-XC

2014-08-19 Thread 鈴木 幸市
XC’s libpq is binary compatible with PG. So as long as Openstack uses libpq to connect to PG, XC should work too. Appreciate if you have a chance to try. Thanks. --- Koichi Suzuki 2014/08/19 20:14、Vivek Singh Raghuwanshi vivekraghuwan...@gmail.commailto:vivekraghuwan...@gmail.com のメール: Hi

Re: [HACKERS] Enable WAL archiving even in standby

2014-08-19 Thread Fujii Masao
On Fri, Aug 15, 2014 at 4:30 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Aug 13, 2014 at 6:42 AM, Fujii Masao masao.fu...@gmail.com wrote: I'd propose the attached WIP patch which allows us to enable WAL archiving even in standby. The patch adds always as the valid value of

[HACKERS] dynahash vs. memory-bounded HashAggregate (hashjoin-style)

2014-08-19 Thread Tomas Vondra
Hi all, while working on a prototype of memory-bounded hash aggregate (alternative to Jeff's patch discussed in [1]), I ran into difficulties when dealing with dynahash. So I'm asking for help ... Some of the difficulties stem from my limited knowledge of dynahash, and complexity in

Re: [HACKERS] [Postgres-xc-developers] Trove with PostgreSQL-XC

2014-08-19 Thread Vivek Singh Raghuwanshi
Thanks, One more question, is this library support multitenancy or we need to launch separate VPC (virtual public cloud) every time for each customer. its good if we have both option. On Tue, Aug 19, 2014 at 5:04 PM, 鈴木 幸市 koi...@intellilink.co.jp wrote: XC’s libpq is binary compatible with

Re: [HACKERS] Reporting the commit LSN at commit time

2014-08-19 Thread Fujii Masao
On Tue, Aug 19, 2014 at 8:22 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-19 19:59:51 +0900, Fujii Masao wrote: On Tue, Aug 19, 2014 at 7:21 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-19 08:21:10 +0800, Craig Ringer wrote: On 08/19/2014 01:03 AM, Robert Haas

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-08-19 Thread Fujii Masao
On Tue, Aug 19, 2014 at 9:52 AM, furu...@pm.nttdata.co.jp wrote: Thank you for updating the patch. I did not get error with applying, and compiling. It works fine. I think this function code has no problem. Could you please submit patch to commit fest app? Thanks for the review! As you

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread MauMau
From: Fujii Masao masao.fu...@gmail.com What's the main purpose of this tool? If it's for WAL archiving, the tool name pg_copy sounds too generic. We already have pg_archivecleanup, so maybe pg_archivecopy or something is better for the consistency? pg_copy in the patch copies the file to the

Re: [HACKERS] proposal for 9.5: monitoring lock time for slow queries

2014-08-19 Thread MauMau
From: Alvaro Herrera alvhe...@2ndquadrant.com Is this supposed to be session-local data, or is it visible from remote sessions too? How durable is it supposed to be? Keep in mind that in case of a crash, all pgstats data is erased. I want it to be visible from other sessions. I'm okay about

Re: [HACKERS] Reporting the commit LSN at commit time

2014-08-19 Thread Craig Ringer
On 08/19/2014 06:21 PM, Andres Freund wrote: What's the problem with the COMMIT WITH (report_lsn on) I've proposed? Reporting the LSN in the command tag? Anything doing transparent failover needs to be aware of transaction boundaries anyway. Tom's objection to a GUC applies there too - a

Re: [HACKERS] Reporting the commit LSN at commit time

2014-08-19 Thread Andres Freund
On 2014-08-19 21:47:26 +0800, Craig Ringer wrote: On 08/19/2014 06:21 PM, Andres Freund wrote: What's the problem with the COMMIT WITH (report_lsn on) I've proposed? Reporting the LSN in the command tag? Anything doing transparent failover needs to be aware of transaction boundaries anyway.

[HACKERS] New PostgreSQL buildfarm client release 4.14 - bug fix for MSVC

2014-08-19 Thread Andrew Dunstan
There is a new release - version 4.14 - of the buildfarm client, now available at http://www.pgbuildfarm.org/downloads/releases/build-farm-4_14.tgz The only change of note is that a bug which only affects MSVC clients (such that the client will not complete a run) and is present in releases

Re: [HACKERS] Enable WAL archiving even in standby

2014-08-19 Thread MauMau
From: Fujii Masao masao.fu...@gmail.com I'd propose the attached WIP patch which allows us to enable WAL archiving even in standby. The patch adds always as the valid value of archive_mode. If it's set to always, the archiver is started when the server is in standby mode and all the WAL files

Re: [HACKERS] WAL format and API changes (9.5)

2014-08-19 Thread Alvaro Herrera
Heikki Linnakangas wrote: Barring objections or better ideas, I'm leaning towards XLogReadBufferForRedo. WFM -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list

Re: [HACKERS] WAL format and API changes (9.5)

2014-08-19 Thread Andres Freund
On 2014-08-19 10:33:29 -0400, Alvaro Herrera wrote: Heikki Linnakangas wrote: Barring objections or better ideas, I'm leaning towards XLogReadBufferForRedo. WFM for me too. Although we could imo strip the 'XLog' in the beginning if we want to make it shorter. The ForRedo is saying that

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Stephen Frost
* Heikki Linnakangas (hlinnakan...@vmware.com) wrote: server_cert_valid: Did the server present a valid certificate? yes or no server_cert_matches_host: Does the Common Name of the certificate match the host connected to? yes or no Aren't these questions addressed by sslmode? Exposing

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Andres Freund
On 2014-08-19 10:48:41 -0400, Stephen Frost wrote: Exposing the SSL information as generic key/value pairs allows adding more attributes in the future, without breaking the ABI, and it also allows exposing implementation-specific information in a generic way. The attributes listed above

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Magnus Hagander
On Tue, Aug 19, 2014 at 4:48 PM, Stephen Frost sfr...@snowman.net wrote: * Heikki Linnakangas (hlinnakan...@vmware.com) wrote: server_cert_valid: Did the server present a valid certificate? yes or no server_cert_matches_host: Does the Common Name of the certificate match the host

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: On 2014-08-19 10:48:41 -0400, Stephen Frost wrote: At first blush, I'd say a whole bunch.. Off the top of my head I can think of: [...] I'm not really sure we need all that. We're not building a general ssl library abstraction here.

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-08-19 Thread Alvaro Herrera
Abhijit Menon-Sen wrote: At 2014-06-29 20:35:04 +0900, maumau...@gmail.com wrote: Thanks, I marked it as ready for committer. I hope Fujii san or another committer will commit this, refining English expression if necessary. Since it was just a matter of editing, I went through the

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Magnus Hagander
On Tue, Aug 19, 2014 at 5:05 PM, Stephen Frost sfr...@snowman.net wrote: * Andres Freund (and...@2ndquadrant.com) wrote: On 2014-08-19 10:48:41 -0400, Stephen Frost wrote: At first blush, I'd say a whole bunch.. Off the top of my head I can think of: [...] I'm not really sure we need

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Stephen Frost
* Magnus Hagander (mag...@hagander.net) wrote: On Tue, Aug 19, 2014 at 4:48 PM, Stephen Frost sfr...@snowman.net wrote: Aren't these questions addressed by sslmode? Not entirely. You can have sslmode=require and have a matching certificate. You don't *have* to have sslmode=verify-full for

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Heikki Linnakangas
On 08/19/2014 05:48 PM, Stephen Frost wrote: * Heikki Linnakangas (hlinnakan...@vmware.com) wrote: server_cert_valid: Did the server present a valid certificate? yes or no server_cert_matches_host: Does the Common Name of the certificate match the host connected to? yes or no Aren't

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Andres Freund
On 2014-08-19 11:05:07 -0400, Stephen Frost wrote: * Andres Freund (and...@2ndquadrant.com) wrote: On 2014-08-19 10:48:41 -0400, Stephen Frost wrote: At first blush, I'd say a whole bunch.. Off the top of my head I can think of: [...] I'm not really sure we need all that. We're

Re: [HACKERS] [Postgres-xc-developers] Trove with PostgreSQL-XC

2014-08-19 Thread Craig Ringer
I replied to the XC-list (only) to ask them to discontinue cross-posting this thread. Replying here just so you know. On 08/19/2014 07:46 PM, Vivek Singh Raghuwanshi wrote: Thanks, One more question, is this library support multitenancy or we need to launch separate VPC (virtual public cloud)

Re: [HACKERS] Hokey wrong versions of libpq in apt.postgresql.org

2014-08-19 Thread Craig Ringer
On 08/19/2014 05:02 AM, Joshua D. Drake wrote: I am finally able to get back around to this and I am still calling: Hokey. I just loaded up a fresh precise (I assume trusty will act the same way) and installed postgresql. I installed it, without the PDGD repository and everything worked

Re: [HACKERS] pg_dump refactor patch to remove global variables

2014-08-19 Thread Craig Ringer
On 08/19/2014 01:40 AM, Robert Haas wrote: Attached is a patch that doesn't add any new functionality or features, all it does is get rid of the global variables that pg_dump.c is full of. I think this is an excellent idea. It's also one small step toward library-ifying pg_dump. Huge +1.

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Stephen Frost
* Heikki Linnakangas (hlinnakan...@vmware.com) wrote: I think it would be nice to be able to query those attributes explicitly, rather than just expect libpq to reject the connection if something's wrong. For example, I'm thinking that an interactive client might present an annoying pop-up

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: No. We should build something that's suitable for postgres, not something general. We'll fail otherwise. For anything fancy the user has to look at the certificate themselves. We should make it easy to get at the whole certificate chain in a

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Andres Freund
On 2014-08-19 11:52:37 -0400, Stephen Frost wrote: * Andres Freund (and...@2ndquadrant.com) wrote: No. We should build something that's suitable for postgres, not something general. We'll fail otherwise. For anything fancy the user has to look at the certificate themselves. We should make

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Alvaro Herrera
Stephen Frost wrote: * Heikki Linnakangas (hlinnakan...@vmware.com) wrote: Indeed, the ODBC driver only uses libpq for authentication, then calls PQgetssl(), and takes over the whole show calling SSL_read() and SSL_write() itself. Ideally, we'd modify psqlodbc to stop doing that, but

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: Per Apache's documentation, mod_ssl and mod_gnutls support the same set of environment variables (with the same names even), so I don't buy this argument either. Gnutls is quite similar from what it provides to openssl. That's not saying

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Stephen Frost
* Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: Stephen Frost wrote: Yeah, that's what I remembered. There was an attempt to make that change at one point, but it was reverted due to the lack of batching ability in libpq (without resorting to cursors, as I recall...), requiring double

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Heikki Linnakangas
On 08/19/2014 06:44 PM, Stephen Frost wrote: Hmm. That seems a bit too much. Perhaps provide just the certificate itself in DER/PEM format, and have the client parse it (using OpenSSL or something else) if it wants more details. I really don't care for that approach. Our SSL support has always

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Heikki Linnakangas
On 08/19/2014 07:10 PM, Alvaro Herrera wrote: Stephen Frost wrote: * Heikki Linnakangas (hlinnakan...@vmware.com) wrote: Indeed, the ODBC driver only uses libpq for authentication, then calls PQgetssl(), and takes over the whole show calling SSL_read() and SSL_write() itself. Ideally, we'd

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes: * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: Um, libpq has recently gained the ability to return result fragments, right? Those didn't exist when libpq-ification of odbc was attempted, as I recall -- perhaps it's possible now. I was trying to

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Heikki Linnakangas
On 08/19/2014 06:52 PM, Stephen Frost wrote: * Andres Freund (and...@2ndquadrant.com) wrote: No. We should build something that's suitable for postgres, not something general. We'll fail otherwise. For anything fancy the user has to look at the certificate themselves. We should make it easy to

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Heikki Linnakangas
On 08/19/2014 06:00 PM, Magnus Hagander wrote: On Tue, Aug 19, 2014 at 4:48 PM, Stephen Frost sfr...@snowman.net wrote: * Heikki Linnakangas (hlinnakan...@vmware.com) wrote: server_cert_valid: Did the server present a valid certificate? yes or no server_cert_matches_host: Does the

Re: [HACKERS] replication commands and log_statements

2014-08-19 Thread Robert Haas
On Sat, Aug 16, 2014 at 10:27 AM, Amit Kapila amit.kapil...@gmail.com wrote: I think ideally it would have been better if we could have logged replication commands under separate log_level, but as still there is no consensus on extending log_statement and nobody is even willing to pursue, it

Re: [HACKERS] bad estimation together with large work_mem generates terrible slow hash joins

2014-08-19 Thread Robert Haas
On Sat, Aug 16, 2014 at 9:31 AM, Tomas Vondra t...@fuzzy.cz wrote: On 12.8.2014 00:30, Tomas Vondra wrote: On 11.8.2014 20:25, Robert Haas wrote: It also strikes me that when there's only 1 batch, the set of bits that map onto the batch number is zero-width, and one zero-width bit range is as

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-08-19 Thread Rahila Syed
So, it seems like you're basically using malloc to work around the fact that a palloc failure is an error, and we can't throw an error in a critical section. I don't think that's good; we want all of our allocations, as far as possible, to be tracked via palloc. It might be a good idea to add a

Re: [HACKERS] bad estimation together with large work_mem generates terrible slow hash joins

2014-08-19 Thread Tomas Vondra
On 19.8.2014 19:05, Robert Haas wrote: On Sat, Aug 16, 2014 at 9:31 AM, Tomas Vondra t...@fuzzy.cz wrote: On 12.8.2014 00:30, Tomas Vondra wrote: On 11.8.2014 20:25, Robert Haas wrote: It also strikes me that when there's only 1 batch, the set of bits that map onto the batch number is

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: Stephen Frost sfr...@snowman.net writes: * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: Um, libpq has recently gained the ability to return result fragments, right? Those didn't exist when libpq-ification of odbc was attempted, as I recall --

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Robert Haas
On Mon, Aug 18, 2014 at 7:54 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: In order to support alternatives to OpenSSL, we need to wean off applications from using PQgetssl(). To do that, we have to provide an alternative API to get the same information. PQgetSSL() returns a pointer

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Magnus Hagander
On Tue, Aug 19, 2014 at 8:49 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Aug 18, 2014 at 7:54 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: In order to support alternatives to OpenSSL, we need to wean off applications from using PQgetssl(). To do that, we have to provide an

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: On Tue, Aug 19, 2014 at 8:49 PM, Robert Haas robertmh...@gmail.com wrote: I have a hard time believing that something like this will really satisfy anyone. Why not just add PQgetSchannelHandleOrWhatever() and call it good? We can try to be

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Magnus Hagander
On Tue, Aug 19, 2014 at 9:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Tue, Aug 19, 2014 at 8:49 PM, Robert Haas robertmh...@gmail.com wrote: I have a hard time believing that something like this will really satisfy anyone. Why not just add

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Stephen Frost
* Heikki Linnakangas (hlinnakan...@vmware.com) wrote: On 08/19/2014 06:44 PM, Stephen Frost wrote: Hmm. That seems a bit too much. Perhaps provide just the certificate itself in DER/PEM format, and have the client parse it (using OpenSSL or something else) if it wants more details. I really

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Stephen Frost
* Heikki Linnakangas (hlinnakan...@vmware.com) wrote: I think you just packed up the goalposts for a one-way trip to Mars, but I wonder: What would you consider proper SSL support? What exactly are we missing? I hit on a few things in my other email, but there is a huge portion of SSL which is

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Robert Haas
On Tue, Aug 19, 2014 at 3:16 PM, Magnus Hagander mag...@hagander.net wrote: On Tue, Aug 19, 2014 at 9:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Tue, Aug 19, 2014 at 8:49 PM, Robert Haas robertmh...@gmail.com wrote: I have a hard time believing

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Peter Eisentraut
On 8/19/14 9:35 AM, MauMau wrote: pg_copy is for copying a file reliably by syncing. If sync is not necessary, people can use cp/copy. I'm getting mixed messages from this thread. I think there could be a fair amount of support for a new tool that can serve as a universal plug-and-play

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Peter Eisentraut
On 8/15/14 10:46 AM, Fujii Masao wrote: At last, the big question is, is there really no OS command which provides the same functionality as pg_copy does? If there is, I'd like to avoid duplicate work basically. If you look hard enough, you can maybe find an OS command that can fsync a file

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: BTW, if we're beating on libpq, I wonder if we shouldn't consider bumping the soversion at some point. I mean, I know that we technically don't need to do that if we're only *adding* functions and not changing any of the existing stuff in

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Josh Berkus
On 08/19/2014 12:37 PM, Peter Eisentraut wrote: On 8/19/14 9:35 AM, MauMau wrote: pg_copy is for copying a file reliably by syncing. If sync is not necessary, people can use cp/copy. I'm getting mixed messages from this thread. I think there could be a fair amount of support for a new

Re: [HACKERS] 9.3: more problems with Could not open file pg_multixact/members/xxxx

2014-08-19 Thread Jeff Janes
On Tue, Jul 15, 2014 at 3:58 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Fri, Jun 27, 2014 at 11:51 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Jeff Janes wrote: This problem was initially fairly easy to reproduce, but since I started adding instrumentation specifically to

Re: [HACKERS] 9.3: more problems with Could not open file pg_multixact/members/xxxx

2014-08-19 Thread Andres Freund
On August 19, 2014 10:24:20 PM CEST, Jeff Janes jeff.ja...@gmail.com wrote: On Tue, Jul 15, 2014 at 3:58 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Fri, Jun 27, 2014 at 11:51 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Jeff Janes wrote: This problem was initially fairly easy to

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Alvaro Herrera
MauMau wrote: With that said, copying to a temporary file like dest.tmp and renaming it to dest sounds worthwhile even as a basic copy utility. I want to avoid copying to a temporary file with a fixed name like _copy.tmp, because some advanced utility may want to run multiple instances of

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Heikki Linnakangas
On 08/19/2014 10:31 PM, Robert Haas wrote: On Tue, Aug 19, 2014 at 3:16 PM, Magnus Hagander mag...@hagander.net wrote: On Tue, Aug 19, 2014 at 9:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Tue, Aug 19, 2014 at 8:49 PM, Robert Haas

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-08-19 Thread Heikki Linnakangas
On 08/20/2014 12:17 AM, John Lumby wrote: I am attaching a new version of the patch for consideration in the current commit fest. Thanks for working on this! Relative to the one I submitted on 25 June in bay175-w412ff89303686022a9f16aa3...@phx.gbl the method for handling aio completion

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Andres Freund
On 2014-08-20 00:58:22 +0300, Heikki Linnakangas wrote: I don't much like adding a separate function for every SSL implementation, but you've got a point that it would be nice to make it difficult to call PQgetSSLstruct() and just assume that the returned struct is e.g an OpenSSL struct, while

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-08-19 Thread Claudio Freire
On Tue, Aug 19, 2014 at 7:27 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Also, please split prefetching of regular index scans into a separate patch. It's orthogonal to doing async I/O; we could prefetch regular index scans with posix_fadvise too, and AIO should be useful for

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-08-20 00:58:22 +0300, Heikki Linnakangas wrote: I don't much like adding a separate function for every SSL implementation, but you've got a point that it would be nice to make it difficult to call PQgetSSLstruct() and just assume that the

Re: [HACKERS] PQgetssl() and alternative SSL implementations

2014-08-19 Thread Andres Freund
On 2014-08-19 19:11:46 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-08-20 00:58:22 +0300, Heikki Linnakangas wrote: I don't much like adding a separate function for every SSL implementation, but you've got a point that it would be nice to make it difficult to

[HACKERS] KNN searches support for SP-GiST [GSOC'14]

2014-08-19 Thread Vladislav Sterzhanov
Hi there, pg-Hackers! Here I go with the patch which brings up the possibility to perform nearest-neighbour searches on SP-GiSTs (as of now includes implementation for quad and kd trees). Pre-reviewed by my GSoC mentor Alexander Korotkov. Sample benchmarking script included in the attachment

Re: [HACKERS] GSoC on WAL-logging hash indexes

2014-08-19 Thread Michael Paquier
On Thu, Jun 19, 2014 at 6:40 PM, Vik Fearing vik.fear...@dalibo.com wrote: On 04/30/2014 11:41 PM, Tom Lane wrote: We really oughta fix the WAL situation, not just band-aid around it. After re-reading this thread, it is not clear that anyone is going to work on it so I'll just ask: Is

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-19 Thread Josh Berkus
On 08/15/2014 04:19 PM, Tom Lane wrote: Personally I'd prefer to go to the all-lengths approach, but a large part of that comes from a subjective assessment that the hybrid approach is too messy. Others might well disagree. In case anyone else wants to do measurements on some more data

Re: [HACKERS] wrapping in extended mode doesn't work well with default pager

2014-08-19 Thread Noah Misch
On Mon, Aug 18, 2014 at 12:30:40PM +0100, Greg Stark wrote: On Tue, Aug 5, 2014 at 3:41 AM, Noah Misch n...@leadboat.com wrote: This remains open for 9.4. Your proposal to revert the feature in 9.4 and fix it in 9.5 sounds reasonable. Ok, I've gone ahead and done this. I'm sorry for

Re: [HACKERS] replication commands and log_statements

2014-08-19 Thread Michael Paquier
On Wed, Aug 20, 2014 at 2:06 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Aug 16, 2014 at 10:27 AM, Amit Kapila amit.kapil...@gmail.com wrote: I think ideally it would have been better if we could have logged replication commands under separate log_level, but as still there is no

Re: [HACKERS] Hokey wrong versions of libpq in apt.postgresql.org

2014-08-19 Thread Joshua D. Drake
On 08/19/2014 08:34 AM, Craig Ringer wrote: I iterate, the current apt.postgresql.org is not doing things correctly. It breaks things and it shouldn't. FWIW, this is inconsistent with what yum.postgresql.org does - it takes the POLA approach of packing the libpq from the major release

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-08-19 Thread Michael Paquier
On Fri, Jul 25, 2014 at 4:45 AM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: Given this is a very small and simple patch I thought it's not necessary... Added to the next CommitFest. I had a look at this patch, and here are a couple of comments: 1) Depending on how ArchiveEntry is