Re: [HACKERS] [RFC, POC] Don't require a NBuffer sized PrivateRefCount array of local buffer pins

2014-08-27 Thread Andres Freund
On 2014-08-26 22:19:47 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: The biggest concern previously were some benchmarks. I'm not entirely sure where to get a good testcase for this that's not completely artificial - most simpler testcases don't pin many buffers.

Re: [HACKERS] [RFC, POC] Don't require a NBuffer sized PrivateRefCount array of local buffer pins

2014-08-27 Thread Andres Freund
On 2014-08-26 22:04:03 -0400, Robert Haas wrote: On Tue, Aug 26, 2014 at 7:52 PM, Andres Freund and...@2ndquadrant.com wrote: Here's the next version of this patch. + *much never requried. So we keep a small array of reference counts Typo. But I think you could just drop the

Re: [HACKERS] [RFC, POC] Don't require a NBuffer sized PrivateRefCount array of local buffer pins

2014-08-27 Thread Andres Freund
On 2014-08-26 20:44:32 -0500, Jim Nasby wrote: These results look very encouraging, especially thinking about the cache impact. Yep. I've seen PrivateRefCount array accesses prominently in the source of cache misses in big servers. It occurs to me that it'd also be nice to have some stats

Re: [HACKERS] REINDEX CONCURRENTLY 2.0

2014-08-27 Thread Andres Freund
On 2014-08-27 11:00:56 +0900, Michael Paquier wrote: On Tue, Jan 21, 2014 at 10:12 PM, Michael Paquier michael.paqu...@gmail.com wrote: I have realigned this patch with latest head (d2458e3)... In case someone is interested at some point... Attached is a patch for REINDEX CONCURRENTLY

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 08:23 AM, Jeff Davis wrote: On Tue, 2014-08-26 at 13:45 +0300, Heikki Linnakangas wrote: Yeah. This patch in the current state is likely much much slower than unpatched master, except in extreme cases where you have thousands of connections and short transactions so that without

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 09:40 AM, Heikki Linnakangas wrote: On 08/27/2014 08:23 AM, Jeff Davis wrote: On Tue, 2014-08-26 at 13:45 +0300, Heikki Linnakangas wrote: Yeah. This patch in the current state is likely much much slower than unpatched master, except in extreme cases where you have thousands of

Re: [HACKERS] REINDEX CONCURRENTLY 2.0

2014-08-27 Thread Michael Paquier
On Wed, Aug 27, 2014 at 3:41 PM, Andres Freund and...@2ndquadrant.com wrote: Can you add it to the next CF? I'll try to look earlier, but can't promise anything. I very much would like this to get committed in some form or another. Added it here to keep track of it:

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-27 Thread Magnus Hagander
On Wed, Aug 27, 2014 at 5:16 AM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Aug 27, 2014 at 6:16 AM, Magnus Hagander mag...@hagander.net wrote: On Tue, Aug 26, 2014 at 10:46 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-26 16:41:44 -0400, Peter Eisentraut wrote: On 8/26/14

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 02:26 AM, Kevin Grittner wrote: spi-tuplestore-registry allows tuplestores, with associated name and TupleDesc, to be registered with the current SPI connection. Queries planned or executed on that connection will recognize the name as a tuplestore relation. It doesn't care who is

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Fabien COELHO
Hello Andres, [...] I think you're misunderstanding how spread checkpoints work. Yep, definitely:-) On the other hand I though I was seeking something simple, namely correct latency under small load, that I would expect out of the box. What you describe is reasonable, and is more or less

Re: [HACKERS] Allow multi-byte characters as escape in SIMILAR TO and SUBSTRING

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 08:13 AM, Jeff Davis wrote: On Mon, 2014-08-25 at 17:41 +0300, Heikki Linnakangas wrote: your patch seems to be about 2x-3x as slow as unpatched master. So this needs some optimization. A couple of ideas: I didn't see anywhere near that kind of regression. On unpatched master,

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 03:47 AM, Rukh Meski wrote: Hi Fabien, On Tue, Aug 26, 2014 at 04:07 AM, Fabien COELHO coe...@cri.ensmp.fr wrote: Please find attached a new version which fixes these two points. Indeed it does. Marking the patch ready for a committer. I find the definition of the latency

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 09:32:16 +0200, Fabien COELHO wrote: Hello Andres, [...] I think you're misunderstanding how spread checkpoints work. Yep, definitely:-) On the other hand I though I was seeking something simple, namely correct latency under small load, that I would expect out of the box.

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Fabien COELHO
[...] What's your evidence the pacing doesn't work? Afaik it's the fsync that causes the problem, not the the writes themselves. Hmmm. My (poor) understanding is that fsync would work fine if everything was already written beforehand:-) that is it has nothing to do but assess that all is

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 11:05:52 +0200, Fabien COELHO wrote: I can test a couple of patches. I already did one on someone advice (make bgwriter round all stuff in 1s instead of 120s, without positive effect. I've quickly cobbled together the attached patch (which at least doesn't seem to crash burn). It

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-27 Thread David Rowley
On Wed, Aug 27, 2014 at 1:40 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 08/26/2014 03:28 PM, David Rowley wrote: Any ideas or feedback on this would be welcome Before someone spends time reviewing this patch, are you sure this is worth the effort? It seems like very narrow

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 11:14:46 +0200, Andres Freund wrote: On 2014-08-27 11:05:52 +0200, Fabien COELHO wrote: I can test a couple of patches. I already did one on someone advice (make bgwriter round all stuff in 1s instead of 120s, without positive effect. I've quickly cobbled together the

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Fabien COELHO
Hello Heikki, I find the definition of the latency limit a bit strange. It's a limit on how late a transaction can *start* compared to it's scheduled starting time, not how long a query is allowed to last. Yes. This is what can be done easily with pgbench under throttling. Note that if

[HACKERS] re-reading SSL certificates during server reload

2014-08-27 Thread Alexey Klyukin
Greetings, Is there a strong reason to disallow reloading server key and cert files during the PostgreSQL reload? Basically, once you run multiple databases in a cluster and use different DNS names to connect to different databases (in order for those databases to be moved somewhere without

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 12:41 PM, Fabien COELHO wrote: Hello Heikki, I find the definition of the latency limit a bit strange. It's a limit on how late a transaction can *start* compared to it's scheduled starting time, not how long a query is allowed to last. Yes. This is what can be done easily

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 11:19:22 +0200, Andres Freund wrote: On 2014-08-27 11:14:46 +0200, Andres Freund wrote: On 2014-08-27 11:05:52 +0200, Fabien COELHO wrote: I can test a couple of patches. I already did one on someone advice (make bgwriter round all stuff in 1s instead of 120s, without

Re: [HACKERS] re-reading SSL certificates during server reload

2014-08-27 Thread Magnus Hagander
On Wed, Aug 27, 2014 at 11:56 AM, Alexey Klyukin al...@hintbits.com wrote: Greetings, Is there a strong reason to disallow reloading server key and cert files during the PostgreSQL reload? Key and cert files are loaded in the postmaster. We'd need to change that. I'm not saying that's not a

Re: [HACKERS] Sample LDIF for pg_service.conf no longer works

2014-08-27 Thread Magnus Hagander
On Sat, Aug 16, 2014 at 5:31 AM, Noah Misch n...@leadboat.com wrote: When using pg_service.conf with LDAP, we document[1] the following sample LDIF for populating the LDAP server: version:1 dn:cn=mydatabase,dc=mycompany,dc=com changetype:add objectclass:top objectclass:groupOfUniqueNames

Re: [HACKERS] Missing comment block at the top of streamutil.h and receivelog.h

2014-08-27 Thread Fujii Masao
On Wed, Aug 27, 2014 at 2:07 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Aug 27, 2014 at 12:31 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Aug 27, 2014 at 10:34 AM, Michael Paquier michael.paqu...@gmail.com wrote: I think that it's better to add an include guard like

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Fabien COELHO
Hello Heikki, [...] With a latency limit on when the query should finish, as opposed to how late it can start, it's a lot easier to give a number. For example, your requirements might state that a user must always get a response to a click on a web page in 200 ms, so you set the limit to

Re: [HACKERS] Audit of logout

2014-08-27 Thread Fujii Masao
On Sat, Aug 23, 2014 at 3:44 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Tue, Aug 5, 2014 at 8:04 PM, Fujii Masao masao.fu...@gmail.com wrote: Yep, the attached patch introduces PGC_SU_BACKEND and changes the contexts of log_connections and log_disconnections to PGC_SU_BACKEND. Review?

Re: [HACKERS] Specifying the unit in storage parameter

2014-08-27 Thread Fujii Masao
On Tue, Aug 26, 2014 at 3:27 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao wrote: On Thu, Aug 21, 2014 at 4:20 PM, Michael Paquier michael.paqu...@gmail.com wrote: Looking at the patch, the parameter fillfactor in the category RELOPT_KIND_HEAP (the first element in

[HACKERS] Simplify calls of pg_class_aclcheck when multiple modes are used

2014-08-27 Thread Michael Paquier
Hi all, In a couple of code paths we do the following to check permissions on an object: if (pg_class_aclcheck(relid, userid, ACL_USAGE) != ACLCHECK_OK pg_class_aclcheck(relid, userid, ACL_UPDATE) != ACLCHECK_OK) ereport(ERROR, blah); Wouldn't it be better to simplify that with a single

Re: [HACKERS] After switching primary server while using replication slot.

2014-08-27 Thread Fujii Masao
On Fri, Aug 22, 2014 at 11:29 PM, Andres Freund and...@2ndquadrant.com wrote: Hi, On 2014-08-20 13:14:30 -0400, Robert Haas wrote: On Tue, Aug 19, 2014 at 6:25 AM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, Aug 18, 2014 at 11:16 PM, Sawada Masahiko sawada.m...@gmail.com wrote: Hi

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Fabien COELHO
Hello Amit, I see there is some merit in your point which is to make bgwriter more useful than its current form. I could see 3 top level points to think about whether improvement in any of those can improve the current situation: a. Scanning of buffer pool to find the dirty buffers that

Re: [HACKERS] Parallel Sequence Scan doubts

2014-08-27 Thread Robert Haas
On Thu, Aug 21, 2014 at 2:47 AM, Haribabu Kommi kommi.harib...@gmail.com wrote: Implementation of Parallel Sequence Scan Approach: 1.Parallel Sequence Scan can achieved by using the background workers doing the job of actual sequence scan including the qualification check also. 2. Planner

Re: [HACKERS] Function to know last log write timestamp

2014-08-27 Thread Fujii Masao
On Tue, Aug 19, 2014 at 1:07 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Aug 15, 2014 at 7:17 AM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Aug 15, 2014 at 3:40 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-14 14:37:22 -0400, Robert Haas wrote: On Thu, Aug 14,

Re: [HACKERS] re-reading SSL certificates during server reload

2014-08-27 Thread Stephen Frost
* Magnus Hagander (mag...@hagander.net) wrote: That's certainly an issue. Potentially bigger ones are that you cannot replace an expired certificate or CRL without a restart. +100. I had forgotten about that issue- but it definitely sucks. :( Some of this is going to have to be at least

Re: [HACKERS] Code bug or doc bug?

2014-08-27 Thread Robert Haas
On Sun, Aug 24, 2014 at 6:26 PM, Josh Berkus j...@agliodbs.com wrote: Quoth our docs (http://www.postgresql.org/docs/9.3/static/sql-alterdatabase.html): The fourth form changes the default tablespace of the database. Only the database owner or a superuser can do this; you must also have

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Claudio Freire
On Wed, Aug 27, 2014 at 6:05 AM, Fabien COELHO coe...@cri.ensmp.fr wrote: [...] What's your evidence the pacing doesn't work? Afaik it's the fsync that causes the problem, not the the writes themselves. Hmmm. My (poor) understanding is that fsync would work fine if everything was already

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Claudio Freire
On Wed, Aug 27, 2014 at 10:10 AM, Claudio Freire klaussfre...@gmail.com wrote: On Wed, Aug 27, 2014 at 6:05 AM, Fabien COELHO coe...@cri.ensmp.fr wrote: [...] What's your evidence the pacing doesn't work? Afaik it's the fsync that causes the problem, not the the writes themselves. Hmmm. My

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 10:10:49 -0300, Claudio Freire wrote: On Wed, Aug 27, 2014 at 6:05 AM, Fabien COELHO coe...@cri.ensmp.fr wrote: [...] What's your evidence the pacing doesn't work? Afaik it's the fsync that causes the problem, not the the writes themselves. Hmmm. My (poor) understanding is

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Claudio Freire
On Wed, Aug 27, 2014 at 10:15 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-27 10:10:49 -0300, Claudio Freire wrote: On Wed, Aug 27, 2014 at 6:05 AM, Fabien COELHO coe...@cri.ensmp.fr wrote: [...] What's your evidence the pacing doesn't work? Afaik it's the fsync that causes the

Re: [HACKERS] WIP Patch for GROUPING SETS phase 1

2014-08-27 Thread Robert Haas
On Mon, Aug 25, 2014 at 1:35 AM, Andrew Gierth and...@tao11.riddles.org.uk wrote: If you look at the latest patch post, there's a small patch in it that does nothing but unreserve the keywords and fix ruleutils to make deparse/parse work. The required fix to ruleutils is an example of

Re: [HACKERS] Code bug or doc bug?

2014-08-27 Thread Bruce Momjian
On Wed, Aug 27, 2014 at 09:05:41AM -0400, Robert Haas wrote: Another idea is to have a command that you can run, while connected to a particular database, that updates the default tablespace for that database without actually moving any data on disk - i.e. it sets pg_database.dattablespace,

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 10:17:06 -0300, Claudio Freire wrote: I think a somewhat smarter version of the explicit flushes in the hack^Wpatch I posted nearby is going to more likely to be successful. That path is dangerous (as in, may not work as intended) if the filesystem doesn't properly

Re: [HACKERS] Set new system identifier using pg_resetxlog

2014-08-27 Thread Robert Haas
On Mon, Aug 25, 2014 at 4:06 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I didn't understand this one. But it seems like the obvious solution is to not use the consumer's system identifier as the slot name. Or rename it afterwards. You can't use the consumer's system identifier as

Re: [HACKERS] Code bug or doc bug?

2014-08-27 Thread David G Johnston
Bruce Momjian wrote On Wed, Aug 27, 2014 at 09:05:41AM -0400, Robert Haas wrote: Another idea is to have a command that you can run, while connected to a particular database, that updates the default tablespace for that database without actually moving any data on disk - i.e. it sets

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Claudio Freire
On Wed, Aug 27, 2014 at 10:20 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-27 10:17:06 -0300, Claudio Freire wrote: I think a somewhat smarter version of the explicit flushes in the hack^Wpatch I posted nearby is going to more likely to be successful. That path is dangerous

Re: [HACKERS] ALTER SYSTEM RESET?

2014-08-27 Thread Fujii Masao
On Mon, Aug 25, 2014 at 1:34 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Jul 30, 2014 at 9:11 AM, Amit Kapila amit.kapil...@gmail.com wrote: I have verified the patch and found that it works well for all scenario's. Few minor suggestions: 1. !values to the

Re: [HACKERS] Hardening pg_upgrade

2014-08-27 Thread Robert Haas
On Mon, Aug 25, 2014 at 3:04 PM, Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: OK, I will move in the direction of removing 8.3 support and use a single query to pull schema information. I was hesistant to remove 8.3 support as I know we have kept pg_dump support

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2014-08-27 Thread Robert Haas
On Mon, Aug 25, 2014 at 8:58 AM, Albe Laurenz laurenz.a...@wien.gv.at wrote: Reading the code, I noticed that the pushed down UPDATE or DELETE statement is executed during postgresBeginForeignScan rather than during postgresIterateForeignScan. It probably does not matter, but is there a

Re: [HACKERS] Specifying the unit in storage parameter

2014-08-27 Thread Alvaro Herrera
Fujii Masao wrote: On Tue, Aug 26, 2014 at 3:27 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao wrote: On Thu, Aug 21, 2014 at 4:20 PM, Michael Paquier michael.paqu...@gmail.com wrote: Looking at the patch, the parameter fillfactor in the category RELOPT_KIND_HEAP

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2014-08-27 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Mon, Aug 25, 2014 at 8:58 AM, Albe Laurenz laurenz.a...@wien.gv.at wrote: Reading the code, I noticed that the pushed down UPDATE or DELETE statement is executed during postgresBeginForeignScan rather than during postgresIterateForeignScan. It

Re: [HACKERS] replication commands and log_statements

2014-08-27 Thread Robert Haas
On Tue, Aug 26, 2014 at 7:17 AM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Aug 20, 2014 at 1:14 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Aug 20, 2014 at 2:06 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Aug 16, 2014 at 10:27 AM, Amit Kapila

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2014-08-27 Thread Tom Lane
I wrote: Robert Haas robertmh...@gmail.com writes: Hmm, I'm worried that may be an API contract violation. Indeed it is. You could get away with it if you check the EXEC_FLAG_EXPLAIN_ONLY flag before doing anything with visible side-effects, but it's still pretty ugly. Actually, there's

Re: [HACKERS] [RFC, POC] Don't require a NBuffer sized PrivateRefCount array of local buffer pins

2014-08-27 Thread Alvaro Herrera
Andres Freund wrote: On 2014-08-26 22:19:47 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: I would say that the issue most deserving of performance testing is your sizing of the linear-search array --- it's not obvious that 8 is a good size. It's about the size of

Re: [HACKERS] Hardening pg_upgrade

2014-08-27 Thread Bruce Momjian
On Wed, Aug 27, 2014 at 09:54:11AM -0400, Robert Haas wrote: Having said that, there are obviously advantages for our users if we don't get too crazy about requiring that. I've used products in the past where to get from version 3 to version 11 you have to upgrade from 3 to 5, then 5 to 7,

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Aidan Van Dyk
On Wed, Aug 27, 2014 at 3:32 AM, Fabien COELHO coe...@cri.ensmp.fr wrote: Hello Andres, [...] I think you're misunderstanding how spread checkpoints work. Yep, definitely:-) On the other hand I though I was seeking something simple, namely correct latency under small load, that I would

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 02:37 PM, Fabien COELHO wrote: As for an actual latency limit under throttling, this is significantly more tricky and invasive to implement... ISTM that it would mean: - if the tx is not stated an the latency is already consummed, SKIP++. - if the tx is after its schedule

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 10:32:19 -0400, Aidan Van Dyk wrote: On Wed, Aug 27, 2014 at 3:32 AM, Fabien COELHO coe...@cri.ensmp.fr wrote: Hello Andres, [...] I think you're misunderstanding how spread checkpoints work. Yep, definitely:-) On the other hand I though I was seeking something

[HACKERS] Re: [BUGS] Re: BUG #9555: pg_dump for tables with inheritance recreates the table with the wrong order of columns

2014-08-27 Thread Bruce Momjian
On Mon, Mar 17, 2014 at 07:12:12PM -0400, Noah Misch wrote: On Fri, Mar 14, 2014 at 12:33:04PM -0300, Alvaro Herrera wrote: Tom Lane wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: I wonder if the real fix here is to have ALTER / INHERIT error out of the columns in B are not a

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Fabien COELHO
Hello, If all you want is to avoid the write storms when fsyncs start happening on slow storage, can you not just adjust the kernel vm.dirty* tunables to start making the kernel write out dirty buffers much sooner instead of letting them accumulate until fsyncs force them out all at once? I

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-08-27 Thread Robert Haas
On Tue, Aug 26, 2014 at 8:14 AM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Aug 19, 2014 at 6:37 PM, Rahila Syed rahilasye...@gmail.com wrote: Hello, Thank you for comments. Could you tell me where the patch for single block in one run is? Please find attached patch for single block

Re: [HACKERS] Hardening pg_upgrade

2014-08-27 Thread Robert Haas
On Wed, Aug 27, 2014 at 10:13 AM, Bruce Momjian br...@momjian.us wrote: On Wed, Aug 27, 2014 at 09:54:11AM -0400, Robert Haas wrote: Having said that, there are obviously advantages for our users if we don't get too crazy about requiring that. I've used products in the past where to get from

Re: [HACKERS] Scaling shared buffer eviction

2014-08-27 Thread Robert Haas
On Tue, Aug 26, 2014 at 11:10 AM, Tom Lane t...@sss.pgh.pa.us wrote: Amit Kapila amit.kapil...@gmail.com writes: On Tue, Aug 5, 2014 at 9:21 PM, Robert Haas robertmh...@gmail.com wrote: I think you should get rid of BufFreelistLock completely and just decide that freelist_lck will protect

Re: [HACKERS] Scaling shared buffer eviction

2014-08-27 Thread Robert Haas
On Tue, Aug 26, 2014 at 10:53 AM, Amit Kapila amit.kapil...@gmail.com wrote: Today, while working on updating the patch to improve locking I found that as now we are going to have a new process, we need a separate latch in StrategyControl to wakeup that process. Another point is I think it

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Fabien COELHO
As for an actual latency limit under throttling, this is significantly more tricky and invasive to implement... ISTM that it would mean: [...] Yeah, something like that. I don't think it would be necessary to set statement_timeout, you can inject that in your script or postgresql.conf if

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 06:08 PM, Fabien COELHO wrote: I've submitted this simple lag limit version because being able to measure quickly and simply (un)responsiveness seems like a good idea, especially given the current state of things. Ok, fair enough. I don't think doing a latency limit would be

Re: [HACKERS] [BUGS] Re: BUG #9555: pg_dump for tables with inheritance recreates the table with the wrong order of columns

2014-08-27 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: I looked at this issue from March and I think we need to do something. In summary, the problem is that tables using inheritance can be dumped and reloaded with columns in a different order from the original cluster. Yeah ... this has been a

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-08-27 Thread Arthur Silva
Em 26/08/2014 09:16, Fujii Masao masao.fu...@gmail.com escreveu: On Tue, Aug 19, 2014 at 6:37 PM, Rahila Syed rahilasye...@gmail.com wrote: Hello, Thank you for comments. Could you tell me where the patch for single block in one run is? Please find attached patch for single block

[HACKERS] Missing plpgsql.o Symbols on OS X

2014-08-27 Thread David E . Wheeler
Hackers, I’m trying to build Pavel’s plpgsql_check against the 9.4 beta on OS X 10.9, but get these errors: make gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv

Re: [HACKERS] Similar to csvlog but not really, json logs?

2014-08-27 Thread Greg Sabino Mullane
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Stephen Frost wrote: To try to clarify that a bit, as it comes across as rather opaque even on my re-reading, consider a case where you can't have the credit_card_number field ever exported to an audit or log file, but you're required to

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-08-27 Thread Alvaro Herrera
Thomas Munro wrote: On 25 August 2014 02:57, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Thomas Munro wrote: The difficulty of course will be testing all these racy cases reproducibly... Does this help? http://www.postgresql.org/message-id/51fb4305.3070...@2ndquadrant.com The

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 04:20 PM, Andres Freund wrote: On 2014-08-27 10:17:06 -0300, Claudio Freire wrote: I think a somewhat smarter version of the explicit flushes in the hack^Wpatch I posted nearby is going to more likely to be successful. That path is dangerous (as in, may not work as intended) if

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 19:23:04 +0300, Heikki Linnakangas wrote: On 08/27/2014 04:20 PM, Andres Freund wrote: On 2014-08-27 10:17:06 -0300, Claudio Freire wrote: I think a somewhat smarter version of the explicit flushes in the hack^Wpatch I posted nearby is going to more likely to be successful.

Re: [HACKERS] Similar to csvlog but not really, json logs?

2014-08-27 Thread Jim Nasby
On 8/26/14, 8:45 PM, Michael Paquier wrote: Hi all, As mentioned here, we support multiple logging format: http://www.postgresql.org/docs/devel/static/runtime-config-logging.html Now what about a json format logging with one json object per log entry? A single json entry would need more space

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread Jim Nasby
On 8/27/14, 2:23 AM, Heikki Linnakangas wrote: Does this make sense? In essence, make the relations work like PL/pgSQL variables do. If you squint a little, the new/old relation is a variable from the function's point of view, and a parameter from the planner/executor's point of view. It's

Re: [HACKERS] Similar to csvlog but not really, json logs?

2014-08-27 Thread Andres Freund
On 2014-08-26 23:04:48 -0500, Jim Nasby wrote: On 8/26/14, 8:45 PM, Michael Paquier wrote: Hi all, As mentioned here, we support multiple logging format: http://www.postgresql.org/docs/devel/static/runtime-config-logging.html Now what about a json format logging with one json object per log

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Fabien COELHO
off: $ pgbench -p 5440 -h /tmp postgres -M prepared -c 16 -j16 -T 120 -R 180 -L 200 number of skipped transactions: 1345 (6.246 %) on: $ pgbench -p 5440 -h /tmp postgres -M prepared -c 16 -j16 -T 120 -R 180 -L 200 number of skipped transactions: 1 (0.005 %) That machine is far from idle

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-08-27 Thread Thomas Munro
On 27 August 2014 17:18, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Thomas Munro wrote: On 25 August 2014 02:57, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Thomas Munro wrote: The difficulty of course will be testing all these racy cases reproducibly... Does this help?

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Fabien COELHO
[...] Yeah, something like that. I don't think it would be necessary to set statement_timeout, you can inject that in your script or postgresql.conf if you want. I don't think aborting a transaction that's already started is necessary either. You could count it as LATE, but let it finish

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 19:00:12 +0200, Fabien COELHO wrote: off: $ pgbench -p 5440 -h /tmp postgres -M prepared -c 16 -j16 -T 120 -R 180 -L 200 number of skipped transactions: 1345 (6.246 %) on: $ pgbench -p 5440 -h /tmp postgres -M prepared -c 16 -j16 -T 120 -R 180 -L 200 number of

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-27 Thread Jim Nasby
On 8/26/14, 8:40 AM, Heikki Linnakangas wrote: Just so everyone is on the same page on what kind of queries this helps with, here are some examples from the added regression tests: -- Test join removals for semi and anti joins CREATE TEMP TABLE b (id INT NOT NULL PRIMARY KEY, val INT); CREATE

Re: [HACKERS] Similar to csvlog but not really, json logs?

2014-08-27 Thread Petr Jelinek
On 27/08/14 18:53, Andres Freund wrote: On 2014-08-26 23:04:48 -0500, Jim Nasby wrote: On 8/26/14, 8:45 PM, Michael Paquier wrote: Hi all, As mentioned here, we support multiple logging format: http://www.postgresql.org/docs/devel/static/runtime-config-logging.html Now what about a json

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread David Fetter
On Wed, Aug 27, 2014 at 11:51:40AM -0500, Jim Nasby wrote: On 8/27/14, 2:23 AM, Heikki Linnakangas wrote: Does this make sense? In essence, make the relations work like PL/pgSQL variables do. If you squint a little, the new/old relation is a variable from the function's point of view, and a

Re: [HACKERS] Parallel Sequence Scan doubts

2014-08-27 Thread Jim Nasby
On 8/24/14, 6:22 AM, Haribabu Kommi wrote: Yes, we are mainly targeting CPU-limited sequential scans, Because of this reason only I want the worker to handle the predicates also not just reading the tuples from disk. In that case, I would suggest focusing on parallel execution of conditions

Re: [HACKERS] Function to know last log write timestamp

2014-08-27 Thread Jim Nasby
On 8/27/14, 7:33 AM, Fujii Masao wrote: On Tue, Aug 19, 2014 at 1:07 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Aug 15, 2014 at 7:17 AM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Aug 15, 2014 at 3:40 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-14 14:37:22

Re: [HACKERS] [RFC, POC] Don't require a NBuffer sized PrivateRefCount array of local buffer pins

2014-08-27 Thread Jim Nasby
On 8/27/14, 1:38 AM, Andres Freund wrote: It occurs to me that it'd also be nice to have some stats available on how this is performing; perhaps a dtrace probe for whenever we overflow to the hash table, and one that shows maximum usage for a statement? (Presumably that's not much extra code or

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-27 Thread Tom Lane
Jim Nasby j...@nasby.net writes: On 8/26/14, 8:40 AM, Heikki Linnakangas wrote: Just so everyone is on the same page on what kind of queries this helps with, here are some examples from the added regression tests: -- Test join removals for semi and anti joins CREATE TEMP TABLE b (id INT

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-27 Thread Arthur Silva
On Wed, Aug 27, 2014 at 1:09 AM, Arthur Silva arthur...@gmail.com wrote: It won't be faster by any means, but it should definitely be incorporated if any format changes are made (like Tom already suggested). I think it's important we gather at least 2 more things before making any calls: *

[HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Pavel Stehule
Hi one user asked about using a partitioning for faster aggregates queries. I found so there is not any optimization. create table x1(a int, d date); create table x_1 ( check(d = '2014-01-01'::date)) inherits(x1); create table x_2 ( check(d = '2014-01-02'::date)) inherits(x1); create table x_3

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Merlin Moncure
On Wed, Aug 27, 2014 at 2:07 PM, Pavel Stehule pavel.steh...@gmail.com wrote: Hi one user asked about using a partitioning for faster aggregates queries. I found so there is not any optimization. create table x1(a int, d date); create table x_1 ( check(d = '2014-01-01'::date))

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Claudio Freire
On Wed, Aug 27, 2014 at 4:41 PM, Merlin Moncure mmonc...@gmail.com wrote: On Wed, Aug 27, 2014 at 2:07 PM, Pavel Stehule pavel.steh...@gmail.com wrote: Hi one user asked about using a partitioning for faster aggregates queries. I found so there is not any optimization. create table x1(a

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Pavel Stehule
2014-08-27 21:41 GMT+02:00 Merlin Moncure mmonc...@gmail.com: On Wed, Aug 27, 2014 at 2:07 PM, Pavel Stehule pavel.steh...@gmail.com wrote: Hi one user asked about using a partitioning for faster aggregates queries. I found so there is not any optimization. create table x1(a int,

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Pavel Stehule
2014-08-27 21:46 GMT+02:00 Claudio Freire klaussfre...@gmail.com: On Wed, Aug 27, 2014 at 4:41 PM, Merlin Moncure mmonc...@gmail.com wrote: On Wed, Aug 27, 2014 at 2:07 PM, Pavel Stehule pavel.steh...@gmail.com wrote: Hi one user asked about using a partitioning for faster aggregates

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Merlin Moncure
On Wed, Aug 27, 2014 at 2:46 PM, Claudio Freire klaussfre...@gmail.com wrote: On Wed, Aug 27, 2014 at 4:41 PM, Merlin Moncure mmonc...@gmail.com wrote: On Wed, Aug 27, 2014 at 2:07 PM, Pavel Stehule pavel.steh...@gmail.com wrote: Hi one user asked about using a partitioning for faster

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-08-27 Thread Alvaro Herrera
Thomas Munro wrote: On 27 August 2014 17:18, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Thomas Munro wrote: Yes it does, thanks Alvaro and Craig. I think the attached spec reproduces the problem using that trick, ie shows NOWAIT blocking, presumably in EvalPlanQualFetch (though I

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Tom Lane
Merlin Moncure mmonc...@gmail.com writes: associative bit just makes it easier (which is important of course!). mean for example can be pushed down if the 'pushed down' aggregates return to the count to the reaggregator so that you can weight the final average. that's a lot more complicated

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-08-27 Thread Pavel Stehule
Hi I chose \? xxx, because it is related to psql features. I wrote commands: \? options \? variables comments? Regards Pavel 2014-08-26 13:48 GMT+02:00 Andres Freund and...@2ndquadrant.com: On 2014-08-26 13:44:16 +0200, Pavel Stehule wrote: 2014-08-26 13:30 GMT+02:00 Petr Jelinek

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Pavel Stehule
2014-08-27 22:27 GMT+02:00 Tom Lane t...@sss.pgh.pa.us: Merlin Moncure mmonc...@gmail.com writes: associative bit just makes it easier (which is important of course!). mean for example can be pushed down if the 'pushed down' aggregates return to the count to the reaggregator so that you

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread Kevin Grittner
Heikki Linnakangas hlinnakan...@vmware.com wrote: On 08/27/2014 02:26 AM, Kevin Grittner wrote: spi-tuplestore-registry allows tuplestores, with associated name and TupleDesc, to be registered with the current SPI connection. Queries planned or executed on that connection will recognize the

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread Kevin Grittner
Jim Nasby j...@nasby.net wrote: Something to keep in mind is that users will definitely think about NEW/OLD as tables. I suspect that it won't be long after release before someone asks why they can't create an index on it. :) I'm comfortable saying No to that.  But it's a good point -- I'll

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread Stephen Frost
* Kevin Grittner (kgri...@ymail.com) wrote: Heikki Linnakangas hlinnakan...@vmware.com wrote: In essence, make the relations work like PL/pgSQL variables do. If you squint a little, the new/old relation is a variable from the function's point of view, and a parameter from the

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread Heikki Linnakangas
On 08/28/2014 12:03 AM, Kevin Grittner wrote: Heikki Linnakangas hlinnakan...@vmware.com wrote: I suggest adding a new hook to the ParseState struct, (p_rangevar_hook ?). The planner calls it whenever it sees a reference to a table, and the hook function returns back some sort of placeholder

  1   2   >