Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
Hello Josh, So I think that you're confusing the roles of bgwriter vs. spread checkpoint. What you're experiencing above is pretty common for nonspread checkpoints on slow storage (and RAID5 is slow for DB updates, no matter how fast the disks are), or for attempts to do spread checkpoint on

Re: [HACKERS] Escaping from blocked send() reprised.

2014-08-26 Thread Kyotaro HORIGUCHI
Sorry, I was absorbed by other tasks.. Thank you for reviewing thiis. On 07/01/2014 06:26 AM, Kyotaro HORIGUCHI wrote: At Mon, 30 Jun 2014 11:27:47 -0400, Robert Haas robertmh...@gmail.com wrote in CA+TgmoZfcGzAEmtbyoCe6VdHnq085x+ox752zuJ2AKN=wc8...@mail.gmail.com 1. I think it's the

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Jeff Janes
On Monday, August 25, 2014, Fabien COELHO coe...@cri.ensmp.fr wrote: I have not found any mean to force bgwriter to send writes when it can. (Well, I have: create a process which sends CHECKPOINT every 0.2 seconds... it works more or less, but this is not my point:-) There is

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
[oops, wrong from, resent...] Hello Jeff, The culprit I found is bgwriter, which is basically doing nothing to prevent the coming checkpoint IO storm, even though there would be ample time to write the accumulating dirty pages so that checkpoint would find a clean field and pass in a blink.

Re: [HACKERS] Escaping from blocked send() reprised.

2014-08-26 Thread Heikki Linnakangas
On 08/26/2014 09:17 AM, Kyotaro HORIGUCHI wrote: but I don't think we want to define the behavior as usually, pq_terminate_backend() will kill a backend that's blocked on sending to the client, but sometimes you have to call it twice (or more!) to really kill it. I agree that it is desirable

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-08-26 Thread Michael Paquier
On Tue, Aug 26, 2014 at 3:48 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: AFAICS, the namespace can never be NULL in any of these. There is a selectSourceSchema(fout, tbinfo-dobj.namespace-dobj.name) call before or after printing the message, so if tbinfo-dobj.namespace is NULL, you'll

Re: [HACKERS] Escaping from blocked send() reprised.

2014-08-26 Thread Kyotaro HORIGUCHI
Hello, I condiered it but select() frequently (rather in most cases when send() blocks by send buffer exhaustion) fails to predict that following send() will be blocked. (If my memory is correct.) So the final problem would be blocked send()... My point was to put the socket in

[HACKERS] Commitfest status

2014-08-26 Thread Heikki Linnakangas
The first week of the commitfest is now behind us. There are still 15 patches in Needs Review state, with no reviewer assigned. Please pick a patch and review! There are 20 patches in Needs Review state, with a reviewer assigned. If you have signed up as the reviewer, please proceed with the

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
Hello Amit, I think another thing to know here is why exactly checkpoint storm is causing tps to drop so steeply. Yep. Actually it is not strictly 0, but a few tps that I rounded to 0. progress: 63.0 s, 47.0 tps, lat 2.810 ms stddev 5.194, lag 0.354 ms progress: 64.1 s, 11.9 tps, lat

Re: [HACKERS] pg_receivexlog and replication slots

2014-08-26 Thread Michael Paquier
On Tue, Aug 19, 2014 at 2:49 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Mon, Aug 18, 2014 at 4:01 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Mon, Aug 18, 2014 at 3:48 PM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, Aug 18, 2014 at 2:38 PM, Michael Paquier

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
Hello again, I have not found any mean to force bgwriter to send writes when it can. (Well, I have: create a process which sends CHECKPOINT every 0.2 seconds... it works more or less, but this is not my point:-) There is scan_whole_pool_milliseconds, which currently forces bgwriter to circle

Re: [HACKERS] Concurrently option for reindexdb

2014-08-26 Thread Andres Freund
On 2014-08-26 12:44:43 +0900, Michael Paquier wrote: On Tue, Aug 26, 2014 at 12:28 PM, Fujii Masao masao.fu...@gmail.com wrote: +many. Although I'm not sure if we managed to find a safe relation swap. Well we didn't AFAIK. With the latest patch provided I could not really find any whole in

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Andres Freund
On 2014-08-26 08:12:48 +0200, Fabien COELHO wrote: As for checkpoint spreading, raising checkpoint_completion_target to 0.9 degrades the situation (20% of transactions are more than 200 ms late instead of 10%, bgwriter wrote less that 1 page per second, on on 500s run). So maybe there is a bug

Re: [HACKERS] Final Patch for GROUPING SETS - unrecognized node type: 347

2014-08-26 Thread Erik Rijkers
On Mon, August 25, 2014 07:21, Andrew Gierth wrote: Here is the new version of our grouping sets patch. This version supersedes the previous post. The patches did not apply anymore so I applied at 73eba19aebe0. There they applied OK, and make make check was OK. drop table if exists

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
Hello Andres, checkpoint when the segments are full... the server is unresponsive about 10% of the time (one in ten transaction is late by more than 200 ms). That's ext4 I guess? Yes! Did you check whether xfs yields a, err, more predictable performance? No. I cannot test that easily

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Andres Freund
On 2014-08-26 10:25:29 +0200, Fabien COELHO wrote: Did you check whether xfs yields a, err, more predictable performance? No. I cannot test that easily without reinstalling the box. I did some quick tests with ZFS/FreeBSD which seemed to freeze the same, but not in the very same conditions.

Re: [HACKERS] pgbench throttling latency limit

2014-08-26 Thread Rukh Meski
Hi Fabien, On Sun, Aug 24, 2014 at 9:16 AM, Fabien COELHO coe...@cri.ensmp.fr wrote: Find attached a new version: - fix dropped percent computation in the final report - simplify progress report code I have reviewed this patch. Is the patch in a patch format which has context? Yes. Does

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Heikki Linnakangas
On 08/16/2014 02:19 AM, Tom Lane wrote: I think the realistic alternatives at this point are either to switch to all-lengths as in my test patch, or to use the hybrid approach of Heikki's test patch. IMO the major attraction of Heikki's patch is that it'd be upward compatible with existing beta

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
What are the other settings here? checkpoint_segments, checkpoint_timeout, wal_buffers? They simply are the defaults: checkpoint_segments = 3 checkpoint_timeout = 5min wal_buffers = -1 I did some test checkpoint_segments = 1, the problem is just more frequent but shorter. I also

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-08-26 Thread Heikki Linnakangas
On 08/26/2014 10:10 AM, Michael Paquier wrote: On Tue, Aug 26, 2014 at 3:48 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: AFAICS, the namespace can never be NULL in any of these. There is a selectSourceSchema(fout, tbinfo-dobj.namespace-dobj.name) call before or after printing the

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Andres Freund
On 2014-08-26 10:49:31 +0200, Fabien COELHO wrote: What are the other settings here? checkpoint_segments, checkpoint_timeout, wal_buffers? They simply are the defaults: checkpoint_segments = 3 checkpoint_timeout = 5min wal_buffers = -1 I did some test checkpoint_segments = 1,

Re: [HACKERS] Final Patch for GROUPING SETS - unrecognized node type: 347

2014-08-26 Thread Andrew Gierth
Erik == Erik Rijkers e...@xs4all.nl writes: Erik The patches did not apply anymore so I applied at 73eba19aebe0. Erik There they applied OK, and make make check was OK. I'll look and rebase if need be. -- WARNING: unrecognized node type: 347 Can't reproduce this - are you sure it's not a

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Jeff Davis
On Fri, 2014-06-13 at 13:24 +0300, Heikki Linnakangas wrote: Attached is a new patch. It now keeps the current pg_clog unchanged, but adds a new pg_csnlog besides it. pg_csnlog is more similar to pg_subtrans than pg_clog: it's not WAL-logged, is reset at startup, and segments older than

Re: [HACKERS] replicating DROP commands across servers

2014-08-26 Thread Andres Freund
On 2014-06-13 15:50:50 -0400, Alvaro Herrera wrote: Here's a patch implementing the proposed idea. This is used in the Bidirectional Replication stuff by Simon/Andres; it works well. I think there's been some changes to this patch since july, care to resend a new version? I think it's

Re: [HACKERS] pgbench throttling latency limit

2014-08-26 Thread Fabien COELHO
Hello Rukh, I have reviewed this patch. Thanks! [...] I get: pgbench: invalid option -- L Which appears to be caused by the fact that the call to getopt_long() has not been updated to reflect the new parameter. Indeed, I only tested/used it with the --limit= syntax. Also this part: +

Re: [HACKERS] Final Patch for GROUPING SETS - unrecognized node type: 347

2014-08-26 Thread Andrew Gierth
Andrew == Andrew Gierth and...@tao11.riddles.org.uk writes: Erik == Erik Rijkers e...@xs4all.nl writes: Erik The patches did not apply anymore so I applied at 73eba19aebe0. Erik There they applied OK, and make make check was OK. Andrew I'll look and rebase if need be. They apply cleanly

Re: [HACKERS] What in the world is happening with castoroides and protosciurus?

2014-08-26 Thread Dave Page
On Tue, Aug 26, 2014 at 1:46 AM, Tom Lane t...@sss.pgh.pa.us wrote: For the last month or so, these two buildfarm animals (which I believe are the same physical machine) have been erratically failing with errors that reflect low-order differences in floating-point calculations. A recent

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
Uh. I'm not surprised you're facing utterly horrible performance with this. Did you try using a *large* checkpoints_segments setting? To achieve high performance I do not seek high performance per se, I seek lower maximum latency. I think that the current settings and parameters are designed

Re: [HACKERS] pgbench throttling latency limit

2014-08-26 Thread Fabien COELHO
Marking Waiting for Author until these small issues have been fixed. I've put it back to Needs review. Feel free to set it to Ready if it is ok for you. -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] 9.5: Memory-bounded HashAgg

2014-08-26 Thread Heikki Linnakangas
Summary of this thread so far: There was a lot of discussion comparing this with Tomas Vondra's Hash Join patch. The conclusion was that while it would be nice to be able to dump transition state to disk, for aggregates like array_agg, the patch is fine as it is. Dumping transition states

Re: [HACKERS] Selectivity estimation for inet operators

2014-08-26 Thread Emre Hasegeli
I agree with you that we can support other join type and anti join later, If others don’t have any objection in doing other parts later I will mark as Ready For Committer. I updated the patch to cover semi and anti joins with eqjoinsel_semi(). I think it is better than returning a constant.

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Andres Freund
On 2014-08-26 11:34:36 +0200, Fabien COELHO wrote: Uh. I'm not surprised you're facing utterly horrible performance with this. Did you try using a *large* checkpoints_segments setting? To achieve high performance I do not seek high performance per se, I seek lower maximum latency. So? I

Re: Compute attr_needed for child relations (was Re: [HACKERS] inherit support for foreign tables)

2014-08-26 Thread Shigeru Hanada
Hi Fujita-san, I reviewed the v4 patch, and here are some comments from me. * After applying this patch, pull_varattnos() should not called in unnecessary places. For developers who want list of columns-to-be-processed for some purpose, it would be nice to mention when they should use

Re: [HACKERS] Final Patch for GROUPING SETS - unrecognized node type: 347

2014-08-26 Thread Erik Rijkers
On Tue, August 26, 2014 11:13, Andrew Gierth wrote: Andrew == Andrew Gierth and...@tao11.riddles.org.uk writes: Erik == Erik Rijkers e...@xs4all.nl writes: Erik The patches did not apply anymore so I applied at 73eba19aebe0. Erik There they applied OK, and make make check was OK.

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Heikki Linnakangas
On 08/26/2014 12:03 PM, Jeff Davis wrote: On Fri, 2014-06-13 at 13:24 +0300, Heikki Linnakangas wrote: Attached is a new patch. It now keeps the current pg_clog unchanged, but adds a new pg_csnlog besides it. pg_csnlog is more similar to pg_subtrans than pg_clog: it's not WAL-logged, is reset

Re: [HACKERS] replication commands and log_statements

2014-08-26 Thread Fujii Masao
On Wed, Aug 20, 2014 at 1:14 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Aug 20, 2014 at 2:06 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Aug 16, 2014 at 10:27 AM, Amit Kapila amit.kapil...@gmail.com wrote: I think ideally it would have been better if we could

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-08-26 Thread Andres Freund
On 2014-06-29 18:54:50 +0530, Abhijit Menon-Sen wrote: At 2014-06-29 20:35:04 +0900, maumau...@gmail.com wrote: Thanks, I marked it as ready for committer. I hope Fujii san or another committer will commit this, refining English expression if necessary. Since it was just a matter of

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Oskari Saarenmaa
On 25/08/14 14:35, Andres Freund wrote: currently pg_basebackup uses fetch mode when only -x is specified - which imo isn't a very good thing to use due to the increased risk of not fetching everything. How about switching to stream mode for 9.5+? +1. I was just wondering why it's not the

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-08-26 Thread Fabrízio de Royes Mello
On Tue, Aug 26, 2014 at 4:10 AM, Michael Paquier michael.paqu...@gmail.com wrote: On Tue, Aug 26, 2014 at 3:48 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: AFAICS, the namespace can never be NULL in any of these. There is a selectSourceSchema(fout, tbinfo-dobj.namespace-dobj.name)

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-08-26 Thread Petr Jelinek
On 26/08/14 13:20, Andres Freund wrote: I'm looking at committing this, but I wonder: Shouldn't this be accessible from inside psql as well? I.e. as a backslash command? +1 -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-08-26 Thread Pavel Stehule
2014-08-26 13:30 GMT+02:00 Petr Jelinek p...@2ndquadrant.com: On 26/08/14 13:20, Andres Freund wrote: I'm looking at committing this, but I wonder: Shouldn't this be accessible from inside psql as well? I.e. as a backslash command? +1 have you idea about command name? \?+ Pavel

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-08-26 Thread Andres Freund
On 2014-08-26 13:44:16 +0200, Pavel Stehule wrote: 2014-08-26 13:30 GMT+02:00 Petr Jelinek p...@2ndquadrant.com: On 26/08/14 13:20, Andres Freund wrote: I'm looking at committing this, but I wonder: Shouldn't this be accessible from inside psql as well? I.e. as a backslash command?

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-08-26 Thread Fujii Masao
On Tue, Aug 19, 2014 at 6:37 PM, Rahila Syed rahilasye...@gmail.com wrote: Hello, Thank you for comments. Could you tell me where the patch for single block in one run is? Please find attached patch for single block compression in one run. Thanks! I ran the benchmark using pgbench and

Re: [HACKERS] Final Patch for GROUPING SETS - unrecognized node type: 347

2014-08-26 Thread Andrew Gierth
Erik == Erik Rijkers e...@xs4all.nl writes: They apply cleanly for me at 2bde297 whether with git apply or patch, except for the contrib one (which you don't need unless you want to run the contrib regression tests without applying the gsp-u patch). Erik Ah, I had not realised that.

[HACKERS] a7ae1dc has broken the windows builds

2014-08-26 Thread David Rowley
I've attached a small patch which should get the windows builds up and running again. They're currently failing from this: c:\prog\bf\root\HEAD\pgsql.build\pgsql.sln (default target) (1) - c:\prog\bf\root\HEAD\pgsql.build\pg_upgrade_support.vcxproj (default target) (67) - (Link target) -

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-26 Thread David Rowley
On Tue, Aug 5, 2014 at 10:35 PM, David Rowley dgrowle...@gmail.com wrote: Currently most of my changes are in analyzejoin.c, but I did also have to make changes to load the foreign key constraints so that they were available to the planner. One thing that is currently lacking, which would

Re: [HACKERS] Final Patch for GROUPING SETS - unrecognized node type: 347

2014-08-26 Thread Erik Rijkers
On Tue, August 26, 2014 14:24, Andrew Gierth wrote: Erik == Erik Rijkers e...@xs4all.nl writes: They apply cleanly for me at 2bde297 whether with git apply or patch, except for the contrib one (which you don't need unless you want to run the contrib regression tests without applying the

Re: [HACKERS] Patch to add support of IF NOT EXISTS to others CREATE statements

2014-08-26 Thread Heikki Linnakangas
On 04/14/2014 10:31 PM, Fabrízio de Royes Mello wrote: On Tue, Apr 1, 2014 at 2:46 PM, Robert Haas robertmh...@gmail.com wrote: Where this is a bit more interesting is in the case of sequences, where resetting the sequence to zero may cause further inserts into an existing table to fail.

Re: [HACKERS] add line number as prompt option to psql

2014-08-26 Thread Andres Freund
On 2014-08-21 11:43:48 +0900, Sawada Masahiko wrote: On Wed, Aug 20, 2014 at 9:00 PM, Jeevan Chalke jeevan.cha...@enterprisedb.com wrote: Hi, I have reviewed this: I have initialize cur_lineno to UINTMAX - 2. And then observed following behaviour to check wrap-around. postgres=#

Re: [HACKERS] a7ae1dc has broken the windows builds

2014-08-26 Thread Andres Freund
On 2014-08-27 00:25:43 +1200, David Rowley wrote: I've attached a small patch which should get the windows builds up and running again. Pushed, thanks. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training

Re: [HACKERS] Patch to add support of IF NOT EXISTS to others CREATE statements

2014-08-26 Thread Fabrízio de Royes Mello
On Tue, Aug 26, 2014 at 10:20 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 04/14/2014 10:31 PM, Fabrízio de Royes Mello wrote: On Tue, Apr 1, 2014 at 2:46 PM, Robert Haas robertmh...@gmail.com wrote: Where this is a bit more interesting is in the case of sequences, where

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-26 Thread Heikki Linnakangas
On 08/26/2014 03:28 PM, David Rowley wrote: Any ideas or feedback on this would be welcome Before someone spends time reviewing this patch, are you sure this is worth the effort? It seems like very narrow use case to me. I understand removing LEFT and INNER joins, but the case for SEMI and

Re: [HACKERS] replicating DROP commands across servers

2014-08-26 Thread Alvaro Herrera
Andres Freund wrote: On 2014-06-13 15:50:50 -0400, Alvaro Herrera wrote: Here's a patch implementing the proposed idea. This is used in the Bidirectional Replication stuff by Simon/Andres; it works well. I think there's been some changes to this patch since july, care to resend a new

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: On 08/16/2014 02:19 AM, Tom Lane wrote: I think the realistic alternatives at this point are either to switch to all-lengths as in my test patch, or to use the hybrid approach of Heikki's test patch. ... Personally I'd prefer to go to the

Re: [HACKERS] Scaling shared buffer eviction

2014-08-26 Thread Amit Kapila
On Tue, Aug 5, 2014 at 9:21 PM, Robert Haas robertmh...@gmail.com wrote: Incidentally, while I generally think your changes to the locking regimen in StrategyGetBuffer() are going in the right direction, they need significant cleanup. Your patch adds two new spinlocks, freelist_lck and

Re: [HACKERS] Scaling shared buffer eviction

2014-08-26 Thread Tom Lane
Amit Kapila amit.kapil...@gmail.com writes: On Tue, Aug 5, 2014 at 9:21 PM, Robert Haas robertmh...@gmail.com wrote: I think you should get rid of BufFreelistLock completely and just decide that freelist_lck will protect everything the freeNext links, plus everything in StrategyControl except

Re: [HACKERS] [BUGS] BUG #11208: Refresh Materialized View Concurrently bug using user Postgres

2014-08-26 Thread Kevin Grittner
Kevin Grittner kgri...@ymail.com wrote: Kevin Grittner kgri...@ymail.com wrote: bemanuel...@gmail.com bemanuel...@gmail.com wrote: tjma_dw= set role user_dw; tjma_dw= CREATE TABLE foo_data AS SELECT i, md5(random()::text) FROM generate_series(1, 10) i; SELECT 10 tjma_dw= CREATE

Re: [HACKERS] Scaling shared buffer eviction

2014-08-26 Thread Amit Kapila
On Tue, Aug 26, 2014 at 8:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: Amit Kapila amit.kapil...@gmail.com writes: Another point is I think it will be better to protect StrategyControl-completePasses with victimbuf_lck rather than freelist_lck, as when we are going to update it we will already

Re: [HACKERS] pg_dump refactor patch to remove global variables

2014-08-26 Thread Peter Eisentraut
On 8/15/14 7:30 PM, Joachim Wieland wrote: Attached is a patch that doesn't add any new functionality or features, all it does is get rid of the global variables that pg_dump.c is full of. I'm getting a compiler error: In file included from pg_dump.c:60: In file included from

Re: [HACKERS] Per table autovacuum vacuum cost limit behaviour strange

2014-08-26 Thread Alvaro Herrera
Mark Kirkwood wrote: On 06/05/14 16:28, Amit Kapila wrote: On Mon, May 5, 2014 at 11:57 AM, Mark Kirkwood mark.kirkw...@catalyst.net.nz wrote: I could think of 2 ways to change this: a. if user has specified cost_limit value for table, then it just uses it rather than rebalancing

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Magnus Hagander
On Mon, Aug 25, 2014 at 1:35 PM, Andres Freund and...@2ndquadrant.com wrote: Hi, currently pg_basebackup uses fetch mode when only -x is specified - which imo isn't a very good thing to use due to the increased risk of not fetching everything. How about switching to stream mode for 9.5+? I

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Andres Freund
On 2014-08-26 18:40:27 +0200, Magnus Hagander wrote: On Mon, Aug 25, 2014 at 1:35 PM, Andres Freund and...@2ndquadrant.com wrote: Hi, currently pg_basebackup uses fetch mode when only -x is specified - which imo isn't a very good thing to use due to the increased risk of not fetching

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Magnus Hagander
On Tue, Aug 26, 2014 at 6:51 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-26 18:40:27 +0200, Magnus Hagander wrote: On Mon, Aug 25, 2014 at 1:35 PM, Andres Freund and...@2ndquadrant.com wrote: Hi, currently pg_basebackup uses fetch mode when only -x is specified - which

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Jeff Davis
On Tue, 2014-08-26 at 13:45 +0300, Heikki Linnakangas wrote: My current thinking is that access to the csnlog will need to be made faster. Currently, it's just another SLRU, but I'm sure we can do better than that. Or add a backend-private cache on top of it; that might already alleviate it

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Greg Stark
On Tue, Aug 26, 2014 at 11:45 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: It appears that this patch weakens the idea of hint bits. Even if HEAP_XMIN_COMMITTED is set, it still needs to find out if it's in the snapshot. That's fast if the xid is less than snap-xmin, but otherwise it

Re: Compute attr_needed for child relations (was Re: [HACKERS] inherit support for foreign tables)

2014-08-26 Thread Tom Lane
Etsuro Fujita fujita.ets...@lab.ntt.co.jp writes: [ attr_needed-v4.patch ] I looked this over, and TBH I'm rather disappointed. The patch adds 150 lines of dubiously-correct code in order to save ... uh, well, actually it *adds* code, because the places that are supposedly getting a benefit are

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Josh Berkus
On 08/26/2014 07:51 AM, Tom Lane wrote: My feeling about it at this point is that the apparent speed gain from using offsets is illusory: in practically all real-world cases where there are enough keys or array elements for it to matter, costs associated with compression (or rather failure to

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: Anyway, I called for feedback on by blog, and have gotten some: http://www.databasesoup.com/2014/08/the-great-jsonb-tradeoff.html I was hoping you'd get some useful data from that, but so far it seems like a rehash of points made in the on-list thread :-(

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Josh Berkus
On 08/26/2014 11:40 AM, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: Anyway, I called for feedback on by blog, and have gotten some: http://www.databasesoup.com/2014/08/the-great-jsonb-tradeoff.html I was hoping you'd get some useful data from that, but so far it seems like a

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: On 08/26/2014 11:40 AM, Tom Lane wrote: I was hoping you'd get some useful data from that, but so far it seems like a rehash of points made in the on-list thread :-( Unfortunately even the outside commentors don't seem to understand that storage size

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Andres Freund
On 2014-08-26 15:01:27 -0400, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: On 08/26/2014 11:40 AM, Tom Lane wrote: I was hoping you'd get some useful data from that, but so far it seems like a rehash of points made in the on-list thread :-( Unfortunately even the outside

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Laurence Rowe
On 26 August 2014 11:34, Josh Berkus j...@agliodbs.com wrote: On 08/26/2014 07:51 AM, Tom Lane wrote: My feeling about it at this point is that the apparent speed gain from using offsets is illusory: in practically all real-world cases where there are enough keys or array elements for it

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-08-26 15:01:27 -0400, Tom Lane wrote: Yeah, exactly. Given current hardware trends, data compression is becoming more of a win not less as time goes on: CPU cycles are cheap even compared to main memory access, let alone mass storage. So

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Claudio Freire
On Tue, Aug 26, 2014 at 4:01 PM, Tom Lane t...@sss.pgh.pa.us wrote: Josh Berkus j...@agliodbs.com writes: On 08/26/2014 11:40 AM, Tom Lane wrote: I was hoping you'd get some useful data from that, but so far it seems like a rehash of points made in the on-list thread :-( Unfortunately even

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Andres Freund
On 2014-08-26 15:17:13 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-08-26 15:01:27 -0400, Tom Lane wrote: Yeah, exactly. Given current hardware trends, data compression is becoming more of a win not less as time goes on: CPU cycles are cheap even compared

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Jeff Davis
On Tue, 2014-08-26 at 19:25 +0100, Greg Stark wrote: I don't immediately see how to make that practical. One thought would be to have a list of xids in the page header with their corresponding csn -- which starts to sound a lot like Oralce's Interested Transaction List. But I don't see how to

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Peter Geoghegan
On Tue, Aug 26, 2014 at 12:27 PM, Andres Freund and...@2ndquadrant.com wrote: Anyway, that's just to say that I don't really agree that CPU overhead is a worthy price to pay for storage efficiency if the gains are small. +1 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list

Re: [HACKERS] New Model For Role Attributes and Fine Grained Permssions

2014-08-26 Thread Robert Haas
On Thu, Aug 21, 2014 at 10:49 PM, Stephen Frost sfr...@snowman.net wrote: * Heikki Linnakangas (hlinnakan...@vmware.com) wrote: On 08/19/2014 04:27 AM, Brightwell, Adam wrote: This is a proof-of-concept patch for a new model around role attributes and fine grained permissions meant to

Re: [HACKERS] 9.5: Memory-bounded HashAgg

2014-08-26 Thread Jeff Davis
On Tue, 2014-08-26 at 12:39 +0300, Heikki Linnakangas wrote: I think this is enough for this commitfest - we have consensus on the design. For the next one, please address those open items, and resubmit. Agreed, return with feedback. I need to get the accounting patch in first, which needs to

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Josh Berkus
On 08/26/2014 12:27 PM, Andres Freund wrote: Anyway, that's just to say that I don't really agree that CPU overhead is a worthy price to pay for storage efficiency if the gains are small. But in this case the gains aren't small; we're talking up to 60% smaller storage. Testing STORAGE EXTENDED

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
Hello Jeff, The culprit I found is bgwriter, which is basically doing nothing to prevent the coming checkpoint IO storm, even though there would be ample time to write the accumulating dirty pages so that checkpoint would find a clean field and pass in a blink. Indeed, at the end of the 500

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-08-26 Thread Robert Haas
On Fri, Aug 22, 2014 at 2:46 PM, Peter Geoghegan p...@heroku.com wrote: On Fri, Aug 22, 2014 at 7:19 AM, Robert Haas robertmh...@gmail.com wrote: Patch 0002 no longer applies; please rebase. I attach rebased patch. Note that there is currently a bug in the master branch: + if (len2 =

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Greg Stark
On Tue, Aug 26, 2014 at 8:32 PM, Jeff Davis pg...@j-davis.com wrote: We are mixing two kinds of data: user data and visibility information. Each is changed under different circumstances and has different characteristics, and I'm beginning to think they shouldn't be mixed at all. What if we

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-08-26 Thread Peter Geoghegan
On Tue, Aug 26, 2014 at 12:59 PM, Robert Haas robertmh...@gmail.com wrote: I have committed a fix for that problem. Let me know if I missed something else. Yes, that's all I meant. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] [v9.5] Custom Plan API

2014-08-26 Thread Robert Haas
On Fri, Aug 22, 2014 at 9:48 PM, Kohei KaiGai kai...@kaigai.gr.jp wrote: One thing I was pointed out, it is the reason why I implemented DDL support, is that intermediation of c-language function also loads the extension module implicitly. It is an advantage, but not sure whether it shall be

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Jeff Davis
On Tue, 2014-08-26 at 21:00 +0100, Greg Stark wrote: Well fundamentally the reason the visibility information is with the user data is so that we don't need to do two i/os to access the data. The whole point of hint bits is to guarantee that after some amount of time you can read data directly

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-08-26 Thread Robert Haas
On Sat, Aug 23, 2014 at 6:39 PM, Greg Stark st...@mit.edu wrote: On Sat, Aug 23, 2014 at 9:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: Ah. Okay, but then what's wrong with the original proposal of use ceil() instead of floor()? Basically I think the idea of treating fractions less than one

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-08-26 Thread Andres Freund
On 2014-08-26 16:16:32 -0400, Robert Haas wrote: On Sat, Aug 23, 2014 at 6:39 PM, Greg Stark st...@mit.edu wrote: On Sat, Aug 23, 2014 at 9:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: Ah. Okay, but then what's wrong with the original proposal of use ceil() instead of floor()? Basically I

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Peter Eisentraut
On 8/26/14 12:40 PM, Magnus Hagander wrote: I think the first reason is gone now, and the risk/damage of the two connections is probably smaller than running out of WAL. -x is a good default for smaller systems, but -X is a safer one for bigger ones. So I agree that changing the default mode

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Andres Freund
On 2014-08-26 16:41:44 -0400, Peter Eisentraut wrote: On 8/26/14 12:40 PM, Magnus Hagander wrote: I think the first reason is gone now, and the risk/damage of the two connections is probably smaller than running out of WAL. -x is a good default for smaller systems, but -X is a safer one for

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-08-26 Thread Peter Eisentraut
On 8/23/14 6:39 PM, Greg Stark wrote: Or make it an error to specify a value that rounds to 0 but isn't 0. That's what I would prefer. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-08-26 Thread Peter Eisentraut
On 8/26/14 4:22 PM, Andres Freund wrote: Is the whole topic actually practically relevant? It's clearly not all that important, or otherwise we'd have heard about before now. I suppose someone could do something like wal_receiver_status_interval = 10ms and end up silently turning the whole

Re: [HACKERS] Missing comment block at the top of streamutil.h and receivelog.h

2014-08-26 Thread Robert Haas
On Sat, Aug 23, 2014 at 11:59 PM, Michael Paquier michael.paqu...@gmail.com wrote: As mentioned in $subject, the header files in src/bin/pg_basebackup do not have a comment block at the top and do not have any copyright text. Any reason for that? Shouldn't we have something for consistency

Re: [HACKERS] Missing comment block at the top of streamutil.h and receivelog.h

2014-08-26 Thread Magnus Hagander
On Tue, Aug 26, 2014 at 11:03 PM, Robert Haas robertmh...@gmail.com wrote: On Sat, Aug 23, 2014 at 11:59 PM, Michael Paquier michael.paqu...@gmail.com wrote: As mentioned in $subject, the header files in src/bin/pg_basebackup do not have a comment block at the top and do not have any copyright

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Magnus Hagander
On Tue, Aug 26, 2014 at 10:46 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-26 16:41:44 -0400, Peter Eisentraut wrote: On 8/26/14 12:40 PM, Magnus Hagander wrote: I think the first reason is gone now, and the risk/damage of the two connections is probably smaller than running

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-08-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I liked David Johnston's even stronger suggestion upthread: make it an error to specify a value requires rounding of any kind. In other words, if the minimum granularity is 1 minute, you can specify that as 60 seconds instead, but if you write 59

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-08-26 Thread David G Johnston
Tom Lane-2 wrote Robert Haas lt; robertmhaas@ gt; writes: I liked David Johnston's even stronger suggestion upthread: make it an error to specify a value requires rounding of any kind. In other words, if the minimum granularity is 1 minute, you can specify that as 60 seconds instead, but

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Tom Lane
I wrote: I wish it were cache-friendly too, per the upthread tangent about having to fetch keys from all over the place within a large JSON object. ... and while I was typing that sentence, lightning struck. The existing arrangement of object subfields with keys and values interleaved is

Re: [HACKERS] Per table autovacuum vacuum cost limit behaviour strange

2014-08-26 Thread Mark Kirkwood
On 27/08/14 10:27, Alvaro Herrera wrote: Alvaro Herrera wrote: So my proposal is a bit more complicated. First we introduce the notion of a single number, to enable sorting and computations: the delay equivalent, which is the cost_limit divided by cost_delay. Here's a patch that implements

Re: [HACKERS] delta relations in AFTER triggers

2014-08-26 Thread Kevin Grittner
Kevin Grittner kgri...@ymail.com wrote: I think this is approaching a committable state, although I think it should probably be broken down to four separate patches. And here they are. This should net to the same set of changes as the prior post, but the changes are logically separated. They

  1   2   >