Re: [PERFORM] Multiple index builds on same table - in one sweep?

2011-04-11 Thread Scott Marlowe
On Sun, Apr 10, 2011 at 8:29 PM, Greg Smith g...@2ndquadrant.com wrote: On 04/09/2011 01:23 PM, Chris Ruprecht wrote: Maybe, in a future release, somebody will develop something that can create indexes as inactive and have a build tool build and activate them at the same time. Food for

Re: [PERFORM] Slow query postgres 8.3

2011-04-11 Thread Anne Rosset
Hi Thomas, Here is the plan after explain. QUERY PLAN

Re: [PERFORM] Multiple index builds on same table - in one sweep?

2011-04-11 Thread Scott Marlowe
On Sun, Apr 10, 2011 at 11:35 PM, Scott Marlowe scott.marl...@gmail.com wrote: On Sun, Apr 10, 2011 at 8:29 PM, Greg Smith g...@2ndquadrant.com wrote: On 04/09/2011 01:23 PM, Chris Ruprecht wrote: Maybe, in a future release, somebody will develop something that can create indexes as inactive

Re: [PERFORM] Multiple index builds on same table - in one sweep?

2011-04-11 Thread Greg Smith
Scott Marlowe wrote: FYI, in 8.3.13 I get this for all but one index: ERROR: deadlock detected DETAIL: Process 24488 waits for ShareLock on virtual transaction 64/825033; blocked by process 27505. Process 27505 waits for ShareUpdateExclusiveLock on relation 297369165 of database 278059474;

Re: [PERFORM] Slow query postgres 8.3

2011-04-11 Thread Kevin Grittner
Anne Rosset aros...@collab.net wrote: - Index Scan using role_oper_obj_oper on role_operation (cost=0.00..93.20 rows=45 width=9) (actual time=0.236..71.291 rows=6108 loops=1) Index Cond: (((object_type_id)::text =

Re: [PERFORM] Multiple index builds on same table - in one sweep?

2011-04-11 Thread Scott Marlowe
On Mon, Apr 11, 2011 at 1:41 AM, Greg Smith g...@2ndquadrant.com wrote: Scott Marlowe wrote: FYI, in 8.3.13 I get this for all but one index: ERROR:  deadlock detected DETAIL:  Process 24488 waits for ShareLock on virtual transaction 64/825033; blocked by process 27505. Process 27505 waits

[PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Glyn Astill
Hi Guys, I'm just doing some tests on a new server running one of our heavy select functions (the select part of a plpgsql function to allocate seats) concurrently.  We do use connection pooling and split out some selects to slony slaves, but the tests here are primeraly to test what an

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Kevin Grittner
Glyn Astill glynast...@yahoo.co.uk wrote: The new server uses 4 x 8 core Xeon X7550 CPUs at 2Ghz Which has hyperthreading. our current servers are 2 x 4 core Xeon E5320 CPUs at 2Ghz. Which doesn't have hyperthreading. PostgreSQL often performs worse with hyperthreading than without.

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Joshua D. Drake
On Mon, 11 Apr 2011 13:09:15 -0500, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Glyn Astill glynast...@yahoo.co.uk wrote: The new server uses 4 x 8 core Xeon X7550 CPUs at 2Ghz Which has hyperthreading. our current servers are 2 x 4 core Xeon E5320 CPUs at 2Ghz. Which

Re: [PERFORM] Multiple index builds on same table - in one sweep?

2011-04-11 Thread Shaun Thomas
On 04/09/2011 11:28 AM, Chris Ruprecht wrote: I'm wondering if there is a way to build these indexes in parallel while reading the table only once for all indexes and building them all at the same time. Is there an index build tool that I missed somehow, that can do this? I threw together a

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Glyn Astill
--- On Mon, 11/4/11, Joshua D. Drake j...@commandprompt.com wrote: From: Joshua D. Drake j...@commandprompt.com Subject: Re: [PERFORM] Linux: more cores = less concurrency. To: Kevin Grittner kevin.gritt...@wicourts.gov Cc: pgsql-performance@postgresql.org, Glyn Astill

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Scott Marlowe
On Mon, Apr 11, 2011 at 12:12 PM, Joshua D. Drake j...@commandprompt.com wrote: On Mon, 11 Apr 2011 13:09:15 -0500, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Glyn Astill glynast...@yahoo.co.uk wrote: The new server uses 4 x 8 core Xeon X7550 CPUs at 2Ghz Which has hyperthreading.

Re: [PERFORM] Multiple index builds on same table - in one sweep?

2011-04-11 Thread Tom Lane
Scott Marlowe scott.marl...@gmail.com writes: On Mon, Apr 11, 2011 at 1:41 AM, Greg Smith g...@2ndquadrant.com wrote: Scott Marlowe wrote: FYI, in 8.3.13 I get this for all but one index: ERROR:  deadlock detected Is that trying to build them by hand?  The upthread request here is actually

Re: [PERFORM] Multiple index builds on same table - in one sweep?

2011-04-11 Thread Scott Marlowe
On Mon, Apr 11, 2011 at 12:39 PM, Tom Lane t...@sss.pgh.pa.us wrote: Scott Marlowe scott.marl...@gmail.com writes: On Mon, Apr 11, 2011 at 1:41 AM, Greg Smith g...@2ndquadrant.com wrote: Scott Marlowe wrote: FYI, in 8.3.13 I get this for all but one index: ERROR:  deadlock detected Is that

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Scott Marlowe
On Mon, Apr 11, 2011 at 12:23 PM, Glyn Astill glynast...@yahoo.co.uk wrote: --- On Mon, 11/4/11, Joshua D. Drake j...@commandprompt.com wrote: From: Joshua D. Drake j...@commandprompt.com Subject: Re: [PERFORM] Linux: more cores = less concurrency. To: Kevin Grittner

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Glyn Astill
--- On Mon, 11/4/11, Scott Marlowe scott.marl...@gmail.com wrote: Just FYI, in synthetic pgbench type benchmarks, a 48 core AMD Magny Cours with LSI HW RAID and 34 15k6 Hard drives scales almost linearly up to 48 or so threads, getting into the 7000+ tps range.  With SW RAID it gets into

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Steve Clark
On 04/11/2011 02:32 PM, Scott Marlowe wrote: On Mon, Apr 11, 2011 at 12:12 PM, Joshua D. Drakej...@commandprompt.com wrote: On Mon, 11 Apr 2011 13:09:15 -0500, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Glyn Astillglynast...@yahoo.co.uk wrote: The new server uses 4 x 8 core Xeon

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Jesper Krogh
On 2011-04-11 21:42, Glyn Astill wrote: I'll have to try with the synthetic benchmarks next then, but somethings definately going off here. I'm seeing no disk activity at all as they're selects and all pages are in ram. Well, if you dont have enough computations to be bottlenecked on the

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Glyn Astill
--- On Mon, 11/4/11, da...@lang.hm da...@lang.hm wrote: From: da...@lang.hm da...@lang.hm Subject: Re: [PERFORM] Linux: more cores = less concurrency. To: Steve Clark scl...@netwolves.com Cc: Scott Marlowe scott.marl...@gmail.com, Joshua D. Drake j...@commandprompt.com, Kevin Grittner

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread david
On Mon, 11 Apr 2011, Steve Clark wrote: On 04/11/2011 02:32 PM, Scott Marlowe wrote: On Mon, Apr 11, 2011 at 12:12 PM, Joshua D. Drakej...@commandprompt.com wrote: On Mon, 11 Apr 2011 13:09:15 -0500, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Glyn Astillglynast...@yahoo.co.uk wrote:

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Scott Marlowe
On Mon, Apr 11, 2011 at 1:42 PM, Glyn Astill glynast...@yahoo.co.uk wrote: A wild guess is something like multiple cores contending for cpu cache, cpu affinity, or some kind of contention in the kernel, alas a little out of my depth. It's pretty sickening to think I can't get anything else

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Kevin Grittner
Glyn Astill glynast...@yahoo.co.uk wrote: The issue I'm seeing is that 8 real cores outperform 16 real cores, which outperform 32 real cores under high concurrency. With every benchmark I've done of PostgreSQL, the knee in the performance graph comes right around ((2 * cores) +

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Glyn Astill
--- On Mon, 11/4/11, Scott Marlowe scott.marl...@gmail.com wrote: From: Scott Marlowe scott.marl...@gmail.com Subject: Re: [PERFORM] Linux: more cores = less concurrency. To: Glyn Astill glynast...@yahoo.co.uk Cc: Kevin Grittner kevin.gritt...@wicourts.gov, Joshua D. Drake

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread James Cloos
GA == Glyn Astill glynast...@yahoo.co.uk writes: GA I was hoping someone had seen this sort of behaviour before, GA and could offer some sort of explanation or advice. Jesper's reply is probably most on point as to the reason. I know that recent Opterons use some of their cache to better

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Kevin Grittner
Kevin Grittner kevin.gritt...@wicourts.gov wrote: I don't know why you were hitting the knee sooner than I've seen in my benchmarks If you're compiling your own executable, you might try boosting LOG2_NUM_LOCK_PARTITIONS (defined in lwlocks.h) to 5 or 6. The current value of 4 means that

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread David Rees
On Mon, Apr 11, 2011 at 6:04 AM, Glyn Astill glynast...@yahoo.co.uk wrote: The new server uses 4 x 8 core Xeon X7550 CPUs at 2Ghz, our current servers are 2 x 4 core Xeon E5320 CPUs at 2Ghz. What I'm seeing is when the number of clients is greater than the number of cores, the new servers

Re: [PERFORM] Postgres 9 slave lagging

2011-04-11 Thread Merlin Moncure
On Mon, Apr 11, 2011 at 9:10 AM, Saurabh Agrawal saurabh.m...@gmail.com wrote: Hi All, I have setup postgres 9 master slave streaming replication but experiencing slave lagging sometimes by 50 min to 60 min. I am not getting exact reason for slave lag delay. Below are the details: 1. Master

[PERFORM] how explain works

2011-04-11 Thread Radhya sahal
Dear ,all plz could any one help me !!! how explian works as math equations to estimate cost with  constatn query parameters such as cpu_tuple cost ,random page cost ...etc  i want maths  expression  in order to know how these parameters will effect in cost ??? please any one can help me ??    

Re: [PERFORM] how explain works

2011-04-11 Thread Nathan Boley
how explian works as math equations to estimate cost with  constatn query parameters such as cpu_tuple cost ,random page cost ...etc  i want maths  expression  in order to know how these parameters will effect in cost ??? The expressions are complicated, and they are certainly not linear as

Re: [PERFORM] how explain works to Mr Nathan Boley

2011-04-11 Thread Radhya sahal
Thanks Mr Nathan Boley , i want these equations to solve thsese equtions of parameters and total time  in order to get each paramter formula i need these formula  in my experiments is very important to know the rate for each parameter in total cost for plan.  Best Radhya..

[PERFORM] Two servers - One Replicated - Same query

2011-04-11 Thread Ozer, Pam
I have two servers one has replication the other does not. The same query on both servers. One takes 225seconds on the replicated server the first time it runs and only 125ms on the other server the first time it runs. The second time you execute the query it drops to the 125ms. They are using

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread mark
-Original Message- From: pgsql-performance-ow...@postgresql.org [mailto:pgsql-performance- ow...@postgresql.org] On Behalf Of Scott Marlowe Sent: Monday, April 11, 2011 1:29 PM To: Glyn Astill Cc: Kevin Grittner; Joshua D. Drake; pgsql-performance@postgresql.org Subject: Re:

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Scott Marlowe
On Mon, Apr 11, 2011 at 6:05 PM, mark dvlh...@gmail.com wrote: Just wondering, which LSI card ? Was this 32 drives in Raid 1+0 with a two drive raid 1 for logs or some other config? We were using teh LSI but I'll be switching back to Areca when we go back to HW RAID. The LSI only

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Scott Marlowe
On Mon, Apr 11, 2011 at 6:18 PM, Scott Marlowe scott.marl...@gmail.com wrote: On Mon, Apr 11, 2011 at 6:05 PM, mark dvlh...@gmail.com wrote: Just wondering, which LSI card ? Was this 32 drives in Raid 1+0 with a two drive raid 1 for logs or some other config? We were using teh LSI but

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread mark
-Original Message- From: Scott Marlowe [mailto:scott.marl...@gmail.com] Sent: Monday, April 11, 2011 6:18 PM To: mark Cc: Glyn Astill; Kevin Grittner; Joshua D. Drake; pgsql- performa...@postgresql.org Subject: Re: [PERFORM] Linux: more cores = less concurrency. On Mon, Apr 11,

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Merlin Moncure
On Mon, Apr 11, 2011 at 5:06 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Glyn Astill glynast...@yahoo.co.uk wrote: The issue I'm seeing is that 8 real cores outperform 16 real cores, which outperform 32 real cores under high concurrency. With every benchmark I've done of

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Scott Marlowe
On Mon, Apr 11, 2011 at 6:50 PM, mark dvlh...@gmail.com wrote: Interesting, thanks for sharing. I guess I have never gotten to the point where I felt I needed more than 2 drives for my xlogs. Maybe I have been dismissing that as a possibility something. (my biggest array is only 24 SFF

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Jesper Krogh
On 2011-04-11 22:39, James Cloos wrote: GA == Glyn Astillglynast...@yahoo.co.uk writes: GA I was hoping someone had seen this sort of behaviour before, GA and could offer some sort of explanation or advice. Jesper's reply is probably most on point as to the reason. I know that recent

[PERFORM] performance problem with LIMIT (order BY in DESC order). Wrong index used?

2011-04-11 Thread Dieter Rehbein
Hi everybody, I have a performance-problem with a query using a LIMIT. There are other threads rergading performance issues with LIMIT, but I didn't find useful hints for our problem and it might be interesting for other postgres-users. There are only 2 simple tables: CREATE TABLE newsfeed (

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Scott Marlowe
On Mon, Apr 11, 2011 at 7:04 AM, Glyn Astill glynast...@yahoo.co.uk wrote: Hi Guys, I'm just doing some tests on a new server running one of our heavy select functions (the select part of a plpgsql function to allocate seats) concurrently.  We do use connection pooling and split out some