Re: [PERFORM] random slow query

2009-06-30 Thread Sean Ma
top - 10:18:58 up 224 days, 15:10, 2 users, load average: 6.27, 7.33, 6 Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie Cpu(s): 5.0%us, 0.7%sy, 0.0%ni, 61.5%id, 32.7%wa, 0.0%hi, 0.1%si, 0 Mem: 32962804k total, 32802612k used, 160192k free, 325360k buffers Swap:

Re: [PERFORM] random slow query

2009-06-30 Thread Sean Ma
Hi Mike, Thanks for the details. Yes, besides another mysql server running on the same server, there is also an homegrown application that frequent read/write the file system. The postgres shared cache is at 4G, is that too big? Thanks Sean On Tue, Jun 30, 2009 at 1:23 PM, Mike

Re: [PERFORM] random slow query

2009-06-30 Thread Scott Marlowe
On Tue, Jun 30, 2009 at 11:49 AM, Sean Masean...@gmail.com wrote: Hi Mike, Thanks for the details. Yes, besides another mysql server running on the same server, there is also an homegrown application that frequent read/write the file system. The postgres shared cache is at 4G, is that too

Re: [PERFORM] random slow query

2009-06-30 Thread Mike Ivanov
Scott Marlowe wrote: The postgres shared cache is at 4G, is that too big? Not for a machine with 32Gig of ram. He could even add some more. Mike -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription:

Re: [PERFORM] random slow query

2009-06-30 Thread Scott Marlowe
On Tue, Jun 30, 2009 at 12:01 PM, Mike Ivanovmi...@activestate.com wrote: Scott Marlowe wrote: The postgres shared cache is at 4G, is that too big? Not for a machine with 32Gig of ram. He could even add some more. Definitely. Really depends on how big his data set is, and how well

Re: [PERFORM] random slow query

2009-06-30 Thread Mike Ivanov
Hi Scott, Well, we can't be sure OP's only got one core. In fact, we can, Sean posted what top -b -n 1 says. There was only one CPU line. the number of cores, it's the IO subsystem is too slow for the load. More cores wouldn't fix that. While I agree on the IO, more cores would

Re: [PERFORM] random slow query

2009-06-30 Thread Alan Hodgson
On Tuesday 30 June 2009, Mike Ivanov mi...@activestate.com wrote: Hi Scott, Well, we can't be sure OP's only got one core. In fact, we can, Sean posted what top -b -n 1 says. There was only one CPU line. Recent versions of top on Linux (on RedHat 5 anyway) may show only one combined CPU

Re: [PERFORM] random slow query

2009-06-30 Thread Mike Ivanov
Scott Marlowe wrote: Also think about it, the OP has 8G of swap and 30Gig cached. How / why would you be caching 30Gigs worth of data when there's only 8G to cache anyway? You're right, I have misread it again :-) Cheers, Mike -- Sent via pgsql-performance mailing list

Re: [PERFORM] random slow query

2009-06-30 Thread Jean-David Beyer
Alan Hodgson wrote: On Tuesday 30 June 2009, Mike Ivanov mi...@activestate.com wrote: Hi Scott, Well, we can't be sure OP's only got one core. In fact, we can, Sean posted what top -b -n 1 says. There was only one CPU line. Recent versions of top on Linux (on RedHat 5 anyway) may show

Re: [PERFORM] random slow query

2009-06-30 Thread Scott Carey
Well, this is going to be a bit redundant but: On 6/30/09 11:22 AM, Mike Ivanov mi...@activestate.com wrote: Hi Scott, Well, we can't be sure OP's only got one core. In fact, we can, Sean posted what top -b -n 1 says. There was only one CPU line. I do not believe that setting means

Re: [PERFORM] random slow query

2009-06-30 Thread Scott Carey
On 6/30/09 12:06 PM, Jean-David Beyer jeandav...@verizon.net wrote: Alan Hodgson wrote: On Tuesday 30 June 2009, Mike Ivanov mi...@activestate.com wrote: Hi Scott, Well, we can't be sure OP's only got one core. In fact, we can, Sean posted what top -b -n 1 says. There was only one CPU

Re: [PERFORM] random slow query

2009-06-30 Thread Scott Carey
On 6/30/09 1:08 PM, Scott Carey sc...@richrelevance.com wrote: A larger shared_buffers size can help if sequential scans are infrequent and kick out pages from the OS page cache. Postgres does not let sequential scans kick out index pages or pages accessed randomly from its buffer cache,

Re: [PERFORM] random slow query

2009-06-30 Thread Mike Ivanov
Scott Carey wrote: 222 / 8 cores = ridiculous 27 processes per core, while the OP has 239 That's not rediculous at all. Modern OS's handle thousands of idle processes just fine. I meant that 27 was a ridiculously small number. Or you can control the behavior with the following kenrnel

Re: [PERFORM] random slow query

2009-06-30 Thread Scott Carey
On 6/30/09 2:39 PM, Mike Ivanov mi...@activestate.com wrote: Scott Carey wrote: 222 / 8 cores = ridiculous 27 processes per core, while the OP has 239 That's not rediculous at all. Modern OS's handle thousands of idle processes just fine. I meant that 27 was a ridiculously small

Re: [PERFORM] random slow query

2009-06-30 Thread Mike Ivanov
Scott Carey wrote: the OS can either quickly allocate to the process or the page cache from the free buffers, or more slowly take from the page cache, or even more slowly page out a process page. Aha, now it all makes sense. I like to use the '5 second rule'. dirty_background_ratio

[PERFORM] random slow query

2009-06-29 Thread Sean Ma
I have a table about 50 million rows. There are a few writers to pump data into the table at the rate of 4 row/hours. Most the time, the SELECT is less than 100 ms. However sometime it is very slow, from 30 seconds to 500 seconds. The database is vacuum analyze regularly. One months ago, this

Re: [PERFORM] random slow query

2009-06-29 Thread Andres Freund
On 06/29/2009 03:33 PM, Sean Ma wrote: I have a table about 50 million rows. There are a few writers to pump data into the table at the rate of 4 row/hours. Most the time, the SELECT is less than 100 ms. However sometime it is very slow, from 30 seconds to 500 seconds. The database is vacuum

Re: [PERFORM] random slow query

2009-06-29 Thread Mike Ivanov
Hi Sean, Sean Ma wrote: One months ago, this type of slow query happened about a few time per day. But recently, the slow query happens more frequent at the rate of once every 10 minutes or less. There seesm not relation to th What is your hardware (memory, CPU type and such)? This seems