[PERFORM] High CPU Utilization

2009-03-16 Thread Joe Uhl
Our production database is seeing very heavy CPU utilization - anyone have any ideas/input considering the following? CPU utilization gradually increases during the day until it approaches 90%-100% at our peak time. When this happens our transactions/sec drops and our site becomes very slo

Re: [PERFORM] High CPU Utilization

2009-03-16 Thread Alan Hodgson
On Monday 16 March 2009, Joe Uhl wrote: > Right now (not under peak load) this server is running at 68% CPU > utilization and its SATA raid 10 is doing about 2MB/s writes and 11MB/ > s reads. When I run dd I can hit 200+MB/s writes and 230+ MB/s reads, > so we are barely using the available IO.

Re: [PERFORM] High CPU Utilization

2009-03-16 Thread Joe Uhl
Here is vmstat 1 30. We are under peak load right now so I can gather information from the real deal :) Had an almost complete lockup a moment ago, number of non-idle postgres connections was 637. Going to drop our JDBC pool sizes a bit and bounce everything. procs ---memory

Re: [PERFORM] High CPU Utilization

2009-03-16 Thread Greg Smith
On Mon, 16 Mar 2009, Joe Uhl wrote: Here is vmstat 1 30. We are under peak load right now so I can gather information from the real deal Quite helpful, reformatting a bit and picking an informative section: procs ---memory-----swap- io--- -system-- cpu r b

Re: [PERFORM] High CPU Utilization

2009-03-16 Thread Joe Uhl
I dropped the pool sizes and brought things back up. Things are stable, site is fast, CPU utilization is still high. Probably just a matter of time before issue comes back (we get slammed as kids get out of school in the US). Now when I run vmtstat 1 30 it looks very different (below). W

Re: [PERFORM] High CPU Utilization

2009-03-16 Thread Scott Marlowe
On Mon, Mar 16, 2009 at 2:50 PM, Joe Uhl wrote: > I dropped the pool sizes and brought things back up.  Things are stable, > site is fast, CPU utilization is still high.  Probably just a matter of time > before issue comes back (we get slammed as kids get out of school in the > US). Yeah, I'm gue

Re: [PERFORM] High CPU Utilization

2009-03-16 Thread Gregory Stark
Greg Smith writes: > On Mon, 16 Mar 2009, Joe Uhl wrote: > >> Here is vmstat 1 30. We are under peak load right now so I can gather >> information from the real deal > > Quite helpful, reformatting a bit and picking an informative section: > > procs ---memory-----swap- io

Re: [PERFORM] High CPU Utilization

2009-03-16 Thread Greg Smith
On Mon, 16 Mar 2009, Joe Uhl wrote: Now when I run vmtstat 1 30 it looks very different (below). That looks much better. Obviously you'd like some more headroom on the CPU situation than you're seeing, but that's way better than having so much time spent waiting for I/O. max_connections

Re: [PERFORM] High CPU Utilization

2009-03-16 Thread Greg Smith
On Tue, 17 Mar 2009, Gregory Stark wrote: Hm, well the tests I ran for posix_fadvise were actually on a Perc5 -- though who knows if it was the same under the hood -- and I saw better performance than this. I saw about 4MB/s for a single drive and up to about 35MB/s for 15 drives. However this w

Re: [PERFORM] High CPU Utilization

2009-03-20 Thread Joe Uhl
On Mar 17, 2009, at 12:19 AM, Greg Smith wrote: On Tue, 17 Mar 2009, Gregory Stark wrote: Hm, well the tests I ran for posix_fadvise were actually on a Perc5 -- though who knows if it was the same under the hood -- and I saw better performance than this. I saw about 4MB/s for a single drive

Re: [PERFORM] High CPU Utilization

2009-03-20 Thread Scott Marlowe
On Fri, Mar 20, 2009 at 2:26 PM, Joe Uhl wrote: > On Mar 17, 2009, at 12:19 AM, Greg Smith wrote: > >> On Tue, 17 Mar 2009, Gregory Stark wrote: >> >>> Hm, well the tests I ran for posix_fadvise were actually on a Perc5 -- >>> though >>> who knows if it was the same under the hood -- and I saw bet

Re: [PERFORM] High CPU Utilization

2009-03-20 Thread Joe Uhl
On Mar 20, 2009, at 4:29 PM, Scott Marlowe wrote: On Fri, Mar 20, 2009 at 2:26 PM, Joe Uhl wrote: On Mar 17, 2009, at 12:19 AM, Greg Smith wrote: On Tue, 17 Mar 2009, Gregory Stark wrote: Hm, well the tests I ran for posix_fadvise were actually on a Perc5 -- though who knows if it was t

Re: [PERFORM] High CPU Utilization

2009-03-20 Thread Scott Marlowe
On Fri, Mar 20, 2009 at 2:49 PM, Joe Uhl wrote: > > On Mar 20, 2009, at 4:29 PM, Scott Marlowe wrote: >> What does the cs entry on vmstat say at this time?  If you're cs is >> skyrocketing then you're getting a context switch storm, which is >> usually a sign that there are just too many things g

Re: [PERFORM] High CPU Utilization

2009-03-20 Thread Joe Uhl
On Mar 20, 2009, at 4:58 PM, Scott Marlowe wrote: On Fri, Mar 20, 2009 at 2:49 PM, Joe Uhl wrote: On Mar 20, 2009, at 4:29 PM, Scott Marlowe wrote: What does the cs entry on vmstat say at this time? If you're cs is skyrocketing then you're getting a context switch storm, which is usually

Re: [PERFORM] High CPU Utilization

2009-03-24 Thread Joe Uhl
On Mar 20, 2009, at 4:58 PM, Scott Marlowe wrote: On Fri, Mar 20, 2009 at 2:49 PM, Joe Uhl wrote: On Mar 20, 2009, at 4:29 PM, Scott Marlowe wrote: What does the cs entry on vmstat say at this time? If you're cs is skyrocketing then you're getting a context switch storm, which is usually

Re: [PERFORM] High CPU Utilization

2009-03-24 Thread Greg Smith
On Tue, 24 Mar 2009, Joe Uhl wrote: Can anyone recommend a whitebox vendor? I dumped a list of recommended vendors from a discussion here a while back at http://wiki.postgresql.org/wiki/SCSI_vs._IDE/SATA_Disks you could get started with. -- * Greg Smith gsm...@gregsmith.com http://www.greg

Re: [PERFORM] High CPU Utilization

2009-03-24 Thread Scott Marlowe
On Tue, Mar 24, 2009 at 1:29 PM, Greg Smith wrote: > On Tue, 24 Mar 2009, Joe Uhl wrote: > >> Can anyone recommend a whitebox vendor? > > I dumped a list of recommended vendors from a discussion here a while back > at http://wiki.postgresql.org/wiki/SCSI_vs._IDE/SATA_Disks you could get > started

Re: [PERFORM] High CPU Utilization

2009-03-24 Thread Ron
At 02:47 PM 3/24/2009, Joe Uhl wrote: Turns out we may have an opportunity to purchase a new database server with this increased load. Seems that the best route, based on feedback to this thread, is to go whitebox, get quad opterons, and get a very good disk controller. Can anyone recommend

Re: [PERFORM] High CPU Utilization

2009-03-24 Thread Scott Marlowe
On Tue, Mar 24, 2009 at 4:58 PM, Ron wrote: > At 02:47 PM 3/24/2009, Joe Uhl wrote: > >> Turns out we may have an opportunity to purchase a new database server >> with this increased load.  Seems that the best route, based on feedback to >> this thread, is to go whitebox, get quad opterons, and ge

Re: [PERFORM] High CPU Utilization

2009-03-24 Thread Scott Carey
On 3/24/09 4:16 PM, "Scott Marlowe" wrote: > On Tue, Mar 24, 2009 at 4:58 PM, Ron wrote: >> At 02:47 PM 3/24/2009, Joe Uhl wrote: >> >>> Turns out we may have an opportunity to purchase a new database server >>> with this increased load.  Seems that the best route, based on feedback to >>> thi