Re: [GENERAL] Ridiculous load

2004-12-13 Thread Peter Haworth 
On Sat, 11 Dec 2004 10:44:59 -0600, Wes wrote:
 On 12/9/04 9:23 AM, Peter Haworth [EMAIL PROTECTED] wrote:
 
  It runs RHEL ES v3, kernel 2.4.21-20.ELsmp
  It's generally a very stable box which runs a number of postgresql
  instances.  But last night we hit very high low averages - 10+, vs the
  normal 0-2.
  The culprit appeared to be kswapd, which was using huge amounts of cpu.
  I'd like to know why!
 
 There were some major changes in swap management somewhere in the linux 2.4
 kernel (also RH 3.0 ES).  I don't off hand remember exactly which level.
 Under a heavy I/O load, we also saw kswapd going nuts with the 2.4.21
 kernel, destroying system performance.  The only solution we found was to
 upgrade to a 2.6 kernel.

We've now upgraded to RedHat's 2.4.21-25.ELsmp, which seems to have fixed
things.

-- 
Peter Haworth   [EMAIL PROTECTED]
Back then, I remember wondering how come he accepted my offer instead
 of doing something sensible like, say, running away screaming.
-- Yanick Champoux, on being a Perl Golf referee

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [GENERAL] Ridiculous load

2004-12-11 Thread Wes 
On 12/9/04 9:23 AM, Peter Haworth [EMAIL PROTECTED] wrote:

 It runs RHEL ES v3, kernel 2.4.21-20.ELsmp
 It's generally a very stable box which runs a number of postgresql
 instances.  But last night we hit very high low averages - 10+, vs the
 normal 0-2.
 The culprit appeared to be kswapd, which was using huge amounts of cpu.
 I'd like to know why!

There were some major changes in swap management somewhere in the linux 2.4
kernel (also RH 3.0 ES).  I don't off hand remember exactly which level.
Under a heavy I/O load, we also saw kswapd going nuts with the 2.4.21
kernel, destroying system performance.  The only solution we found was to
upgrade to a 2.6 kernel.  The problem has not reoccurred since then.  Our
database is currently about 67 gigabytes and is growing at about 1 million
records per day.  We are using Postgresql 7.4.x.

Wes


---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

[GENERAL] Ridiculous load

2004-12-10 Thread Peter Haworth 
On Monday we upgraded one of our PostgreSQL instances from v7.2 to v7.4.

Yesterday the box (sabine) on which this runs became very unhappy:

It runs RHEL ES v3, kernel 2.4.21-20.ELsmp
It's generally a very stable box which runs a number of postgresql
instances.  But last night we hit very high low averages - 10+, vs the
normal 0-2.
The culprit appeared to be kswapd, which was using huge amounts of cpu.
I'd like to know why!

We resolved the problem last night via a reboot.  And so far all is well,
but I am concerned that the problem may reoccur.

I wondered if you could help.  In particular the only recent change is the
postgresql upgrade above, so the timing seems suspect.

We used to run two postgresql instances, each with the following
parameters:
Command line was:
/usr/bin/postmaster '-p' '5679' '-i' '-N' '256' '-B' '128000' '-o' '-S
64000'

As we only upgraded one database we split up the 2nd instance into two,
each with the following parameters:
/usr/bin/postmaster '-p' '5679' '-i' '-N' '256' '-B' '64000' '-o' '-S
32000'
(different ports, obviously)

The ram on the box should be generous.  Right now (whilst it's behaving,
and all 3 instances are running) free gives the following output:
sabine% free
~
 total   used   free sharedbuffers cached
Mem:   38573123834676  22636  0  508803339568
-/+ buffers/cache: 4442283413084
Swap:  2096472 3194601777012
sabine%

We plan to revert back to two instances when we upgrade the remaining
databases next week.  Should we change the amount of ram we allocate to
these with the move to v7.4?
Are there other settings we should be altering?

All this is assuming that it's a postgres problem, rather than something
else. Another leading candidate is this, which we are looking into:
  https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=132155
It would be good to eliminate all possible causes, though.


-- 
Peter Haworth   [EMAIL PROTECTED]
Of course, if you were to print(1..Inf), you'd have plenty of time to go and
 get a cup of coffee.  And even then (given the comparatively imminent heat
 death of the universe) that coffee would be really cold before the output
 was complete. So there will probably be a warning when you try to do that.
-- Damian Conway in Exegesis 3

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings