On Mon, Jan 26, 2004 at 10:09:20AM -0800, Aaron Bannert wrote:
> On Thu, Jan 15, 2004 at 04:04:38PM +0000, Colm MacCarthaigh wrote:
> > There were other changes co-incidental to that, like going to 12Gb
> > of RAM, which certainly helped, so it's hard to narrow it down too
> > much.
>
> Ok with 18,000 or so child processes (all in the run queue) what does
> your load look like? Also, what kind of memory footprint are you seeing?
At the time, we were seeing a laod of between 8 and 15, varying like a
sawtooth waveform. It would climb and climb, there'd be a steady sharp
decrease and the cycle would star again. At one point I miscompiled in
the Linux pre-empt options into the Kernel, and that made things very
intresting. Load was much more radically in it's mood-swings then.
There would be points when it would slow to a crawl, and the ammount
of data we shipped was down - we only managed to peak at 200Mbit/sec
during the heaviest part of it. Our daily peak is about 380Mbit,
but hopefully we'll be more ready next time. I've managed to commision
the second server, and move the updates to it, see:
http://ftp.heanet.ie/about/
for an idea of the architecture. As for memory footprint, it wasn't
too bad, I actually put the system into 4Gb mode to avoid bounce-buffering
- something I hadn't fully mapped out yet. We were using all of the RAM,
but that's not unusual for us, we aggressively cache as much of the
filesystem as XFS lets us. All of the Apache instances added up to
about 165Mb of RAM.
> > I don't use worker because it still dumps an un-backtracable corefile
> > within about 5 minutes for me. I still have no idea why, though plenty
> > of corefiles. I havn't tried a serious analysis yet, becasue I've been
> > moving house, but I hope to get to it soon. Moving to worker would be
> > a good thing :)
>
> I'd love to find out what's causing your worker failures. Are you using
> any thread-unsafe modules or libraries?
Not to my knowledge, I wasn't planning to do this till later, but
I've bumped to 2.1, I'll try out the forensic_id and backtrace
modules right now, and see how that goes.
--
Colm MacCárthaigh Public Key: [EMAIL PROTECTED]
