On Wed, Jul 19, 2006 at 04:58:59PM +1000, Carlo Sogono wrote:
> Does RHEL or Linux in general limit the amount of memory being used by a
> single process?
It can do, via setrlimit(2) (or see ulimit in bash(1) for the Limits for
Dummies version).
> If so, how do I get around this?
You don't, without finding a bug to exploit. Limits tend to be there for a
good reason.
> After an application performing a lot of small mallocs (160 bytes each)
> nears about 1 GB worth of allocated memory, RHEL seems to be slowing down
> the allocation speed of the application. In the first maybe 5 million
> mallocs it can do about 100,000 mallocs per second, however after more
> than 1 GB worth it slows down to just a few thousand per second. CPU
> consumption also slowed down from ~12% (1 CPU full load) to just about
> 1-3%.
If there's an RSS limit in place, then you can see that with ulimit -m.
According to setrlimit, though, that was only available before 2.4.30, so
unless you're running Ye Olde RHEL (my commiserations) then that ain't it.
All the other memory-oriented limits I can find with cause ENOMEM failures
if you exceed them, so it's unlikely to be any of those.
It's possible, though unlikely, that there's some sort of scaling issue that
you're hitting with all of those little mallocs, in that once you exceed
some internal limit the allocation management algorithm starts performing
poorly. I say that's unlikely simply because that sort of thing just isn't
done any more (poor allocation management, that is), and I'd expect the
performance to drop off a bit more gradually.
What I suspect might be happening (and which is supported by the top output
you gave; see below) is that you're hitting swap and the system is grinding
to a halt. What's really odd is that you've still got umpteen GB of free
memory -- I'd expect the system to use that before hitting swap. Maybe your
top output is anomalous, though.
> Is there something I can do on Linux or RHEL, or maybe something else I
> should do in my coding? Some stats...
"Don't allocate so much memory" seems like the best strategy. If you have
to, try to do it in bigger blocks, so that you'll waste less memory in
allocation metadata. Don't try to write your own block allocator to save
memory, though; it's unlikely you'll do better than glibc at saving memory,
but you might be able to get some good savings if you can (for instance)
using big-arse arrays instead of allocating every instance of the struct
separately.
Swap evidence:
> CPU states: cpu user nice system irq softirq iowait idle
> total 0.4% 0.0% 0.2% 0.0% 0.0% 12.1% 87.0%
> cpu00 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
> cpu01 0.9% 0.0% 0.0% 0.0% 0.0% 0.0% 99.0%
> cpu02 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
> cpu03 0.0% 0.0% 1.9% 0.0% 0.0% 0.0% 98.0%
> cpu04 2.9% 0.0% 0.0% 0.0% 0.0% 97.0% 0.0%
^^^^^
Massive iowait is a very common sign of massive swapping. It's not the only
possible cause, but it's certainly the most common IME...
> Mem: 9879976k av, 2391196k used, 7488780k free, 0k shrd, 35592k
> buff
> 2009064k actv, 134928k in_d, 83160k in_c
> Swap: 2044072k av, 1022036k used, 1022036k free 246204k
... especially when you're 1G into hock.
The odd bit is the 7.4GB of free memory that's sitting there. I can't
imagine why the system would be shoving stuff into swap when there was large
gobs of available memory, but it's possible that a *very* large process just
terminated when you snapped this output. I'd look at top output more
closely. There's also various options to sar (-R, -W) that'll show you how
quickly pages are being brought in and out.
- Matt
--
Talk about unlucky. D'you know, if I fell in a barrel of tits I'd come out
sucking me thumb.
-- Seen on the 'net:
http://thelawwestofealingbroadway.blogspot.com/2006/01/bang-to-rights.html
_______________________________________________
coders mailing list
[email protected]
http://lists.slug.org.au/listinfo/coders
