To sum up what I have learned so far and also for the benefit of other
programmers:
* The reason why malloc was slowing down was because Linux was paging
only after allocating a mere 1-2GB even with another 7GB of physical
memory free.
* The reason why Linux was paging was because it was running on NUMA
hardware with NUMA kernel support on RHEL. With NUMA support enabled,
Linux will not treat the NUMA memory nodes as "equals" and will try to
match them with certain close proximity processors. If a memory node
gets full, the kernel (at least 2.4) will force the process to page. In
effect, the largest memory available to a process is the memory size of
a particular NUMA memory node. This is probably a limitation of the NUMA
implementation of RHEL3.
* To force a process not to page a particular memory segment, I
discovered one can use the mlock() or mlockall() system calls. However,
when I did this on a NUMA kernel, I got "out of memory" errors after
just 2GB because my memory node was getting exhausted. To fix this, I
disabled NUMA in the kernel which means Linux will treat them as one
entity regardless of whether you have NUMA hardware (seperate buses for
memory nodes). Performance will be slower for accessing remote physical
nodes but will fix my particular problem.
* I have chosen to use mlockall() over Matt's madvice() suggestion
because we really have to work with our data in near-real-time (we're in
the SMS/MMS gateway business). I have also done research on madvice and
learned that in Linux it's more like a command than an advice compared
to other UNIX systems. However using mlockall won't be a problem since
we will running it as root in a totally isolated network.
* After devouring a few NUMA sites I have learned that it's probably
just my kernel that's causing my memory allocation problems. While I had
NUMA generally enabled, I had NUMA-aware memory allocation disabled. I
have really enjoyed reading this paper:
http://archive.linuxsymposium.org/ols2003/Proceedings/All-Reprints/Reprint-Gaughen-OLS2003.pdf
and it explains quite clearly what was going on. NUMA seems to be an
interesting research field. And as I have told my peers time and again I
should have probably taken up computer engineering and not computer
science. =P
* I did not know much about the system I'm working on as it's a one of a
few remote development machines assigned to us by our client. Had I
known it was NUMA earlier I would have came up with a solution sooner.
* If I have more time I will look into fine tuning NUMA and maybe trying
out some of the NUMA API's...might be something there that I can use to
my advantage.
Thank you to all that helped! Have a good weekend!
Carlo
_______________________________________________
coders mailing list
[email protected]
http://lists.slug.org.au/listinfo/coders
