Dear Alex,
Didn't you mention earlier, that the bloat happens only on
one of your machines? Was this a wrong indication?
If you have a well behaved system, you can increase "maxconnections" to
e.g. 10000 (the parameter denotes the number of connections to be
served by a connection thread before it restarts) and set minthreads to eg.
5. This does not solve the memory-leak problem but should
slowdown the memory hunger, if connection thread restart
causes the growth. I assume, you are using the standard
zippy memory allocator from tcl.
I would not recommend to restart based on the size, since the server
will do this at a time you might not want it (e.g. during a demo). A
nightly reboot is the better option. The size certainly depends on
the amount of tcl code times number of threads and the state management
(use of connection-spanning tcl vars, or use of nsv).
Concerning tcllib and MD5:
tcllib contains actually 3 versions of md5:
a) a tcl-based code
b) a version based on critcl (see tcllib*/md5/md5c.tcl)
c) a version based on trf
During startup, it tries to use the "best" version.
(b) and (c) are accelerated versions using c-code, this might
explain the speed differences you are experiencing. I would
recommend to test first with the tcl-only version and check
the memory growth. If it has no effect, it seems safe to
use the accelerated versions.
Concerning xotcl:
- there is no special handling needed concening memory freeing.
as long you take care, that only those objects survive a connection
that you want. Typical applications load the frequently uses
classes during startup to speed processing of requests.
- xotcl-core of OpenACS has support to ease deallocation
of temporary objects at the end of a connection (e.g.
Object o -destroy_on_cleanup)
- [Class allinstances] returns the classes. To check the objects,
use [Object allinstances]
Concerning Sig 11:
This seems to come from libnssession.so. I am not sure,
how frequently this is used (we don't use it). It seems as
if the source for the crash is there.
Is it possible for you to test without nssession (crashes,
memory consumption)?
-gustaf
Hello Tom, Gustaf
I am still trying to pin down the memory leak problem. Could you
answer few more questions, please?
1. It looks like that nsd process grows ~1.5 - 2.0 MB every time a
thread dies due to timeout or number of connections (and new one
starts).
Changing configuration just makes the bloating process slower of faster.
I restart the nsd process when it reaches 0.5GB in size, but I hope
that's a reasonable number.
I tried valgrind it, but couldn't make any conclusions. Any better
ideas how to debug it?
2. TclLib shall not harm IMHO, it's Tcl- only, no binary code. Please
confirm that.
Besides, I need it only for MD5, and standalone md5 Tcl
implementations for some reason seems to be much slower.
If TclLib is risky, what would be the best package/module for MD5 ?
mhash may be?
3. XoTcl - I verified that I do not leave any objects which I created,
they all destroyed. (at least that's what [Class allinstances]
reports)
However, Gustaf noted that it still may need special handling. What
kind of handling? Could you point me to the code in openacs that does
that?
For me it would be a big hit if I need to rewrite without XoTcl, so I
would like to make it working.
4. Memory leaks aside, nsd quite frequently dies on Sig 11. Sometimes
just a few seconds after being started.
I remember having similar issues some year+ ago, but those were fixed
quickly. Anyone experiences the same?
I didnt' debug it in depth yet, that's what gdb said when I quickly tried it:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1296041040 (LWP 32284)]
0xb7338ba5 in Ns_CookieGet () from
/aolserver_bin/aolserver4.5/lib/libnssession.so
Thanks,
~ Alex.
On Tue, Mar 4, 2008 at 3:29 AM, Gustaf Neumann <[EMAIL PROTECTED]> wrote:
Dear Alex,
The good news is that it bloats only on one server,
so the problem must on that configuration.
There were a couple of memory leaks reported for postgres 7.4,
not sure if these hit you (google around):
http://www.postgresql.org/docs/7.4/static/release-6-5.html
do you have a chance to install on the affected machine
as well postgres 8.2 (maybe in addition to 7.4)?
it would be interesting to see, if you still see a bloat.
i found a small glitch in your "crude script":
The last line of the xotcl install should read:
execute make install-aol
instead of
execute make install
-gustaf neumann
Alex schrieb:
Tom, Gustaf, All
>
> Yes, that happens even if the same page is requested.
> Could you tell me more about pre-queue filters?
>
> The interesting thing is that I only see this memory leak issue on one
> of my servers.
> All servers run x86 Debian, though might be slightly differently
> dist-upgraded, Postgres version differs 7.4 vs 8.2
> The website code itself, nsd config files are exactly the same on all
servers.
>
> Over the weekend I tried to rebuild tcl, tcllib, xotcl, aolserver.
> The details how I did it can be found here
> http://alex-andr.org/notes/tiki-index.php?page=AOLServer+setup+with+XOTcl
> I used latest versions however, tcl 8.4.18, tcllib 1.10, xotcl 1.6.0
>
> I even made a crude script that fetches the sources and builds a
> stand-alone aolserver installation,
> in order to make sure the build results are reproducible.
> http://alex-andr.org/notes/tiki-index.php?page=nsd+setup+script
>
> Still, even when all [hopefully] binaries are being copied to all
> servers, only one of them
> experience the same issue with huge memory leak on each page request.
>
> Any insightful thoughts?
>
> Thanks,
> ~ Alex.
>
>
> On Fri, Feb 29, 2008 at 6:23 PM, Tom Jackson <[EMAIL PROTECTED]> wrote:
>
>> Actually 2MB / request sounds like a massive memory leak. Maybe something
>> isn't getting cleaned up?
>>
>> Question: does this happen if you request the same page over and over?
>>
>> I discovered one way to get a massive leak in 4.5: register a prequeue
filter.
>>
>> Note that both XoTCL and tcllib are compiled third party modules, which is
>> usually where you get memory leaks, not in the modules which have been used
>> with AOLserver for years. My guess would be tcllib, or just failing to
>> release something big.
>>
>> tom jackson
>>
>>
>> On Friday 29 February 2008 11:47, Alex wrote:
>> > What would be a "normal" size of 32 bit nsd process? How can it be
reduced?
>> > One of my servers only has 1G or ram, and I am forced to restart nsd
>> > every so often, when it uses up almost all the memory.
>> > Running 4.5 + XoTCL + tcllib, it seams to grow up to 2MB on each page
>> > request .
>>
>>
>>
>>
>> --
>> AOLserver - http://www.aolserver.com/
>>
>> To Remove yourself from this list, simply send an email to <[EMAIL
PROTECTED]> with the
>> body of "SIGNOFF AOLSERVER" in the email message. You can leave the
Subject: field of your email blank.
>>
>>
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to <[EMAIL
PROTECTED]> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
>
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
