On 21/10/2015 17:35, Dan McDonald wrote:
On Oct 21, 2015, at 6:08 AM, Al Slater <al.sla...@scluk.com>
wrote:
Hi,
I am running omnios r151014 on a couple of machines with a couple
of zones each. 1 zone runs apache as an SSL reverse proxy, the
other runs ILB for load balancing web to app tier connections.
I noticed that in the ILB zone, the ilbd process memory grows to
about 2Gb. Restarting ILB releases the memory, and then the
memory usage gradually increases again, with each memory increase
approximately 2 * the size of the previous one. I run a cronjob
twice a day ( 8am and 8pm) which restarts the ilb service and
releases the memory.
A graph of memory usage is available at
https://www.dropbox.com/s/zaz51apxslnivlq/ILB_Memory_2_days.png?dl=0
>> There are currently 62 rules in the load balancer, with a
>> total
of 664 server/port pairs.
Is there anything I can provide that would help track this down?
You can use svccfg(1M) to enable user-level memory debugging on ilb.
It may cause the ilb daemon to dump core. (And you're just noticing
this in the process, not kernel memory consumption, correct?)
I am seeing kernel memory consumption increasing as well, but that may
be a different issue. The ilbd process memory is definitely growing.
As root:
svcadm disable -t ilb svccfg -s ilb setenv LD_PRELOAD libumem.so
svccfg -s ilb setenv UMEM_DEBUG default svccfg -s ilb refresh svcadm
enable ilb
That should enable user-level memory debugging. If you get a
coredump, save it and share it. If you don't and the ilb daemon
keeps running, eventually please:
gcore `pgrep ilbd`
and share THAT corefile. You can also do this by youself:
mdb <ilbd-core> > ::findleaks
and share ::findleaks.
Once you're done generating corefiles, repeat the steps above, but
use "unsetenv LD_PRELOAD" and "unsetenv UMEM_DEBUG" instead of the
setenv lines.
Thanks Dan. As we are talking about production boxes here, I will have
to try and reproduce on another box and then I will give the process
above a go and see what we come up with.
--
Al Slater
Technical Director
SCL
Phone : +44 (0)1273 666607
Fax : +44 (0)1273 666601
email : al.sla...@scluk.com
Stanton Consultancy Ltd
Park Gate, 161 Preston Road, Brighton, East Sussex, BN1 6AU
Registered in England Company number: 1957652 VAT number: GB 760 2433 55
_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
