Re: [OmniOS-discuss] ILB memory leak?

2015-11-10 Thread Al Slater
On 10/11/15 15:26, Dan McDonald wrote: > >> On Nov 10, 2015, at 2:50 AM, Al Slater wrote: >> >> On 10/11/2015 07:40, Al Slater wrote: >>> It seems to me that ilbd_run_probe just needs to call >>> posix_spawn_file_actions_destroy appropriately. >> >> And probably posix_spawnattr_destroy as well? >

Re: [OmniOS-discuss] ILB memory leak?

2015-11-10 Thread Dan McDonald
> On Nov 10, 2015, at 2:50 AM, Al Slater wrote: > > On 10/11/2015 07:40, Al Slater wrote: >> It seems to me that ilbd_run_probe just needs to call >> posix_spawn_file_actions_destroy appropriately. > > And probably posix_spawnattr_destroy as well? Wow! Great catch. I'll bet a small sum you n

Re: [OmniOS-discuss] ILB memory leak?

2015-11-09 Thread Al Slater
On 10/11/2015 07:40, Al Slater wrote: It seems to me that ilbd_run_probe just needs to call posix_spawn_file_actions_destroy appropriately. And probably posix_spawnattr_destroy as well? -- Al Slater Technical Director SCL Phone : +44 (0)1273 07 Fax : +44 (0)1273 01 email : al.sla..

Re: [OmniOS-discuss] ILB memory leak?

2015-11-09 Thread Al Slater
On 06/11/2015 18:33, Bob Friesenhahn wrote: Tracing mmap() calls on the program while is is running might reveal something. Tracing mmap calls with dtrace gives me the following stack, called roughly the same number of times as there are doubled-in-size anon mappings in the mmap output.

Re: [OmniOS-discuss] ILB memory leak?

2015-11-09 Thread Al Slater
On 09/11/15 15:43, Dan McDonald wrote: > >> On Nov 9, 2015, at 8:39 AM, Al Slater wrote: >> >> Attached is a compressed file with 5hrs or so of 10s pmaps. >> Hopefully not too big for the list. > > It compressed nicely. I'm noticing a pattern: > > Mon Nov 9 08:21:45 UTC 2015 total Kb 134008

Re: [OmniOS-discuss] ILB memory leak?

2015-11-09 Thread Dan McDonald
> On Nov 9, 2015, at 8:39 AM, Al Slater wrote: > > Attached is a compressed file with 5hrs or so of 10s pmaps. Hopefully not > too big for the list. It compressed nicely. I'm noticing a pattern: Mon Nov 9 08:21:45 UTC 2015 total Kb 134008 133504 131416 - Mon Nov 9 08:50:21 UTC 20

Re: [OmniOS-discuss] ILB memory leak?

2015-11-09 Thread Al Slater
Hi Dan, On 06/11/2015 18:31, Dan McDonald wrote: You said you had a test box, right? Yes. Can you: - Disable UMEM_DEBUG - RESTART the service. - IMMEDIATELY after restart do pmap, and do pmap once per (sec, 10 sec, something) to see how it grows? Attached is a compressed file with 5hrs o

Re: [OmniOS-discuss] ILB memory leak?

2015-11-06 Thread Bob Friesenhahn
On Fri, 6 Nov 2015, Dan McDonald wrote: More huge anonymous mappings (1G, 512MB, 256MB, 128MB). I don't know pmap as well as I should. I don't see anything in the man page to give me further insight into why these chunks of memory are being eaten. It is pretty common for memory allocators

Re: [OmniOS-discuss] ILB memory leak?

2015-11-06 Thread Dan McDonald
> On Nov 6, 2015, at 11:25 AM, Dan McDonald wrote: > >> On Nov 6, 2015, at 10:57 AM, Al Slater wrote: >> >> >> 7D80 1048576 1048576 1048576 - rwx--[ anon ] >> BDA0 524288 524288 524288 - rwx--[ anon ] >> DDC0 262144 262144 262144 - rwx--[ anon ]

Re: [OmniOS-discuss] ILB memory leak?

2015-11-06 Thread Dan McDonald
> On Nov 6, 2015, at 10:57 AM, Al Slater wrote: > > > 7D80 1048576 1048576 1048576 - rwx--[ anon ] > BDA0 524288 524288 524288 - rwx--[ anon ] > DDC0 262144 262144 262144 - rwx--[ anon ] > EDE0 131072 131072 131072 - rwx--[ anon ]

Re: [OmniOS-discuss] ILB memory leak?

2015-11-06 Thread Al Slater
On 06/11/15 15:43, Dan McDonald wrote: > >> On Nov 6, 2015, at 10:38 AM, Al Slater wrote: >> >> 3D60 1048576 1048576 1048576 - rwx--[ anon ] >> 7D80 1048576 1048576 1048576 - rwx--[ anon ] > > Hmmm. > > I wonder if those are pre-allocated by libumem? They're HUGE (1

Re: [OmniOS-discuss] ILB memory leak?

2015-11-06 Thread Dan McDonald
> On Nov 6, 2015, at 10:38 AM, Al Slater wrote: > > 3D60 1048576 1048576 1048576 - rwx--[ anon ] > 7D80 1048576 1048576 1048576 - rwx--[ anon ] Hmmm. I wonder if those are pre-allocated by libumem? They're HUGE (1G) and more around them are smaller Do you think y

Re: [OmniOS-discuss] ILB memory leak?

2015-11-06 Thread Al Slater
On 06/11/15 14:51, Dan McDonald wrote: > >> On Nov 6, 2015, at 9:39 AM, Dan McDonald wrote: >> >> Lots of LARGE anonymous mappings. I wonder why that happened? I'll dig into >> that a bit more. > > pmap(1) works even better on running processes. Could you run, say "pmap -xa > `pgrep ilbd`" o

Re: [OmniOS-discuss] ILB memory leak?

2015-11-06 Thread Dan McDonald
> On Nov 6, 2015, at 9:39 AM, Dan McDonald wrote: > > Lots of LARGE anonymous mappings. I wonder why that happened? I'll dig into > that a bit more. pmap(1) works even better on running processes. Could you run, say "pmap -xa `pgrep ilbd`" on your running machine? Dan

Re: [OmniOS-discuss] ILB memory leak?

2015-11-06 Thread Dan McDonald
> On Nov 6, 2015, at 3:11 AM, Al Slater wrote: > > On 05/11/2015 14:57, Dan McDonald wrote: >> >>> On Nov 5, 2015, at 6:38 AM, Al Slater wrote: >>> >>> I have the 4Gb core file. Is there anything useful I can extract from >>> it to try and spot where the problem is? >> >> Your one ::findlea

Re: [OmniOS-discuss] ILB memory leak?

2015-11-06 Thread Al Slater
On 05/11/2015 14:57, Dan McDonald wrote: On Nov 5, 2015, at 6:38 AM, Al Slater wrote: I have the 4Gb core file. Is there anything useful I can extract from it to try and spot where the problem is? Your one ::findleaks showed nothing. Did your 4GB corefile have ::findleaks show nothing as

Re: [OmniOS-discuss] ILB memory leak?

2015-11-05 Thread Al Slater
Hi Dan, On 05/11/2015 14:57, Dan McDonald wrote: On Nov 5, 2015, at 6:38 AM, Al Slater wrote: I have the 4Gb core file. Is there anything useful I can extract from it to try and spot where the problem is? Your one ::findleaks showed nothing. Did your 4GB corefile have ::findleaks show no

Re: [OmniOS-discuss] ILB memory leak?

2015-11-05 Thread Dan McDonald
> On Nov 5, 2015, at 6:38 AM, Al Slater wrote: > > I have the 4Gb core file. Is there anything useful I can extract from > it to try and spot where the problem is? Your one ::findleaks showed nothing. Did your 4GB corefile have ::findleaks show nothing as well? ::umausers may be helpful. S

Re: [OmniOS-discuss] ILB memory leak?

2015-11-05 Thread Al Slater
To the mailing list as well... On 22/10/2015 09:43, Al Slater wrote: > On 21/10/2015 17:35, Dan McDonald wrote: >> >>> On Oct 21, 2015, at 6:08 AM, Al Slater >>> wrote: >>> >>> Hi, >>> >>> I am running omnios r151014 on a couple of machines with a couple >>> of zones each. 1 zone runs apache as

Re: [OmniOS-discuss] ILB memory leak?

2015-10-23 Thread Richard PALO
Le 22/10/15 10:43, Al Slater a écrit : > I am seeing kernel memory consumption increasing as well, but that may > be a different issue. The ilbd process memory is definitely growing. > this is indeed probably a different issue, but it would be useful to create a thread on illumos discuss as I'

Re: [OmniOS-discuss] ILB memory leak?

2015-10-22 Thread Ryan Zezeski
Al Slater writes: > On 21/10/2015 17:35, Dan McDonald wrote: >> >> That should enable user-level memory debugging. If you get a >> coredump, save it and share it. If you don't and the ilb daemon >> keeps running, eventually please: >> >> gcore `pgrep ilbd` >> >> and share THAT corefile. You ca

Re: [OmniOS-discuss] ILB memory leak?

2015-10-22 Thread Al Slater
On 21/10/2015 17:35, Dan McDonald wrote: On Oct 21, 2015, at 6:08 AM, Al Slater wrote: Hi, I am running omnios r151014 on a couple of machines with a couple of zones each. 1 zone runs apache as an SSL reverse proxy, the other runs ILB for load balancing web to app tier connections. I notic

Re: [OmniOS-discuss] ILB memory leak?

2015-10-21 Thread Bob Friesenhahn
On Wed, 21 Oct 2015, Dan McDonald wrote: You can use svccfg(1M) to enable user-level memory debugging on ilb. It may cause the ilb daemon to dump core. (And you're just noticing this in the process, not kernel memory consumption, correct?) As root: svcadm disable -t ilb svc

Re: [OmniOS-discuss] ILB memory leak?

2015-10-21 Thread Dan McDonald
> On Oct 21, 2015, at 6:08 AM, Al Slater wrote: > > Hi, > > I am running omnios r151014 on a couple of machines with a couple of zones > each. 1 zone runs apache as an SSL reverse proxy, the other runs ILB for > load balancing web to app tier connections. > > I noticed that in the ILB zone,

[OmniOS-discuss] ILB memory leak?

2015-10-21 Thread Al Slater
Hi, I am running omnios r151014 on a couple of machines with a couple of zones each. 1 zone runs apache as an SSL reverse proxy, the other runs ILB for load balancing web to app tier connections. I noticed that in the ILB zone, the ilbd process memory grows to about 2Gb. Restarting ILB re