Re: [Ganglia-general] Ganglia gmond memory leak?

2012-02-23 Thread Martin Knoblauch
Hi Aidan,  if possible for you, I would suggest running the "gmond" in foreground under the control of "valgrind" or a similar tool. Send us the report generated by the tool. Cheers Martin -- Martin Knoblauch email: k n o b i AT knobisoft D

Re: [Ganglia-general] Ganglia gmond memory leak?

2012-02-23 Thread Jesse Becker
How many metrics are you monitoring? gmond must allocated memory for each metric, from each host. If you are using multicast, each gmond instance will get metrics from all other instances. If you run gmond in isolation--no traffic to/from other gmond instances--does memory usage still go up? On

Re: [Ganglia-general] Ganglia gmond memory leak?

2012-02-23 Thread Martin Knoblauch
Hi Jesse,  but in that case the memory footprint of "gmond" would approach a maximum after some time - correct? Aidan did not say whether it grows forever or goes asymptotic. Aidan?   Cheers Martin -- Martin Knoblauch email: k n o b i AT knob

Re: [Ganglia-general] Ganglia gmond memory leak?

2012-02-23 Thread svd.ganglia
i observed this in the past as well. running valgrind for days did not yeild any clue. i had a hunch that remote spoofed metrics were involved, as the leak seemed to get better when i had coincidentally disabled the sending of some of those spoof metrics. but, we never found anything conclus

Re: [Ganglia-general] Ganglia gmond memory leak?

2012-02-23 Thread svd.ganglia
makes sense, but i know in my case the number of metrics was constant after the server gmond had been started for about 10 minutes all gmetric crons had a chance to submit an initial value. -scott On Thu, 23 Feb 2012, Matt Massie wrote: Each unique metric (keyed on metric name) requires memo

Re: [Ganglia-general] Ganglia gmond memory leak?

2012-02-23 Thread Aidan Wong
That one node that recently had the running away memory leak was sending 253 metrics. I'm using unicast sending all metrics to a specific host where I have configured the "udp_send_channel" with the "host" and "port" attributes defined. On 2/23/12 5:36 AM, "Jesse Becker" wrote: >How many metric

Re: [Ganglia-general] Ganglia gmond memory leak?

2012-02-23 Thread Aidan Wong
To me it looks like gmond memory usage goes on as long as there is memory resource left and I've seen some nodes with gmond causing swapping. Before restart of gmond: $ free -m total used free sharedbuffers cached Mem: 18038 13217 4820

Re: [Ganglia-general] Ganglia gmond memory leak?

2012-02-23 Thread Matt Massie
Each unique metric (keyed on metric name) requires memory space in gmond. A good test is to peek at the number of metrics in gmond over time, e.g. $ telnet localhost 8649 | grep METRIC | wc -l If the number of metrics over time increases, so will the memory use. Ganglia will release the metric

[Ganglia-general] Failing multicast for 1 host

2012-02-23 Thread Jeff Blaine
Hi all, We've got an existing Ganglia set up that is working fine. One new host is giving us trouble though. Although one of its hostname resolves to 2 IP addresses (something I am looking into the history of to see if it's sane), I would like to get the problem out to the list in case someone se

Re: [Ganglia-general] Ganglia gmond memory leak?

2012-02-23 Thread Robin Humble
On Thu, Feb 23, 2012 at 07:22:36PM +, Aidan Wong wrote: >That one node that recently had the running away memory leak was sending >253 metrics. I'm using unicast sending all metrics to a specific host >where I have configured the "udp_send_channel" with the "host" and "port" >attributes define