Unfortunately there appears to be several problems, but they don't all
correlate with Error 1 sending the modular data. There were reporting
problems with approximately 75 gmond instances around midnight local
time last night (which is still suspicious). Some examples:
One case box #82:
- Sending to 3 udp unicast channels, which are polled by 2 gmetads.
- Nothing in /var/log/messages near when it sopped reporting.
- Both gmetad's agree it was down (they poll the same aggregator).
- Host did not appear in 2/3 gmond aggregators.
- Restart of gmond on #82 did not fix problem.
- Restart of the relevant aggregator did not fix the problem.
- Restart of aggregator with debug 0 did fix this problem. Obviously
I missed something.
- meta data send interval is 120 seconds. Does that impact when HOST
NAME= update?
- The aggregator (for old memory leak fighting) has a cron to restart
at midnight.
In another case (for the other 75) a gmetad instance stopped updating
after getting an error like:
Aug 14 23:59:01 host /usr/sbin/gmetad[5999]: Process XML (Cluster 1):
XML_ParseBuffer() error at line 22388:#012no element found#012
Restarting that gmetad caused it to start polling again
I found
http://bugzilla.ganglia.info/cgi-bin/bugzilla/show_bug.cgi?id=189 but
that seems to refer to an un-escaped name, as opposed to a transient but
not recovered error.
On 08/13/2012 01:29 PM, Chris Burroughs wrote:
So for background, my original problem is that load_one will not be
updated by gmetad for a period of over 600 seconds (an arbitrary timeout
signifying that gmond/the host is probably down). It occurs a few
times/day across hundreds of hosts, and often occurs near midnight
localtime. This *appears* to correlate with messages along the lines of
the following (I didn't see anything else suspicious in syslog):
Aug 12 23:53:26 adq82 /usr/sbin/gmond[28637]: Error 1 sending the
modular data for entropy_avail#012
Aug 12 23:59:00 adq82 /usr/sbin/gmond[28637]: Error 1 sending the
modular data for mem_cached#012
Aug 12 23:59:10 adq82 /usr/sbin/gmond[28637]: Error 1 sending the
modular data for diskstat_sda_write_bytes_per_sec#012
Since it occurs infrequently running in debug mode on every server is
not a good option. But false positives that keep people from sleeping
are bad. First of all, does a correlation between these messages and all
metrics not reporting for a period of time make sense? If not what
should I be looking at?
Second, if this is anything other than a red herring, I'm totally
confused how to debug it. Even if debug was enabled the debug message
[1] does doesn't seem to include any additional information. Also 1
seems like it could be two different errors [2] [3].
System information:
- gmond 3.4.0
- centos6
- using send channels
[1]
https://github.com/ganglia/monitor-core/blob/release/3.4/gmond/gmond.c#L2735
[2]
https://github.com/ganglia/monitor-core/blob/release/3.4/lib/libgmond.c#L575
[3]
https://github.com/ganglia/monitor-core/blob/release/3.4/lib/libgmond.c#L517
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general
