Hi all, I have recently installed ganglia on a small cluster with 4 servers (h101, h102, h103, h104) and am having an issue whereby the 3 slaves are reported as being down (even though they are up). In fact, I can make it work for a short while (see below on changing the directory owner) and then they appear as dead.
gmond is running all four machines and gmetad is running on the server (h101). The web interface is also working. >From what I can see, the slaves appear down from master's view because TN is high: [root@h101 ~]# telnet h101 8649 | grep HOST | grep TN <HOST NAME="h102" IP="" REPORTED="1396176378" TN="174355" TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1396175888"> <HOST NAME="h103" IP="" REPORTED="1396176382" TN="174351" TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1396179776"> <HOST NAME="h104" IP="" REPORTED="1396176379" TN="174355" TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1396176191"> <HOST NAME="h101" IP="" REPORTED="1396350726" TN="8" TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1396176013"> However if I perform the same command from any of the slaves, the see their own TN low and the others high, eg: [root@h101 ~]# telnet h102 8649 | grep HOST | grep TN <HOST NAME="h102" IP="hidden" REPORTED="1396350629" TN="2" TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1396284414"> <HOST NAME="h103" IP="hidden" REPORTED="1396284601" TN="66030" TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1396181187"> <HOST NAME="h104" IP="hidden" REPORTED="1396284597" TN="66034" TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1396177590"> <HOST NAME="h101" IP="hidden" REPORTED="1396284599" TN="66032" TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1396176013"> I have tried restarting gmond on all machines and gmetad on the server but it doesn't help. I went through the FAQs - here are the results: - For gmond: - See if the gmond service is running, issue the *ps aux|grep gmond* command. Confirmed - Stop the gmond service and run it by hand with debug mode. */etc/init.d/gmond stop; /usr/sbin/gmond -d 2*. Look for errors near the top. No errors - Attempt to retrieve the XML data by netcatting to the gmond daemon. *nc <hostname> 8649 *Works for all hosts - Confirm that UDP connections can be established between the gmetad and gmond(or gmond and other gmond's for multicast purposes) by running *nc -u -l 8653* on the host in question, then *echo "hello"|nc -u <hostname> 8653* from the gmetad or another gmond. This works - but only for the first echo. If I try to send another message I get 'connection refused'. I have to stop and restart nc -u -l for it to receive the another message. Not sure if this is expected behaviour - Check gmond data using /usr/bin/gstat -a Each machine only sees itself - For gmetad: - See if the gmetad service is running, issue the *ps aux|grep gmetad* command. Confirmed - Check syslog for errors. *tail /var/log/messages *No errors - Stop the gmetad service and run it by hand with debug mode. */etc/init.d/gmetad stop; /usr/sbin/gmetad -d 2*. Look for errors near the top. It starts with no errors, but I don't see data from the other hosts coming in - Ensure that */var/lib/ganglia* and it's children are owned and writable by the *nobody* user (*ganglia* user on Debian/Ubuntu). I'm on RHEL and the user was set to ganglia. I changed it to nobody and restarted all daemons but now getting There was an error collecting ganglia data (127.0.0.1:8652): fsockopen error: Connection refused on the web interface. I changed it back to owner ganglia and restarted and suddenly the web page has data from all clusters - but only for a short while. I monitored using telnet as above and the TNs were being reset to low numbers for a short while, before increasing again and the hosts appeared dead again - Retrieve the XML data by netcatting to the gmetad daemon. *nc <hostname> 8650*. This information is useful for submitting bug reports. This returns with no output thanks Peter
------------------------------------------------------------------------------
_______________________________________________ Ganglia-general mailing list Ganglia-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-general