Re: [Ganglia-general] [Ganglia-developers] Ganglia not recognizing other nodes?

Jeff Layton Sat, 21 Jul 2012 08:44:38 -0700

 Jesse,

[Note - I changed groups to ganglia-general since this isn't
a developers issue - just a silly user issue].


I must admit that networking is one of my weak areas but here
is are some relevant sections of output from netstat and lsof
on the master node.

Netstat:

[root@test1 ~]# netstat | more
Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address ForeignAddress Statetcp 0 0 192.168.1.250:53687<http://192.168.1.250:53687> ord08s06-in-f15.1e100:httpsESTABLISHEDtcp 0 0 10.1.0.250:shelln0001:48199 ESTABLISHEDtcp 0 0 192.168.1.250:41461<http://192.168.1.250:41461> ord08s08-in-f21.1e100:httpsESTABLISHEDtcp 0 0 192.168.1.250:41476<http://192.168.1.250:41476> ord08s08-in-f21.1e100:httpsESTABLISHEDtcp 0 0 10.1.0.250:8649 <http://10.1.0.250:8649>10.1.0.250:49899 <http://10.1.0.250:49899> TIME_WAITtcp 0 0 10.1.0.250:nfsn0001:imaps ESTABLISHEDtcp 0 0 192.168.1.250:50191<http://192.168.1.250:50191> den03s05-in-f16.1e100:https TIME_WAITtcp 0 0 10.1.0.250:8649 <http://10.1.0.250:8649>10.1.0.250:49905 <http://10.1.0.250:49905> TIME_WAITtcp 0 0 192.168.1.250:52133<http://192.168.1.250:52133> ord08s07-in-f14.1e100:httpsESTABLISHEDtcp 0 0 192.168.1.250:50500<http://192.168.1.250:50500> ord08s09-in-f22.1e100:httpsESTABLISHEDtcp 0 0 10.1.0.250:8649 <http://10.1.0.250:8649>10.1.0.250:49904 <http://10.1.0.250:49904> TIME_WAITtcp 0 0 10.1.0.250:8649 <http://10.1.0.250:8649>10.1.0.250:49903 <http://10.1.0.250:49903> TIME_WAITtcp 0 0 10.1.0.250:43479<http://10.1.0.250:43479> n0001:sshESTABLISHEDtcp 0 0 192.168.1.250:52134<http://192.168.1.250:52134> ord08s07-in-f14.1e100:https TIME_WAITtcp 0 0 192.168.1.250:53686<http://192.168.1.250:53686> ord08s06-in-f15.1e100:httpsESTABLISHEDudp 0 0 192.168.1.250:52035 <http://192.168.1.250:52035>239.2.11.71:8649 <http://239.2.11.71:8649> ESTABLISHED



lsof output:
[root@test1 ~]# lsof | grep gmond

gmond 1948 nobody cwd DIR 8,24096 2 /gmond 1948 nobody rtd DIR 8,24096 2 /gmond 1948 nobody txt REG 8,2111475 2527491 /usr/sbin/gmondgmond 1948 nobody mem REG 8,2161084 1318670 /lib/libexpat.so.1.5.2gmond 1948 nobody mem REG 8,243916 2527474 /usr/lib/libconfuse.so.0.0.0gmond 1948 nobody mem REG 8,267920 3165062 /usr/lib/ganglia/modcpu.sogmond 1948 nobody mem REG 8,267461 3165065 /usr/lib/ganglia/modload.sogmond 1948 nobody mem REG 8,2131044 1318655 /lib/libpthread-2.12.so <http://libpthread-2.12.so>gmond 1948 nobody mem REG 8,21876580 1318624 /lib/libc-2.12.so <http://libc-2.12.so>gmond 1948 nobody mem REG 8,2113908 1318641 /lib/libnsl-2.12.so <http://libnsl-2.12.so>gmond 1948 nobody mem REG 8,267515 3165068 /usr/lib/ganglia/modnet.sogmond 1948 nobody mem REG 8,2190604 1318668 /lib/libpcre.so.0.0.1gmond 1948 nobody mem REG 8,2142480 1318667 /lib/ld-2.12.so <http://ld-2.12.so>gmond 1948 nobody mem REG 8,267469 3165063 /usr/lib/ganglia/moddisk.sogmond 1948 nobody mem REG 8,258704 1318647 /lib/libnss_files-2.12.so <http://libnss_files-2.12.so>gmond 1948 nobody mem REG 8,267885 3165066 /usr/lib/ganglia/modmem.sogmond 1948 nobody mem REG 8,2184012 2527486 /usr/lib/libapr-1.so.0.3.9gmond 1948 nobody mem REG 8,267760 3165070 /usr/lib/ganglia/modsys.sogmond 1948 nobody mem REG 8,238376 1318635 /lib/libcrypt-2.12.so <http://libcrypt-2.12.so>gmond 1948 nobody mem REG 8,2103384 1318657 /lib/libresolv-2.12.so <http://libresolv-2.12.so>gmond 1948 nobody mem REG 8,215200 1318672 /lib/libuuid.so.1.3.0gmond 1948 nobody mem REG 8,225592 1318645 /lib/libnss_dns-2.12.so <http://libnss_dns-2.12.so>gmond 1948 nobody mem REG 8,2300676 1318666 /lib/libfreebl3.sogmond 1948 nobody mem REG 8,217892 1318637 /lib/libdl-2.12.so <http://libdl-2.12.so>gmond 1948 nobody mem REG 8,224382 3165067 /usr/lib/ganglia/modmulticpu.sogmond 1948 nobody mem REG 8,2202573 2527488 /usr/lib/libganglia-3.4.0.so.0.0.0gmond 1948 nobody mem REG 8,267397 3165069 /usr/lib/ganglia/modproc.sogmond 1948 nobody mem REG 8,226048 2646130 /usr/lib/gconv/gconv-modules.cachegmond 1948 nobody mem REG 8,299158704 2520564 /usr/lib/locale/locale-archivegmond 1948 nobody 0r CHR 1,30t0 3652 /dev/nullgmond 1948 nobody 1w CHR 1,30t0 3652 /dev/nullgmond 1948 nobody 2w CHR 1,30t0 3652 /dev/nullgmond 1948 nobody 3u REG 0,90 3650 anon_inodegmond 1948 nobody 4u IPv4 119480t0 UDP 239.2.11.71:8649 <http://239.2.11.71:8649>gmond 1948 nobody 5u IPv4 119520t0 TCP *:8649 (LISTEN)gmond 1948 nobody 6u IPv4 119550t0 UDP 192.168.1.250:52035->239.2.11.71:8649<http://239.2.11.71:8649>




To me it looks correct except I'm worried about that list bit from
lsof with 192.168.1.250 pointing to multicast address. But again,
I'm not entirely sure what I'm looking at :)

Thanks!

Jeff

What does netstat or lsof say about gmond interface binding?

(in haste, sorry for the brevity)

On Sat, Jul 21, 2012 at 10:50 AM, Jeff Layton<layto...@att.net>  wrote:

Good morning,

Apologies for the simple question. I've got a simple cluster
with a master node and one compute node. I installed the
latest Ganglia on the master node (3.4.0) - libganglia,
ganglia-gmond, ganglia-metad, ganglia-web (3.5.1). I can
use ganglia-web to "see" the master node with no problems.

I'm using Warewulf for the compute node and I installed
libganglia and ganglia-gmond in the VNFS and rebooted the
node. When the node comes back up, I tested ganglia via
"gstat -all" on the compute node and it seems to work
correctly.

However, ganglia-web doesn't display anything for the compute
node even though I've added it to the data_source line in
gmetad.conf:


data_source "my cluster" 10.1.0.250 10.1.0.1:8649


I also checked if the master node could access the data from
the compute node via "gstat -all" and I only get data from the
master node (i.e. no compute node).

I checked the Ethernet interfaces on both nodes and both
are listed as MULTICAST. iptabbles on the master node and the
compute node are off and the services are not running (checked
that 3 times).

There is a simple Netgear GigE switch between the nodes
(unmanaged). I don't think that's a problem.

One thing I think is interesting is that the master node has
eth0 with an IP of 192.168.1.250 which is to the outside world
and eth1 is 10.1.0.250 which is the cluster network. The compute
node has eth0 as 10.1.0.1. But when I go to http://localhost/ganglia
I can only access the master node as 192.168.1.250, not
10.1.0.250 (i.e. the list of nodes is only 192.168.1.250).

Otherwise i can login into the compute node, ping it, etc. It works
fine but somehow I'm missing a configuration piece for ganglia.

TIA!

Jeff



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Ganglia-developers mailing list
ganglia-develop...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general

Re: [Ganglia-general] [Ganglia-developers] Ganglia not recognizing other nodes?

Reply via email to