Hi Sid, Yusaku
I have observed 2 things on the namenode which also runs datanode:
*1. ps out only has HDPSlaves:*
ps -ef | grep gmond
nobody 14840 1 1 06:14 ? 00:00:06 /usr/sbin/gmond
--conf=/etc/ganglia/hdp/HDPSlaves/gmond.core.conf
--pid-file=/var/run/ganglia/hdp/HDPSlaves/gmond.pid
*2. /etc/ganglia/hdp only has folder HDPSlaves*
ls -l /etc/ganglia/hdp
drwxr-xr-x 3 root hadoop 4096 Oct 17 08:13 HDPSlaves
I think this is a bug in the ganglia_monitor.py. The generate_slave_configs
method only generates the HDPSlaves config files. It should generate slave
config file for other services running on this node.
Below is the existing code snippet:
def generate_slave_configs(self):
import params
generate_daemon("gmond",
name = "HDPSlaves",
role = "monitor",
owner = "root",
group = params.user_group)
I added the following code and restart the ganglia monitor, then the
load/cpu/memory/network are now displayed under HDPNameNode
if params.is_namenode_master:
generate_daemon("gmond",
name = "HDPNameNode",
role = "monitor",
owner = "root",
group = params.user_group)
See below:
What's your view on this?
On Tue, Oct 21, 2014 at 2:13 PM, Siddharth Wagle <swa...@hortonworks.com>
wrote:
> Hi Ming,
>
> NN sends metrics to gmond running on the *Ganglia server* host.
>
> Ambari, configures the /etc/ganglia/hdp/<packages> automatically, can you
> restart Ganglia service and check if there are errors in the agent log.
>
> Check, ps aux | grep gmond, on ganglia host, also check the debug log after
> restarting hdp-gmond service.
>
> -Sid
>
> On Mon, Oct 20, 2014 at 11:00 PM, Mingjiang Shi <m...@pivotal.io> wrote:
>
> > Hi Siddharth,
> > Thanks for your reply.
> >
> > I checked the namenode machine, it doesn't have HDPNamenode related
> > gmond. Also, under the /etc/ganglia/hdp, it only has HDPSlaves folder
> (No
> > HDPNamenode folder).
> >
> > I think the HDPNameNode related config files should be pushed to
> > /etc/ganglia/hdp folder of the namenode machine. This should also apply
> to
> > other services.
> >
> > Let me know what you think. Thanks!
> >
> > On Tue, Oct 21, 2014 at 11:51 AM, Siddharth Wagle <
> swa...@hortonworks.com>
> > wrote:
> >
> >> Hi Ming,
> >>
> >> 1. Check of gmond daemon is running on the Ganglia server host with
> >> HDPNamenode in the path. Namenode pushes metrics to that gmond process.
> >>
> >> 2. Edit /etc/ganglia/hdp/HDPNamenode/gmond.core.conf, set debug = 10.
> >>
> >> Restart gmond on ganglia server host, #] service hdp-gmond restart.
> >>
> >> This will start the proc in foreground and print logs to stdout.
> >>
> >> -Sid
> >>
> >>
> >> On Oct 20, 2014 6:33 PM, "Mingjiang Shi" <m...@pivotal.io> wrote:
> >>
> >> > Hi Yusaku,
> >> > Yes, the Load, Memory, CPU and network are not showing for any of HDP*
> >> > except HDPSlaves.
> >> >
> >> > The cluster is deployed using the ambari-vagrant scripts, so the
> >> /etc/hosts
> >> > are are correct.
> >> >
> >> >
> >> > On Tue, Oct 21, 2014 at 6:30 AM, Yusaku Sako <yus...@hortonworks.com>
> >> > wrote:
> >> >
> >> > > I presume Load, Memory, CPU, and Network are not showing for any of
> >> > > HDPNameNode, HDPDataNode, HDPResourceManager, HDP*?
> >> > > Are you seeing any metrics getting captured, and if so, what are
> >> those?
> >> > > How is the /etc/hosts set up? With Ganglia, based on my experience,
> >> > > things don't work well unless /etc/hosts are set up in the following
> >> way:
> >> > > <ip> <fqdn> <shortname>
> >> > >
> >> > > Something like:
> >> > > 192.168.64.101 c6401.ambari.apache.org c6401
> >> > > 192.168.64.102 c6402.ambari.apache.org c6402
> >> > >
> >> > > On Mon, Oct 20, 2014 at 3:05 AM, Mingjiang Shi <m...@pivotal.io>
> >> wrote:
> >> > >
> >> > >> Hi There,
> >> > >> I deployed an HDP-2.1 cluster with Ganglia installed, but I don't
> see
> >> > any
> >> > >> Load, Memory, CPU and network information for HDPNamenode. Is this
> as
> >> > >> designed? Thanks!
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Cheers
> >> > >> -MJ
> >> > >>
> >> > >
> >> > >
> >> > > CONFIDENTIALITY NOTICE
> >> > > NOTICE: This message is intended for the use of the individual or
> >> entity
> >> > > to which it is addressed and may contain information that is
> >> > confidential,
> >> > > privileged and exempt from disclosure under applicable law. If the
> >> reader
> >> > > of this message is not the intended recipient, you are hereby
> notified
> >> > that
> >> > > any printing, copying, dissemination, distribution, disclosure or
> >> > > forwarding of this communication is strictly prohibited. If you have
> >> > > received this communication in error, please contact the sender
> >> > immediately
> >> > > and delete it from your system. Thank You.
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Cheers
> >> > -MJ
> >> >
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> >> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> >> immediately
> >> and delete it from your system. Thank You.
> >>
> >
> >
> >
> > --
> > Cheers
> > -MJ
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Cheers
-MJ
