Hi Sid, Yusaku,
The above changes could make the load/cpu/memory/network (let's call it
LCMN) metrics be displayed for HDPNameNode. However, the current behavior
that the LCMN metrics of a node is only reported via the HDPSlaves might be
as designed.
This reason is, in ganglia, the LCMN metric for HDP_GRID is the sum of the
LCMN metric of the HDPNameNode and other HDP*. Multiple services could be
installed on the same machine. In that case, if the LCMN metric is
reported via all the HDP*, then the LCMN metric of a node could be reported
multiple times and cause the LCMN metric for the HDP_GRID to be incorrect.
Would you please confirm on this? Thanks!
On Tue, Oct 21, 2014 at 2:35 PM, Mingjiang Shi <m...@pivotal.io> wrote:
> Hi Sid, Yusaku
> I have observed 2 things on the namenode which also runs datanode:
> *1. ps out only has HDPSlaves:*
> ps -ef | grep gmond
> nobody 14840 1 1 06:14 ? 00:00:06 /usr/sbin/gmond
> --conf=/etc/ganglia/hdp/HDPSlaves/gmond.core.conf
> --pid-file=/var/run/ganglia/hdp/HDPSlaves/gmond.pid
> *2. /etc/ganglia/hdp only has folder HDPSlaves*
> ls -l /etc/ganglia/hdp
> drwxr-xr-x 3 root hadoop 4096 Oct 17 08:13 HDPSlaves
>
> I think this is a bug in the ganglia_monitor.py. The
> generate_slave_configs method only generates the HDPSlaves config files.
> It should generate slave config file for other services running on this
> node.
>
> Below is the existing code snippet:
> def generate_slave_configs(self):
> import params
>
> generate_daemon("gmond",
> name = "HDPSlaves",
> role = "monitor",
> owner = "root",
> group = params.user_group)
>
> I added the following code and restart the ganglia monitor, then the
> load/cpu/memory/network are now displayed under HDPNameNode
> if params.is_namenode_master:
> generate_daemon("gmond",
> name = "HDPNameNode",
> role = "monitor",
> owner = "root",
> group = params.user_group)
> See below:
>
>
> What's your view on this?
>
> On Tue, Oct 21, 2014 at 2:13 PM, Siddharth Wagle <swa...@hortonworks.com>
> wrote:
>
>> Hi Ming,
>>
>> NN sends metrics to gmond running on the *Ganglia server* host.
>>
>> Ambari, configures the /etc/ganglia/hdp/<packages> automatically, can you
>> restart Ganglia service and check if there are errors in the agent log.
>>
>> Check, ps aux | grep gmond, on ganglia host, also check the debug log
>> after
>> restarting hdp-gmond service.
>>
>> -Sid
>>
>> On Mon, Oct 20, 2014 at 11:00 PM, Mingjiang Shi <m...@pivotal.io> wrote:
>>
>> > Hi Siddharth,
>> > Thanks for your reply.
>> >
>> > I checked the namenode machine, it doesn't have HDPNamenode related
>> > gmond. Also, under the /etc/ganglia/hdp, it only has HDPSlaves folder
>> (No
>> > HDPNamenode folder).
>> >
>> > I think the HDPNameNode related config files should be pushed to
>> > /etc/ganglia/hdp folder of the namenode machine. This should also apply
>> to
>> > other services.
>> >
>> > Let me know what you think. Thanks!
>> >
>> > On Tue, Oct 21, 2014 at 11:51 AM, Siddharth Wagle <
>> swa...@hortonworks.com>
>> > wrote:
>> >
>> >> Hi Ming,
>> >>
>> >> 1. Check of gmond daemon is running on the Ganglia server host with
>> >> HDPNamenode in the path. Namenode pushes metrics to that gmond process.
>> >>
>> >> 2. Edit /etc/ganglia/hdp/HDPNamenode/gmond.core.conf, set debug = 10.
>> >>
>> >> Restart gmond on ganglia server host, #] service hdp-gmond restart.
>> >>
>> >> This will start the proc in foreground and print logs to stdout.
>> >>
>> >> -Sid
>> >>
>> >>
>> >> On Oct 20, 2014 6:33 PM, "Mingjiang Shi" <m...@pivotal.io> wrote:
>> >>
>> >> > Hi Yusaku,
>> >> > Yes, the Load, Memory, CPU and network are not showing for any of
>> HDP*
>> >> > except HDPSlaves.
>> >> >
>> >> > The cluster is deployed using the ambari-vagrant scripts, so the
>> >> /etc/hosts
>> >> > are are correct.
>> >> >
>> >> >
>> >> > On Tue, Oct 21, 2014 at 6:30 AM, Yusaku Sako <yus...@hortonworks.com
>> >
>> >> > wrote:
>> >> >
>> >> > > I presume Load, Memory, CPU, and Network are not showing for any of
>> >> > > HDPNameNode, HDPDataNode, HDPResourceManager, HDP*?
>> >> > > Are you seeing any metrics getting captured, and if so, what are
>> >> those?
>> >> > > How is the /etc/hosts set up? With Ganglia, based on my
>> experience,
>> >> > > things don't work well unless /etc/hosts are set up in the
>> following
>> >> way:
>> >> > > <ip> <fqdn> <shortname>
>> >> > >
>> >> > > Something like:
>> >> > > 192.168.64.101 c6401.ambari.apache.org c6401
>> >> > > 192.168.64.102 c6402.ambari.apache.org c6402
>> >> > >
>> >> > > On Mon, Oct 20, 2014 at 3:05 AM, Mingjiang Shi <m...@pivotal.io>
>> >> wrote:
>> >> > >
>> >> > >> Hi There,
>> >> > >> I deployed an HDP-2.1 cluster with Ganglia installed, but I don't
>> see
>> >> > any
>> >> > >> Load, Memory, CPU and network information for HDPNamenode. Is
>> this as
>> >> > >> designed? Thanks!
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >> --
>> >> > >> Cheers
>> >> > >> -MJ
>> >> > >>
>> >> > >
>> >> > >
>> >> > > CONFIDENTIALITY NOTICE
>> >> > > NOTICE: This message is intended for the use of the individual or
>> >> entity
>> >> > > to which it is addressed and may contain information that is
>> >> > confidential,
>> >> > > privileged and exempt from disclosure under applicable law. If the
>> >> reader
>> >> > > of this message is not the intended recipient, you are hereby
>> notified
>> >> > that
>> >> > > any printing, copying, dissemination, distribution, disclosure or
>> >> > > forwarding of this communication is strictly prohibited. If you
>> have
>> >> > > received this communication in error, please contact the sender
>> >> > immediately
>> >> > > and delete it from your system. Thank You.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Cheers
>> >> > -MJ
>> >> >
>> >>
>> >> --
>> >> CONFIDENTIALITY NOTICE
>> >> NOTICE: This message is intended for the use of the individual or
>> entity
>> >> to
>> >> which it is addressed and may contain information that is confidential,
>> >> privileged and exempt from disclosure under applicable law. If the
>> reader
>> >> of this message is not the intended recipient, you are hereby notified
>> >> that
>> >> any printing, copying, dissemination, distribution, disclosure or
>> >> forwarding of this communication is strictly prohibited. If you have
>> >> received this communication in error, please contact the sender
>> >> immediately
>> >> and delete it from your system. Thank You.
>> >>
>> >
>> >
>> >
>> > --
>> > Cheers
>> > -MJ
>> >
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Cheers
> -MJ
>
--
Cheers
-MJ
