[Ganglia-general] gmond occasionally doesn't connect up in unicast

[Sam Barham]

We've got about 100 machines running on AWS EC2s, with Ganglia for
monitoring.  Because we are on Amazon, we can't use multicast, so the
architecture we have is each cluster has a Bastion machine, and each other
machine in the cluster has gmond send its' data to the bastion, which
gmetad then queries.  All standard and sensible and it works just fine.

Except that occasionally, when I redeploy the machines in a cluster (but
not the bastion - that stays running through this operation), just one of
the machines will not send data through to the bastion or something.  All I
can say for sure is that gmond is running OK on the problem machine, there
are no error logs on the problem machine, the bastion or the gmetad
machine, but the machine doesn't appear in gmetad.  If I go into the
problem machine and restart gmond, it reconnects just fine and appears in
gmetad.

Which machine has the error is random - it's not a particular type of
machine or anything.  Because the error only shows up rarely, and only at
deployment time, I can't really turn on debug_level to investigate.

Also, some of the configuration values in gmond.conf are filled in when the
userdata is run.  I've edited /etc/init.d/ganglia-monitor so that it starts
up immediately after the userdata has run, just in case that matters.

Any ideas?

Sam
--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general

Re: [Ganglia-general] gmond occasionally doesn't connect up in unicast

[Sam Barham]

Until recently I wasn't controlling the start order of ec2-run-user-data
and ganglia-monitor, so they were starting at the same 'time'.  Yesterday I
fixed that, so that now ec2-run-user-data starts at S02 and ganglia-monitor
at S03.  I thought the issue might be exactly what you describe -
ganglia-monitor starting before ec2-run-user-data has finished altering the
gmond.conf, but the error still happened today.

Also, I suspect (but don't know for sure) that the gmond.conf will actually
be invalid before ec2-run-user-data has run - I've altered it to have flags
that get replaced with valid values.

On Thu, Nov 13, 2014 at 12:20 PM, Joe Gracyk  wrote:

> Hi, Sam -
>
> We've got a similar deployment (EC2 instances unicasting to a per-AZ
> gmetad) that we're managing with Puppet, and I can't say we've seen
> anything like that.
>
> How are you automating your redeployments and gmond configurations? Could
> your gmond instances be starting up before their unicast configurations
> have been applied? If you had some sort of race condition where gmond could
> be installed and started, and *then *getting the conf file written, I'd
> expect gmond to merrily chug along, fruitlessly trying to multicast into
> the void.
>
> Good luck!
>
> On Wed, Nov 12, 2014 at 2:41 PM, Sam Barham 
> wrote:
>
>> We've got about 100 machines running on AWS EC2s, with Ganglia for
>> monitoring.  Because we are on Amazon, we can't use multicast, so the
>> architecture we have is each cluster has a Bastion machine, and each other
>> machine in the cluster has gmond send its' data to the bastion, which
>> gmetad then queries.  All standard and sensible and it works just fine.
>>
>> Except that occasionally, when I redeploy the machines in a cluster (but
>> not the bastion - that stays running through this operation), just one of
>> the machines will not send data through to the bastion or something.  All I
>> can say for sure is that gmond is running OK on the problem machine, there
>> are no error logs on the problem machine, the bastion or the gmetad
>> machine, but the machine doesn't appear in gmetad.  If I go into the
>> problem machine and restart gmond, it reconnects just fine and appears in
>> gmetad.
>>
>> Which machine has the error is random - it's not a particular type of
>> machine or anything.  Because the error only shows up rarely, and only at
>> deployment time, I can't really turn on debug_level to investigate.
>>
>> Also, some of the configuration values in gmond.conf are filled in when
>> the userdata is run.  I've edited /etc/init.d/ganglia-monitor so that it
>> starts up immediately after the userdata has run, just in case that matters.
>>
>> Any ideas?
>>
>> Sam
>>
>>
>> --
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
>> ___
>> Ganglia-general mailing list
>> Ganglia-general@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/ganglia-general
>>
>>
>
>
> --
>
> [image: logo] 
>
> Joe Gracyk | *DevOps Developer*
> 707-780-1848 | jgra...@marketlive.com
>
> [image: Follow us on Facebook] 
> 
> 
>  
>
--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general

Re: [Ganglia-general] gmond occasionally doesn't connect up in unicast

[Joe Gracyk]

Hi, Sam -

We've got a similar deployment (EC2 instances unicasting to a per-AZ
gmetad) that we're managing with Puppet, and I can't say we've seen
anything like that.

How are you automating your redeployments and gmond configurations? Could
your gmond instances be starting up before their unicast configurations
have been applied? If you had some sort of race condition where gmond could
be installed and started, and *then *getting the conf file written, I'd
expect gmond to merrily chug along, fruitlessly trying to multicast into
the void.

Good luck!

On Wed, Nov 12, 2014 at 2:41 PM, Sam Barham 
wrote:

> We've got about 100 machines running on AWS EC2s, with Ganglia for
> monitoring.  Because we are on Amazon, we can't use multicast, so the
> architecture we have is each cluster has a Bastion machine, and each other
> machine in the cluster has gmond send its' data to the bastion, which
> gmetad then queries.  All standard and sensible and it works just fine.
>
> Except that occasionally, when I redeploy the machines in a cluster (but
> not the bastion - that stays running through this operation), just one of
> the machines will not send data through to the bastion or something.  All I
> can say for sure is that gmond is running OK on the problem machine, there
> are no error logs on the problem machine, the bastion or the gmetad
> machine, but the machine doesn't appear in gmetad.  If I go into the
> problem machine and restart gmond, it reconnects just fine and appears in
> gmetad.
>
> Which machine has the error is random - it's not a particular type of
> machine or anything.  Because the error only shows up rarely, and only at
> deployment time, I can't really turn on debug_level to investigate.
>
> Also, some of the configuration values in gmond.conf are filled in when
> the userdata is run.  I've edited /etc/init.d/ganglia-monitor so that it
> starts up immediately after the userdata has run, just in case that matters.
>
> Any ideas?
>
> Sam
>
>
> --
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
> ___
> Ganglia-general mailing list
> Ganglia-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ganglia-general
>
>


-- 

[image: logo] 

Joe Gracyk | *DevOps Developer*
707-780-1848 | jgra...@marketlive.com

[image: Follow us on Facebook] 


 
--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general