Re: [Ganglia-developers] UDP / mute=yes and high cpu usage
Oh yes - setting deaf=yes is the solution - I mentioned that in my first
email (sorry that I was not very clear about that).
My question however was - is that a bug or a feature? It can be tricky
as in default configuration deaf=no and one could not simply get to the
source of problem. In "Monitoring with Ganglia" book hovewer it is
mentioned (page 22) in the picture that gmond nodes are deaf; but maybe
(just maybe) gmond could not listen to anything if it has no
recv_channels defined? :)
>
> On 02/07/2014 09:59 AM, Maciej Lasyk wrote:
> >Sure, it's not that long so I'm posting it in-place:
> >
> >globals {
> > daemonize = yes
> > setuid = yes
> > user = ganglia
> > debug_level = 0
> > max_udp_msg_len = 1472
> > mute = no
> > deaf = no
> > allow_extra_data = yes
> > host_dmax = 0 /*secs */
> > cleanup_threshold = 300 /*secs */
> > gexec = no
> > send_metadata_interval = 60 /*secs */
> >}
> >
> >cluster {
> > name = "somecluster"
> > owner = "someowner"
> > latlong = "unspecified"
> > url = "unspecified"
> >}
> >
> >host {
> > location = "host"
> >}
> >
> >udp_send_channel {
> > bind_hostname = yes
> > port = 8649
> > ttl = 2
> > host = 192.168.1.23
> >}
> >
> >.. and here go modules / metrics
> >
> >On Fri, Feb 07, 2014 at 09:15:56AM -0500, Vladimir Vuksan wrote:
> >>Maciej,
> >>
> >>can you post top 100 lines or so of your config ie. with all the udp
> >>channels etc.
> >>
> >>Thanks
> >>
> >>On 02/07/2014 08:52 AM, Maciej Lasyk wrote:
> >>
> >> Hi guys,
> >>
> >> I've been struggling with very high cpu usage of my gmond daemons
> >> lately. I've been using UDP unicast topology. I just couldn't find the
> >> source of my problem. All my gmonds on my servers were generating
> >> ~99-100% of procs usage.
> >>
> >> stracing gmond processes revealed zounds of epools and gettimeofdays
> >> syscalls (like very many every second) and that was all.
> >>
> >> I tried to start from scratch - there was no problem when using default
> >> config (mutlicast topo, deaf/mute=no). So after playing a while I
> >> thought that _maybe_ deaf=no on gmond which is only sending data to some
> >> aggregator (only udp_send_channel, no rcv channels or tcp channels)
> >> generates some negative energy here. And that was it.
> >>
> >> My mistake was that I thought that when deaf=no is set than it doesn't
> >> matter as in UDP unicast we just don't listen - unless we have
> >> recv_channel configured.
> >>
> >> So I think that this could be a bug - not a feature. What's your
> >> thoughts on this?
> >>
> >>
> >> --
> >> Managing the Performance of Cloud-Based Applications
> >> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
> >> Read the Whitepaper.
> >>
> >> [1]http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
> >>
> >> ___
> >> Ganglia-developers mailing list
> >> [2]Ganglia-developers@lists.sourceforge.net
> >> [3]https://lists.sourceforge.net/lists/listinfo/ganglia-developers
> >>
> >>References
> >>
> >>Visible links
> >>1.
> >> http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
> >>2. mailto:Ganglia-developers@lists.sourceforge.net
> >>3. https://lists.sourceforge.net/lists/listinfo/ganglia-developers
> >>/usr/bin/xdg-open: line 402: htmlview: command not found
> >>/usr/bin/xdg-open: line 402: firefox: command not found
> >>/usr/bin/xdg-open: line 402: mozilla: command not found
> >>/usr/bin/xdg-open: line 402: netscape: command not found
>
--
--
pozdrawiam,
Maciej Lasyk
GPG key ID: FFA8AEEC
GPG info: http://maciek.lasyk.info/gpg.txt
pgpIVn3sZ3v_l.pgp
Description: PGP signature
--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] UDP / mute=yes and high cpu usage
Set deaf=yes. Let me know if that lowers the CPU usage.
Vladimir
On 02/07/2014 09:59 AM, Maciej Lasyk wrote:
> Sure, it's not that long so I'm posting it in-place:
>
> globals {
>daemonize = yes
>setuid = yes
>user = ganglia
>debug_level = 0
>max_udp_msg_len = 1472
>mute = no
>deaf = no
>allow_extra_data = yes
>host_dmax = 0 /*secs */
>cleanup_threshold = 300 /*secs */
>gexec = no
>send_metadata_interval = 60 /*secs */
> }
>
> cluster {
>name = "somecluster"
>owner = "someowner"
>latlong = "unspecified"
>url = "unspecified"
> }
>
> host {
>location = "host"
> }
>
> udp_send_channel {
>bind_hostname = yes
>port = 8649
>ttl = 2
>host = 192.168.1.23
> }
>
> .. and here go modules / metrics
>
> On Fri, Feb 07, 2014 at 09:15:56AM -0500, Vladimir Vuksan wrote:
>> Maciej,
>>
>> can you post top 100 lines or so of your config ie. with all the udp
>> channels etc.
>>
>> Thanks
>>
>> On 02/07/2014 08:52 AM, Maciej Lasyk wrote:
>>
>> Hi guys,
>>
>> I've been struggling with very high cpu usage of my gmond daemons
>> lately. I've been using UDP unicast topology. I just couldn't find the
>> source of my problem. All my gmonds on my servers were generating
>> ~99-100% of procs usage.
>>
>> stracing gmond processes revealed zounds of epools and gettimeofdays
>> syscalls (like very many every second) and that was all.
>>
>> I tried to start from scratch - there was no problem when using default
>> config (mutlicast topo, deaf/mute=no). So after playing a while I
>> thought that _maybe_ deaf=no on gmond which is only sending data to some
>> aggregator (only udp_send_channel, no rcv channels or tcp channels)
>> generates some negative energy here. And that was it.
>>
>> My mistake was that I thought that when deaf=no is set than it doesn't
>> matter as in UDP unicast we just don't listen - unless we have
>> recv_channel configured.
>>
>> So I think that this could be a bug - not a feature. What's your
>> thoughts on this?
>>
>>
>> --
>> Managing the Performance of Cloud-Based Applications
>> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
>> Read the Whitepaper.
>>
>> [1]http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
>>
>> ___
>> Ganglia-developers mailing list
>> [2]Ganglia-developers@lists.sourceforge.net
>> [3]https://lists.sourceforge.net/lists/listinfo/ganglia-developers
>>
>> References
>>
>> Visible links
>> 1.
>> http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
>> 2. mailto:Ganglia-developers@lists.sourceforge.net
>> 3. https://lists.sourceforge.net/lists/listinfo/ganglia-developers
>> /usr/bin/xdg-open: line 402: htmlview: command not found
>> /usr/bin/xdg-open: line 402: firefox: command not found
>> /usr/bin/xdg-open: line 402: mozilla: command not found
>> /usr/bin/xdg-open: line 402: netscape: command not found
--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] UDP / mute=yes and high cpu usage
Sure, it's not that long so I'm posting it in-place:
globals {
daemonize = yes
setuid = yes
user = ganglia
debug_level = 0
max_udp_msg_len = 1472
mute = no
deaf = no
allow_extra_data = yes
host_dmax = 0 /*secs */
cleanup_threshold = 300 /*secs */
gexec = no
send_metadata_interval = 60 /*secs */
}
cluster {
name = "somecluster"
owner = "someowner"
latlong = "unspecified"
url = "unspecified"
}
host {
location = "host"
}
udp_send_channel {
bind_hostname = yes
port = 8649
ttl = 2
host = 192.168.1.23
}
.. and here go modules / metrics
On Fri, Feb 07, 2014 at 09:15:56AM -0500, Vladimir Vuksan wrote:
>Maciej,
>
>can you post top 100 lines or so of your config ie. with all the udp
>channels etc.
>
>Thanks
>
>On 02/07/2014 08:52 AM, Maciej Lasyk wrote:
>
> Hi guys,
>
> I've been struggling with very high cpu usage of my gmond daemons
> lately. I've been using UDP unicast topology. I just couldn't find the
> source of my problem. All my gmonds on my servers were generating
> ~99-100% of procs usage.
>
> stracing gmond processes revealed zounds of epools and gettimeofdays
> syscalls (like very many every second) and that was all.
>
> I tried to start from scratch - there was no problem when using default
> config (mutlicast topo, deaf/mute=no). So after playing a while I
> thought that _maybe_ deaf=no on gmond which is only sending data to some
> aggregator (only udp_send_channel, no rcv channels or tcp channels)
> generates some negative energy here. And that was it.
>
> My mistake was that I thought that when deaf=no is set than it doesn't
> matter as in UDP unicast we just don't listen - unless we have
> recv_channel configured.
>
> So I think that this could be a bug - not a feature. What's your
> thoughts on this?
>
>
> --
> Managing the Performance of Cloud-Based Applications
> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
> Read the Whitepaper.
>
> [1]http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
>
> ___
> Ganglia-developers mailing list
> [2]Ganglia-developers@lists.sourceforge.net
> [3]https://lists.sourceforge.net/lists/listinfo/ganglia-developers
>
> References
>
>Visible links
>1.
> http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
>2. mailto:Ganglia-developers@lists.sourceforge.net
>3. https://lists.sourceforge.net/lists/listinfo/ganglia-developers
> /usr/bin/xdg-open: line 402: htmlview: command not found
> /usr/bin/xdg-open: line 402: firefox: command not found
> /usr/bin/xdg-open: line 402: mozilla: command not found
> /usr/bin/xdg-open: line 402: netscape: command not found
--
--
pozdrawiam,
Maciej Lasyk
GPG key ID: FFA8AEEC
GPG info: http://maciek.lasyk.info/gpg.txt
pgpkzEkJhqmTF.pgp
Description: PGP signature
--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] UDP / mute=yes and high cpu usage
Maciej, can you post top 100 lines or so of your config ie. with all the udp channels etc. Thanks On 02/07/2014 08:52 AM, Maciej Lasyk wrote: Hi guys, I've been struggling with very high cpu usage of my gmond daemons lately. I've been using UDP unicast topology. I just couldn't find the source of my problem. All my gmonds on my servers were generating ~99-100% of procs usage. stracing gmond processes revealed zounds of epools and gettimeofdays syscalls (like very many every second) and that was all. I tried to start from scratch - there was no problem when using default config (mutlicast topo, deaf/mute=no). So after playing a while I thought that _maybe_ deaf=no on gmond which is only sending data to some aggregator (only udp_send_channel, no rcv channels or tcp channels) generates some negative energy here. And that was it. My mistake was that I thought that when deaf=no is set than it doesn't matter as in UDP unicast we just don't listen - unless we have recv_channel configured. So I think that this could be a bug - not a feature. What's your thoughts on this? -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
