On 12/07/2013 02:23 PM, Chris Burroughs wrote: > On 12/06/2013 03:36 PM, Vladimir Vuksan wrote: >> The Ganglia core is comprised of two daemons, `gmond` and `gmetad`. >> `Gmond` is >> primarily responsible for sending and receiving metrics; `gmetad` >> carries the >> hefty task of summarizing / aggregating the information, writing the >> metrics >> information to graphing utilities (such as RRD), and reporting >> summary metrics >> to the web front-end. Due to growth some metrics were never updating, >> and >> front-end response time was abysmal. These issues are tied directly >> to `gmetad`. >> > > Were these failures totally random or grouped in some way? (Same > cluster, type, etc).
We run multiple dozens of clusters and some of the larger clusters ie. clusters that had 2-3x machines that other clusters would exhibit either gaps, slower updating ie. data points would update on 3 or 4x poll period if you looked on graph details. Also Grid summary views disappeared altogether. >> In our Ganglia setup, we run a `gmond` to collect data for every >> machine and >> several `gmetad` processes: >> >> * An interactive `gmetad` process is responsible solely for >> reporting summary >> statistics to the web interface. >> * Another `gmetad` process is responsible for writing graphs. > > > Are these two gmetad process co-located on the same server? I think > this is an interesting option that I at least was not aware of. This set up is very similar to what you have. Basically have one gmetad that polls all the same gmonds however has write_rrd off and is used for both alerting and feeding the web interface. > Did you go with this setup to alleviate the problems described above > or for other reasons? It was mostly to improve web interface responsiveness. Perhaps with the latest changes we no longer have to do it but we haven't tested that yet. > Frankly I don't understand rrdcached. The OS already has a fancy > substytem for keeping frequencly accessed data in memory. If we are > dealing with a lot of files (instead of a database with indexes where > the applicaiton might have more information than the OS) why fight > with it? (Canonical rant: > https://www.varnish-cache.org/trac/wiki/ArchitectNotes) > > Anyway we have had much better luck with tuning the page cache and > disabling fsync for gmetad. > http://oss.oetiker.ch/rrdtool-trac/wiki/TuningRRD Adminitedly at > least some of the problems we had with rrdcached could have been due > to the issues you have identified. > >> In the process of doing this, I noticed that ganglia used a >> particularly poor >> method for reading its XML metrics from gmond: It initialized a >> 1024-byte >> buffer, read into it, and if it would overflow, it would realloc the >> buffer with >> an additional 1024 bytes and try reading again. When dealing with XML >> files many >> megabytes in size, this caused many unnecessary reallocations. I >> modified this >> code to start with a 128KB buffer and double the buffer size when it >> runs out of >> space. (I made a similar change to the code for decompressing gzip'ed >> data that >> used a similar buffer sizing paradigm). > > This sounds like a solid find. I'm a little worried about the > doubling though since as you said the responses can get quiet large. > Is there a max buffer size? > > Does your fix also handle the case of gmetad polling other gmetad? I don't think we know yet as we don't have that kind of a setup. >> After all these changes, both the interactive and RRD-writing >> processes spend >> most of their time in the hash table. I can continue improving Ganglia >> performance, but most of the low hanging fruit is now gone; at some >> me point it >> will require: >> >> * writing a version of librrd (this probably also means changing >> the rrd file >> format), > > I think something got cut off here. It meant to say an improved librrd. > * refactoring gmetad and gmond into a single process that shares > memory > > I'm not sure I folow this one. While the node with gmetad likely also > has gmond, gmond typically runs alone. The local gmond is also not > necessarily reporting directly to the co-located gmetad. > Idea is that for certain scenarios e.g. centralized metrics host where all gmonds run it makes some sense to have gmetad accept all metrics over UDP instead of doing the dance of downloading XML tree of metrics from gmond, parsing and populating it's own hash. It's unnecessary overhead. Vladimir ------------------------------------------------------------------------------ Sponsored by Intel(R) XDK Develop, test and display web and hybrid apps with a single code base. Download it for free now! http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk _______________________________________________ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
