Well, from this project's standpoint I wish I had access to a big
installation to test on still, but I don't (and I'm happy to be doing what
I'm doing).
The scalability issue I've seen happened in a grid with a few dozen
clusters; each cluster had a few dozen nodes in it, and we monitored a
couple hundred metrics per node (yes, I'm being deliberately vague). In that
case, the typical failure was that the grid summary metrics would "gap",
even though the cluster summary metrics and the underlying host metrics were
updated. We were running on a plenty-big ramdisk (8GB), and the actual CPU
utilization was under 50%, IIRC.
We traced the problem to lock contention in gmetad. Gmetad needs to take a
lock per cluster as it collects data for the grid summary metrics.
Otherwise, each cluster just runs in its own thread, with its own sampling
against the gmonds.
By reducing the number of clusters per grid (and moving to a grid-of-grid)
deployment, we were able to handle the same number of machines & metrics
with much less probability of gapping.
My suspicion is that a naive attempt to do the kind of pivoting you're
talking about in this email will require holding those cluster-query locks
longer.
My advice would be to treat each cluster as if it was its own gmond when
doing that. I.e., the aggregating thread should simply read the data into
its own workspace as quickly as possible without doing any arithmetic (in
fact, a copy-on-write clone would be great) so as to avoid holding those
cluster locks too long. Then the arithmetic/aggregation thread can run
independently. If it can't do arithmetic fast enough, well, change the RRAs
to reflect how fast the aggregation thread can run. At all costs, avoid
getting in the way of recording the host & cluster metrics; they're the raw
material. Folks can always derive their aggregates from that data later or
out-of-band if necessary.
-- ReC
On Tue, Nov 30, 2010 at 10:54 AM, Bernard Li <bern...@vanhpc.org> wrote:
> Hi Rick:
>
> On Tue, Nov 30, 2010 at 10:38 AM, Rick Cobb <rick_c...@ieee.org> wrote:
>
> > On your idea, Bernard -- I don't think it would necessarily require gmond
>
> You're right, I meant to say gmetad... it was late last night :)
>
> > changes. OTOH, I think it would require very interesting gmetad changes
> to
> > do a good job. In particular, the ability to summarize by different
> > aggregations seems like the scalability wall; it's already hard to get
> grid
> > summaries updated (the only inter-thread update in the current code).
>
> Can you elaborate on the last point? Depending on what you mean
> exactly, I might have a solution for that...
>
> I think going forward the gmetad-python code could be easier to
> maintain and add new features, and not to mention it supports the
> plug-in interface. However, it hasn't really been tested in a large
> production environment for scalability and stability, so that's why I
> really want to get it released in some form or manner ASAP.
>
> This is a call for help -- if you have a large installation and would
> like to help test some cutting edge code, please take gmetad-python
> out for a spin and report back any issues you may have. Hopefully
> we'll have it packaged up and released as 3.2.0 by early 2011 :-)
>
> Thanks all!
>
> Bernard
>
------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
