-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 guys-
i wanted to get you up to speed on where g3 is right now. i can't remember what i've talked about before.. so if i repeat myself i apologize. i apologize. as far as testing the code goes.. i've been testing it by running it through valgrind and mpatrol (even though i start the tests when i leave at night and get the results in the morning). i also make sure it compiles and runs correctly on linux and cygwin (and when i have little extra time i run it on solaris). i'm pretty sure all the other unix flavors out there will be fine too but it would be nice to have someone confirm it. if you to take a look at the latest snapshot of the code.. visit http://matt-massie.com/g3/ganglia-3.0.0.tar.gz you can run the g3 gmetad on real g2 data sources to get a feel for how it will work and to get some basic benchmarks on the disk i/o savings you'll find (among other things). here's how to install g3 gmetad to play around with it. % gunzip < ganglia-3.0.0.tar.gz | tar -xvf - % cd ganglia-3.0.0 % ./configure configure will give you helpful? directions about any missing libraries.. like where to get them from and how to install them in non-default locations (say a home directory or something). g3 statically links against all libraries so you only need the libraries on your compile box.. not on every node in the cluster (that would be a bother!). % cd apps % vi gmetad.c you need to define different data sources at the top of the source file (of course later.. both commandline and configuration files will replace this). i would just set the data source to be 127.0.0.1:8649 and install g3 gmetad on a node running the old gmond. % cd .. % make % cd apps % ./gmetad and gmetad is running. it will spit out disk operations to stderr for you to watch. it will give you an idea of how much disk i/o is going to be reduced. i ran gmetad will two different data sources here at berkeley for a total of 134 machines (90 for one and 44 the other). one of the clusters is reporting pretty standard ganglia metrics but the other has lots of custom metrics (cpu temperature, etc). here is what i found. when g3 gmetad starts up it polls each data source to get all the data that is available. after that.. it only operates on the changes that have occurred since the last poll. da numbers poll | rrds updated (g3) | rrds updated (old g2) ===========+====================+============================ 1 (all) | 5245 | 5245 2 (diff) | 606 | 5245 3 (diff) | 513 | 5245 4 (diff) | 518 | 5245 5 (diff) | 409 | 5245 6 (diff) | 448 | 5245 7 (diff) | 346 | 5245 8 (diff) | 834 | 5245 9 (diff) | 403 | 5245 10 (diff) | 513 | 5245 total 9835 52450 per second 65 350 [these stats are pretty conservative since i ran the test during a very busy time for the clusters.. now that it has calmed down i'm getting numbers like 9, 82, 317, 15, 260, 41] based on this simple test.. you can expect g3 to use about 18% the disk i/o that g2 gmetad does. hell we might even be able to run it on a "real" filesystem instead of a ram-based one. CPU benchmarks show that it is on the CPU about 1 minute every hour overall for a whopping 1.6% of CPU (keep in mind that includes constructing summary data records every sample step). one of the big limitations of the old gmetad was that it was not interactive. to be honest.. the largest reason i didn't want to make it interactive was securitive. an interactive daemon is much more hackable than one that just writes to a socket and closes. another reason it wasn't interactive was because i didn't have a way to convert xml to an internal data storage structure and back again. the old gmetad just stored the raw XML in memory without processing it (aside from writing the data to rrds). that is no longer a problem. i have an API in libg3 that allows me to go from xml to mem to xml. i didn't want to reinvent any wheels in the way that gmetad interacts with the clients. i really really wanted gmetad to just do http. i spent about three days researching ways to make that happen. there were two approaches i looked at: http libraries (for embedding web server code into gmetad) and making gmetad an apache module. in the end both approaches (i think) won't work. the http libraries out there suck. some are not portable, some have a horrible api, some are built for completely different applications, and some require about a bajillion other libraries be available. i think the apache developers did a great job on making apache modular. running gmetad as a module would relieve alot of the development burden... gmetad would get a digested http request structure and answer it. unfortunately... i think that approach is not light-weight enough. imagine needing to install apache on all nodes in your cluster .. along with mod_ganglia. i hope in the future will have use http.. for now.. i have another idea (which i think will evolve into http in the future..) gmetad accept key/value pairs delimited by ':'. an empty line signals the end of data. currently gmetad understands the following keys... ou_regex: a case insensitive extended regular expression for the "ou"/"host" mu_regex: a case insensitive extended regular expression for the "mu"/"metric" depth: how deep in the tree to recurse client_time: timestamp from the client for dealing with time skew time_range: number of secs old the data can be to be returned (i didn't say that right). it means.. give me the data that has changed in the last x secs. all keys are optional. if you just send an empty line, you get everything (just like the old gmetad). gmetad simply ignores any keys that it doesn't understand. this will allow us to add new keys in the future without breaking old servers. they will just not handle the new key. say you want to know all cpu information for all hosts mu_regex:^/cpu say you want just the summary data mu_regex:^/cpu depth:1 say you want just the cpu data from a particular cluster ou_regex:My Particular Cluster mu_regex:^/cpu depth:1 say you want just the cpu data from a particular cluster over the last minute ou_regex:My particular cluster mu_regex:^/cpu depth:1 time_range:60 right now all operators are joined by an AND... OR is just around the corner. all the querying works in g3 gmetad so feel free to try it out. ok.. it's getting late in the day and i have too much more to talk about.. i'll just add one more thing... history. currently we use rrdtool to save/print historical information. in the near future.. i'd like to have gmetad serve history data in the xml.. for example, <ou id="Cluster Foo"> <host id="node01.foo.baz" ip="1.1.1.1"...> <mu id="cpu"> <metric id="user"...> <history time="1058487650" value="23"/> <history time="1058487000" value="21"/> <history time="1058486343" value="18"/> </metric> </mu> </host> </ou> right now gmetad does have that code.. i think it will be missing from 3.0.0. the advantage of that approach is that... we can add a new request key.. say history_range that will set the amount of history to return. that history information negates the need to move images over the network.. instead the client can easily convert the history xml to svg xml. much more efficient and it means that gmetad can easily share its history data with anyone in the world. currently .. you have to install the web frontend on the same machine running gmetad in order to have direct access to the historical information. that will disappear with this new approach. - -matt ps. i'll talk about gmond in g3 next time.. it's late and i have some 1942 to play. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (GNU/Linux) iD8DBQE/F0CAVmIXr0CKtmERAuxbAJ46CGOgC0iXheljk4lYXkxQbHD4BwCeJqcF FMqujS48Vina3uYLuBtjSz0= =szG2 -----END PGP SIGNATURE-----
