i've thought more about what steve said earlier and i think we might want
to use a simple timestamp instead of human-readable 8601 timestamps.
there are libraries that handle standard 8601 date strings but i can't
believe that it's faster than atoi(). we'll see .. i'll do some
benchmarks later.
i wanted to outline where i'm going with ganglia 3 as far as the overall
design and to share a cool little trick. i'll start with the cool trick.
one concern i have with a non-threaded design is blocking. of course
we'll use non-blocking i/o but there's still the metric collection
functions. if we have a function that gets stuck or is in an infinite
loop then it could gum up the works. (actually even our current design
w/threads can block too).
one solution is to use sigsetjump and siglongjump. this code snippet
runs on linux, cygwin, solaris, macos x (i've tested it). it _seems_
portable.
-------- code snippet ------------------
#include <stdio.h>
#include <setjmp.h>
#include <signal.h>
#include <unistd.h>
sigjmp_buf jmpbuf;
/* My signal handler */
void
sig_alrm( int signo )
{
/* Make bad_function look like it returned 1 */
siglongjmp(jmpbuf, 1);
}
/*
* I'm a bad function that just blocks forever
*/
int
bad_function( void )
{
for (;;)
{
fprintf(stderr,"bad_function running and running\n");
}
}
int
main ( void )
{
signal(SIGALRM, sig_alrm);
/* Give the bad_function 2 secs to run */
alarm(2);
if( sigsetjmp(jmpbuf, 1) )
{
/* We are returning from a siglongjmp */
fprintf(stderr,"\nTimeout\n");
return 1;
}
bad_function();
/* Turn off the alarm */
alarm(0);
return 0;
}
--------- end code snippet ---------------
we could use this approach to put a boundary on the amount of time they
get to run. it they exceed the time, an alert is sent and gmond keeps
right on chugging. we have to be smart though since on many operating
systems use SIGALRM for sleep(), etc. we'll see.
about the overall design of ganglia. this is what i propose and i'm
looking for feeedback.
g3mond -
acts just like a deaf 2.x gmond. it just loads all the metric modules
and send measurements on the wire. it will also have
a local unix socket open for other apps/daemons to bootstrap the
registered metric attributes. this will use the xml-> data functions
and is necessary to allow condensed network messages to be sent when
modules are obiquitous.
g3listend -
acts just like a mute 2.x gmond. it just listens to all the traffic
from one or more data sources, saves it, and exports it in different
formats (xml, ldif, sexpressions). we'll start with xml alone (since
it's already written :).
output modules will pass a file descriptor, callbacks to tree foreach
which recurses the tree structure.
if you run g3mond with a single data source being a multicast channel and
g3listend listening to the same multicast channel you have a 2.x gmond.
i don't want to lock people into multicast (although i personally think
it's the best way to do business) in g3. support for tcp and unicast udp
will be there (it's already in the library).
g3rrdd -
it just listens to one or more data sources and saves the data
to round-robin databases. (it could also be made to "serve"
historical images but i'm not sure it that is wise or not. thoughts?)
if you run g3xmld, with multiple data sources (to other g3xmld for
example) and g3rrdd on a machine.. you have a 2.x gmetad.
g3sqld -
it just listen to one or more data sources and save the data to a
sql database (i haven't written this.. i'm just throwing it out there).
all apps/daemons will get there configuration information from a single
/etc/ganglia.conf configuration file which i would like to have written
in.. you guessed it.. XML. again it'll just use the tree library. it
would look something like.
<daemon name="g3listend">
<section name="tranport">
<module name="g3XML" path="/var/lib/g3/output/g3XML.so"/>
<module name="globus_LDIF" path="/var/lib/g3/output/globus.so"/>
</section>
<section name="input">
<source name="Cluster A" ip="1.2.1.2" port="8649" format="XML" />
<source name="Cluster B" ip="1.2.1.4" port="8649" format="XML" />
</section>
<section name="output">
<export name="ganglia_port" protocol="TCP" port="8649" format="ganglia_XML"
/>
<export name="globus_port" protocol="TCP" port="8651"
format="globus_LDIF"/>
</section>
</daemon>
<daemon name="g3mond">
<section name="transport">
<module name="Sexp" path="/var/lib/g3/output/g3sexp.so"/>
</section>
<section name="metrics">
<module_dir name="g3 dir" path="/var/lib/ganglia/modules"/>
<module name="Single Module" path="/home/foobar/special/foo.so"/>
</section>
<section name="output">
<export name="Wire" protocol="UDP" ip="239.2.11.71" port="8649"
format="Sexp"/>
</section>
</daemon>
i just made all that XML up as an example (it's nothing solid). the nice
thing is that whatever the format it can be slurped in very easily. the
only assumption the tree library makes is that:
1. all tags have at least one attribute
2. the first attribute is the index attribute
3. the value of the attribute and the value of the tag combined must be
unique for each level of the xml tree.
that's it. it doesn't enforce any more structure than that so it'll
handle many different xml sources.
does this seem like a sane design? since we are moving to an event-driven
model i thought i was wise to break out the functions into separate
daemons.
what do you guys think?
the tree library also allows g3listend to quickly bootstrap from a data source.
i'll stop rambling now and get back to the code. what are your guys
schedules looking like in the next few weeks? can any of your contribute
code time to g3? i feel like i'm almost to the point that asking for some
time wouldn't be a waste of your time.
--
matt
