Today, Federico Sacerdoti wrote forth saying...
> I dont understand. Floats and doubles often will be generated in binary.
> For example a detailed running time from MPI_Wtime() or a GPS
> coordinate. No matter how you slice it, storing it in ascii means a
> conversion. That conversion looses precision. We could store the
> exponent and mantissa as separate integers in ascii, but that is a
> different strategy with its own problems.
it's true that many of the system/library calls we use to collect metrics
return floats/doubles in binary form. even if we send/save the data in
binary form we still have to convert it once we want to output xml (since
xml is text). if we use ascii on the wire then each node is responsible
to convert its binary to ascii instead of having a single machine convert
every value for every host on the fly. the only difference is where the
conversion is happening... not if it happens or not. it makes more since
to distribute the conversion since it will make gmond answer MUCH faster.
your benchmarks on gmond demonstrated that gmond could only provide about
6 samples/sec for a cluster of 100 nodes. that's largely because of the
thousands of binary conversions that it needs to make. if everything in
memory is ascii.. no conversions are necessary.
keep in mind too that on linux everything is ascii thanks to /proc.
> > i forgot to mention that a host token can be empty for summary
> > information. for example...
> >
> > World:California:::cpu:number
> > ^^-------------- empty host token
> >
> > would point to the number of cpus available in California... in xml...
> >
>
> I still dont see the need for the unique organizational unit delimeter.
> Perhaps I'm missing something.
i'm probably not explaining myself very well.
i really don't like the CLUSTER and GRID tags. are a federation of
clusters really a Grid? is a federation of grids still a Grid? i'm sure
that over time the vocabulary will change. what about planetary-scale
systems or network overlays.. i mean.. it's just vocabulary and i want the
ganglia admins to define that vocabulary for themselves.
as far as the shorthand delimiters, i'm just looking for a simple way to
translate a block of xml to a filesytem like format. this is important
for the location of round-robin databases (or other database names for
that matter). it would be trivial to take
World:California:Berkeley:Millennium::mm24:cpu:user
and write it to
${rrd_root}/World/Calfornia/Berkeley/Millennium/mm24/cpu/user.rrd
for example or to a sql database name
World_California_Berkeley_Millennium_mm24_cpu column "user".
in a sense what i want is a quick way to collapse the xml to make it flat
and vise versa.
maybe if i explain why the tree library (as i demo'd to you) wouldn't
work.
the tree library is completely hash table based. there are no linked
lists. i wrote it that way intensionally to make it fast for
insert/reads/lookups etc. the only problem with using hash tables is that
you have to have a unique "key" at each level of the tree path.
for example,
when you insert data at
/processes/1034/cpu/user
in the tree. it creates an entry in the root level hash with the key
"processes" and a value which is a data structure. the value structure
has the attributes, name, value and children (all hash tables) for
"processes". the child hash table then gets a "1034" key pointing to
another value structure and then the "1034" child hash gets the "cpu" key
and so on.
root hash
|
+ key (processes) -> attrs
value
name = "processes"
children
|
+ key (1034) -> attrs
value
name = "1034"
children
|
+ key (cpu) -> attrs
value
name
children
say i wanted to do a tree_foreach on the tree it could be something like
this...
<processes ...>
<1034 ...>
<cpu ...>
<user .../>
</cpu>
</1034>
</processes>
but that doesn't work. what we want is
<mu name="processes">
<mu name="1034">
<mu name="cpu">
<metric name="user" .../>
</mu>
</mu>
</mu>
that's not a hard conversion to make but what happens when we throw in the
hosts...
/mm34/cpu/user
again.. we don't want
<mm34>
<cpu>
<user .../>
</cpu>
</mm34>
which is not what we want... but a simply conversion doesn't work
<mu name="mm34">
<mu name="cpu">
... etc etc etc...
i'm really sukin' at explaining myself here. i guess what i'm trying to
say is this... my tree library will not work for XML in general. it's not
a DOM processor. it works under certain assumptions that it needs to
work. the biggest assumption is that there is a unique index value
("name") at each level. that is the same assumption that databases make
as well.. just like a primary index in MySQL. without a primary index, we
don't have a database really.. we just have a list.
i can't use "host" or "mu" or "ou" as index values because they are not
unique...
my shorthand is just a way to simultaneously express primary index values
AND the xml context at the same time.
http://ganglia.sf.net/doc/index.html
for example does the same thing. "http" is a unique index for the
protocol. i know it's the protocol because it's the text up to ":".
ganglia.sf.net is the unique value for the host because it's trapped
between // and /. (actually 'ganglia' and 'sf' and 'net' are all unique
indexes in the dns database). i know that what follows are unique
directories that point to a unique file. it's a route to data from top to
bottom.
the same thing with
California:UCSD:NPACI Rocks::rock1:cpu:system
it is a unique path pointing to a value structure and can be expanded into
valid ganglia xml.
<ou name="California">
<ou name="UCSD">
<ou name="NPACI Rocks">
<host name="rock1">
<mu name="cpu">
<metric name="system" value.../>
</mu>
</host>
</ou>
</ou>
</ou>
here's an idea too... we _could_ even match the organization units to DNS
domains.
EDU:Berkeley:Millennium:Monitor::mm56:memory:total
for example. :) but we'd need to delimit the DNS portion somehow because
say...
ORG:RocksClusters:Meta:San Diego Grid:SDSC Rocks Grid:Meteor::computer-0-2
doesn't work... only part of the route (Meta.RocksClusters.ORG) points to
the machine with the data.
it's just an idea and i'm not sure how well that will work for us.
something to think about...
it's late and i'm tired and i think i might have just succeeded in
muddying up the waters even more.
-matt
ps. have you guys seen the latest pirate movie?
pps. it's rated "Aaaarrrrrr"
