Re: [Ganglia-developers] gmond python modules location
IIRC at least some of the motivation behind the current split had to do with separating out modules which aren't generally platform agnostic since platform specific stuff sometimes causes problems with downstream linux distro adoption/packaging. I.E. the official python modules should be expected to work on the full set of platforms that gmond works on. -Dave On 02/05/2015 11:21 AM, Chris Burroughs wrote: Currently python modules are bundled in monitor-core, some are in gmond_python_modules (or os specific repos), and a few are in both. Ones in monitor-core (at least in RedHat-land) get their own package which makes them easy to install. I suspect they are also presumed to by users to be somehow more 'official' or at least that's how I thought of them. I don't think gmond_python_modules has releases or os packaging for example. I suspect some of this might reflect a pre-git situation where it was harder to contribute to monitoring-core. I'd like to propose the following for now: * For modules that are in both monitor-core and somewhere else, make the monitor-core version canonical and remove the other. * Everything else stays where it is. This will make it for to contribute patches are report issues without looking in two different places and manually syncing files. Longer term there is probably something fancy we could do such as build time flags to opt modules in, or maybe switch to a deb/rpm per module. That way 'is this module good enough' can be decoupled from which repo is it in. -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] Gmetad bottlenecks
On 01/15/2014 06:42 AM, Jesse Becker wrote: On Wed, Jan 15, 2014 at 8:41 AM, Nicholas Satterly nfsatte...@gmail.com wrote: If we are to look at redoing the XML parsing next then the two contenders that come to mind are gzipped JSON and Google Protocol Buffers. PB is meant to be very efficient and therefore faster, however it seems people have gotten comparable results with gzipped JSON. An obvious advantage of gzipped JSON is that it would be simple to make the output human readable though we could easily develop a CLI tool that allowed us to query and decode ganglia PB data for testing. I think there are large advantages to using standard and widely adopted formats, so JSON gets my vote there. That's not to say PB isn't widely used, but I suspect there are a lot more tools and programming language bindings to read and process JSON data. Or...why not just use XDR for bulk data transport, instead of introducing a third format (XML, XDR, and JSON/PB)? I'm not hugely familiar with XDR, so apologies if I'm missing an obvious reason why it wouldn't work here. +1 vote for XDR, it works great it is pretty damned efficient and the code to use it is ubiquitous. -Dave -- CenturyLink Cloud: The Leader in Enterprise Cloud Services. Learn Why More Businesses Are Choosing CenturyLink Cloud For Critical Workloads, Development Environments Everything In Between. Get a Quote or Start a Free Trial Today. http://pubads.g.doubleclick.net/gampad/clk?id=119420431iu=/4140/ostg.clktrk ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] system UUID (dmidecode)
Thanks for the suggestion, but dmidecode is not portable and is super
unreliable since it depends on upstream hardware vendors setting
reasonable values. Even running on a sample of 8-10 models of
motherboards all from the same vendor I get vastly different results
from dmidecode. I'd strongly discourage anybody from trying to integrate
it into other software for anything aside from general informational use.
-Dave
On 08/01/2013 01:50 AM, Adrian Sevcenco wrote:
Hi! Regarding the identification of hosts based on motherboard UUID i
thought to post some info to spare developers of ganglia of some wasted
seconds :)
So : on rhel systems you must do :
yum install python-dmidecode (so gmond should have dependency on this
package that is found in distros repo)
and the small python script (taken from the example of developers of
python-demidecode) would be like this (removed all comments for space
reasons) :
import dmidecode
import sys, os
from pprint import pprint
def print_warnings():
Simple function, dumping out warnings with a prefix if warnings
are found and clearing warning buffer
warn = dmidecode.get_warnings()
if warn:
print ### WARNING: %s % warn
dmidecode.clear_warnings()
root_user = (os.getuid() == 0 and True or False)
if not root_user:
print NOT RUNNING AS ROOT
dmixml = dmidecode.dmidecodeXML()
dmixml.SetResultType(dmidecode.DMIXML_DOC)
xmldoc = dmixml.QuerySection('all')
dmixp = xmldoc.xpathNewContext()
data = dmixp.xpathEval('/dmidecode/SystemInfo/SystemUUID')
for d in data:
print %s % (d.get_content())
del dmixp
del xmldoc
HTH,
Adrian
--
Get your SQL database under version control now!
Version control is standard for application code, but databases havent
caught up. So what steps can you take to put your SQL databases under
version control? Why should you start doing it? Read more to find out.
http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk
___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
--
Get your SQL database under version control now!
Version control is standard for application code, but databases havent
caught up. So what steps can you take to put your SQL databases under
version control? Why should you start doing it? Read more to find out.
http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk
___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] system UUID (dmidecode)
I 100% agree that we should be able to use some sort of unique and persistent way to identify hosts, but I don't think that dmidecode is the way to do it. Generating a UUID once and then persisting it in a file would be my preference, but I think that more generally if we were to allow a user specified callback to provide the ID string would be the ideal situation and would be the most flexible. Some users may want to use the UUID from a disk or a filesystem label, others may find that their vendors provide the info in something usable via dmidecode, and yet others may just want to populate hostname as the identifier. Moving away from DNS would be a big win, IMHO, for a large percentage of users, but lets not adopt something that is equally troublesome for some other equally large percentage of users... -Dave On 08/01/2013 10:24 AM, Chris Burroughs wrote: I think the more general point here (and has come up in other contexts such as mobile phones) is that sometimes neither IP nor reverse DNS make sense and people would like sending something else to be 'smoother' (as opposed to hacky/spoofy). My experience with vendor data hasn't given me warm and fuzzies either (TO BE ENTERED BY O.E.M) but I could see something like UUID on startup and 'custom bits on the motherboard' as reasonable -- if not common -- use cases. On 08/01/2013 12:20 PM, Dave Rawks wrote: Thanks for the suggestion, but dmidecode is not portable and is super unreliable since it depends on upstream hardware vendors setting reasonable values. Even running on a sample of 8-10 models of motherboards all from the same vendor I get vastly different results from dmidecode. I'd strongly discourage anybody from trying to integrate it into other software for anything aside from general informational use. -Dave -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] system UUID (dmidecode)
On 08/01/2013 10:23 AM, Peter Phaal wrote: Adrian, The Host sFlow agent exports UUIDs (and is compatible with Ganglia 3.2+). http://blog.sflow.com/2011/07/ganglia-32-released.html The daemon relies on command utilities invoked at startup to find the UUID (for reasons of portability that Dave mentioned). You could take a look at the logic in the Linux Host sFlow agent startup script and turn it into a Ganglia metric. http://sourceforge.net/p/host-sflow/code/413/tree/trunk/src/Linux/scripts/hsflowd Dave, the script uses the /usr/sbin/dmidecode command line tool. If it can't find a valid UUID, it uses the UUID of the first local disk. The UUID of the first disk is a completely different level of persistence than something less likely to be replaced during normal operation such as a CPU serial. But, realistically we aren't going to get something that is derived from hardware and meets everyone's definition of persistent and unique. What are the issues with the vendor supplied UUIDs? Are they all zeros? Not set? Not unique? All of the above. Do you have a better way of retrieving persistent UUIDs to Linux systems? Part of the problem is that people have differing levels of their need for persistence AND we need something agnostic of the underlying OS and host architecture. Not everyone running ganglia is running X86/AMD64 linux. Other operating systems provide different mechanism to read the UUID. You can browse the Host sFlow source code for the different supported platforms. Peter -Dave On Aug 1, 2013, at 9:20 AM, Dave Rawks d...@pandora.com mailto:d...@pandora.com wrote: Thanks for the suggestion, but dmidecode is not portable and is super unreliable since it depends on upstream hardware vendors setting reasonable values. Even running on a sample of 8-10 models of motherboards all from the same vendor I get vastly different results from dmidecode. I'd strongly discourage anybody from trying to integrate it into other software for anything aside from general informational use. -Dave -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] Possibility of using different serialization format than XDR
On 07/30/2013 05:53 AM, Chris Burroughs wrote: On 07/30/2013 02:45 AM, Nicholas Satterly wrote: Could you expand on what you mean by multi-tenancy, please? I'm curious. There is currently no application or other namespacing. If I have a bunch of java apps on the same box and they are all emitting jvm.memory.heap_usage, those metrics will clobber each other. Also you cannot have single node rolled up into more than one cluster. Clusters are defined purely by whatever the final gmond reports to gmetad. So individual nodes must all belong to a single cluster per sender/receiver config, so if for instance I have a cluster of UNIXHOSTS, and some of those are WEBHOSTS, DBHOSTS, MAILHOSTS, etc... You incur the duplication of network complexity and bandwidth in order (and RRDs and I/O if you use gmetad as well) to allow a node to participate in more than one of those clusters. Being able to arbitrarily tag nodes as members of clusters and then having gmetad smartly handle aggregate graphs driven one those tags would be a huge improvement. -Dave -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] Possibility of using different serialization format than XDR
I'm still trying to figure out what you're trying to improve here? XDR seems like a fine, standard, lightweight serialization protocol to use. It is already implemented and we've already got some protocol handling for backwards compat for really old ganglia monitor clients. What is there to gain from switching aside from having some new and shiny that needs to be supported in addition to the existing stuff? We aren't serializing any custom data types or references or anything aside from some floats, ints, and a couple of strings. XDR compute overhead is not hurting performance, especially on modern hardware, the payloads aren't very big and the tuning of various check timings and metric validity timings further reduces the amount of chatter on the wire. If you want to introduce some more modern code to ganglia I think adding support for pushing gmond communications into a modern pub/sub message queue framework. I've never heard anybody have problems with our serialization, but there is frequent and often confusing troubleshooting around multicast vs unicast and the various infrastructural/configuration tweaks to make the most out of those. Further improvements could probably be had in the arena of node multi-tenancy and/or arbitrary node grouping/clustering. Maybe I'm missing something that you've said or implied already, but this just seems like change for the sake of change. -Dave On 07/28/2013 02:09 PM, Nikhil wrote: Hi, Thanks for response. I see there is no averseness to the idea of considering different serialization format/protocol. Before we have any contribution in terms of code/specifications, what would be the ideal choice among these : http://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats for choosing the serialization format over the current XDR implementation in ganglia? As in like what is the current payload by XDR and what we should not intend to cross over, the performance overhead in processing and storing, the availability of libraries and ease of use being some of them that comes to thought of discussion. As Dave also mentions platform agnostic, portability (endianness?) and efficiency are also of the critical things to be considered. While ASN.1 does offer all of this, some of the others that I wanted to consider are : MessagePack and UBJson. Formats specs are described here for MessagePack http://wiki.msgpack.org/display/MSGPACK/Format+specification and for UBJson http://ubjson.org http://ubjson.org/ Let me know what do you all think would be the ideal choice. Thanks. On Sat, Jul 27, 2013 at 6:13 AM, Vladimir Vuksan vli...@veus.hr mailto:vli...@veus.hr wrote: I am not necessarily opposed to it if it's implemented in such a way not to break backwards compatibility. Someone would need to contribute some code. Vladimir On Fri, 26 Jul 2013, Dave Rawks wrote: I'm curious to hear what you think is going to be more efficient, platform agnostic and portable than XDR? ASN1 would be the only thing I would even consider using instead, but it is arguable whether it would be worth the pain of supporting more than one serialization format and it certainly doesn't seem sane to break all backwards compatibility to switch to something new unilaterally. ASN1 /might/ be a reasonable alternative to XDR, but I don't see what advantages this could possibly bring. -Dave On 7/26/13 10:46 AM, Nikhil wrote: Hi, Considering that we have better and compute efficient and binary serialization open formats out there . How hard would it to make Ganglia use them instead of XDR ? Can the serialization format engines be pluggable, instead of being closely integrated with XDR? Is it still worth continuing to stick with XDR? The intention is to understand and see the possibility and have a discussion what could be best to go with, if its appropriate. I am really hoping to see the reply from the authors of ganglia core :-) Thanks, Nikhil -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] Possibility of using different serialization format than XDR
I'm curious to hear what you think is going to be more efficient, platform agnostic and portable than XDR? ASN1 would be the only thing I would even consider using instead, but it is arguable whether it would be worth the pain of supporting more than one serialization format and it certainly doesn't seem sane to break all backwards compatibility to switch to something new unilaterally. ASN1 /might/ be a reasonable alternative to XDR, but I don't see what advantages this could possibly bring. -Dave On 7/26/13 10:46 AM, Nikhil wrote: Hi, Considering that we have better and compute efficient and binary serialization open formats out there . How hard would it to make Ganglia use them instead of XDR ? Can the serialization format engines be pluggable, instead of being closely integrated with XDR? Is it still worth continuing to stick with XDR? The intention is to understand and see the possibility and have a discussion what could be best to go with, if its appropriate. I am really hoping to see the reply from the authors of ganglia core :-) Thanks, Nikhil -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] web and debian/* files
It becomes problematic when the two /debian subdirs diverge and then you may end up with official debian packages owning a different set of files or having different install locations than ones built directly from upstream. Other packages which depend on or work in concert with packages that provide ganglia-web may break. And the support of two different sources of debian packages claiming to provide ganglia-web may be more than anyone really wants to deal with. -Dave On 07/19/2012 01:55 PM, Vladimir Vuksan wrote: Exactly. The other day I was actually missing the debian/ directory in ganglia-3.4.0 and as a result couldn't build it due to short timeline. Can we put stuff like that back in. If Debian guys don't like it we can strip it out for them. Vladimir On Thu, 19 Jul 2012, Bernard Li wrote: Hi Daniel: One reason why we would want to keep the debian/ directory in our repo is if for whatever reason the upstream Debian package doesn't get updated, a user could still download our official tarball and build Debian packages directly. Cheers, Bernard On Thu, Jul 19, 2012 at 7:41 AM, Daniel Pocockdan...@pocock.com.au wrote: The web download includes a debian/ directory with files for building a Debian package Debian also keeps a separate set of files for the same purpose in the Debian git VCS: git.debian.org/git/collab-maint/ganglia-web.git When importing release tarballs into the Debian VCS, it is necessary to filter out the upstream version of the debian/* files, they don't get used at all: git-import-orig -u 3.5.2 --filter='debian/*' ~/Downloads/ganglia-web_3.5.2.orig.tar.gz To simplify things and avoid duplication, debian/* could be omitted from future gweb releases (and even removed from the git repo) Would anyone object to that? I believe that the main advantage of tracking these files in Debian's git is that any Debian developer can update them and update an emergency bug fix release at any time. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] Trac Wiki, Bugzilla and GitHub
FWIW, the guys at github are eager to assist people with bulk importing of issues via their json issue format. And they have been trying to encourage people to write export tools for bugzilla and other services/software. I'd be a fan of moving from bugtracker to github just for the one-stop simplicity of it. However I think that tackling the export/import of migration may buy our project a bit of goodwill from others who would like to make that switch as well... -Dave On 05/10/2012 10:01 AM, Brad Nicholes wrote: +1 for sticking with bugzilla. If we can move it to somewhere that is more maintainable, that would be better. But I would hate to just abandon everything there. Brad On 5/10/2012 at 10:01 AM, in message ca+3xn_lykk8gveq0itiak980t7w4hk+awrpy0fk39bvhgpg...@mail.gmail.com, Bernard Libern...@vanhpc.org wrote: Hi Daniel: Just for the record, I actually like Bugzilla and would like to keep using it. However because we do not have direct ownership to the server (it is being hosted at UC Berkeley) it makes it hard to maintain. For instance it has currently been down for at least two days and so far I have not been able to get ahold of the admins who could tell us what's going on. This is not the first time it has happened. So either we move the Bugzilla instance to somewhere we have more control or we move them to GitHub Issues, it just can't stay where it is. I agree however that there are probably more bugs in Bugzilla than GitHub Issues so perhaps moving from GitHub Issues - Bugzilla and disabling GitHub Issues is the way to go. But I am also under the impression some folks like GitHub Issues better. Anybody else have any comments? Thanks! Bernard On Thursday, May 10, 2012, Daniel Pocock wrote: This is our request for help. We need someone to take charge of managing our documentation making sure they are up to date and in one canonical location. We'll also need someone to help with importing the bugs in Bugzilla to GitHub Issues. We definitely have to abandon bugzilla? Can we just turn off the issue tracker in github to avoid people opening issues in the wrong place? -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.netjavascript:; https://lists.sourceforge.net/lists/listinfo/ganglia-developers -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] bootstrap / autotools (renamed)
On 04/04/2012 09:51 AM, Daniel Pocock wrote: On 04/04/12 18:35, Dave Rawks wrote: Not to speak out of turn, but making the build process dependent on a very specific version of libtools and autotools seems like a really bad plan. I've never run into a debian source package before that demands to Only the `bootstrap' step recommends a particular version of autotools. There would be more QA risk if every revision (e.g. 3.3.5, 3.3.6, 3.3.7) was bootstrapped on a different autotools. Given the limited resources of the project, the easiest way to ensure consistency between releases is to make sure that we always bootstrap on a specified platform before making a release tarball. This only impacts people who want to work with source code directly from git (rather than using a tarball) Well, if you've ever tried to build the debian source package, in the middle of whatever is called from debian/rules build there is echo'd a stern warning about autotools versions and it looks like a few git commands are attempted but fail without stopping the build. -- Better than sec? Nothing is better than sec when it comes to monitoring Big Data applications. Try Boundary one-second resolution app monitoring today. Free. http://p.sf.net/sfu/Boundary-dev2dev ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
[Ganglia-developers] 3.3.4 tag doesn't build debian packages
Am I missing something? I just downloaded the tarball tagged 3.3.4 from github and attempted to build debian packages from it. 1. The debian changelog is not updated to reflect the version tag. 2. the build dependencies appear to be broken, as the build requires libtools to be installed but that isn't called out in the control file. 3. the build fails when it attempts to descend into the web subdirectory of the build tree. I've been away from this project for a few weeks now and I was under the impression for the traffic on this list that some of the version tagging/workflow questions have been heavily discussed. However it seems that the actual function of the source tree may have become slowly and subtly broken at some point in the recent past (circa 3.2~) At any rate I've not got a lot of time on my hands to fix this at the moment, but figured I'd at least say something in case that might rouse some interest in fixing it. -Dave -- Better than sec? Nothing is better than sec when it comes to monitoring Big Data applications. Try Boundary one-second resolution app monitoring today. Free. http://p.sf.net/sfu/Boundary-dev2dev ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] pyc support in mod_python
I don't think this is a good patch. .pyc files are not meant to be portable or stable. If at some later date we are meant to troubleshoot a problem with a module that is loaded as byte compiled python it becomes very difficult to gurantee that the bytecode is sane since the actual code is not evaluated. -Dave On 03/05/2012 01:47 AM, Ankit Srivastava wrote: Hi, I have added the changes in mod_python.c to support '.pyc' files present in /usr/lib/ganglia/python_modules directory. Please tell me the mailing list email address, where I can submit my changes for review. Changes are attached with this mail. regards, Ankit -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] Protocol Efficiency Ideas
On 1/27/12 6:59 AM, Im Root wrote: I believe that adding json would be a mistake. The reason is that when users install the main package there would be now a dependency on having json installed. It just adds to the complexity and helps to perpetuate RPM hell. I've had to deal with installing json in the past and it's been awful. It may be nice for a developer but not so nice for the end users. I'm not a Redhat user, but from looking at the json-c package requirements I'm not sure I see anything that wouldn't already be installed in even the smallest redhat install. Unless you've got some specific example of what dependency you are trying to avoid I'd have to say this claim is unsupported. -Dave -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] Protocol Efficiency Ideas
On 1/27/12 7:24 AM, Gladish, Jacob wrote: If you want access to that metric data in JSON format, the quickest way, by far, is to write a small conversion utility that fetches data from the gmetad. I can imagine a perl script running in Apache/CGI that does it on demand. You'd then have access to apache's caching capability. And if necessary, you could even make it so that you can query for scalars instead of the entire dataset. Imagine something like: wget http://my-gmetad-proxy:8080/gxml-to-json.pl?cluster=foohost=barmetric=baz Someone who's PHP savvy could possibly write this so that it fit into the ganglia web. I'm not sure RPM dependencies are an issue at this point as this could be something optionally installed or simply disabled if JSON packages where not present. Quickest maybe if your metric is time to implement without writing C code for the ganglia project... However, what I'm proposing is offering a tcp listener port to gmond, not gmetad, which could be queried directly and in a light weight fashion and would be entirely independent from even having a running gmetad. Also, ANY system which takes one format then parses it and reformats it into a another output format is not only going to reduce transport and computational load but it will greatly increase those loads. I can say this with 100% confidence as I've implemented such systems for not only ganglia, but also for opennms, and nagios. I don't want something that builds on top of gmetad (honestly I'm not even convinced that gmetad has much of a future in ganglia-monitor-core but that is a whole other discussion) rather I'm proposing that gmond be made more flexible and developer friendly as well as increasing the general efficiency of it's method of communication. gmond already has a nice lightweight binary protocol for nodes to communicate with one another, one which it is possible to code against directly, but it requires independently tracking the metadata hash as well as maintaining a state machine to honor DMAX and TMAX policy. XML OTOH is self describing and schemaless, but it is wildly inefficient in byte count and parsing it is quite computationally expensive especially in the case of large clusters where the XML from gmond might weigh in at 10's of MB per transaction (I measured 30MB on a cluster yesterday). Generating json in the same sort of streaming fashion that the current xml code works in should be mostly trivial and the end result would serve provide more options to both endusers and developers. -Dave -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] Protocol Efficiency Ideas
On 1/27/12 7:43 AM, Alex Dean wrote: 1. As a user, I sometimes want to consume ganglia data in other contexts (not just in gweb). I'd much prefer working with JSON instead of XML. I think that most developers would prefer json to xml, if for no other reason than json libraries/modules are available for all major languages AND they are much much much simpler to write against. 2. gweb currently allows users to download metrics as JSON. This is done by getting XML from gmetad, and transforming it into JSON in PHP. If gmetad could create JSON natively, gweb code could probably be streamlined. I agree! Using PHP to read XML and output json is a ridiculous inefficiency on a few levels. It means that people who want json have to run an entire webserver stack to generate the json, consumers must request the json via http instead of writing simple direct socket operations, the results of the fetch are proxied through whatever caching mechanisms which may be present in the webserver/php config and therefore may not come with any gurantee of freshness, the ultimate source of the data ends up being rrd files on disk which means even a single what is the current value of metric X could be a very slow operation instead of the quick memory based hash lookup that would be available if the json came from lower in the stack -Dave -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] Protocol Efficiency Ideas
On 1/27/12 2:05 PM, Im Root wrote: I forgot to add that by adding json, you will be restricting the types of Linux that this platform runs on. Although it may be a nice to have feature for a few developers who may want to customize things, this puts an additional dependency on the platform. You will be restricted to using only the flavors of linux that the keepers of json feel like using. Sorry, but adding the json dependency is a completely boneheaded move. I'm up for whatever sort of lively and informed debate you'd like to offer. Would you care to explain to me on which platforms json-c isn't available on? Maybe instead of calling ideas boneheaded you could be a bit more constructive in your criticism. FWIW to help inform the conversation... json-c, the proposed library for supplying json features is available under a standard MIT license free for use without restriction. It is available pre-packaged on all major linux, bsd, and solaris distributions and I'm fairly certin it build on 32 and 64 bit windows as well. It provides an open unrestricted and common interchange format for data that is available in every major programming language. Sooo.. Mister Root, where is the restriction that creates a platform dependency? P.S. typically this is something that would be build time configurable as well for those who don't care to include it in their build. AND the lib is smally enough that those who care to statically link their bins could easily do that too... -Dave -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
[Ganglia-developers] Protocol Efficiency Ideas
Hey All,
We've been talking about adding json in addition to xml for the tcp
listen port exchange format. And I was curious if the EXTRA_DATA
subtree to the XML ever contains something aside from EXTRA_ELEMENTS
and if the EXTRA_ELEMENTS ever have attributes aside from NAME and
VAL.
Just doing some back of napkin calculations it looks like reducing this
portion of the xml from:
METRIC NAME=swap_free VAL=47872928 TYPE=float UNITS=KB TN=24
TMAX=180 DMAX=0 SLOPE=both
EXTRA_DATA
EXTRA_ELEMENT NAME=GROUP VAL=memory/
EXTRA_ELEMENT NAME=DESC VAL=Amount of available swap memory/
EXTRA_ELEMENT NAME=TITLE VAL=Free Swap Space/
/EXTRA_DATA
/METRIC
to:
METRIC NAME=swap_free VAL=47872928 TYPE=float UNITS=KB TN=24
TMAX=180 DMAX=0 SLOPE=both
EXTRA_DATA GROUP=memory DESC=Amount of available swap memory
TITLE=Free Swap Space/
/METRIC
would be quite a savings over the wire. Of course this would break
compatibility with anything that currently exchanges xml with ganglia
monitor. But... That gets me back to json...
The current data structure from xml to json is something like this:
{metric:{name:'swap_free',val:47872928,type:'float',units:'KB',tn:24,tmax:180,dmax:0,slope:'both',extra_data:{extra_element:[{name:'GROUP',val:'memory'},{name:'DESC',val:'Amount
of available swap memory'},{name:'TITLE',val:'Free Swap Space'}]}}}
While the collapsed version ends up being this tiny json blob:
{metric:{name:'swap_free',val:47872928,type:'float',units:'KB',tn:24,tmax:180,dmax:0,slope:'both',extra_data:{group:'memory',desc:'Amount
of available swap memory',title:'Free Swap Space'}}}
On a relatively small cluster with a dozen metrics and a handful of
hosts the savings are minor. However on a cluster of hundreds of hosts
with perhaps dozens of metrics the savings would equate to MBs of data
per tcp fetch. And the parse speed of the json /should/ be much faster
as well.
Any comments/questions/ideas?
-Dave
--
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] Protocol Efficiency Ideas
On 01/26/2012 12:07 PM, Bryan Thompson wrote: Dave, There is undoubtably fat in the XML. However, I think that you would get far more savings by giving the client a means to request which data they are interested in. Right now it sends everything each time, right? It would be nice if you could provide a regex for what you needed. One option would be to put up an http end point which allowed query parameters that describe the data of interest. I've done this in a similar system in Java. Not sure what options are out there in C for embedded httpd. Bryan Such a thing already exists for requesting data from gmetad, the ganglia-monitor xml exchange however doesn't have any notion of limiting scope. Typically there is only one consumer of data from ganglia-monitor via xml, which is gmetad AND gmetad wants all the data every time, so I don't know that any sort of scope limit on request would be that great of a gain really. The json bit OTOH doesn't currently exist AND as such doesn't have any known consumers or consumption patterns. Adding some provider side filtering may be desirable, but I would speculate that most of the time the ganglia-monitor access pattern will be tell me everything and the per grid/cluster/host/group/metric filtering would happen up at the gmetad or gmetad-equivalent layer. I.E. maybe you replace gmetad with a message queue, the filtering would be provided by the topic subscription mechanism of your message queue system. -Dave -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] Protocol Efficiency Ideas
On 01/26/2012 01:16 PM, Jeff Buchbinder wrote: On Thu, Jan 26, 2012 at 4:09 PM, Bryan Thompson br...@systap.com wrote: Dave, I thought that the web UI was driven this way. Or is it gmond providing the XML output for the web UI (rather than gmetad)? The web UI is driven that way, but gmetad is providing the xml for that. I'm proposing changes to gmond not gmetad. -Dave -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] Protocol Efficiency Ideas
On 01/26/2012 02:02 PM, Bryan Thompson wrote: Ok. Not sure if this matters to your proposal, but GROUP is not always a single name/value pair. There can be multiple GROUPs for a metadata record. If there is more than one value for GROUP it could be an array instead of a string, That'd bump the efficiency a bit higher on that case, but cost a few bytes for the regular one or none case. Is gmetad really pulling the XML from gmond? Why not just embed the logic to listen for UDP packets and develop a soft state model of the cluster metrics in gmetad? Jeff, is that what you meant by a major rewrite of gmetad? Yeah gmetad hits each source to collect data, the source in this case is the tcp listener port of gmond, which only knows how to spew a full xml state table. I'm personally in favor of entirely killing off gmetad in favor of a more extensible gmond. If you could bolt on a storage, transport, or query plugin as easily as you could a metric module it'd be completely killer. -Dave -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
