Hey Anthony,
I'd add you can look at our Ganglia stats:
http://rcf.unl.edu/ganglia/?c=red-workers
http://rcf.unl.edu/ganglia/?c=red-workers&h=node123&m=load_one&r=hour&s=descending&hc=4
The second one is the node-specific view, and you can see all the
Hadoop metrics integrated nicely with the system metrics.
Brian
On Jun 5, 2009, at 11:33 AM, Matt Massie wrote:
Anthony-
The ganglia web site is at http://ganglia.info/ with documentation
in a wiki at http://ganglia.wiki.sourceforge.net/. There is also a
good wiki page at IBM as well http://www.ibm.com/developerworks/wikis/display/WikiPtype/ganglia
. Ganglia packages are available for most distributions to help
with installation so make sure to grep for ganglia with your
favorite package manager (e.g. aptitude, yum, etc). Ganglia will
give you more information about your cluster than just Hadoop
metrics. You'll get CPU, load, memory, disk and network monitoring
as well for free.
You can see live demos of ganglia at http://ganglia.info/?page_id=69.
Good luck.
-Matt
On Jun 5, 2009, at 7:10 AM, Brian Bockelman wrote:
Hey Anthony,
Look into hooking your Hadoop system into Ganglia; this produces
about 20 real-time statistics per node.
Hadoop also does JMX, which hooks into more "enterprise"-y
monitoring systems.
Brian
On Jun 5, 2009, at 8:55 AM, Anthony McCulley wrote:
Hey all,
I'm currently tasked to come up with a web/flex-based
visualization/monitoring system for a cloud system using hadoop as
part of a
university research project. I was wondering if I could elicit some
feedback from all of you with regards to:
- If you were an engineer of a cloud system running hadoop, what
information would you be interested in capturing, viewing,
monitoring, etc?
- Is there any sort of real-time stats or monitoring currently
available
for hadoop? if so, is in a web-friendly format?
Thanks in advance,
- Anthony
