Brian Bockelman wrote:
;) Unfortunately, I'm going to go out on a limb and guess that we don't
want to add OpenGL to the dependency list for the namenode... The viz
application actually doesn't depend on the namenode, it uses the datanodes.
Here's the source:
svn://t2.unl.edu/brian/HadoopViz/trunk
The server portion is a bit hardcoded to our site (simply a python
server); the client application is pretty cross-platform. I actually
compile and display the application on my Mac.
Here's how it works:
1) Client issues read() request
2) Datanode services it. Logs it with log4j
3) One of the log4j appenders is syslog pointing at a separate server
4) Separate log server recieves UDP packets; one packet per read()
5) Log server parses packets and decides whether they are within the
cluster or going to the internet
- Currently a Pentium 4 throw-away machine; handles up to 4-5k packets
per second before it starts dropping
6) Each client opens a TCP stream to the server and receives the
transfer type, source, and dest, then renders appropriately
It's pretty danged close to real-time; the time the client issues the
read() request to seeing something plotted is on the order of 1 second.
I'd really like to see this on a big (Yahoo, Facebook, any takers?)
cluster.
Brian
Ok, so this is really an example of a datacentre back-end for Log4J,
pushing out UDP packets to something else in the datacentre. A nice
side-line to the classic hadoop management displays. Add something about
jobs executing and you are laughing. Do it all in Java3D and you even
have cross platformness
