(my apologies for those who have received this already. i posted this
mail a few days back on the common-dev list, as this is more a
development related mail; but one of the original authors/maintainers
suggested to also post this here)
hi all,
i am a system administrator/user support person/... for the HPC team at
Ghent University (Ghent, Flanders, Belgium).
recently we have been asked to look into support for hadoop. for the
moment we are holding off on a dedicated cluster (esp dedicated hdfs setup).
but as all our systems are torque/pbs based, we looked into HOD to help
out our users.
we have started from the HOD code that was part of the hadoop 1.0.0
release (in the contrib part).
at first it was not working, but we have been patching and cleaning up
the code for a a few weeks and now have a version that works for us (we
had to add some features besides fixing a few things).
it looks sufficient for now, although we will add some more features
soon to get the users started.
my question is the following: what is the state of HOD atm? is it still
maintained/supported? are there forks somewhere that have more
up-to-date code?
what we are now missing most is the documentation (eg
http://hadoop.apache.org/common/docs/r0.16.4/hod.html) so we can update
this with our extra features. is the source available somewhere?
i could contribute back all patches, but a few of them are identation
fixes (to use 4 space indentation throughout the code) and other
cosmetic changes, so this messes up patches a lot.
i have also shuffled a bit with the options (rename and/or move to other
sections) so no 100% backwards compatibility with the current HOD code.
current main improvements:
- works with python 2.5 and up (we have been testing with 2.7.2)
- set options through environment variables
- better default values (we can now run with empty hodrc file)
- support for mail and nodes:ppn for pbs
- no deprecation warnings from hadoop (nearly finished)
- host-mask to bind xrs addr on non-default ip (in case you have
non-standard network on the compute nodes)
- more debug statements
- gradual code cleanup (using pylint)
on the todo list:
- further tuning of hadoop parameters (i'm not a hadoop user myself, so
this will take some time)
- 0.23.X support
many thanks,
stijn
