It wouldn't be too much of a stretch to use Lustre directly...
although it isn't trivial either.
You'd need to implement the 'FileSystem' interface for Lustre, define
a URI scheme (e.g. lfs://) etc. Please take a take a look at the KFS/
S3 implementations.
Arun
On Aug 21, 2008, at 9:59 AM, Joel Welling wrote:
Hi folks;
I'm new to Hadoop, and I'm trying to set it up on a cluster for which
almost all the disk is mounted via the Lustre filesystem. That
filesystem is visible to all the nodes, so I don't actually need
HDFS to
implement a shared filesystem. (I know the philosophical reasons why
people say local disks are better for Hadoop, but that's not the
situation I've got). My system is failing, and I think it's because
the
different nodes are tripping over each other when they try to run HDFS
out of the same directory tree.
Is there a way to turn off HDFS and just let Lustre do the
distributed
filesystem? I've seen discussion threads about Hadoop with NFS which
said something like 'just specify a local filesystem and everything
will
be fine', but I don't know how to do that. I'm using Hadoop 0.17.2.
Thanks, I hope;
-Joel Welling
[EMAIL PROTECTED]
