Have you looked into MogileFS already? Seems like a good fit, based on your description. This question has come up more than once here, and MogileFS is an oft-recommended solution.
Norbert On 3/26/09, phil cryer <p...@cryer.us> wrote: > > When you say that you have huge images, how big is "huge?" > > > Yes, we're looking at some images that are 100Megs in size, but > nothing like what you're speaking of. This helps me understand > Hadoop's usage better and unfortunately it won't be the fit I was > hoping for. > > > > You can use the API or the FUSE module to mount hadoop but that is not > > a direct goal of hadoop. Hope that helps. > > > Very interesting, and yes, that indeed does help, not to veer off > thread too much, but does Sun's Lustre follow in the steps of Gluster > then? I know Lustre requires kernel patches to install, so it's at a > different level than the others, but I have seen some articles about > large scale clusters built with Lustre and want to look at that as > another option. > > Again, thanks for the info, if anyone has general information on > cluster software, or know of a more appropriate list, I'd appreciate > the advice. > > Thanks > > > P > > > On Thu, Mar 26, 2009 at 12:32 PM, Edward Capriolo <edlinuxg...@gmail.com> > wrote: > > It is a little more natural to connect to HDFS from apache tomcat. > > This will allow you to skip the FUSE mounts and just use the HDFS-API. > > > > I have modified this code to run inside tomcat. > > http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample > > > > I will not testify to how well this setup will perform under internet > > traffic, but it does work. > > > > GlusterFS is more like a traditional POSIX filesystem. It supports > > locking and appends and you can do things like put the mysql data > > directory on it. > > > > GLUSTERFS is geared for storing data to be accessed with low latency. > > Nodes (Bricks) are normally connected via GIG-E or infiniban. The > > GlusterFS volume is mounted directly on a unix system. > > > > Hadoop is a user space file system. The latency is higher. Nodes are > > connected by GIG-E. It is closely coupled with MAP/REDUCE. > > > > You can use the API or the FUSE module to mount hadoop but that is not > > a direct goal of hadoop. Hope that helps. > > >
