Hi, Came across this thread in which some ideas are discussed.
http://www.quora.com/What-is-a-good-choice-for-storing-blob-like-files-in-a-distributed-environment# Some quote from the thread There are two major classes of solutions for this. One is a distributed filesystem - but do yourself a favor and avoid Lustre. It has the same SPOF/bottleneck you mention for HDFS, and the general effort to keep it running is probably higher than you want. I'd suggest GlusterFS, OrangeFS, ExtreemFS or MooseFS as better alternatives, depending on your specific needs. (Disclaimer: I've been working with the Gluster team for years, and my company recently acquired theirs.) The other main type of solution is a dedicated "blob store" like OpenStack Storage (a.k.a. Swift) or S3. Multiple implementations of the S3 API exist, including "Walrus" in Eucalyptus, "tabled" in Project Hail, and "UFO" in GlusterFS. There are also in-between solutions that you have more of a filesystem interface (rather than HTTP-based) but implement blob-store semantics underneath. Examples of this include MogileFS and KosmosFS. Any of these would probably work for you. Since you're also looking for a webserver, though, I think I'd lean toward the Swift/S3 solutions that are already basically webservers themselves and don't require a separate one. Thanks. On Saturday, August 3, 2013, Prabath Abeysekera wrote: > Hi Sanjiva, > > On Sat, Aug 3, 2013 at 11:32 PM, Sanjiva Weerawarana > <sanj...@wso2.com<javascript:_e({}, 'cvml', 'sanj...@wso2.com');> > > wrote: > >> On Sat, Aug 3, 2013 at 11:06 PM, Prabath Abeysekera >> <praba...@wso2.com<javascript:_e({}, 'cvml', 'praba...@wso2.com');> >> > wrote: >> >>> >>>> I'm wondering whether using a relational DB blob store is good enough >>>> for a start. >>>> >>> >>> IMO, HDFS too doesn't have any added advantage over a Relational/Any >>> other blob store with respect to this particular use case as it's just >>> going to be used as a container for those media, etc files. However, >>> considering the OOTB capabilities of scalabilities and support for fault >>> tolerance, etc maybe HDFS/Cassandra would be a good choice? >>> >> >> Yes what they need is a scalable blob store. If Cassandra is good for >> that we can go with that since that's older and better understood. >> >> The main thing is that storing the file in the regular file system >> doesn't work in an (elastic) cluster environment. >> > > Understood. I believe, additionally, we'll also have to consider how the > data access is optimized (availability of streaming, specially since the > files can be quite larger), whether the storage option itself is optimized > for storing a few GBs of data (if that is part of the requirement), etc as > well. > > So, first, we'd have to check whether Cassandra is capable of handling > data contents of few hundreds of MBs (or maybe more than that). I believe, > that would/wouldn't make it a valid reason to use HDFS or any other > scalable blob store. However, it would be good to check on this a bit more > in detail, considering the aspects such as the availability of > multitenancy, etc so we can come up with the best suited solution for the > problem at hand. Will do a bit of researching and update the thread. > > > Cheers, > Prabath > > >> >> Sanjiva. >> -- >> Sanjiva Weerawarana, Ph.D. >> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >> email: sanj...@wso2.com <javascript:_e({}, 'cvml', 'sanj...@wso2.com');>; >> phone: +94 11 763 9614; cell: +94 77 787 6880 | +1 650 265 8311 >> blog: http://sanjiva.weerawarana.org/ >> >> Lean . Enterprise . Middleware >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org <javascript:_e({}, 'cvml', >> 'Architecture@wso2.org');> >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Prabath Abeysekara > Associate Technical Lead, Data TG. > WSO2 Inc. > Email: praba...@wso2.com <javascript:_e({}, 'cvml', > 'harsha...@wso2.com');> > Mobile: +94774171471 > > <http://harshana05.blogspot.com/> > -- *Hasitha Abeykoon* Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>* * * *
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
