A DFS is merely a convenience for getting data to all the nodes of your mesos cluster. If you want to have all nodes retrieve data from HTTP servers (like S3), this is feasible as well. If you want your nodes to store data, you can have them write to any database (although I'd recommend something scalable, like Riak, rather than a DB like an unsharded MySQL)
On Mon, Oct 20, 2014 at 11:33 AM, CCAAT <cc...@tampabay.rr.com> wrote: > If one is building a mesos-cluster, then is a DFS mandatory, or is there > combinations of other codes that suffice for the needs of a mesos cluster? > > So what is the list of available Distributed File Systems that are > generally available and mesos is know to work on top of? > > Glusterfs, Lustrefs, FhGFS (BeeGFS), Ceph > > [1] http://en.wikipedia.org/wiki/Comparison_of_distributed_file_systems > > > An explicit list of other DFS's or combinations of codes that provide > these required 'feature sets' is of interest to many. > > > James > > > > > On 10/20/14 07:08, David Greenberg wrote: > >> You certainly don't need hdfs if you've got other infrastructure that's >> providing the same sorts of features. We don't run hdfs on our mesos >> cluster, and it's fine. >> >> On Monday, October 20, 2014, Ankur Chauhan <an...@malloc64.com >> <mailto:an...@malloc64.com>> wrote: >> >> __ >> Hi all, >> >> I am trying to setup a new mesos cluster and I so far I have a set >> of master and slave nodes working and I can get everything running. >> I am able to install and run a couple of sample apps, hookup jenkins >> etc. My main question now is Do I really need HDFS? All my artifacts >> (for apps) are on a protected S3 bucket or in a private docker >> registry. >> >> If I need HDFS, do I need to go "all in" even when I am not using >> hdfs as a data store but rather as a simple way to fetch files from >> s3; or can I get away with putting the correct core-site.xml and >> hdfs-site.xml in HADOOP_HOME and get away with it? >> >> It would really help how other have their mesos setup in production >> or what they would recommend regarding my setup? >> >> >> -- >> Ankur Chauhan >> >> >
