What workloads are you attempting to run exactly? Is this for an actual implementation or a research project?
Thanks, Aaron -- Aaron Carey Production Engineer - Cloud Pipeline Industrial Light & Magic London 020 3751 9150 ________________________________________ From: james [gar...@verizon.net] Sent: 11 May 2016 17:46 To: user@mesos.apache.org Subject: Re: distributed file systems On 05/11/2016 10:09 AM, Aaron Carey wrote: > What exactly do you mean by deploying a mesos cluster to run on ceph etc? > > Do you mean having a clustered file system mounted via nfs to the hosts which > contains the > mesos binaries? That would be one way to use a DFS, but low latency on a variety of different linux kernels will be evaluated too. So will the testing of ulibc-ng, musl and other fundamental components, as to the trade offs of performance vs robustness of supported frameworks. > Or something to do with how jobs are executed? YES, exactly. In fact, it is most interesting to test with Hi Performance Computing (HPC) on top of Mesos. The simple way to think about one of the ultimate goals of the research is how to completely replace Hadoop with Mesos, a DFS and many other components. Portability, not limited to Arm64v8 is a companion track related to HPC on clusters and bare metal hardware. But most HPC installations moving forward, will have to also support some typical (admin/web/security/AI/etc) types of workloads, as part of the evaluation. Mesos does seem to be flexible enough for these sorts of explorations and experiments. There is no inclination for constricting ideas and testing of competing components; only the desire to experiment, share and refine the component mix (aka the cluster-stack), all in open forums. I. E. Cluster-Stack_A vs Cluster-Stack_B etc etc for various workloads, on fixed, modest cluster sizes. Think of it as experimentation that will eventually lead to published results, such as what mips/mops and various benchmarks have done for hardware, only the cluster-stack is the variable(s) under study. All ideas, comments and criticisms are most welcome. HDFS is mostly described as the main bottleneck to HPC efforts and much of what folks are doing with various DFS replacements is being too strictly constrained, imho. James > Aaron Carey > Production Engineer - Cloud Pipeline > Industrial Light & Magic > London > 020 3751 9150 > > ________________________________________ > From: james [gar...@verizon.net] > Sent: 11 May 2016 17:08 > To: user@mesos.apache.org > Subject: Re: distributed file systems > > Hello Rodrick, > > That EFS looks interesting, but I did not find the location for the > source-code/git download? I do not remember the (linux) kernel hooks > for that Distributed File System, or is completely on top of the systems > codes? > > License details and I'm not sure if it's 100% opensource? > > Beegfs [A] is partially opensource, but that does not fit what is needed > for experimentation. A robust community around open sources and tools, > such as github, should have been mentioned. Equally important > is a community keen on sharing and supporting other efforts to replicate > and use the components of these cluster centric codes. [B,C] > > > James > > [A] http://www.beegfs.com/content/ > > [B] https://forums.aws.amazon.com/thread.jspa?threadID=217783 > > [C] > http://searchaws.techtarget.com/news/4500272286/Amazon-EFS-stuck-in-beta-lacks-native-Windows-support > > > > > On 05/11/2016 01:07 AM, Rodrick Brown wrote: >> Does EFS count? :-) >> >> https://aws.amazon.com/efs/ >> >> >> -- >> >> *Rodrick Brown* / Systems Engineer >> >> +1 917 445 6839 / rodr...@orchardplatform.com >> <mailto:char...@orchardplatform.com> >> >> *Orchard Platform* >> >> 101 5th Avenue, 4th Floor, New York, NY 10003 >> >> http://www.orchardplatform.com <http://www.orchardplatform.com/> >> >> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace >> Lending Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/> >> >> >> On May 10 2016, at 9:07 pm, james <gar...@verizon.net> wrote: >> >> Hello, >> >> >> Has anyone customized/compiled mesos and successfully deployed a mesos >> cluster to run on cephfs, orangefs [1], or any other distributed file >> systems? >> >> If so, some detail on your setup would be appreciated. >> >> >> [1] >> >> http://www.phoronix.com/scan.php?page=news_item&px=OrangeFS-Lands-Linux-4.6 >> >> >> *NOTICE TO RECIPIENTS*: This communication is confidential and intended >> for the use of the addressee only. If you are not an intended recipient >> of this communication, please delete it immediately and notify the >> sender by return email. Unauthorized reading, dissemination, >> distribution or copying of this communication is prohibited. This >> communication does not constitute an offer to sell or a solicitation of >> an indication of interest to purchase any loan, security or any other >> financial product or instrument, nor is it an offer to sell or a >> solicitation of an indication of interest to purchase any products or >> services to any persons who are prohibited from receiving such >> information under applicable law. The contents of this communication may >> not be accurate or complete and are subject to change without notice. As >> such, Orchard App, Inc. (including its subsidiaries and affiliates, >> "Orchard") makes no representation regarding the accuracy or >> completeness of the information contained herein. The intended recipient >> is advised to consult its own professional advisors, including those >> specializing in legal, tax and accounting matters. Orchard does not >> provide legal, tax or accounting advice. > >
