Mesos does not explicitly support HDFS and S3. Rather, Mesos will assume you have a hadoop binary and use it (blindly) for certain types of URIs. If the hadoop binary is not present, the mesos-fetcher will fail to fetch your HDFS or S3 URIs.
Mesos does not ship/package hadoop, so these URIs are not expected to work out of the box (for plain Mesos distributions). In all cases, the operator must preconfigure hadoop on each node (similar to how Docker in Mesos works). Here's the epic tracking the modularization of the mesos-fetcher (I estimate it'll be done by 0.30): https://issues.apache.org/jira/browse/MESOS-3918 ^ Once done, it should be easier to plug in more fetchers, such as one for your use-case. On Tue, May 10, 2016 at 11:21 AM, Briant, James < james.bri...@thermofisher.com> wrote: > I’m happy to have default IAM role on the box that can read-only fetch > from my s3 bucket. s3a gets the credentials from AWS instance metadata. It > works. > > If hadoop is gone, does that mean that hfds: URIs don’t work either? > > Are you saying dcos and mesos are diverging? Mesos explicitly supports > hdfs and s3. > > In the absence of S3, how do you propose I make large binaries available > to my cluster, and only to my cluster, on AWS? > > Jamie > > From: Cody Maloney <c...@mesosphere.io> > Reply-To: "user@mesos.apache.org" <user@mesos.apache.org> > Date: Tuesday, May 10, 2016 at 10:58 AM > To: "user@mesos.apache.org" <user@mesos.apache.org> > Subject: Re: Enable s3a for fetcher > > The s3 fetcher stuff inside of DC/OS is not supported. The `hadoop` binary > has been entirely removed from DC/OS 1.8 already. There have been various > proposals to make it so the mesos fetcher is much more pluggable / > extensible (https://issues.apache.org/jira/browse/MESOS-2731 for > instance). > > Generally speaking people want a lot of different sorts of fetching, and > there are all sorts of questions of how to properly get auth to the various > chunks (if you're using s3a:// presumably you need to get credentials there > somehow. Otherwise you could just use http://). Need to design / build > that into Mesos and DC/OS to be able to use this stuff. > > Cody > > On Tue, May 10, 2016 at 9:55 AM Briant, James < > james.bri...@thermofisher.com> wrote: > >> I want to use s3a: urls in fetcher. I’m using dcos 1.7 which has hadoop >> 2.5 on its agents. This version has the necessary hadoop-aws and aws-sdk: >> >> hadoop--afadb46fe64d0ee7ce23dbe769e44bfb0767a8b9]$ ls >> usr/share/hadoop/tools/lib/ | grep aws >> aws-java-sdk-1.7.4.jar >> hadoop-aws-2.5.0-cdh5.3.3.jar >> >> What config/scripts do I need to hack to get these guys on the classpath >> so that "hadoop fs -copyToLocal” works? >> >> Thanks, >> Jamie > >
