No, it just means you need the utility libraries to access the path. ----- Original Message ----- > From: "Ankur Chauhan" <an...@malloc64.com> > To: user@mesos.apache.org > Sent: Tuesday, October 21, 2014 11:18:11 AM > Subject: Re: Do i really need HDFS? > > So that means even if I don't use the dfs I would need HDFS namenode and data > node and related config to fetch s3 and s3n urns. > > Sent from my iPhone > > > On Oct 21, 2014, at 8:40 AM, Tim St Clair <tstcl...@redhat.com> wrote: > > > > Ankur - > > > > To answer your specific question re: > > Q: Is a s3 path considered non-hdfs? > > A: At this time no, it uses the hdfs layer to resolve (for better or > > worse). > > > > --------------------------------------------------------------------- > > // Grab the resource using the hadoop client if it's one of the known > > schemes > > // TODO(tarnfeld): This isn't very scalable with hadoop's pluggable > > // filesystem implementations. > > // TODO(matei): Enforce some size limits on files we get from HDFS > > if (strings::startsWith(uri, "hdfs://") || > > strings::startsWith(uri, "hftp://";) || > > strings::startsWith(uri, "s3://") || > > strings::startsWith(uri, "s3n://")) { > > Try<string> base = os::basename(uri); > > if (base.isError()) { > > LOG(ERROR) << "Invalid basename for URI: " << base.error(); > > return Error("Invalid basename for URI"); > > } > > string path = path::join(directory, base.get()); > > > > HDFS hdfs; > > > > LOG(INFO) << "Downloading resource from '" << uri > > << "' to '" << path << "'"; > > Try<Nothing> result = hdfs.copyToLocal(uri, path); > > if (result.isError()) { > > LOG(ERROR) << "HDFS copyToLocal failed: " << result.error(); > > return Error(result.error()); > > } > > --------------------------------------------------------------------- > > > > ----- Original Message ----- > > > >> From: "Ankur Chauhan" <an...@malloc64.com> > >> To: user@mesos.apache.org > >> Sent: Tuesday, October 21, 2014 10:28:50 AM > >> Subject: Re: Do i really need HDFS? > > > >> This is what I also intend to do. Is a s3 path considered non-hdfs? If so, > >> how does it know the credentials to use to fetch the file. > > > >> Sent from my iPhone > > > >> On Oct 21, 2014, at 5:16 AM, David Greenberg < dsg123456...@gmail.com > > >> wrote: > > > >>> We use spark without HDFS--in our case, we just use ansible to copy the > >>> spark > >>> executors onto all hosts at the same path. We also load and store our > >>> spark > >>> data from non-HDFS sources. > > > >>> On Tue, Oct 21, 2014 at 4:57 AM, Dick Davies < d...@hellooperator.net > > >>> wrote: > > > >>>> I think Spark needs a way to send jobs to/from the workers - the Spark > >> > >>>> distro itself > >> > >>>> will pull down the executor ok, but in my (very basic) tests I got > >> > >>>> stuck without HDFS. > > > >>>> So basically it depends on the framework. I think in Sparks case they > >> > >>>> assume most > >> > >>>> users are migrating from an existing Hadoop deployment, so HDFS is > >> > >>>> sort of assumed. > > > >>>> On 20 October 2014 23:18, CCAAT < cc...@tampabay.rr.com > wrote: > >> > >>>>> On 10/20/14 11:46, Steven Schlansker wrote: > >> > >> > >> > >>>>>> We are running Mesos entirely without HDFS with no problems. We use > >> > >>>>>> Docker to distribute our > >> > >>>>>> application to slave nodes, and keep no state on individual nodes. > >> > >> > >> > >> > >>>>> Background: I'm building up a 3 node cluster to run mesos and spark. No > >> > >>>>> legacy Hadoop needed or wanted. I am using btrfs for the local file > >>>>> system, > >> > >>>>> with (2) drives set up for raid1 on each system. > >> > >> > >>>>> So you are suggesting that I can install mesos + spark + docker > >> > >>>>> and not a DFS on these (3) machines? > >> > >> > >> > >>>>> Will I need any other softwares? My application is a geophysical > >> > >>>>> fluid simulator, so scala, R, and all sorts of advanced math will > >> > >>>>> be required on the cluster for the Finite Element Methods. > >> > >> > >> > >>>>> James > >> > >> > > > > -- > > > > -- > > Cheers, > > Timothy St. Clair > > Red Hat Inc. >
-- Cheers, Timothy St. Clair Red Hat Inc.
