I tried hdfs:/// and hdfs://cldbnode:7222/ Neither worked (examples below) I really think the hdfs vs other prefixes should be looked at. Like I said above, the tachyon project just added a env variable to address this.
hdfs://cldbnode:7222/ WARNING: Logging before InitGoogleLogging() is written to STDERR I0815 19:14:17.101666 22022 fetcher.cpp:76] Fetching URI 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' I0815 19:14:17.101780 22022 fetcher.cpp:105] Downloading resource from 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' to '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0003/executors/executor_Task_Tracker_5/runs/b3174e72-75ea-48be-bbb8-a9a6cc605018/hadoop-0.20.2-mapr-4.0.0.tgz' E0815 19:14:17.778833 22022 fetcher.cpp:109] HDFS copyToLocal failed: hadoop fs -copyToLocal 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0003/executors/executor_Task_Tracker_5/runs/b3174e72-75ea-48be-bbb8-a9a6cc605018/hadoop-0.20.2-mapr-4.0.0.tgz' WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. -copyToLocal: Wrong FS: maprfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz, expected: hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz Usage: hadoop fs [generic options] -copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst> Failed to fetch: hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz Failed to synchronize with slave (it's probably exited) hdfs:/// I0815 19:10:45.006803 21508 fetcher.cpp:76] Fetching URI 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' I0815 19:10:45.007099 21508 fetcher.cpp:105] Downloading resource from 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' to '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0002/executors/executor_Task_Tracker_2/runs/22689054-aff6-4f7c-9746-a068a11ff000/hadoop-0.20.2-mapr-4.0.0.tgz' E0815 19:10:45.681922 21508 fetcher.cpp:109] HDFS copyToLocal failed: hadoop fs -copyToLocal 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0002/executors/executor_Task_Tracker_2/runs/22689054-aff6-4f7c-9746-a068a11ff000/hadoop-0.20.2-mapr-4.0.0.tgz' WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. -copyToLocal: Wrong FS: maprfs:/mesos/hadoop-0.20.2-mapr-4.0.0.tgz, expected: hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz Usage: hadoop fs [generic options] -copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst> Failed to fetch: hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz Failed to synchronize with slave (it's probably exited) On Fri, Aug 15, 2014 at 5:38 PM, John Omernik <j...@omernik.com> wrote: > I am away from my cluster right now, I trued doing a hadoop fs -ls > maprfs:// and that worked. When I tries hadoop fs -ls hdfs:/// it failed > with wrong fs type. With that error I didn't try it in the mapred-site. I > will try it. Still...why hard code the file prefixes? I guess I am curious > on how glusterfs would work, or others as they pop up. > On Aug 15, 2014 5:04 PM, "Adam Bordelon" <a...@mesosphere.io> wrote: > >> Can't you just use the hdfs:// protocol for maprfs? That should work just >> fine. >> >> >> On Fri, Aug 15, 2014 at 2:50 PM, John Omernik <j...@omernik.com> wrote: >> >>> Thanks all. >>> >>> I realized MapR has a work around for me that I will try soon in that I >>> have MapR fs NFS mounted on each node, I.e. I should be able to get the tar >>> from there. >>> >>> That said, perhaps someone with better coding skills than me could >>> provide an env variable where a user could provide the HDFS prefixes to >>> try. I know we did that with the tachyon project and it works well for >>> other HDFS compatible fs implementations, perhaps that would work here? >>> Hard coding a pluggable system seems like a long term issue that will keep >>> coming up. >>> On Aug 15, 2014 4:02 PM, "Tim St Clair" <tstcl...@redhat.com> wrote: >>> >>>> The uri doesn't currently start with any of the known types (at least >>>> on 1st grok). >>>> You could redirect via a proxy that does the job for you. >>>> >>>> | if you had some fuse mount that would work too. >>>> >>>> Cheers, >>>> Tim >>>> >>>> ------------------------------ >>>> >>>> *From: *"John Omernik" <j...@omernik.com> >>>> *To: *user@mesos.apache.org >>>> *Sent: *Friday, August 15, 2014 3:55:02 PM >>>> *Subject: *Alternate HDFS Filesystems + Hadoop on Mesos >>>> >>>> I am on a wonderful journey trying to get hadoop on Mesos working with >>>> MapR. I feel like I am close, but when the slaves try to run the packaged >>>> Hadoop, I get the error below. The odd thing is, I KNOW I got Spark >>>> running on Mesos pulling both data and the packages from MapRFS. So I am >>>> confused why there is and issue with the fetcher.cpp here. Granted, when I >>>> got spark working, it was on 0.19.0, and I am trying a "fresh" version from >>>> git (0.20.0?) that I just pulled today. I am not sure if that work, but >>>> when I have more time I will try spark again. >>>> >>>> Any thoughts on this error? Thanks. >>>> >>>> >>>> >>>> >>>> Error: >>>> >>>> >>>> >>>> >>>> >>>> WARNING: Logging before InitGoogleLogging() is written to STDERR >>>> I0815 15:48:35.446071 20636 fetcher.cpp:76] Fetching URI >>>> 'maprfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' >>>> E0815 15:48:35.446184 20636 fetcher.cpp:161] A relative path was passed >>>> for the resource but the environment variable MESOS_FRAMEWORKS_HOME is not >>>> set. Please either specify this config option or avoid using a relative >>>> path >>>> Failed to fetch: maprfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz >>>> Failed to synchronize with slave (it's probably exited) >>>> >>>> >>>> >>>> >>>> -- >>>> Cheers, >>>> Timothy St. Clair >>>> Red Hat Inc. >>>> >>> >>
