Spark And Mapr
Hi We were using Horton 2.4.1 as our Hadoop distribution and now switched to MapR Previously to read a text file we would use : test = sc.textFile(\hdfs://10.48.101.111:8020/user/hdfs/test\) What would be the equivalent of the same for Mapr. Best Regards Santosh
Re: Spark And Mapr
There is doc on MapR: http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Applications -Vladimir Rodionov On Wed, Oct 1, 2014 at 3:00 PM, Addanki, Santosh Kumar santosh.kumar.adda...@sap.com wrote: Hi We were using Horton 2.4.1 as our Hadoop distribution and now switched to MapR Previously to read a text file we would use : test = sc.textFile(\hdfs://10.48.101.111:8020/user/hdfs/test\ http://10.48.101.111:8020/user/hdfs/test%5C) What would be the equivalent of the same for Mapr. Best Regards Santosh
RE: Spark And Mapr
Hi We would like to do this in PySpark Environment i.e something like test = sc.textFile(maprfs:///user/root/test) or test = sc.textFile(hdfs:///user/root/test) or Currently when we try test = sc.textFile(maprfs:///user/root/test) It throws the error No File-System for scheme: maprfs Best Regards Santosh From: Vladimir Rodionov [mailto:vrodio...@splicemachine.com] Sent: Wednesday, October 01, 2014 3:59 PM To: Addanki, Santosh Kumar Cc: user@spark.apache.org Subject: Re: Spark And Mapr There is doc on MapR: http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Applications -Vladimir Rodionov On Wed, Oct 1, 2014 at 3:00 PM, Addanki, Santosh Kumar santosh.kumar.adda...@sap.commailto:santosh.kumar.adda...@sap.com wrote: Hi We were using Horton 2.4.1 as our Hadoop distribution and now switched to MapR Previously to read a text file we would use : test = sc.textFile(\hdfs://10.48.101.111:8020/user/hdfs/test\http://10.48.101.111:8020/user/hdfs/test%5C) What would be the equivalent of the same for Mapr. Best Regards Santosh
Re: Spark And Mapr
Yes.. you should use maprfs:// I personally haven't used pyspark, I just used scala shell or standalone with MapR. I think you need to set classpath right, adding jar like /opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar to the classpath in the classpath. Sungwook On Wed, Oct 1, 2014 at 4:09 PM, Addanki, Santosh Kumar santosh.kumar.adda...@sap.com wrote: Hi We would like to do this in PySpark Environment i.e something like test = sc.textFile(maprfs:///user/root/test) or test = sc.textFile(hdfs:///user/root/test) or Currently when we try test = sc.textFile(maprfs:///user/root/test) It throws the error No File-System for scheme: maprfs Best Regards Santosh *From:* Vladimir Rodionov [mailto:vrodio...@splicemachine.com] *Sent:* Wednesday, October 01, 2014 3:59 PM *To:* Addanki, Santosh Kumar *Cc:* user@spark.apache.org *Subject:* Re: Spark And Mapr There is doc on MapR: http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Applications -Vladimir Rodionov On Wed, Oct 1, 2014 at 3:00 PM, Addanki, Santosh Kumar santosh.kumar.adda...@sap.com wrote: Hi We were using Horton 2.4.1 as our Hadoop distribution and now switched to MapR Previously to read a text file we would use : test = sc.textFile(\hdfs://10.48.101.111:8020/user/hdfs/test\ http://10.48.101.111:8020/user/hdfs/test%5C) What would be the equivalent of the same for Mapr. Best Regards Santosh
Re: Spark And Mapr
It should just work in PySpark, the same way it does in Java / Scala apps. Matei On Oct 1, 2014, at 4:12 PM, Sungwook Yoon sy...@maprtech.com wrote: Yes.. you should use maprfs:// I personally haven't used pyspark, I just used scala shell or standalone with MapR. I think you need to set classpath right, adding jar like /opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar to the classpath in the classpath. Sungwook On Wed, Oct 1, 2014 at 4:09 PM, Addanki, Santosh Kumar santosh.kumar.adda...@sap.com wrote: Hi We would like to do this in PySpark Environment i.e something like test = sc.textFile(maprfs:///user/root/test) or test = sc.textFile(hdfs:///user/root/test) or Currently when we try test = sc.textFile(maprfs:///user/root/test) It throws the error No File-System for scheme: maprfs Best Regards Santosh From: Vladimir Rodionov [mailto:vrodio...@splicemachine.com] Sent: Wednesday, October 01, 2014 3:59 PM To: Addanki, Santosh Kumar Cc: user@spark.apache.org Subject: Re: Spark And Mapr There is doc on MapR: http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Applications -Vladimir Rodionov On Wed, Oct 1, 2014 at 3:00 PM, Addanki, Santosh Kumar santosh.kumar.adda...@sap.com wrote: Hi We were using Horton 2.4.1 as our Hadoop distribution and now switched to MapR Previously to read a text file we would use : test = sc.textFile(\hdfs://10.48.101.111:8020/user/hdfs/test\) What would be the equivalent of the same for Mapr. Best Regards Santosh
Re: Spark And Mapr
As Sungwook said, the classpath pointing to the mapr jar is the key for that error. MapR has a Spark install that hopefully makes it easier. I don't have the instructions handy but you can asking their support about it. -Suren On Wed, Oct 1, 2014 at 7:18 PM, Matei Zaharia matei.zaha...@gmail.com wrote: It should just work in PySpark, the same way it does in Java / Scala apps. Matei On Oct 1, 2014, at 4:12 PM, Sungwook Yoon sy...@maprtech.com wrote: Yes.. you should use maprfs:// I personally haven't used pyspark, I just used scala shell or standalone with MapR. I think you need to set classpath right, adding jar like /opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar to the classpath in the classpath. Sungwook On Wed, Oct 1, 2014 at 4:09 PM, Addanki, Santosh Kumar santosh.kumar.adda...@sap.com wrote: Hi We would like to do this in PySpark Environment i.e something like test = sc.textFile(maprfs:///user/root/test) or test = sc.textFile(hdfs:///user/root/test) or Currently when we try test = sc.textFile(maprfs:///user/root/test) It throws the error No File-System for scheme: maprfs Best Regards Santosh *From:* Vladimir Rodionov [mailto:vrodio...@splicemachine.com] *Sent:* Wednesday, October 01, 2014 3:59 PM *To:* Addanki, Santosh Kumar *Cc:* user@spark.apache.org *Subject:* Re: Spark And Mapr There is doc on MapR: http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Applications -Vladimir Rodionov On Wed, Oct 1, 2014 at 3:00 PM, Addanki, Santosh Kumar santosh.kumar.adda...@sap.com wrote: Hi We were using Horton 2.4.1 as our Hadoop distribution and now switched to MapR Previously to read a text file we would use : test = sc.textFile(\hdfs://10.48.101.111:8020/user/hdfs/test\ http://10.48.101.111:8020/user/hdfs/test%5C) What would be the equivalent of the same for Mapr. Best Regards Santosh -- SUREN HIRAMAN, VP TECHNOLOGY Velos Accelerating Machine Learning 440 NINTH AVENUE, 11TH FLOOR NEW YORK, NY 10001 O: (917) 525-2466 ext. 105 F: 646.349.4063 E: suren.hiraman@v suren.hira...@sociocast.comelos.io W: www.velos.io
