Re: spark with cdh 5.2.1

2015-02-04 Thread Mohit Jaggi
yep...it was unnecessary to create a 2.5 profile. i struggled a bit because it wasn't clear that i *need* to select a profile using -P option. i didn't have to do that for earlier hadoop versions. On Fri, Jan 30, 2015 at 12:11 AM, Sean Owen so...@cloudera.com wrote: There is no need for a 2.5

Re: spark with cdh 5.2.1

2015-01-30 Thread Sean Owen
There is no need for a 2.5 profile. The hadoop-2.4 profile is for Hadoop 2.4 and beyond. You can set the particular version you want with -Dhadoop.version= You do not need to make any new profile to compile vs 2.5.0-cdh5.2.1. Again, the hadoop-2.4 profile is what you need. On Thu, Jan 29, 2015

spark with cdh 5.2.1

2015-01-29 Thread Mohit Jaggi
Hi All, I noticed in pom.xml that there is no entry for Hadoop 2.5. Has anyone tried Spark with 2.5.0-cdh5.2.1? Will replicating the 2.4 entry be sufficient to make this work? Mohit. - To unsubscribe, e-mail:

Re: spark with cdh 5.2.1

2015-01-29 Thread Nobuhiro Sue
Mohit, I'm using spark modules provided by Cloudera repos, it works fine. Please add Cloudera maven repo, and specify dependencies with CDH version, like spark-core_2.10-1.1.0-cdh5.2.1. To add Cloudera maven repo, see: