Re: Re: Classpath hell and Elasticsearch 2.3.2...

Hi, Sorry to hear about your troubles. Not sure whether you are aware of the ES-Hadoop docs [1]. I've raised an issue [2] to better clarify the usage of elasticsearch-hadoop vs elasticsearch-spark jars. Apologies for the delayed response, for ES-Hadoop questions/issues it's best to use the

Re: Classpath hell and Elasticsearch 2.3.2...

i recently powered through this Spark + ElasticSearch integration, as well. you see this + many other Spark integrations with the PANCAKE STACK here: https://github.com/fluxcapacitor/pipeline all configs found here:

Re: Classpath hell and Elasticsearch 2.3.2...

Fair enough. However, if you take a look at the deployment guide ( http://spark.apache.org/docs/latest/submitting-applications.html#bundling-your-applications-dependencies) you will see that the generally advised approach is to package your app dependencies into a fat JAR and submit (possibly

Re: Classpath hell and Elasticsearch 2.3.2...

Yeah.. thanks Nick. Figured that out since your last email... I deleted the 2.10 by accident but then put 2+2 together. Got it working now. Still sticking to my story that it's somewhat complicated to setup :) Kevin On Thu, Jun 2, 2016 at 3:59 PM, Nick Pentreath

Re: Classpath hell and Elasticsearch 2.3.2...

Which Scala version is Spark built against? I'd guess it's 2.10 since you're using spark-1.6, and you're using the 2.11 jar for es-hadoop. On Thu, 2 Jun 2016 at 15:50 Kevin Burton wrote: > Thanks. > > I'm trying to run it in a standalone cluster with an existing / large 100

Re: Classpath hell and Elasticsearch 2.3.2...

Thanks. I'm trying to run it in a standalone cluster with an existing / large 100 node ES install. I'm using the standard 1.6.1 -2.6 distribution with elasticsearch-hadoop-2.3.2... I *think* I'm only supposed to use the elasticsearch-spark_2.11-2.3.2.jar with it... but now I get the following

Re: Classpath hell and Elasticsearch 2.3.2...

Hey there When I used es-hadoop, I just pulled in the dependency into my pom.xml, with spark as a "provided" dependency, and built a fat jar with assembly. Then with spark-submit use the --jars option to include your assembly jar (IIRC I sometimes also needed to use --driver-classpath too, but

Classpath hell and Elasticsearch 2.3.2...

I'm trying to get spark 1.6.1 to work with 2.3.2... needless to say it's not super easy. I wish there was an easier way to get this stuff to work.. Last time I tried to use spark more I was having similar problems with classpath setup and Cassandra. Seems a huge opportunity to make this easier