I have been using the spark from the last 6 months with the version 1.2.0.
I am trying to migrate to the 1.3.0 but the same problem i have written is
not wokring.
Its giving class not found error when i try to load some dependent jars
from the main program.
This use to work in 1.2.0 when set all
Try sc.addJar instead of setJars
Thanks
Best Regards
On Mon, Jun 22, 2015 at 8:24 PM, Murthy Chelankuri
wrote:
> I have been using the spark from the last 6 months with the version 1.2.0.
>
> I am trying to migrate to the 1.3.0 but the same problem i have written is
> not wokring.
>
> Its givin
Yes.
Thanks
Best Regards
On Mon, Jun 22, 2015 at 8:33 PM, Murthy Chelankuri
wrote:
> I have more than one jar. can we set sc.addJar multiple times with each
> dependent jar ?
>
> On Mon, Jun 22, 2015 at 8:30 PM, Akhil Das
> wrote:
>
>> Try sc.addJar instead of setJars
>>
>> Thanks
>> Best Rega
I am using spark streaming. what i am trying to do is sending few messages
to some kafka topic. where its failing.
java.lang.ClassNotFoundException: com.abc.mq.msg.ObjectEncoder
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.jav
Do you have Kafka producer in your classpath? If so how are adding that
library? Are you running on YARN, or Mesos or Standalone or local. These
details will be very useful.
On Mon, Jun 22, 2015 at 8:34 AM, Murthy Chelankuri
wrote:
> I am using spark streaming. what i am trying to do is sending
Yes I have the producer in the class path. And I am using in standalone mode.
Sent from my iPhone
> On 23-Jun-2015, at 3:31 am, Tathagata Das wrote:
>
> Do you have Kafka producer in your classpath? If so how are adding that
> library? Are you running on YARN, or Mesos or Standalone or local.
How are you adding that to the classpath? Through spark-submit or otherwise?
On Mon, Jun 22, 2015 at 5:02 PM, Murthy Chelankuri
wrote:
> Yes I have the producer in the class path. And I am using in standalone
> mode.
>
> Sent from my iPhone
>
> On 23-Jun-2015, at 3:31 am, Tathagata Das wrote:
>
I am invoking it from the java application by creating the sparkcontext
On Tue, Jun 23, 2015 at 12:17 PM, Tathagata Das wrote:
> How are you adding that to the classpath? Through spark-submit or
> otherwise?
>
> On Mon, Jun 22, 2015 at 5:02 PM, Murthy Chelankuri
> wrote:
>
>> Yes I have the pro
So you have Kafka in your classpath in you Java application, where you are
creating the sparkContext with the spark standalone master URL, right?
The recommended way of submitting spark applications to any cluster is
using spark-submit. See
https://spark.apache.org/docs/latest/submitting-applicati
yes , in spark standalone mode witht the master URL.
Jar are copying to execeutor and the application is running fine but its
failing at some point when kafka is trying to load the classes using some
reflection mechanisims for loading the Encoder and Partitioner classes.
Here are my finding so fa
This could be because of some subtle change in the classloaders used by
executors. I think there has been issues in the past with libraries that
use Class.forName to find classes by reflection. Because the executors load
classes dynamically using custom class loaders, libraries that use
Class.forNa
11 matches
Mail list logo