Re: spark-streaming "Could not compute split" exception

s/latest/running-on-yarn.html). On Tue, Sep 9, 2014 at 1:13 PM, Penny Espinoza mailto:pesp...@societyconsulting.com>> wrote: Hey - I have a Spark 1.0.2 job (using spark-streaming-kafka) that runs successfully using master = local[4]. However, when I run it on a Hadoop 2.2 EMR cluster using ma

Re: prepending jars to the driver class path for spark-submit on YARN

embly, eliminating all exclusions I had previously included to force use of org.apache.httpcomponents 4.1 On Sep 8, 2014, at 12:03 PM, Penny Espinoza mailto:pesp...@societyconsulting.com>> wrote: I have tried using the spark.files.userClassPathFirst option (which, incidentally, is document

spark-streaming "Could not compute split" exception

Hey - I have a Spark 1.0.2 job (using spark-streaming-kafka) that runs successfully using master = local[4]. However, when I run it on a Hadoop 2.2 EMR cluster using master yarn-client, it fails after running for about 5 minutes. My main method does something like this: 1. gets streaming

RE: prepending jars to the driver class path for spark-submit on YARN

?VIctor - Not sure what you mean. Can you provide more detail about what you did? From: Victor Tso-Guillen Sent: Saturday, September 06, 2014 5:13 PM To: Penny Espinoza Cc: Spark Subject: Re: prepending jars to the driver class path for spark-submit on YARN I

RE: prepending jars to the driver class path for spark-submit on YARN

: Sunday, September 07, 2014 11:40 PM To: Victor Tso-Guillen Cc: Penny Espinoza; Spark Subject: Re: prepending jars to the driver class path for spark-submit on YARN There is an undocumented configuration to put users jars in front of spark jar. But I'm not very certain that it works as expected

RE: prepending jars to the driver class path for spark-submit on YARN

I don't understand what you mean. Can you be more specific? From: Victor Tso-Guillen Sent: Saturday, September 06, 2014 5:13 PM To: Penny Espinoza Cc: Spark Subject: Re: prepending jars to the driver class path for spark-submit on YARN I ran into the

prepending jars to the driver class path for spark-submit on YARN

Hey - I’m struggling with some dependency issues with org.apache.httpcomponents httpcore and httpclient when using spark-submit with YARN running Spark 1.0.2 on a Hadoop 2.2 cluster. I’ve seen several posts about this issue, but no resolution. The error message is this: Caused by: java.lang.

spark-streaming-kafka with broadcast variable

I need to use a broadcast variable inside the Decoder I use for class parameter T in org.apache.spark.streaming.kafka.KafkaUtils.createStream. I am using the override with this signature: createStream