I finally seem to have gotten past this issue. Here’s what I did:
* rather than using the binary distribution, I built Spark from scratch to
eliminate the 4.1 version of org.apache.httpcomponents from the assembly
* git clone https://github.com/apache/spark.git
* cd spark
There is an undocumented configuration to put users jars in front of
spark jar. But I'm not very certain that it works as expected (and
this is why it is undocumented). Please try turning on
spark.yarn.user.classpath.first . -Xiangrui
On Sat, Sep 6, 2014 at 5:13 PM, Victor Tso-Guillen
I don't understand what you mean. Can you be more specific?
From: Victor Tso-Guillen v...@paxata.com
Sent: Saturday, September 06, 2014 5:13 PM
To: Penny Espinoza
Cc: Spark
Subject: Re: prepending jars to the driver class path for spark-submit on YARN
I ran
When you submit the job to yarn with spark-submit, set --conf
spark.yarn.user.classpath.first=true .
On Mon, Sep 8, 2014 at 10:46 AM, Penny Espinoza
pesp...@societyconsulting.com wrote:
I don't understand what you mean. Can you be more specific?
From: Victor
I have tried using the spark.files.userClassPathFirst option (which,
incidentally, is documented now, but marked as experimental), but it just
causes different errors. I am using spark-streaming-kafka. If I mark
spark-core and spark-streaming as provided and also exclude them from the
?VIctor - Not sure what you mean. Can you provide more detail about what you
did?
From: Victor Tso-Guillen v...@paxata.com
Sent: Saturday, September 06, 2014 5:13 PM
To: Penny Espinoza
Cc: Spark
Subject: Re: prepending jars to the driver class path for
I ran into the same issue. What I did was use maven shade plugin to shade
my version of httpcomponents libraries into another package.
On Fri, Sep 5, 2014 at 4:33 PM, Penny Espinoza
pesp...@societyconsulting.com wrote:
Hey - I’m struggling with some dependency issues with
