Thanks a lot. I will give a try!
On Monday, March 16, 2015, Adam Lewandowski adam.lewandow...@gmail.com
wrote:
Prior to 1.3.0, Spark has 'spark.files.userClassPathFirst' for non-yarn
apps. For 1.3.0, use 'spark.executor.userClassPathFirst'.
See
https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3CCALrvLxdWwSByxNvcZtTVo8BsNRR_7tbPzWdUiAV8Ps8H1oAayQ%40mail.gmail.com%3E
On Fri, Mar 13, 2015 at 1:04 PM, Shuai Zheng szheng.c...@gmail.com
javascript:_e(%7B%7D,'cvml','szheng.c...@gmail.com'); wrote:
Hi All,
I am running spark to deal with AWS.
And aws sdk latest version is working with httpclient 3.4+. Then but
spark-assembly-*-.jar file has packaged an old httpclient version which
cause me: ClassNotFoundException for
org/apache/http/client/methods/HttpPatch
Even when I put the right httpclient jar there, it won’t help because
spark always take the class from same packaging first.
I don’t know why spark only provide a big package which doesn’t allow us
to customize the library loading sequence. I know I can just rebuild the
spark, but this is very troublesome, and it should not be a general
solution for long term (I can’t rebuild spark jar every time when have a
jar conflict as spark is supposed to be a cluster).
In hadoop, we have “mapreduce.job.user.classpath.first=true”. But
“spark.yarn.user.classpath.first” only work for Yarn.
I think I am not the one who face this issue. Anyone has a more general
solution for this?
Regards,
Shuai
