Re: prepending jars to the driver class path for spark-submit on YARN

2014-09-09 Thread Penny Espinoza
m: Xiangrui Meng mailto:men...@gmail.com>> Sent: Sunday, September 07, 2014 11:40 PM To: Victor Tso-Guillen Cc: Penny Espinoza; Spark Subject: Re: prepending jars to the driver class path for spark-submit on YARN There is an undocumented configuration to put users jars in front of spark jar.

RE: prepending jars to the driver class path for spark-submit on YARN

2014-09-08 Thread Penny Espinoza
?VIctor - Not sure what you mean. Can you provide more detail about what you did? From: Victor Tso-Guillen Sent: Saturday, September 06, 2014 5:13 PM To: Penny Espinoza Cc: Spark Subject: Re: prepending jars to the driver class path for spark-submit on YARN I

RE: prepending jars to the driver class path for spark-submit on YARN

2014-09-08 Thread Penny Espinoza
: Sunday, September 07, 2014 11:40 PM To: Victor Tso-Guillen Cc: Penny Espinoza; Spark Subject: Re: prepending jars to the driver class path for spark-submit on YARN There is an undocumented configuration to put users jars in front of spark jar. But I'm not very certain that it works as expected

Re: prepending jars to the driver class path for spark-submit on YARN

2014-09-08 Thread Xiangrui Meng
Tso-Guillen > Sent: Saturday, September 06, 2014 5:13 PM > To: Penny Espinoza > Cc: Spark > Subject: Re: prepending jars to the driver class path for spark-submit on > YARN > > I ran into the same issue. What I did was use maven shade plugin to shade my > version of httpcompone

RE: prepending jars to the driver class path for spark-submit on YARN

2014-09-08 Thread Penny Espinoza
I don't understand what you mean. Can you be more specific? From: Victor Tso-Guillen Sent: Saturday, September 06, 2014 5:13 PM To: Penny Espinoza Cc: Spark Subject: Re: prepending jars to the driver class path for spark-submit on YARN I ran into the

Re: prepending jars to the driver class path for spark-submit on YARN

2014-09-07 Thread Xiangrui Meng
There is an undocumented configuration to put users jars in front of spark jar. But I'm not very certain that it works as expected (and this is why it is undocumented). Please try turning on spark.yarn.user.classpath.first . -Xiangrui On Sat, Sep 6, 2014 at 5:13 PM, Victor Tso-Guillen wrote: > I

Re: prepending jars to the driver class path for spark-submit on YARN

2014-09-06 Thread Victor Tso-Guillen
I ran into the same issue. What I did was use maven shade plugin to shade my version of httpcomponents libraries into another package. On Fri, Sep 5, 2014 at 4:33 PM, Penny Espinoza < pesp...@societyconsulting.com> wrote: > Hey - I’m struggling with some dependency issues with > org.apache.http