Re: Add jar files on classpath when submitting tasks to Spark

2016-11-02 Thread Mich Talebzadeh
PM > > *To:* Jan Botorek <jan.boto...@infor.com> > *Cc:* Vinod Mangipudi <vinod...@gmail.com>; user <user@spark.apache.org> > *Subject:* Re: Add jar files on classpath when submitting tasks to Spark > > > > If you are using local mode then there is

RE: Add jar files on classpath when submitting tasks to Spark

2016-11-02 Thread Jan Botorek
il.com>; user <user@spark.apache.org> Subject: Re: Add jar files on classpath when submitting tasks to Spark If you are using local mode then there is only one JVM. In Linux as below mine looks like this ${SPARK_HOME}/bin/spark-submit \ --packages ${PACKAGES} \

Re: Add jar files on classpath when submitting tasks to Spark

2016-11-01 Thread Mich Talebzadeh
submit --class com.infor.skyvault.tests.LinearRegressionTest > --master local C:\_resources\spark-1.0-SNAPSHOT.jar > -DtrainDataPath="/path/to/model/data" > > > > > > > > *From:* Mich Talebzadeh [mailto:mich.talebza...@gmail.com] > *Sent:* Tuesday, November 1, 2016 2:

RE: Add jar files on classpath when submitting tasks to Spark

2016-11-01 Thread Jan Botorek
2016 2:51 PM To: Jan Botorek <jan.boto...@infor.com> Cc: Vinod Mangipudi <vinod...@gmail.com>; user <user@spark.apache.org> Subject: Re: Add jar files on classpath when submitting tasks to Spark Are you submitting your job through spark-submit? Dr Mich Talebzadeh LinkedI

Re: Add jar files on classpath when submitting tasks to Spark

2016-11-01 Thread Mich Talebzadeh
alebzadeh [mailto:mich.talebza...@gmail.com] > *Sent:* Tuesday, November 1, 2016 2:18 PM > *To:* Vinod Mangipudi <vinod...@gmail.com> > > *Cc:* user <user@spark.apache.org> > *Subject:* Re: Add jar files on classpath when submitting tasks to Sp

RE: Add jar files on classpath when submitting tasks to Spark

2016-11-01 Thread Jan Botorek
yan guha [mailto:guha.a...@gmail.com<mailto:guha.a...@gmail.com>] Sent: Tuesday, November 1, 2016 1:49 PM To: Jan Botorek <jan.boto...@infor.com<mailto:jan.boto...@infor.com>> Cc: user <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: Add jar files on class

Re: Add jar files on classpath when submitting tasks to Spark

2016-11-01 Thread Mich Talebzadeh
s >> „in global“ - in the spark-defaults.conf?? >> >> >> >> >> >> *From:* ayan guha [mailto:guha.a...@gmail.com] >> *Sent:* Tuesday, November 1, 2016 1:49 PM >> *To:* Jan Botorek <jan.boto...@infor.com> >> *Cc:* user <user@s

Re: Add jar files on classpath when submitting tasks to Spark

2016-11-01 Thread Vinod Mangipudi
location (directory) for jars > „in global“ - in the spark-defaults.conf?? > > > > > > *From:* ayan guha [mailto:guha.a...@gmail.com] > *Sent:* Tuesday, November 1, 2016 1:49 PM > *To:* Jan Botorek <jan.boto...@infor.com> > *Cc:* user <user@spark.apache.org> &g

RE: Add jar files on classpath when submitting tasks to Spark

2016-11-01 Thread Jan Botorek
, November 1, 2016 1:49 PM To: Jan Botorek <jan.boto...@infor.com> Cc: user <user@spark.apache.org> Subject: Re: Add jar files on classpath when submitting tasks to Spark There are options to specify external jars in the form of --jars, --driver-classpath etc depending on spark version

Re: Add jar files on classpath when submitting tasks to Spark

2016-11-01 Thread ayan guha
There are options to specify external jars in the form of --jars, --driver-classpath etc depending on spark version and cluster manager.. Please see spark documents for configuration sections and/or run spark submit help to see available options. On 1 Nov 2016 23:13, "Jan Botorek"

Add jar files on classpath when submitting tasks to Spark

2016-11-01 Thread Jan Botorek
Hello, I have a problem trying to add jar files to be available on classpath when submitting task to Spark. In my spark-defaults.conf file I have configuration: spark.driver.extraClassPath = path/to/folder/with/jars all jars in the folder are available in SPARK-SHELL The problem is that jars