Re: Passing multiple jar files to spark-shell

[Mich Talebzadeh]

 

Thanks Ted. I will have a look. 

regards 

On 15/02/2016 14:34, Ted Yu wrote: 

> Mich: 
> You can pass jars for driver using:
> 
> spark.driver.extraClassPath 
> 
> Cheers 
> 
> On Mon, Feb 15, 2016 at 1:05 AM, Mich Talebzadeh <m...@peridale.co.uk> wrote:
> 
> Thanks Deng. Unfortunately it seems that it looks for driver-class-path as 
> well L 
> 
> For example with -jars alone I get 
> 
> spark-shell --master spark://50.140.197.217:7077 [1] --jars 
> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar 
> 
> s: org.apache.spark.sql.DataFrame = [AMOUNT_SOLD: decimal(10,0), TIME_ID: 
> timestamp, CHANNEL_ID: bigint] 
> 
> warning: there were 1 deprecation warning(s); re-run with -deprecation for 
> details 
> 
> JAVA.SQL.SQLEXCEPTION: NO SUITABLE DRIVER FOUND FOR 
> JDBC:ORACLE:THIN:@RHES564:1521:MYDB 
> 
> Dr Mich Talebzadeh 
> 
> LinkedIn _ 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  [2]_ 
> 
> http://talebzadehmich.wordpress.com [3] 
> 
> NOTE: The information in this email is proprietary and confidential. This 
> message is for the designated recipient only, if you are not the intended 
> recipient, you should destroy it immediately. Any information in this message 
> shall not be understood as given or endorsed by Peridale Technology Ltd, its 
> subsidiaries or their employees, unless expressly so stated. It is the 
> responsibility of the recipient to ensure that this email is virus free, 
> therefore neither Peridale Technology Ltd, its subsidiaries nor their 
> employees accept any responsibility. 
> 
> FROM: odeach...@gmail.com [mailto:odeach...@gmail.com] ON BEHALF OF Deng 
> Ching-Mallete
> SENT: 15 February 2016 03:27
> TO: Mich Talebzadeh <m...@peridale.co.uk>
> CC: user <user@spark.apache.org>
> SUBJECT: Re: Passing multiple jar files to spark-shell 
> 
> Hi Mich, 
> 
> For the --jars parameter, just pass in the jars as comma-delimited. As for 
> the --driver-class-path, make it colon-delimited -- similar to how you set 
> multiple paths for an environment variable (e.g. --driver-class-path 
> /home/hduser/jars/jconn4.jar:/home/hduse/jars/ojdbc6.jar). But if you're 
> already passing the jar files via the --jars param, you shouldn't need to set 
> them in the driver-class-path though, since they should already be 
> automatically added to the classpath. 
> 
> HTH, 
> 
> Deng 
> 
> On Mon, Feb 15, 2016 at 7:35 AM, Mich Talebzadeh <m...@peridale.co.uk> wrote: 
> 
> Hi, 
> 
> Is there anyway one can pass multiple --driver-class-path and multiple -jars 
> to spark shell. 
> 
> For example something as below with two jar files entries for Oracle 
> (ojdbc6.jar) and Sybase IQ (jcoon4,jar) 
> 
> spark-shell --master spark://50.140.197.217:7077 [1] --driver-class-path 
> /home/hduser/jars/jconn4.jar --driver-class-path /home/hduser/jars/ojdbc6.jar 
> --jars /home/hduser/jars/jconn4.jar --jars /home/hduser/jars/ojdbc6.jar 
> 
> This works for one jar file only and you need to add the jar file to both 
> --driver-class-path and -jars. I have not managed to work for more than one 
> type of JAR file 
> 
> Thanks, 
> 
> Dr Mich Talebzadeh 
> 
> LinkedIn _ 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  [2]_ 
> 
> http://talebzadehmich.wordpress.com [3] 
> 
> NOTE: The information in this email is proprietary and confidential. This 
> message is for the designated recipient only, if you are not the intended 
> recipient, you should destroy it immediately. Any information in this message 
> shall not be understood as given or endorsed by Peridale Technology Ltd, its 
> subsidiaries or their employees, unless expressly so stated. It is the 
> responsibility of the recipient to ensure that this email is virus free, 
> therefore neither Peridale Technology Ltd, its subsidiaries nor their 
> employees accept any responsibility.

-- 

Dr Mich Talebzadeh

LinkedIn
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

http://talebzadehmich.wordpress.com

NOTE: The information in this email is proprietary and confidential.
This message is for the designated recipient only, if you are not the
intended recipient, you should destroy it immediately. Any information
in this message shall not be understood as given or endorsed by Cloud
Technology Partners Ltd, its subsidiaries or their employees, unless
expressly so stated. It is the responsibility of the recipient to ensure
that this email is virus free, therefore neither Cloud Technology
partners Ltd, its subsidiaries nor their employees accept any
responsibility.

 

Links:
--
[1] http://50.140.197.217:7077
[2]
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
[3] http://talebzadehmich.wordpress.com/

Re: Passing multiple jar files to spark-shell

[Ted Yu]

Mich:
You can pass jars for driver using:
  spark.driver.extraClassPath

Cheers

On Mon, Feb 15, 2016 at 1:05 AM, Mich Talebzadeh <m...@peridale.co.uk>
wrote:

> Thanks Deng. Unfortunately it seems that it looks for driver-class-path as
> well L
>
>
>
> For example with –jars alone I get
>
>
>
> spark-shell --master spark://50.140.197.217:7077 --jars
> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar
>
>
>
>
>
> s: org.apache.spark.sql.DataFrame = [AMOUNT_SOLD: decimal(10,0), TIME_ID:
> timestamp, CHANNEL_ID: bigint]
>
> warning: there were 1 deprecation warning(s); re-run with -deprecation for
> details
>
> *java.sql.SQLException: No suitable driver found for
> jdbc:oracle:thin:@rhes564:1521:mydb*
>
>
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Technology Ltd, its subsidiaries nor their
> employees accept any responsibility.
>
>
>
>
>
> *From:* odeach...@gmail.com [mailto:odeach...@gmail.com] *On Behalf Of *Deng
> Ching-Mallete
> *Sent:* 15 February 2016 03:27
> *To:* Mich Talebzadeh <m...@peridale.co.uk>
> *Cc:* user <user@spark.apache.org>
> *Subject:* Re: Passing multiple jar files to spark-shell
>
>
>
> Hi Mich,
>
>
>
> For the --jars parameter, just pass in the jars as comma-delimited. As for
> the --driver-class-path, make it colon-delimited -- similar to how you set
> multiple paths for an environment variable (e.g. --driver-class-path
> /home/hduser/jars/jconn4.jar:/home/hduse/jars/ojdbc6.jar). But if you're
> already passing the jar files via the --jars param, you shouldn't need to
> set them in the driver-class-path though, since they should already be
> automatically added to the classpath.
>
>
>
> HTH,
>
> Deng
>
>
>
> On Mon, Feb 15, 2016 at 7:35 AM, Mich Talebzadeh <m...@peridale.co.uk>
> wrote:
>
> Hi,
>
>
>
> Is there anyway one can pass multiple --driver-class-path and multiple
> –jars to spark shell.
>
>
>
> For example something as below with two jar files entries for Oracle
> (ojdbc6.jar) and Sybase IQ (jcoon4,jar)
>
>
>
> spark-shell --master spark://50.140.197.217:7077 --driver-class-path
> /home/hduser/jars/jconn4.jar  --driver-class-path
> /home/hduser/jars/ojdbc6.jar --jars /home/hduser/jars/jconn4.jar --jars
> /home/hduser/jars/ojdbc6.jar
>
>
>
>
>
> This works for one jar file only and you need to add the jar file to both 
> --driver-class-path
> and –jars. I have not managed to work for more than one type of JAR file
>
>
>
> Thanks,
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Technology Ltd, its subsidiaries nor their
> employees accept any responsibility.
>
>
>
>
>
>
>
>
>

RE: Passing multiple jar files to spark-shell

[Mich Talebzadeh]

Thanks Deng. Unfortunately it seems that it looks for driver-class-path as well 
:(

 

For example with –jars alone I get

 

spark-shell --master spark://50.140.197.217:7077 --jars 
/home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar

 

 

s: org.apache.spark.sql.DataFrame = [AMOUNT_SOLD: decimal(10,0), TIME_ID: 
timestamp, CHANNEL_ID: bigint]

warning: there were 1 deprecation warning(s); re-run with -deprecation for 
details

java.sql.SQLException: No suitable driver found for 
jdbc:oracle:thin:@rhes564:1521:mydb

 

 

 

Dr Mich Talebzadeh

 

LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> 

 

NOTE: The information in this email is proprietary and confidential. This 
message is for the designated recipient only, if you are not the intended 
recipient, you should destroy it immediately. Any information in this message 
shall not be understood as given or endorsed by Peridale Technology Ltd, its 
subsidiaries or their employees, unless expressly so stated. It is the 
responsibility of the recipient to ensure that this email is virus free, 
therefore neither Peridale Technology Ltd, its subsidiaries nor their employees 
accept any responsibility.

 

 

From: odeach...@gmail.com [mailto:odeach...@gmail.com] On Behalf Of Deng 
Ching-Mallete
Sent: 15 February 2016 03:27
To: Mich Talebzadeh <m...@peridale.co.uk>
Cc: user <user@spark.apache.org>
Subject: Re: Passing multiple jar files to spark-shell

 

Hi Mich,

 

For the --jars parameter, just pass in the jars as comma-delimited. As for the 
--driver-class-path, make it colon-delimited -- similar to how you set multiple 
paths for an environment variable (e.g. --driver-class-path 
/home/hduser/jars/jconn4.jar:/home/hduse/jars/ojdbc6.jar). But if you're 
already passing the jar files via the --jars param, you shouldn't need to set 
them in the driver-class-path though, since they should already be 
automatically added to the classpath.

 

HTH,

Deng

 

On Mon, Feb 15, 2016 at 7:35 AM, Mich Talebzadeh <m...@peridale.co.uk 
<mailto:m...@peridale.co.uk> > wrote:

Hi,

 

Is there anyway one can pass multiple --driver-class-path and multiple –jars to 
spark shell.

 

For example something as below with two jar files entries for Oracle 
(ojdbc6.jar) and Sybase IQ (jcoon4,jar)

 

spark-shell --master spark://50.140.197.217:7077 <http://50.140.197.217:7077>  
--driver-class-path /home/hduser/jars/jconn4.jar  --driver-class-path 
/home/hduser/jars/ojdbc6.jar --jars /home/hduser/jars/jconn4.jar --jars 
/home/hduser/jars/ojdbc6.jar

 

 

This works for one jar file only and you need to add the jar file to both 
--driver-class-path and –jars. I have not managed to work for more than one 
type of JAR file

 

Thanks,

 

 

Dr Mich Talebzadeh

 

LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> 

 

NOTE: The information in this email is proprietary and confidential. This 
message is for the designated recipient only, if you are not the intended 
recipient, you should destroy it immediately. Any information in this message 
shall not be understood as given or endorsed by Peridale Technology Ltd, its 
subsidiaries or their employees, unless expressly so stated. It is the 
responsibility of the recipient to ensure that this email is virus free, 
therefore neither Peridale Technology Ltd, its subsidiaries nor their employees 
accept any responsibility.

 

 

 

 

Re: Passing multiple jar files to spark-shell

[Deng Ching-Mallete]

Hi Mich,

For the --jars parameter, just pass in the jars as comma-delimited. As for
the --driver-class-path, make it colon-delimited -- similar to how you set
multiple paths for an environment variable (e.g. --driver-class-path
/home/hduser/jars/jconn4.jar:/home/hduse/jars/ojdbc6.jar). But if you're
already passing the jar files via the --jars param, you shouldn't need to
set them in the driver-class-path though, since they should already be
automatically added to the classpath.

HTH,
Deng

On Mon, Feb 15, 2016 at 7:35 AM, Mich Talebzadeh 
wrote:

> Hi,
>
>
>
> Is there anyway one can pass multiple --driver-class-path and multiple
> –jars to spark shell.
>
>
>
> For example something as below with two jar files entries for Oracle
> (ojdbc6.jar) and Sybase IQ (jcoon4,jar)
>
>
>
> spark-shell --master spark://50.140.197.217:7077 --driver-class-path
> /home/hduser/jars/jconn4.jar  --driver-class-path
> /home/hduser/jars/ojdbc6.jar --jars /home/hduser/jars/jconn4.jar --jars
> /home/hduser/jars/ojdbc6.jar
>
>
>
>
>
> This works for one jar file only and you need to add the jar file to both 
> --driver-class-path
> and –jars. I have not managed to work for more than one type of JAR file
>
>
>
> Thanks,
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Technology Ltd, its subsidiaries nor their
> employees accept any responsibility.
>
>
>
>
>

Re: Passing multiple jar files to spark-shell

[Sathish Kumaran Vairavelu]

--jars takes comma separated values.

On Sun, Feb 14, 2016 at 5:35 PM Mich Talebzadeh  wrote:

> Hi,
>
>
>
> Is there anyway one can pass multiple --driver-class-path and multiple
> –jars to spark shell.
>
>
>
> For example something as below with two jar files entries for Oracle
> (ojdbc6.jar) and Sybase IQ (jcoon4,jar)
>
>
>
> spark-shell --master spark://50.140.197.217:7077 --driver-class-path
> /home/hduser/jars/jconn4.jar  --driver-class-path
> /home/hduser/jars/ojdbc6.jar --jars /home/hduser/jars/jconn4.jar --jars
> /home/hduser/jars/ojdbc6.jar
>
>
>
>
>
> This works for one jar file only and you need to add the jar file to both 
> --driver-class-path
> and –jars. I have not managed to work for more than one type of JAR file
>
>
>
> Thanks,
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Technology Ltd, its subsidiaries nor their
> employees accept any responsibility.
>
>
>
>
>

Passing multiple jar files to spark-shell

[Mich Talebzadeh]

Hi,

 

Is there anyway one can pass multiple --driver-class-path and multiple -jars
to spark shell.

 

For example something as below with two jar files entries for Oracle
(ojdbc6.jar) and Sybase IQ (jcoon4,jar)

 

spark-shell --master spark://50.140.197.217:7077 --driver-class-path
/home/hduser/jars/jconn4.jar  --driver-class-path
/home/hduser/jars/ojdbc6.jar --jars /home/hduser/jars/jconn4.jar --jars
/home/hduser/jars/ojdbc6.jar

 

 

This works for one jar file only and you need to add the jar file to both
--driver-class-path and -jars. I have not managed to work for more than one
type of JAR file

 

Thanks,

 

 

Dr Mich Talebzadeh

 

LinkedIn

https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUr
V8Pw

 

  http://talebzadehmich.wordpress.com

 

NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Technology Ltd, its subsidiaries nor their
employees accept any responsibility.