How about this: ADD_JARS="/home/hduser/jars/ojdbc6.jar" spark-shell
Thanks Deepak On Tue, Dec 27, 2016 at 5:04 PM, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Ok I tried this but no luck > > spark-shell --jars /home/hduser/jars/ojdbc6.jar > Spark context Web UI available at http://50.140.197.217:4041 > Spark context available as 'sc' (master = local[*], app id = > local-1482838526271). > Spark session available as 'spark'. > Welcome to > ____ __ > / __/__ ___ _____/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 2.0.0 > /_/ > Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java > 1.8.0_77) > Type in expressions to have them evaluated. > Type :help for more information. > scala> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) > warning: there was one deprecation warning; re-run with -deprecation for > details > HiveContext: org.apache.spark.sql.hive.HiveContext = > org.apache.spark.sql.hive.HiveContext@ad0bb4e > scala> //val sqlContext = new HiveContext(sc) > scala> println ("\nStarted at"); spark.sql("SELECT > FROM_unixtime(unix_timestamp(), 'dd/MM/yyyy HH:mm:ss.ss') > ").collect.foreach(println) > Started at > [27/12/2016 11:36:26.26] > scala> // > scala> var _ORACLEserver= "jdbc:oracle:thin:@rhes564:1521:mydb12" > _ORACLEserver: String = jdbc:oracle:thin:@rhes564:1521:mydb12 > scala> var _username = "scratchpad" > _username: String = scratchpad > scala> var _password = "oracle" > _password: String = oracle > scala> // > scala> val s = HiveContext.read.format("jdbc").options( > | Map("url" -> _ORACLEserver, > | "dbtable" -> "(SELECT ID, CLUSTERED, SCATTERED, RANDOMISED, > RANDOM_STRING, SMALL_VC, PADDING FROM scratchpad.dummy)", > | "partitionColumn" -> "ID", > | "lowerBound" -> "1", > | "upperBound" -> "100000000", > | "numPartitions" -> "10", > | "user" -> _username, > | "password" -> _password)).load > java.sql.SQLException: No suitable driver > at java.sql.DriverManager.getDriver(DriverManager.java:315) > at org.apache.spark.sql.execution.datasources.jdbc. > JdbcUtils$$anonfun$2.apply(JdbcUtils.scala:54) > at org.apache.spark.sql.execution.datasources.jdbc. > JdbcUtils$$anonfun$2.apply(JdbcUtils.scala:54) > at scala.Option.getOrElse(Option.scala:121) > at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$. > createConnectionFactory(JdbcUtils.scala:53) > at org.apache.spark.sql.execution.datasources.jdbc. > JDBCRDD$.resolveTable(JDBCRDD.scala:123) > at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>( > JDBCRelation.scala:117) > at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider. > createRelation(JdbcRelationProvider.scala:53) > at org.apache.spark.sql.execution.datasources. > DataSource.resolveRelation(DataSource.scala:315) > at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149) > at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:122) > ... 56 elided > > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 27 December 2016 at 11:23, Deepak Sharma <deepakmc...@gmail.com> wrote: > >> I meant ADD_JARS as you said --jars is not working for you with >> spark-shell. >> >> Thanks >> Deepak >> >> On Tue, Dec 27, 2016 at 4:51 PM, Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> Ok just to be clear do you mean >>> >>> ADD_JARS="~/jars/ojdbc6.jar" spark-shell >>> >>> or >>> >>> spark-shell --jars $ADD_JARS >>> >>> >>> Thanks >>> >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> On 27 December 2016 at 10:30, Deepak Sharma <deepakmc...@gmail.com> >>> wrote: >>> >>>> It works for me with spark 1.6 (--jars) >>>> Please try this: >>>> ADD_JARS="<<PATH_TO_JAR>>" spark-shell >>>> >>>> Thanks >>>> Deepak >>>> >>>> On Tue, Dec 27, 2016 at 3:49 PM, Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> Thanks. >>>>> >>>>> The problem is that with spark-shell --jars does not work! This is >>>>> Spark 2 accessing Oracle 12c >>>>> >>>>> spark-shell --jars /home/hduser/jars/ojdbc6.jar >>>>> >>>>> It comes back with >>>>> >>>>> java.sql.SQLException: No suitable driver >>>>> >>>>> unfortunately >>>>> >>>>> and spark-shell uses spark-submit under the bonnet if you look at the >>>>> shell file >>>>> >>>>> "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main >>>>> --name "Spark shell" "$@" >>>>> >>>>> >>>>> hm >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Dr Mich Talebzadeh >>>>> >>>>> >>>>> >>>>> LinkedIn * >>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>> >>>>> >>>>> >>>>> http://talebzadehmich.wordpress.com >>>>> >>>>> >>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>>> any loss, damage or destruction of data or any other property which may >>>>> arise from relying on this email's technical content is explicitly >>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> On 27 December 2016 at 09:52, Deepak Sharma <deepakmc...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Mich >>>>>> You can copy the jar to shared location and use --jars command line >>>>>> argument of spark-submit. >>>>>> Who so ever needs access to this jar , can refer to the shared path >>>>>> and access it using --jars argument. >>>>>> >>>>>> Thanks >>>>>> Deepak >>>>>> >>>>>> On Tue, Dec 27, 2016 at 3:03 PM, Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>>> When one runs in Local mode (one JVM) on an edge host (the host user >>>>>>> accesses the cluster), it is possible to put additional jar file say >>>>>>> accessing Oracle RDBMS tables in $SPARK_CLASSPATH. This works >>>>>>> >>>>>>> export SPARK_CLASSPATH=~/user_jars/ojdbc6.jar >>>>>>> >>>>>>> Normally a group of users can have read access to a shared directory >>>>>>> like above and once they log in their shell will invoke an environment >>>>>>> file >>>>>>> that will have the above classpath plus additional parameters like >>>>>>> $JAVA_HOME etc are set up for them. >>>>>>> >>>>>>> However, if the user chooses to run spark through spark-submit with >>>>>>> yarn, then the only way this will work in my research is to add the jar >>>>>>> file as follows on every node of Spark cluster >>>>>>> >>>>>>> in $SPARK_HOME/conf/spark-defaults.conf >>>>>>> >>>>>>> Add the jar path to the following: >>>>>>> >>>>>>> spark.executor.extraClassPath /user_jars/ojdbc6.jar >>>>>>> >>>>>>> Note that setting both spark.executor.extraClassPath and >>>>>>> SPARK_CLASSPATH >>>>>>> will cause initialisation error >>>>>>> >>>>>>> ERROR SparkContext: Error initializing SparkContext. >>>>>>> org.apache.spark.SparkException: Found both >>>>>>> spark.executor.extraClassPath and SPARK_CLASSPATH. Use only the former. >>>>>>> >>>>>>> I was wondering if there are other ways of making this work in YARN >>>>>>> mode, where every node of cluster will require this JAR file? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks >>>>>> Deepak >>>>>> www.bigdatabig.com >>>>>> www.keosha.net >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks >>>> Deepak >>>> www.bigdatabig.com >>>> www.keosha.net >>>> >>> >>> >> >> >> -- >> Thanks >> Deepak >> www.bigdatabig.com >> www.keosha.net >> > > -- Thanks Deepak www.bigdatabig.com www.keosha.net
