Mich Talebzadeh, Technologist | Architect | Data Engineer | Generative AI | FinCrime London United Kingdom
view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". ---------- Forwarded message --------- From: Mich Talebzadeh <mich.talebza...@gmail.com> Date: Tue, 20 Oct 2020 at 16:50 Subject: Why spark-submit works with package not with jar To: user @spark <u...@spark.apache.org> Hi, I have a scenario that I use in Spark submit as follows: spark-submit --driver-class-path /home/hduser/jars/ddhybrid.jar --jars /home/hduser/jars/spark-bigquery-latest.jar,/home/hduser/jars/ddhybrid.jar, */home/hduser/jars/spark-bigquery_2.11-0.2.6.jar* As you can see the jar files needed are added. This comes back with error message as below Creating model test.weights_MODEL java.lang.NoClassDefFoundError: com/google/api/client/http/HttpRequestInitializer at com.samelamin.spark.bigquery.BigQuerySQLContext.bq$lzycompute(BigQuerySQLContext.scala:19) at com.samelamin.spark.bigquery.BigQuerySQLContext.bq(BigQuerySQLContext.scala:19) at com.samelamin.spark.bigquery.BigQuerySQLContext.runDMLQuery(BigQuerySQLContext.scala:105) ... 76 elided Caused by: java.lang.ClassNotFoundException: com.google.api.client.http.HttpRequestInitializer at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) So there is an issue with finding the class, although the jar file used /home/hduser/jars/spark-bigquery_2.11-0.2.6.jar has it. Now if *I remove the above jar file and replace it with the same version but package* it works! spark-submit --driver-class-path /home/hduser/jars/ddhybrid.jar --jars /home/hduser/jars/spark-bigquery-latest.jar,/home/hduser/jars/ddhybrid.jar *-**-packages com.github.samelamin:spark-bigquery_2.11:0.2.6* I have read the write-ups about packages searching the maven libraries etc. Not convinced why using the package should make so much difference between a failure and success. In other words, when to use a package rather than a jar. Any ideas will be appreciated. Thanks *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
