Are you sure com.google.api.client.http.HttpRequestInitialize is in the spark-bigquery-latest.jar or it may be in the transitive dependency of spark-bigquery_2.11?
On Sat, May 4, 2024 at 7:43 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > > Mich Talebzadeh, > Technologist | Architect | Data Engineer | Generative AI | FinCrime > London > United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* The information provided is correct to the best of my > knowledge but of course cannot be guaranteed . It is essential to note > that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von > Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > > > ---------- Forwarded message --------- > From: Mich Talebzadeh <mich.talebza...@gmail.com> > Date: Tue, 20 Oct 2020 at 16:50 > Subject: Why spark-submit works with package not with jar > To: user @spark <u...@spark.apache.org> > > > Hi, > > I have a scenario that I use in Spark submit as follows: > > spark-submit --driver-class-path /home/hduser/jars/ddhybrid.jar --jars > /home/hduser/jars/spark-bigquery-latest.jar,/home/hduser/jars/ddhybrid.jar, > */home/hduser/jars/spark-bigquery_2.11-0.2.6.jar* > > As you can see the jar files needed are added. > > > This comes back with error message as below > > > Creating model test.weights_MODEL > > java.lang.NoClassDefFoundError: > com/google/api/client/http/HttpRequestInitializer > > at > com.samelamin.spark.bigquery.BigQuerySQLContext.bq$lzycompute(BigQuerySQLContext.scala:19) > > at > com.samelamin.spark.bigquery.BigQuerySQLContext.bq(BigQuerySQLContext.scala:19) > > at > com.samelamin.spark.bigquery.BigQuerySQLContext.runDMLQuery(BigQuerySQLContext.scala:105) > > ... 76 elided > > Caused by: java.lang.ClassNotFoundException: > com.google.api.client.http.HttpRequestInitializer > > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > > So there is an issue with finding the class, although the jar file used > > > /home/hduser/jars/spark-bigquery_2.11-0.2.6.jar > > has it. > > > Now if *I remove the above jar file and replace it with the same version > but package* it works! > > > spark-submit --driver-class-path /home/hduser/jars/ddhybrid.jar --jars > /home/hduser/jars/spark-bigquery-latest.jar,/home/hduser/jars/ddhybrid.jar > *-**-packages com.github.samelamin:spark-bigquery_2.11:0.2.6* > > > I have read the write-ups about packages searching the maven > libraries etc. Not convinced why using the package should make so much > difference between a failure and success. In other words, when to use a > package rather than a jar. > > > Any ideas will be appreciated. > > > Thanks > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > -- Best Regards Jeff Zhang
