Hi Morven, We use —packages for all of our spark jobs. Spark downloads the specified jar and all of its dependencies from a Maven repository.
This means we never have to build fat or uber jars. It does mean that the Apache Ivy configuration has to be set up correctly though. Cheers, Steve C > On 15 Feb 2022, at 5:58 pm, Morven Huang <morven.hu...@gmail.com> wrote: > > I wrote a toy spark job and ran it within my IDE, same error if I don’t add > spark-avro to my pom.xml. After putting spark-avro dependency to my pom.xml, > everything works fine. > > Another thing is, if my memory serves me right, the spark-submit options for > extra jars is ‘--jars’ , not ‘--packages’. > > Regards, > > Morven Huang > > > On 2022/02/10 03:25:28 "Karanika, Anna" wrote: >> Hello, >> >> I have been trying to use spark SQL’s operations that are related to the >> Avro file format, >> e.g., stored as, save, load, in a Java class but they keep failing with the >> following stack trace: >> >> Exception in thread "main" org.apache.spark.sql.AnalysisException: Failed >> to find data source: avro. Avro is built-in but external data source module >> since Spark 2.4. Please deploy the application as per the deployment section >> of "Apache Avro Data Source Guide". >> at >> org.apache.spark.sql.errors.QueryCompilationErrors$.failedToFindAvroDataSourceError(QueryCompilationErrors.scala:1032) >> at >> org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:666) >> at >> org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:720) >> at >> org.apache.spark.sql.DataFrameWriter.lookupV2Provider(DataFrameWriter.scala:852) >> at >> org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:256) >> at >> org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239) >> at xsys.fileformats.SparkSQLvsAvro.main(SparkSQLvsAvro.java:57) >> at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) >> at >> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.base/java.lang.reflect.Method.invoke(Method.java:564) >> at >> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) >> at >> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955) >> at >> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) >> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) >> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) >> at >> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> For context, I am invoking spark-submit and adding arguments --packages >> org.apache.spark:spark-avro_2.12:3.2.0. >> Yet, Spark responds as if the dependency was not added. >> I am running spark-v3.2.0 (Scala 2.12). >> >> On the other hand, everything works great with spark-shell or spark-sql. >> >> I would appreciate any advice or feedback to get this running. >> >> Thank you, >> Anna >> >> > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > This email contains confidential information of and is the copyright of Infomedia. It must not be forwarded, amended or disclosed without consent of the sender. If you received this message by mistake, please advise the sender and delete all copies. Security of transmission on the internet cannot be guaranteed, could be infected, intercepted, or corrupted and you should ensure you have suitable antivirus protection in place. By sending us your or any third party personal details, you consent to (or confirm you have obtained consent from such third parties) to Infomedia’s privacy policy. http://www.infomedia.com.au/privacy-policy/
