Hi, I am puzzled with this issue of Spark 3.1 version to read avro file. Everything is done on my local mac laptop so far, and I really don't know where the issue comes from, and I googled a lot and cannot find any clue.
I am always using Spark 2.4 version, as it is really mature. But for a new project, I want to taste Spark 3.1, which needs to read AVRO file. To my surprise, on my local, the Spark 3.1.3 throws error when trying to read the avro files. * I download the Spark 3.1.2 and 3.1.3 with Hadoop2 or 3 from https://spark.apache.org/downloads.html * Use JDK "1.8.0_321" on the Mac * Untar the spark 3.1.x local * And follow https://spark.apache.org/docs/3.1.3/sql-data-sources-avro.html Start the spark-shell in the following exactly command: spark-3.1.3-bin-hadoop3.2/bin/spark-shell --packages org.apache.spark:spark-avro_2.12:3.1.3 * And I always get the following error when read the existing test AVRO files: scala> val pageview = spark.read.format("avro").load("/Users/user/output/raw/") java.lang.NoClassDefFoundError: org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2 I tried different version of Spark 3.x, from Spark 3.1.2 -> 3.1.3 -> 3.2.1, and I believe they are all under Scala 2.12, and I start the spark-shell with "--packages org.apache.spark:spark-avro_2.12:x.x.x", which x.x.x matches the Spark version, but I got the above wired "NoClassDefFoundError" in all cases. Meantime, download Spark2.4.8 and start spark-shell with "--packages org.apache.spark:spark-avro_2.11:2.4.3", I can read the exactly same ARVO file without any issue. I am thinking it must be done wrongly on my end, but: * I downloaded several versions of Spark and untar them directly. * I DIDN'T have any custom "spark-env.sh/spark-default.conf" file to include any potential jar files to mess up things * Straight creating a spark session under spark-shell with the correct package and try to read avro files. Nothing more. I have to doubt there are something wrong with the Spark 3.x avro package releases, but I know that possiblity is very low, especailly for multi different veresions. But the class "org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2" existed under "spark-sql_2.12-3.1.3.jar", as blow: jar tvf spark-sql_2.12-3.1.3.jar | grep FileDataSourceV2 15436 Sun Feb 06 22:54:00 EST 2022 org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.class So what could be wrong? Thanks Yong