gudladona opened a new issue, #20601:
URL: https://github.com/apache/hudi/issues/20601
### Bug Description
**What happened:**
`hudi-spark-common` ships its own copy of
`org.apache.spark.sql.avro.AvroUtils`
(`hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala`,
"borrowed from Spark 3.3.0", consolidated into this module in #19147). Spark
4.0.x ships its own `AvroUtils` in the same package inside `spark-sql`. Both
compile to the same class, `org.apache.spark.sql.avro.AvroUtils$`, so whichever
comes first on the classpath is the one loaded.
Spark 4.0's `SchemaConverters` calls
`AvroUtils.nonNullUnionBranches(Schema)`. Hudi's copy doesn't define that
method. Whenever Hudi's copy comes first on the classpath, any Avro schema
conversion that goes through Spark's `SchemaConverters` fails with
`NoSuchMethodError`. Examples are a Maven test run of `hudi-spark4.0.x`, whose
own dependencies come before `spark-sql`, or
`spark.{driver,executor}.userClassPathFirst=true` with the bundle.
Evidence, from `javap` on the published `spark-sql_2.13` jars from Maven
Central:
| Spark | `SchemaConverters$` calls
`AvroUtils$.nonNullUnionBranches(Lorg/apache/avro/Schema;)Lscala/collection/immutable/Seq;`
| Spark's `AvroUtils$` defines it |
|---|---|---|
| 4.0.0 | yes | yes |
| 4.0.1 | yes | yes |
| 4.0.2 | yes | yes |
Hudi's `AvroUtils` on master (`a65a573a49ff`) doesn't define
`nonNullUnionBranches`, and `hudi-spark4.0.x` depends on `hudi-spark-common`.
This probably went unnoticed because the Spark 4.0 unit-test lanes in
`.github/workflows/bot.yml` were commented out in #19514. The remaining Spark
4.0 jobs (`java_ci_engines.yml`, `release_candidate_validation.yml`) run the
bundle through `spark-submit` on a 4.0.0 runtime, where Spark's own jars come
first and Spark's `AvroUtils` wins.
**What you expected:**
Avro schema conversion on Spark 4.0 works whatever the classpath order:
Hudi's `AvroUtils` either matches what Spark 4.0's `SchemaConverters` calls, or
doesn't shadow Spark's.
**Steps to reproduce:**
1. Build with `-Pspark4.0,scala-2.13` (JDK 17).
2. In `hudi-spark4.0.x`, run a test that converts an Avro schema containing
a union to a Spark type through `SchemaConverters.toSqlType`, for example via
`AvroConversionUtils.convertAvroSchemaToStructType`.
3. Observe `java.lang.NoSuchMethodError: 'scala.collection.immutable.Seq
org.apache.spark.sql.avro.AvroUtils$.nonNullUnionBranches(org.apache.avro.Schema)'`.
Possible fixes: add `nonNullUnionBranches` to the vendored copy (same
signature as Spark 4.0's), or stop vendoring `AvroUtils` for the Spark 4.x
modules and use Spark's.
### Environment
**Hudi version:** master (`a65a573a49ff`); also seen on a fork of 1.1.1
built against Spark 4.0.1
**Query engine:** Spark 4.0.x (Scala 2.13)
**Relevant configs:** none beyond the Spark 4.0 profile; the outcome depends
on classpath order
### Logs and Stack Trace
```
java.lang.NoSuchMethodError: 'scala.collection.immutable.Seq
org.apache.spark.sql.avro.AvroUtils$.nonNullUnionBranches(org.apache.avro.Schema)'
```
(Hit in a Spark 4.0.1 test run on a 1.1.1-based build. On master the
evidence is the bytecode check above; I have not run master's 4.0 module tests.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]