JkSelf opened a new pull request, #13215:
URL: https://github.com/apache/gluten/pull/13215
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid
duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR
title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
This adds a test that reproduces the `java.lang.NoSuchFieldError: chunkSize`
PySpark hits when
Arrow is enabled (`spark.sql.execution.arrow.pyspark.enabled=true`, e.g.
`createDataFrame(pandas_df)`
or `toPandas()`) and Gluten runs on a Spark 3.5 distribution whose Netty was
upgraded to 4.1.118 or
later.
Root cause: Spark 3.5 ships `arrow-memory-netty` 12.0.1. Its allocator reads
a Netty internal field
(`chunkSize`) that Netty 4.1.118 removed. Standalone allocation checks show:
| arrow-memory-netty | Netty 4.1.96 | Netty >= 4.1.118 (incl. 4.2.x) |
|---|---|---|
| 12.0.1 (Spark 3.5) / 14.0.2 | OK | `NoSuchFieldError: chunkSize` |
| 15.x (Gluten) / 18.1.0 (Spark 4.0) | OK | OK |
An in-process test cannot catch this, because Maven resolves Gluten's
`arrow-memory-netty` 15.0.0
with Spark's Netty 4.1.96 for `spark-3.5`, and those two are compatible.
`ArrowNettyCompatSuite`
therefore runs a small probe (`new RootAllocator()` plus a few allocations,
as Spark's
`ArrowUtils.rootAllocator` does) in a child JVM. That JVM's classpath
mirrors a deployment: Gluten's
`arrow-memory-core` first, then Spark 3.5's `arrow-memory-netty` 12.0.1 and
a chosen Netty version.
The allocation manager type is left to Arrow's classpath detection.
`maven-dependency-plugin` copies
the jars into `gluten-ut/test/target/netty-compat/` during
`generate-test-resources`. It is skipped
with `-DskipTests`, and the jars never enter the module's own classpath.
Cases:
- Netty 4.1.96: allocation succeeds. This is the control that shows the
harness is sound.
- Netty 4.1.118: fails with `NoSuchFieldError: chunkSize`.
- Netty 4.2.12: fails with `NoSuchFieldError: chunkSize`.
The two failure cases document the current behavior. Once a fix makes
Spark's `RootAllocator` work
with new Netty, they should be changed to assert success. The suite cancels
itself when Gluten's
Arrow is >= 16 (Spark 4.x builds), where the scenario does not exist.
## How was this patch tested?
The new suite was run locally on JDK 17:
./build/mvn test -pl gluten-ut/test -Pspark-ut,backends-velox,spark-3.5 \
-DwildcardSuites=none
-Dsuites=org.apache.gluten.memory.arrow.alloc.ArrowNettyCompatSuite
Result: `Tests: succeeded 3, failed 0, canceled 0`. Running the probe by
hand with the same
classpaths gives the outcomes in the table above. The cancel path for Arrow
>= 16 has not been run
yet.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-5-5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
