wangyum opened a new issue, #12564:
URL: https://github.com/apache/gluten/issues/12564
## Description
The shaded \`gluten-package.jar\` bundles
\`org.apache.arrow.memory.DefaultAllocationManagerFactory.class\` (and the
\`unsafe/\` / \`netty/\` subpath variants in newer Arrow versions) because the
shade plugin's relocation excludes \`org.apache.arrow.memory.**\` from
relocation but still includes the class files in the bundle:
\`\`\`xml
<relocation>
<pattern>org.apache.arrow</pattern>
<shadedPattern>${gluten.shade.packageName}.org.apache.arrow</shadedPattern>
<excludes>
<exclude>org.apache.arrow.memory.**</exclude> <!-- excluded from
relocation, but still bundled -->
...
</excludes>
</relocation>
\`\`\`
This bundled class is **redundant** — the same class is always available on
the runtime classpath via the upstream \`arrow-memory-unsafe\` /
\`arrow-memory-netty\` jar (a declared dependency of \`gluten-arrow\`). Worse,
starting with **Arrow 16.0.0**,
\`org.apache.arrow.memory.CheckAllocator.check()\` inspects the URL of
\`DefaultAllocationManagerFactory.class\` and throws
\`\`\`
java.lang.IllegalStateException: Unknown allocation manager type to infer.
Current:
file:/.../gluten-package_2.12-1.7.0-SNAPSHOT.jar!/org/apache/arrow/memory/DefaultAllocationManagerFactory.class
at org.apache.arrow.memory.CheckAllocator.check(CheckAllocator.java:57)
\`\`\`
when the URL does not contain \`memory-core\`, \`memory-unsafe\`, or
\`memory-netty\` (which it does not when the class is loaded from inside
\`gluten-package.jar\`).
## When this manifests
\`CheckAllocator\`'s URL check was added in **Arrow 16.0.0**. The bundled
class lives at the CORE path
\`org/apache/arrow/memory/DefaultAllocationManagerFactory.class\` in Arrow <=
15.x, and at subpaths (\`unsafe/\` / \`netty/\`) in Arrow >= 16.x. So the
failure occurs when **all** of the following are true:
1. Gluten is built with \`arrow.version\` <= 15.x (default for
spark-3.3/3.4/3.5 profiles → bundles the CORE-path class)
2. The runtime Arrow version on the Spark classpath is >= 16.0.0 (which has
the new URL-aware \`CheckAllocator\`)
3. \`arrow.allocation.manager.type\` is not explicitly set (the default
\`Unknown\` case falls through to \`CheckAllocator.check()\`)
This combination happens when a Spark distribution ships a newer Arrow than
what Gluten was built against. For example, a downstream Spark 3.5 fork that
upgrades Arrow from 12.0.1 to 18.3.0 will trigger this on every Gluten query,
because Gluten's \`spark-3.5\` profile still uses \`arrow.version=15.0.0\`.
## Why upstream CI does not currently catch this
- Spark 3.3/3.4/3.5 community → Arrow 12.0.1 → \`CheckAllocator\` predates
the URL check → works
- Spark 4.0/4.1 → \`spark-4.0\` profile overrides \`arrow.version\` to
18.1.0 → gluten-package bundles the \`unsafe/\`-subpath class → URL contains
\`/org/apache/arrow/memory/unsafe/\` → matches → works
So the issue is silent on upstream today but breaks for any custom Spark
distribution that bumps Arrow past 16.0.0 without also bumping Gluten's
\`arrow.version\`.
## Reproducer
\`\`\`
java.lang.ExceptionInInitializerError
at
org.apache.gluten.memory.arrow.alloc.ArrowBufferAllocators.<clinit>(ArrowBufferAllocators.java:44)
at
org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance.<init>(...)
Caused by: java.lang.IllegalStateException: Unknown allocation manager type
to infer.
Current:
file:/.../gluten-package_2.12-1.7.0-SNAPSHOT.jar!/org/apache/arrow/memory/DefaultAllocationManagerFactory.class
at org.apache.arrow.memory.CheckAllocator.check(CheckAllocator.java:57)
at
org.apache.arrow.memory.DefaultAllocationManagerOption.getDefaultAllocationManagerFactory(DefaultAllocationManagerOption.java:80)
at
org.apache.arrow.memory.AllocationManager.<clinit>(AllocationManager.java:39)
at org.apache.arrow.memory.BaseAllocator.<clinit>(BaseAllocator.java:46)
at org.apache.arrow.memory.RootAllocator.<init>(RootAllocator.java:43)
at
org.apache.gluten.memory.arrow.alloc.ArrowBufferAllocators.<clinit>(ArrowBufferAllocators.java:44)
\`\`\`
## Proposed fix
Exclude \`DefaultAllocationManagerFactory.class\` from the shaded bundle so
the classloader always finds it in the upstream Arrow jar — whose URL contains
the recognized \`memory-unsafe\` or \`memory-netty\` substring. The class is
functionally identical (same \`FACTORY\` field providing an
\`AllocationManager.Factory\`); the only behavioral change is that Gluten no
longer forces the *unsafe* allocator and lets Arrow's own default selection
apply (which is \`netty\` per \`DefaultAllocationManagerOption.java:80\`).
Alternative mitigation: set \`-Darrow.allocation.manager.type=Unsafe\` (or
\`Netty\`) on driver/executor JVM options to bypass \`CheckAllocator.check()\`
entirely. This works as a runtime workaround but requires every Spark
submission to set the JVM option.
The build-time shade exclusion is preferable because:
- It's a one-time fix — no JVM options needed in every Spark submission
- Works for driver and executor uniformly
- Targets the root cause (redundant bundled class) instead of papering over
it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
