Hi Alex, +1 for option 2 for the reason mention by Yufei.
I looked at the approach in PR #6053 and I like it. The new module has no runtime dependencies: Iceberg and SLF4J are compileOnly, since iceberg-spark-runtime and Spark provide them, and Guava is replaced with JDK code. As a side effect, the Spark bundle LICENSE/NOTICE becomes easier to audit. The bundle is mostly Iceberg runtime plus our own client code. The package change is fine. Our evolution guidelines already say Java classes can change in any release, whatever the module. Maybe worth to mention in CHANGELOG though. About the release, polaris-core was already bundled this way in 1.8.0, so this is not a regression. I don't think we should block 1.9.0. I suggest we land it on main for 1.10.0. I will review the PR in details. Regards JB On Tue, Nov 10, 2026 at 5:30 PM Alex Dutra via dev <[email protected]> wrote: > Hi all, > > While verifying the 1.9.0-rc0 vote I found a few LICENSE issues in the > Spark > bundle JARs, which I'm fixing separately. Tracing one of them raised a > broader > question that I'd like to discuss here. > > Background: > > The Spark plugin includes polaris-core as: > > implementation(project(":polaris-core")) { isTransitive = false } > > See [1] for example. This was introduced in #1933 to "avoid pulling > unused dependencies". The bundle JAR then contains all polaris-core classes > but none of their runtime dependencies. There is a TODO next to the > line about extracting a thin "polaris-rest" module for clients. > > What I observed: > > Running jdeps on the bundled org.apache.polaris.core classes shows > references to > packages that are not in the bundle, e.g. Jackson 3 (tools.jackson.*), > Guava, > Immutables, the AWS/Azure/GCP SDKs, and org.apache.polaris.core.admin.model > (0 classes in the bundle, ~100 references to it). Spark provides some of > these > at runtime (Jackson 2 annotations, SLF4J), but not Jackson 3 or the cloud > SDKs. > > I have NOT determined which of these classes the Spark client actually > loads at > runtime, so this may be harmless today. But it means the bundle's > correctness > depends on the client never touching a class whose dependency is missing, > and > nothing in the build enforces that. > > Why not just drop isTransitive = false? > > polaris-core's runtime classpath resolves to ~300 modules. Bundling them > all > would make the JAR much larger, risk conflicts with Spark's own > Jackson/Guava/ > Netty, and require a full review of the bundle LICENSE/NOTICE. > > Here are our options: > > 1. Keep as is, and document/test the supported surface (e.g. an integration > test that loads every polaris-core class the client uses from the > bundle). > 2. Extract a thin client-facing module (the existing TODO) containing only > what > the Spark client needs, with its own explicit dependencies, and bundle > that. > 3. Drop isTransitive = false and handle conflicts via > relocation/exclusions, > re-auditing LICENSE/NOTICE. > > What are your thoughts? In particular, if anyone knows which polaris-core > classes the Spark client loads at runtime, that would help scope this. > > Thanks, > Alex > > [1]: > https://github.com/apache/polaris/blob/65befed8407eb8e6a1a5bb0d1a242fcdf5e82c2d/plugins/spark/v3.5/spark/build.gradle.kts#L55-L57 >
