Hi all,
While verifying the 1.9.0-rc0 vote I found a few LICENSE issues in the Spark
bundle JARs, which I'm fixing separately. Tracing one of them raised a broader
question that I'd like to discuss here.
Background:
The Spark plugin includes polaris-core as:
implementation(project(":polaris-core")) { isTransitive = false }
See [1] for example. This was introduced in #1933 to "avoid pulling
unused dependencies". The bundle JAR then contains all polaris-core classes
but none of their runtime dependencies. There is a TODO next to the
line about extracting a thin "polaris-rest" module for clients.
What I observed:
Running jdeps on the bundled org.apache.polaris.core classes shows references to
packages that are not in the bundle, e.g. Jackson 3 (tools.jackson.*), Guava,
Immutables, the AWS/Azure/GCP SDKs, and org.apache.polaris.core.admin.model
(0 classes in the bundle, ~100 references to it). Spark provides some of these
at runtime (Jackson 2 annotations, SLF4J), but not Jackson 3 or the cloud SDKs.
I have NOT determined which of these classes the Spark client actually loads at
runtime, so this may be harmless today. But it means the bundle's correctness
depends on the client never touching a class whose dependency is missing, and
nothing in the build enforces that.
Why not just drop isTransitive = false?
polaris-core's runtime classpath resolves to ~300 modules. Bundling them all
would make the JAR much larger, risk conflicts with Spark's own Jackson/Guava/
Netty, and require a full review of the bundle LICENSE/NOTICE.
Here are our options:
1. Keep as is, and document/test the supported surface (e.g. an integration
test that loads every polaris-core class the client uses from the bundle).
2. Extract a thin client-facing module (the existing TODO) containing only what
the Spark client needs, with its own explicit dependencies, and bundle that.
3. Drop isTransitive = false and handle conflicts via relocation/exclusions,
re-auditing LICENSE/NOTICE.
What are your thoughts? In particular, if anyone knows which polaris-core
classes the Spark client loads at runtime, that would help scope this.
Thanks,
Alex
[1]:
https://github.com/apache/polaris/blob/65befed8407eb8e6a1a5bb0d1a242fcdf5e82c2d/plugins/spark/v3.5/spark/build.gradle.kts#L55-L57