And there it is: https://github.com/apache/polaris/pull/6053
Thanks, Alex On Wed, Nov 11, 2026 at 9:26 AM Alex Dutra <[email protected]> wrote: > > Hi Yufei, > > Thanks, that's very helpful. Since it's just 2 classes, I think it > makes sense to favor option 2. I will open a PR shortly. > > Thanks, > Alex > > On Tue, Nov 10, 2026 at 6:38 PM Yufei Gu <[email protected]> wrote: > > > > Hi Alex, > > > > The Spark client uses only two classes from polaris-core: > > GenericTableEndpoints and PolarisResourcePaths. > > > > I'd favor option 2: move them into a small polaris-api-catalog-common > > module under the dir "api", with explicit dependencies. Both Spark and the > > server could use it, and Spark would no longer depend on core. > > > > Yufei > > > > > > Yufei > > > > > > On Tue, Nov 10, 2026 at 8:30 AM Alex Dutra via dev <[email protected]> > > wrote: > >> > >> Hi all, > >> > >> While verifying the 1.9.0-rc0 vote I found a few LICENSE issues in the > >> Spark > >> bundle JARs, which I'm fixing separately. Tracing one of them raised a > >> broader > >> question that I'd like to discuss here. > >> > >> Background: > >> > >> The Spark plugin includes polaris-core as: > >> > >> implementation(project(":polaris-core")) { isTransitive = false } > >> > >> See [1] for example. This was introduced in #1933 to "avoid pulling > >> unused dependencies". The bundle JAR then contains all polaris-core classes > >> but none of their runtime dependencies. There is a TODO next to the > >> line about extracting a thin "polaris-rest" module for clients. > >> > >> What I observed: > >> > >> Running jdeps on the bundled org.apache.polaris.core classes shows > >> references to > >> packages that are not in the bundle, e.g. Jackson 3 (tools.jackson.*), > >> Guava, > >> Immutables, the AWS/Azure/GCP SDKs, and org.apache.polaris.core.admin.model > >> (0 classes in the bundle, ~100 references to it). Spark provides some of > >> these > >> at runtime (Jackson 2 annotations, SLF4J), but not Jackson 3 or the cloud > >> SDKs. > >> > >> I have NOT determined which of these classes the Spark client actually > >> loads at > >> runtime, so this may be harmless today. But it means the bundle's > >> correctness > >> depends on the client never touching a class whose dependency is missing, > >> and > >> nothing in the build enforces that. > >> > >> Why not just drop isTransitive = false? > >> > >> polaris-core's runtime classpath resolves to ~300 modules. Bundling them > >> all > >> would make the JAR much larger, risk conflicts with Spark's own > >> Jackson/Guava/ > >> Netty, and require a full review of the bundle LICENSE/NOTICE. > >> > >> Here are our options: > >> > >> 1. Keep as is, and document/test the supported surface (e.g. an integration > >> test that loads every polaris-core class the client uses from the > >> bundle). > >> 2. Extract a thin client-facing module (the existing TODO) containing only > >> what > >> the Spark client needs, with its own explicit dependencies, and bundle > >> that. > >> 3. Drop isTransitive = false and handle conflicts via > >> relocation/exclusions, > >> re-auditing LICENSE/NOTICE. > >> > >> What are your thoughts? In particular, if anyone knows which polaris-core > >> classes the Spark client loads at runtime, that would help scope this. > >> > >> Thanks, > >> Alex > >> > >> [1]: > >> https://github.com/apache/polaris/blob/65befed8407eb8e6a1a5bb0d1a242fcdf5e82c2d/plugins/spark/v3.5/spark/build.gradle.kts#L55-L57
