And there it is:

https://github.com/apache/polaris/pull/6053

Thanks,
Alex

On Wed, Nov 11, 2026 at 9:26 AM Alex Dutra <[email protected]> wrote:
>
> Hi Yufei,
>
> Thanks, that's very helpful. Since it's just 2 classes, I think it
> makes sense to favor option 2. I will open a PR shortly.
>
> Thanks,
> Alex
>
> On Tue, Nov 10, 2026 at 6:38 PM Yufei Gu <[email protected]> wrote:
> >
> > Hi Alex,
> >
> > The Spark client uses only two classes from polaris-core: 
> > GenericTableEndpoints and PolarisResourcePaths.
> >
> > I'd favor option 2: move them into a small polaris-api-catalog-common 
> > module under the dir "api", with explicit dependencies. Both Spark and the 
> > server could use it, and Spark would no longer depend on core.
> >
> > Yufei
> >
> >
> > Yufei
> >
> >
> > On Tue, Nov 10, 2026 at 8:30 AM Alex Dutra via dev <[email protected]> 
> > wrote:
> >>
> >> Hi all,
> >>
> >> While verifying the 1.9.0-rc0 vote I found a few LICENSE issues in the 
> >> Spark
> >> bundle JARs, which I'm fixing separately. Tracing one of them raised a 
> >> broader
> >> question that I'd like to discuss here.
> >>
> >> Background:
> >>
> >> The Spark plugin includes polaris-core as:
> >>
> >>   implementation(project(":polaris-core")) { isTransitive = false }
> >>
> >> See [1] for example. This was introduced in #1933 to "avoid pulling
> >> unused dependencies". The bundle JAR then contains all polaris-core classes
> >> but none of their runtime dependencies. There is a TODO next to the
> >> line about extracting a thin "polaris-rest" module for clients.
> >>
> >> What I observed:
> >>
> >> Running jdeps on the bundled org.apache.polaris.core classes shows 
> >> references to
> >> packages that are not in the bundle, e.g. Jackson 3 (tools.jackson.*), 
> >> Guava,
> >> Immutables, the AWS/Azure/GCP SDKs, and org.apache.polaris.core.admin.model
> >> (0 classes in the bundle, ~100 references to it). Spark provides some of 
> >> these
> >> at runtime (Jackson 2 annotations, SLF4J), but not Jackson 3 or the cloud 
> >> SDKs.
> >>
> >> I have NOT determined which of these classes the Spark client actually 
> >> loads at
> >> runtime, so this may be harmless today. But it means the bundle's 
> >> correctness
> >> depends on the client never touching a class whose dependency is missing, 
> >> and
> >> nothing in the build enforces that.
> >>
> >> Why not just drop isTransitive = false?
> >>
> >> polaris-core's runtime classpath resolves to ~300 modules. Bundling them 
> >> all
> >> would make the JAR much larger, risk conflicts with Spark's own 
> >> Jackson/Guava/
> >> Netty, and require a full review of the bundle LICENSE/NOTICE.
> >>
> >> Here are our options:
> >>
> >> 1. Keep as is, and document/test the supported surface (e.g. an integration
> >>    test that loads every polaris-core class the client uses from the 
> >> bundle).
> >> 2. Extract a thin client-facing module (the existing TODO) containing only 
> >> what
> >>    the Spark client needs, with its own explicit dependencies, and bundle 
> >> that.
> >> 3. Drop isTransitive = false and handle conflicts via 
> >> relocation/exclusions,
> >>    re-auditing LICENSE/NOTICE.
> >>
> >> What are your thoughts? In particular, if anyone knows which polaris-core
> >> classes the Spark client loads at runtime, that would help scope this.
> >>
> >> Thanks,
> >> Alex
> >>
> >> [1]: 
> >> https://github.com/apache/polaris/blob/65befed8407eb8e6a1a5bb0d1a242fcdf5e82c2d/plugins/spark/v3.5/spark/build.gradle.kts#L55-L57

Reply via email to