Hi Dmitri,

You are right that creating a module just for reusing API URI paths
looks overkill, especially given that the new module needs
iceberg-core, and thus is not a "lightweight" module.

I created an alternative PR without the separate module (but with a
bit of code duplication instead):

https://github.com/apache/polaris/pull/6053 : separate module
https://github.com/apache/polaris/pull/6084 : no separate module

Which one does the community prefer?

Thanks,
Alex

On Wed, Nov 11, 2026 at 5:59 PM Dmitri Bourlatchkov via dev
<[email protected]> wrote:
>
> Hi Alex,
>
> Sorry for coming late to this discussion.
>
> I left some comments in GH. However, I would not mind if PR 6053 got merged
> "as is". I think it is still a net positive change.
>
> More specifically (repeating my points from GH), I do not see much value in
> reusing API URI paths across independent runtime artifacts.
>
> The Spark Plugin may run against any server in practice. Reusing Polaris
> modules does not guarantee that paths are aligned.
>
> On the other hand, API paths are expected to remain the same within the API
> major version (v1), so copying constants will not add maintenance overhead.
>
> Cheers,
> Dmitri.
>
> On Wed, Nov 11, 2026 at 4:10 AM Alex Dutra via dev <[email protected]>
> wrote:
>
> > And there it is:
> >
> > https://github.com/apache/polaris/pull/6053
> >
> > Thanks,
> > Alex
> >
> > On Wed, Nov 11, 2026 at 9:26 AM Alex Dutra <[email protected]> wrote:
> > >
> > > Hi Yufei,
> > >
> > > Thanks, that's very helpful. Since it's just 2 classes, I think it
> > > makes sense to favor option 2. I will open a PR shortly.
> > >
> > > Thanks,
> > > Alex
> > >
> > > On Tue, Nov 10, 2026 at 6:38 PM Yufei Gu <[email protected]> wrote:
> > > >
> > > > Hi Alex,
> > > >
> > > > The Spark client uses only two classes from polaris-core:
> > GenericTableEndpoints and PolarisResourcePaths.
> > > >
> > > > I'd favor option 2: move them into a small polaris-api-catalog-common
> > module under the dir "api", with explicit dependencies. Both Spark and the
> > server could use it, and Spark would no longer depend on core.
> > > >
> > > > Yufei
> > > >
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Tue, Nov 10, 2026 at 8:30 AM Alex Dutra via dev <
> > [email protected]> wrote:
> > > >>
> > > >> Hi all,
> > > >>
> > > >> While verifying the 1.9.0-rc0 vote I found a few LICENSE issues in
> > the Spark
> > > >> bundle JARs, which I'm fixing separately. Tracing one of them raised
> > a broader
> > > >> question that I'd like to discuss here.
> > > >>
> > > >> Background:
> > > >>
> > > >> The Spark plugin includes polaris-core as:
> > > >>
> > > >>   implementation(project(":polaris-core")) { isTransitive = false }
> > > >>
> > > >> See [1] for example. This was introduced in #1933 to "avoid pulling
> > > >> unused dependencies". The bundle JAR then contains all polaris-core
> > classes
> > > >> but none of their runtime dependencies. There is a TODO next to the
> > > >> line about extracting a thin "polaris-rest" module for clients.
> > > >>
> > > >> What I observed:
> > > >>
> > > >> Running jdeps on the bundled org.apache.polaris.core classes shows
> > references to
> > > >> packages that are not in the bundle, e.g. Jackson 3
> > (tools.jackson.*), Guava,
> > > >> Immutables, the AWS/Azure/GCP SDKs, and
> > org.apache.polaris.core.admin.model
> > > >> (0 classes in the bundle, ~100 references to it). Spark provides some
> > of these
> > > >> at runtime (Jackson 2 annotations, SLF4J), but not Jackson 3 or the
> > cloud SDKs.
> > > >>
> > > >> I have NOT determined which of these classes the Spark client
> > actually loads at
> > > >> runtime, so this may be harmless today. But it means the bundle's
> > correctness
> > > >> depends on the client never touching a class whose dependency is
> > missing, and
> > > >> nothing in the build enforces that.
> > > >>
> > > >> Why not just drop isTransitive = false?
> > > >>
> > > >> polaris-core's runtime classpath resolves to ~300 modules. Bundling
> > them all
> > > >> would make the JAR much larger, risk conflicts with Spark's own
> > Jackson/Guava/
> > > >> Netty, and require a full review of the bundle LICENSE/NOTICE.
> > > >>
> > > >> Here are our options:
> > > >>
> > > >> 1. Keep as is, and document/test the supported surface (e.g. an
> > integration
> > > >>    test that loads every polaris-core class the client uses from the
> > bundle).
> > > >> 2. Extract a thin client-facing module (the existing TODO) containing
> > only what
> > > >>    the Spark client needs, with its own explicit dependencies, and
> > bundle that.
> > > >> 3. Drop isTransitive = false and handle conflicts via
> > relocation/exclusions,
> > > >>    re-auditing LICENSE/NOTICE.
> > > >>
> > > >> What are your thoughts? In particular, if anyone knows which
> > polaris-core
> > > >> classes the Spark client loads at runtime, that would help scope this.
> > > >>
> > > >> Thanks,
> > > >> Alex
> > > >>
> > > >> [1]:
> > https://github.com/apache/polaris/blob/65befed8407eb8e6a1a5bb0d1a242fcdf5e82c2d/plugins/spark/v3.5/spark/build.gradle.kts#L55-L57
> >

Reply via email to