HI EJ, Thanks for following up on PR 5052!
> 1. The API/SPI split hinges on "when multiple alternatives exist",[...] I totally agree with your point. Alternatives here would mean something that can be a plugin. Even if Polaris itself has only one implementation, if we expect downstream builds to provide an alternative impl., it will fit the "alternatives exist" principle. > 2. Same question on the placement rule, you allow the SPI to live inside > X when X is already guaranteed to be a dependency of its own > implementations, curious what governs it when that's true for some > implementers and not others. >From my POV, both putting SPI class in X or in X-spi is ok. An SPI implementation works only when invoked from X. Therefore, X must be present at runtime anyway. Having a dependency from the SPI implementation on X at compile time is mildly intrusive, but not too critical from my POV. The positive side is that it reduces the number of modules in the system. I think in the current codebase, this principle applies mainly to runtime/service. If X has a lot of external dependencies that do not make sense in all SPI implementations, then of course, having a separate X-spi module will make more sense. Also if the SPI can be called from multiple modules, a separate X-spi module is advisable too (although I do not think we have these cases in current code). > 3. And on point 4, the compile-error safety net assumes downstream > rebuilds against new releases before it matters, not every consumer does. > Probably its own thread though, I'm thinking of more on the client and > server side (schema versioning on the wire, not something to hash out here I do not know how it might be possible to provide a custom Polaris SPI impl. without a downstream build, ATM. Polaris Servers are based on Quarkus, which performs all introspection tasks for CDI purposes at build time. Plugging something in without recompilation is possible via ServiceLoader, and Polaris has a few use cases for it. Still, a custom SPI provider would do well to cover those implementations by integration tests. The test should catch incompatibilities even if recompilation was not performed in the downstream build. > Polaris's own dependency graph, every extensions/* implementation > already has to depend on polaris-core, lands on keeping SPIs like > PolarisAuthorizer and PolarisMetaStoreManager inside core rather than > split out. >From my POV the dependency chain has to start from the SPI caller. PolarisMetaStoreManager is called from other core classes in a few places, so I guess it fits into polaris-core. Plus it was there historically. PolarisAuthorizer is not called from core. So by my logic it is not a "core SPI", indeed :) PolarisAuthorizer is a "service SPI" (used by runtime/service). I do not mind keeping it in polaris-core for now since it has historically been there. I mean this discussion to apply to new code mostly. > And on GenericTableEndpoints, the thing I raised on the same PR, > it's not an SPI either [...] Could you give me a pointer to that? I might have lost context here :) > [...] Quarkus being today's default impl, a hypothetical Spring > binding being another, it points at a real in-between layer: > framework-agnostic REST-serving logic that isn't quite core and > isn't quite runtime either. Good point. Much depends on how we expect custom builds to function. Currently Quarkus is assumed to be the platform for Polaris. If we (hypothetically) wanted to support different platforms, I'd guess specific REST API service code would not be 100% portable due to framework differences. A sub-set of frameworks may be interchangeable, but not all, I'm sure. (side note: this was very apparent in the migration from Dropwizard to Quarkus early in the project's history. The old code is still in git). So the REST service layer binding to HTTP would have to be implemented separately in each case and then call into a shared layer through an API. The shared layer would call Persistence and AuthZ through an SPI. The role of polaris-core is to provide a shared Entity model (with its own API) and a shared SPI for persisting those entities. Does that make sense? > [...] this would be a build-time, different-deployable choice, > a different axis than what you described [...] Loading SPI implementations without rebuilding is possible. I've done that before :) However, I can say from experience that it is a lot more difficult to support in a backward-compatible way and usually not worth the effort. Requiring a rebuild (with CI) to catch SPI discrepancies is more robust and usually leads to more clarity in downstream components. Check out the related message from Anand [1]. Quarkus enhances this approach by handling CDI wiring and related build-time checks. [1] https://lists.apache.org/thread/s26pj6vp95gq13ltmcvj72s9f1870foj Cheers, Dmitri. On Mon, Jul 20, 2026 at 12:42 PM EJ Wang <[email protected]> wrote: > Hi Dmitri, > > Following up here since it connects with PR5052, which just merged. > CatalogConfigEndpointContributor landed in runtime/service for now. On that > PR I argued core is too broad and runtime is too Quarkus-specific for it, > and suggested punting the real "where does this go long-term" question to a > separate issue rather than blocking the merge on it. You agreed, merged as > is. I don't see that issue filed yet, so figured this thread works as well > as any. > > A few things weren't fully clear to me in the writeup: > > 1. The API/SPI split hinges on "when multiple alternatives exist", is > that about how many alternatives exist today, or design intent to allow > more later? A plugin point with exactly one default implementation, > built that way on purpose, still reads like an SPI to me even with a > single > implementer. > 2. Same question on the placement rule, you allow the SPI to live inside > X when X is already guaranteed to be a dependency of its own > implementations, curious what governs it when that's true for some > implementers and not others. > 3. And on point 4, the compile-error safety net assumes downstream > rebuilds against new releases before it matters, not every consumer > does. > Probably its own thread though, I'm thinking of more on the client and > server side (schema versioning on the wire, not something to hash out > here). > > Separately, curious how general the placement rule is meant to be. > Polaris's own dependency graph, every extensions/* implementation already > has to depend on polaris-core, lands on keeping SPIs like PolarisAuthorizer > and PolarisMetaStoreManager inside core rather than split out. > > Is "ideally X-spi per module" the default shape with the in-X case being > rare, or are both equally "ideal" depending on which way the dependency > graph points, no inherent preference either way? > And on GenericTableEndpoints, the thing I raised on the same PR, it's not > an SPI either, nobody implements a wire-format constant. Does your model > have a read on where that kind of thing goes, or is it genuinely outside > what you're describing here? > > One more thing, more analogy than a literal claim, not saying the > REST-serving layer is an SPI. But if you squint at it that way, Quarkus > being today's default impl, a hypothetical Spring binding being another, it > points at a real in-between layer: framework-agnostic REST-serving logic > that isn't quite core and isn't quite runtime either, the same shape as the > GenericTableEndpoints question, just one layer up. Your examples are all > in-process, CDI-selected at runtime, this would be a build-time, > different-deployable choice, a different axis than what you described. Not > asking you to settle whether it's an SPI, more whether that in-between > layer is something your placement thinking has a slot for. > > -ej > > On Mon, Jul 13, 2026 at 5:42 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi All, > > > > I'm sending this email hoping to confirm alignment of the Polaris > community > > members on SPI development principles in this project. I do not think we > > need strong and formal rules / guidelines as long as the community has > > consensus on the general approach. > > > > Recap: SPI = Service Provider Interface > > > > * What is the difference between API and SPI? > > > > API is meant to be invoked by a client on a library (or service) in order > > to cause the library (service) to perform a certain action. > > > > SPI is meant to be implemented by a library / service plugin in order to > > provide a method for performing certain actions when multiple > alternatives > > exist. > > > > Example from Polaris: > > > > IcebergRestCatalogApiService is an API that represents the IRC spec in > > terms of java classes. > > > > Authenticator is an SPI that defines what Polaris code requires an > > authorization framework to implement. The default impl. > > is DefaultAuthenticator, OPA and Ranger implementation and alternatives > > that can be plugged in via CDI + configuration. > > > > * Where should SPI classes be located normally? > > > > Without considering the legacy code in Polaris, I believe SPI classes > > should be located such that they require minimal depdency leaks into the > > module implementing the SPI. Ideally all SPI classes used by module X > > should be located in a module "X-spi" to convey that they are associated > > with X. > > > > This is not to say that existing code must be refactored immediately to > fit > > these patterns. This is a proposal for future evolution. > > > > If module X can be expected to be a dependency of its SPI > implementations, > > the SPI classes can be placed inside X itself. > > > > * Should SPI classe be in polaris-core? > > > > If polaris-core requires defines a plugin point, the SPI for that can > > certainly be in polaris-core. > > > > BasePersistence is an example of a core SPI. > > > > * Can SPI interfaces have breaking changes from the java language > > perspective? > > > > Yes. Unline API, SPI interfaces define what plugins must implement. not > > what clients rely on. > > > > For example, adding a parameter to an SPI method will cause a compilation > > error downstream, but it's ok. This is a signal to the downstream SPI > > implementation that it is expected to process the new parameter. It is a > > good breakage in the sense that it will inform the downstram > implementation > > and prevent invalid behaviour that might occur as a result of ignoring > the > > new parameter. > > > > In many cases this is not a hard failure downstream. Normally, the > > downstream project will get those compilation errors inside a PR that > > updates its Polaris dependencies. It will not be a deployment time error > > (which is common with service API changes). > > > > Thoughts? > > > > Thanks, > > Dmitri. > > >
