Hi All, Focusing on the technical capabilities of PR [4812], I think it provides a valuable enhancement and we should merge it.
I do not see any adverse effects of this PR on existing functionality, but it certainly opens new use cases like H2 inside pre-built docker images and MySQL (potentially). [4812] https://github.com/apache/polaris/pull/4812 Cheers, Dmitri. On Fri, Jun 19, 2026 at 4:59 AM Alexandre Dutra <[email protected]> wrote: > Hi all, > > Although the conversation has shifted toward multi-datasource support, > the primary focus in this thread remains [4812]: the ability to > activate a datasource at runtime, even in single-source scenarios. > > I want to emphasize that this functionality is essential independently > of the multi-datasource debate. For example, it is required to provide > out-of-the-box support for JDBC + H2 within Polaris images, and also > to provide support for more databases (e.g. MySQL). > > Thanks, > Alex > > [4812]: https://github.com/apache/polaris/pull/4812 > > On Thu, Jun 18, 2026 at 9:28 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > > Hi Romain, > > > > Thanks for the info! > > > > Metrics requests generally read from the MetaStore but write only in the > > Metrics datasource. > > > > The latest thinking about Events is to isolate them from the initial > > request via the Quarkus Event Bus (async delivery), IIRC. > > > > The new Idempotency feature [3] might be the first case where we (could) > do > > writes into multiple datasources. However, the way it is designed does > not > > assume or require coordination of changes / commits between the MetaStore > > and the Idempotency data store, AFAIK. > > > > In general, this is a very interesting topic. It has implications for our > > (MetaStore) Persistence interface contracts too. IIRC, previous [1] > > discussions generally settled on the idea that each Persistence API call > > forms one atomic (and durable) change. Yet, I do not think we have > > completely ironed out all wrinkles [2] there. The tricky aspect is that > > Persistence may be backed by a non-transactional database. Currently we > > have JDBC and MongoDB, which could be said to be transactional, still the > > NoSQL Persistence in Polaris is built without assuming Tx capabilities, > so > > it will also work on non-transactional databases. > > > > This is just for context :) > > > > [1] https://lists.apache.org/thread/fh6t3mgzdb5ft0dt5q7qxm1pj1x5po75 > > > > [2] https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > > > > [3] https://lists.apache.org/thread/ksf5b5y3jt8dmft5kgblbwms8dhs4fn9 > > > > Cheers, > > Dmitri. > > > > On Thu, Jun 18, 2026 at 3:03 PM Romain Manni-Bucau < > [email protected]> > > wrote: > > > > > Hi Dmitri, > > > > > > Assuming you put metadata and metrics/events (maybe metrics but these > ones > > > where not under my radar) in different databases to leverage different > > > tuning and aggregation level but still have exactly once guarantee. > > > I assume at some point there are or will be some writes in the same > request > > > (stat updates, meta refresh/audit etc). > > > > > > While a single request/background task is guaranteed to be single > > > datasource backed there is no real topic if it is what you have in > mind. > > > > > > Romain Manni-Bucau > > > @rmannibucau <https://x.com/rmannibucau> | .NET Blog > > > <https://dotnetbirdie.github.io/> | Blog < > https://rmannibucau.github.io/> > > > | Old > > > Blog <http://rmannibucau.wordpress.com> | Github > > > <https://github.com/rmannibucau> | LinkedIn > > > <https://www.linkedin.com/in/rmannibucau> | Book > > > < > > > > https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064 > > > > > > > Javaccino founder (Java/.NET service - contact via linkedin) > > > > > > > > > Le jeu. 18 juin 2026 à 19:58, Dmitri Bourlatchkov <[email protected]> a > > > écrit : > > > > > > > Hi Romain, > > > > > > > > In what cases / API calls do you envision XA to be used in Polaris? > > > > > > > > I'm asking just for my education, as I do not see how / where it > could be > > > > relevant ATM. > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > On Thu, Jun 18, 2026 at 1:11 PM Romain Manni-Bucau < > > > [email protected]> > > > > wrote: > > > > > > > > > Ok, good it was already thought. > > > > > > > > > > Think worse case an user can set the 3 same or enable XA using > agroal > > > > > capabilities ([1]) > > > > > > > > > > Thanks > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > https://quarkus.io/guides/datasource#quarkus-agroal_quarkus-datasource-jdbc-transactions > > > > > > > > > > Romain Manni-Bucau > > > > > @rmannibucau <https://x.com/rmannibucau> | .NET Blog > > > > > <https://dotnetbirdie.github.io/> | Blog < > > > https://rmannibucau.github.io/ > > > > > > > > > > | Old > > > > > Blog <http://rmannibucau.wordpress.com> | Github > > > > > <https://github.com/rmannibucau> | LinkedIn > > > > > <https://www.linkedin.com/in/rmannibucau> | Book > > > > > < > > > > > > > > > > > > > https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064 > > > > > > > > > > > Javaccino founder (Java/.NET service - contact via linkedin) > > > > > > > > > > > > > > > Le jeu. 18 juin 2026 à 18:17, Alexandre Dutra <[email protected]> > a > > > > écrit > > > > > : > > > > > > > > > > > Hi Romain, > > > > > > > > > > > > You raise good points. > > > > > > > > > > > > Datasource multitenancy in Quarkus would have helped a lot during > > > this > > > > > > old discussion: [1] where we eventually decided to go with one > > > > > > datasource for N realms, out of better choices by then. But that > ship > > > > > > has sailed. > > > > > > > > > > > > And good question about XA: the proposal in [3960] does not > include > > > > > > any transactional coordination between datasources, afaict, and > the > > > > > > outbox pattern doesn't fit anymore in the design we picked for > events > > > > > > (and probably for metrics). This pattern was considered initially > > > > > > iirc, as it would have allowed exactly-once semantics for > events; but > > > > > > in the end, we went with separate tables updated independently > (~= > > > > > > at-most-once semantics). > > > > > > > > > > > > Thanks, > > > > > > Alex > > > > > > > > > > > > [1]: > > > https://lists.apache.org/thread/kvt3gd29hyxlhnwxr9b674s555jlvjjs > > > > > > [3960]: https://github.com/apache/polaris/pull/3960 > > > > > > > > > > > > On Thu, Jun 18, 2026 at 8:38 AM Romain Manni-Bucau > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > Hi Alexandre, > > > > > > > > > > > > > > Can be worth shiming in the quarkus thread about it: > > > > > > > https://github.com/quarkusio/quarkus/issues/11861 before > merging a > > > > new > > > > > > way > > > > > > > to do it (even if it delays from a few weeks the feature) I > think. > > > > > > > > > > > > > > Technically one thing the 1 database -> N databases (not > speaking > > > > about > > > > > > > tenants where there is no challenge but meta+metrics+events) > brings > > > > the > > > > > > > transactional challenge, do you go XA with this shift or do you > > > plan > > > > > some > > > > > > > recovery (kind of outbox pattern with some challenges)? > > > > > > > > > > > > > > Romain Manni-Bucau > > > > > > > @rmannibucau <https://x.com/rmannibucau> | .NET Blog > > > > > > > <https://dotnetbirdie.github.io/> | Blog < > > > > > https://rmannibucau.github.io/> > > > > > > | Old > > > > > > > Blog <http://rmannibucau.wordpress.com> | Github > > > > > > > <https://github.com/rmannibucau> | LinkedIn > > > > > > > <https://www.linkedin.com/in/rmannibucau> | Book > > > > > > > < > > > > > > > > > > > > > > > > > > > https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064 > > > > > > > > > > > > > > Javaccino founder (Java/.NET service - contact via linkedin) > > > > > > > > > > > > > > > > > > > > > Le mer. 17 juin 2026 à 21:37, Alexandre Dutra < > [email protected]> > > > a > > > > > > écrit : > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I'd like to elaborate on the "pluggable persistence" concept > > > > defined > > > > > > > > my previous message, and how it might evolve in the future: > > > > > > > > > > > > > > > > The proposal from PR [3960] originally suggested three > distinct > > > > > "store > > > > > > > > types": metastore, metrics, and events. Although that PR was > > > closed > > > > > > > > because of inactivity, the concept remains relevant. If we > decide > > > > to > > > > > > > > support multiple datasources for each store type, I envision > the > > > > > > > > following approach: > > > > > > > > > > > > > > > > Polaris would provide datasource definitions for every > > > combination > > > > of > > > > > > > > store type and JDBC driver (such as "postgresql-metastore" or > > > > > > > > "h2-metrics"). The current "enabler property" would be > replaced > > > by > > > > a > > > > > > > > map of properties keyed by the store type. For instance: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > polaris.persistence.relational.jdbc.metastore.datasource=postgresql-metastore > > > > > > > > > > > > > > > > > > > polaris.persistence.relational.jdbc.metrics.datasource=postgresql-metrics > > > > > > > > > > > > > > polaris.persistence.relational.jdbc.events.datasource=postgresql-events > > > > > > > > > > > > > > > > In this configuration, three separate PostgreSQL datasources > > > would > > > > be > > > > > > > > enabled, each managing its own connection pool and settings. > > > > > > > > > > > > > > > > Alternatively, if a user wants to use a single datasource > for all > > > > > > > > data, they could simply map every store type to the same name > > > > (e.g., > > > > > > > > "postgresql-metastore"). This would result in only one active > > > > > > > > datasource and a single connection pool for the metastore, > > > metrics, > > > > > > > > and events. > > > > > > > > > > > > > > > > It is still open for debate whether the default setup should > > > > > > > > prioritize a single shared datasource or separate ones for > each > > > > store > > > > > > > > type. > > > > > > > > > > > > > > > > While this is primarily speculative at this point, since the > > > > feature > > > > > > > > hasn't been finalized, it demonstrates that we have a viable > path > > > > > > > > forward should we choose to implement this architecture. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Alex > > > > > > > > > > > > > > > > [3960]: https://github.com/apache/polaris/pull/3960 > > > > > > > > > > > > > > > > On Wed, Jun 17, 2026 at 9:26 PM Alexandre Dutra < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > I have recently submitted a PR [4812] that introduces > support > > > for > > > > > > > > > multiple datasources with possibility of runtime > activation. > > > > > > > > > > > > > > > > > > I would like to open a discussion regarding the intent > behind > > > > this > > > > > > > > > proposal and the advantages it offers. > > > > > > > > > > > > > > > > > > While Quarkus is a powerful platform, it presents certain > > > > > challenges, > > > > > > > > > particularly with JDBC datasources. Currently, the > datasource > > > > type > > > > > > > > > must be defined at build time using the "db-kind" property. > > > This > > > > > > makes > > > > > > > > > managing multiple datasources difficult, especially for > > > Polaris, > > > > > > where > > > > > > > > > users should ideally be able to select the appropriate > > > datasource > > > > > > (and > > > > > > > > > driver) based on their specific requirements. > > > > > > > > > > > > > > > > > > The core of this proposal relies on a key fact: while > "db-kind" > > > > is > > > > > > > > > fixed at build time, the "active" property can be toggled > at > > > > > runtime. > > > > > > > > > Combined with named datasources, this property becomes the > > > > missing > > > > > > > > > lever to achieve runtime datasource activation. > > > > > > > > > > > > > > > > > > Under this change, Polaris must include predefined named > > > > > datasources > > > > > > > > > for every supported driver. Users simply choose which ones > to > > > > > > activate > > > > > > > > > at runtime using an enabler property: > > > > > > > > > "polaris.persistence.relational.jdbc.datasource", which > > > > identifies > > > > > > the > > > > > > > > > named datasource for activation. (Worth noting: an inactive > > > > > > datasource > > > > > > > > > does not consume any resources at runtime.) > > > > > > > > > > > > > > > > > > As a result, the server will now include the H2 driver by > > > > default, > > > > > as > > > > > > > > > it is a prerequisite for selecting an H2 datasource at > runtime. > > > > We > > > > > > > > > will need to follow a similar approach for any other JDBC > > > drivers > > > > > we > > > > > > > > > intend to support in the future. > > > > > > > > > > > > > > > > > > This architectural shift is necessary for several reasons: > > > > > > > > > > > > > > > > > > 1. Flexible defaults: It allows us to use JDBC + H2 as the > > > > default > > > > > > for > > > > > > > > > server images (see these related discussions for details: > [1] > > > > [2]) > > > > > > > > > while enabling users to switch to PostgreSQL or other > drivers > > > > *via > > > > > > > > > configuration alone*, thus eliminating the need to rebuild > > > > Polaris > > > > > > > > > entirely. > > > > > > > > > > > > > > > > > > 2. Pluggable persistence: It enables the persistence layer > to > > > > > connect > > > > > > > > > to multiple datasources simultaneously. For instance, one > could > > > > > use a > > > > > > > > > specific datasource for main persistence and another for > > > metrics > > > > – > > > > > a > > > > > > > > > concept previously discussed in [3960] that lacked an easy > > > > > > > > > implementation path in my opinion. > > > > > > > > > > > > > > > > > > I look forward to hearing your thoughts on this proposal. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > [4812]: https://github.com/apache/polaris/pull/4812 > > > > > > > > > [3960]: https://github.com/apache/polaris/pull/3960 > > > > > > > > > > > > > > > > > > [1]: > > > > > > https://lists.apache.org/thread/yw8l026g2smdk7gdg7k61tdcvdwcncqw > > > > > > > > > [2]: > > > > > > https://lists.apache.org/thread/nzoljc1ohnsq4f5o28dh4opqkqw3p09h > > > > > > > > > > > > > > > > > > > > > > > > > > >
