Hi Dmitri, hi all, Unfortunately I'm not sure your interpretation matches mine, so let's try to wrap up this discussion methodically.
We agreed to deprecate TreeMapMetaStore for removal. But we haven't reached consensus on two questions: 1) What should be the default metastore in the server image? 2) How to replace TreeMapMetaStore in tests? Question 1) is tricky and needs a clear decision. We have only two options for a smooth UX (aka, "docker run apache/polaris" just works): - Option 1a: JDBC+H2 - Option 1b: NoSQL+InMemory Option 1a is hard because it requires either my work on [4812] (runtime activated datasources) or Yufei's work on [4984] (unmanaged datasources). It also *requires* the H2 driver to be bundled by default, which some people are not comfortable with. In short, I don't see this option being feasible any time soon. Therefore, I think option 1b is much easier, and that would be my preference. Question 2) is not as urgent, as I'm fairly sure we can find a solution as we go. I can see a few options: - Option 2a: migrate to JDBC+H2 - Option 2b: migrate to NoSQL+InMemory - Option 2c: decide on a case-by-case basis My personal preference would be 2c. Can we try to reach consensus on the two questions please? Thanks, Alex [4812]: https://github.com/apache/polaris/pull/4812 [4984]: https://github.com/apache/polaris/pull/4984 On Mon, Jul 6, 2026 at 8:29 PM Dmitri Bourlatchkov <[email protected]> wrote: > > Hi All, > > My interpretation of the discussion so far is that: > > 1) We do want to deprecate the TreeMap MetaStore for removal > > 2) JDBC + PostgreSQL should be used for testing as well as NoSQL + MongoDB > (I believe it is in fact the case in current CI). > > 3) Instead of the TreeMap MetaStore, we will use the in-memory NoSQL > MetaStore (for lightweight tests) > > Now, H2 becomes irrevant, IMHO, but removing it is being discussed > separately [1]. > > How does that sound? > > [1] https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39 > > Thanks, > Dmitri. > > On Fri, Jul 3, 2026 at 5:25 AM Alexandre Dutra <[email protected]> wrote: > > > Hi all, > > > > I'm confused: security concerns were previously raised here [1] and > > here [2] about bundling the H2 driver due to its past record of > > critical CVEs. > > > > As a result, I assumed the JDBC+H2 option had been abandoned. Has this > > proposal been brought back into consideration? > > > > Thanks, > > Alex > > > > [1]: https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj > > [2]: https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r > > > > > > On Fri, Jul 3, 2026 at 8:54 AM Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > > Hi > > > > > > Yeah fully agree and it was my point when I replied on the thread > > > initially. > > > > > > Regards > > > JB > > > > > > Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <[email protected]> a écrit : > > > > > > > If our goal is to make the default experience closer to production > > > > while removing TreeMap, then JDBC with H2 seems like a more natural > > > > choice. It exercises the same persistence layer, schema management, > > > > and configuration that users will encounter in production, while still > > > > keeping setup lightweight. > > > > > > > > Yufei > > > > > > > > Yufei > > > > > > > > > > > > On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <[email protected]> > > wrote: > > > > > > > > > > Hi Russell, > > > > > > > > > > The existing default metastore shouldn't be viewed merely as a > > > > > "relatively simple, self-contained implementation": it has unique > > > > > quirks, such as returning result types that no other metastore does. > > > > > > > > > > Choosing the status quo simply to avoid making changes is imho the > > > > > worst solution. > > > > > > > > > > Transitioning to the NoSQL metastore as the default remains a net > > > > > benefit, even if it is only used by 30% of the user base. > > > > > > > > > > However, I agree with keeping the test suite on JDBC + H2 by default > > > > > if that ensures a higher level of confidence in our tests ability to > > > > > reproduce real production setups. > > > > > > > > > > Thanks, > > > > > Alex > > > > > > > > > > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer > > > > > <[email protected]> wrote: > > > > > > > > > > > > I agree TreeMap and its associated transactional stack should be > > > > deprecated. > > > > > > It’s test-only, confusing, and not used by JDBC production paths. > > > > > > > > > > > > I’m less convinced that NoSQL + InMemory is a clearly superior > > > > replacement > > > > > > for TreeMap in tests. At the storage layer it’s essentially the > > same > > > > class > > > > > > of > > > > > > thing: an in-memory, non-durable backend. The difference is that > > it > > > > > > exercises > > > > > > a completely different MetaStoreManager/persistence stack > > > > > > (NoSqlMetaStoreManager > > > > > > vs TransactionalMetaStoreManagerImpl), not the JDBC path most > > > > deployments > > > > > > use. It > > > > > > replaces a relatively simple, self-contained implementation with > > > > something > > > > > > that is > > > > > > part of a much more complicated persistence stack, while remaining > > > > > > non-durable > > > > > > and test-only at the storage layer. > > > > > > > > > > > > For the problems originally raised (test-grade defaults, dead > > > > production > > > > > > path, developer confusion), > > > > > > JDBC + H2 seems like an actual fix. > > > > > > > > > > > > > > > > > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > > > I tend to agree that the DataSource discussion related to H2 > > became > > > > a bit > > > > > > > convoluted, and it's a lot simpler to use the NoSQL in-memory > > > > persistence > > > > > > > instead. > > > > > > > > > > > > > > Cheers, > > > > > > > Dmitri. > > > > > > > > > > > > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra < > > [email protected]> > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > Some updates on this: > > > > > > > > > > > > > > > > I recently started a separate discussion regarding dynamic > > > > datasource > > > > > > > > activation at runtime [1]. This was intended as a prerequisite > > for > > > > > > > > adopting H2 as the default JDBC driver, as suggested on this > > > > thread. > > > > > > > > > > > > > > > > However, that proposal may end up being rejected. > > > > > > > > > > > > > > > > Given this situation, we should reconsider our options. If the > > > > > > > > PostgreSQL + H2 approach is no longer a viable replacement to > > the > > > > > > > > TreeMapMetaStore, I propose an alternative: let's use the NoSQL > > > > > > > > metastore with an InMemory backend. > > > > > > > > > > > > > > > > My reasoning is as follows: > > > > > > > > > > > > > > > > 1. It remains a production-ready metastore, so strictly > > superior to > > > > > > > > TreeMapMetaStore. > > > > > > > > > > > > > > > > 2. The metastore state update logic is the same regardless of > > the > > > > > > > > backend, so we're still close to a real production setup (same > > as > > > > with > > > > > > > > an in-memory JDBC driver). > > > > > > > > > > > > > > > > 3. It doesn't require any external dependencies (no bundled > > > > driver). > > > > > > > > > > > > > > > > I'd also note that while the NoSQL metastore is already > > included > > > > > > > > today, it is not actionable because no backend is available. > > This > > > > goes > > > > > > > > against the onboarding UX that we're trying to achieve, as > > users > > > > > > > > interested in Polaris with the NoSQL metastore cannot test this > > > > setup > > > > > > > > with the official image. > > > > > > > > > > > > > > > > Therefore, I think NoSQL + InMemory would serve as a more > > > > appropriate > > > > > > > > default setup for the official Polaris image. > > > > > > > > > > > > > > > > From what I see, it would boil down to adding one line to > > > > > > > > polaris-server or polaris-service: > > > > > > > > > > > > > > > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory")) > > > > > > > > > > > > > > > > What are your thoughts on this? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Alex > > > > > > > > > > > > > > > > [1]: > > > > https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > I wanted to give an update on this topic: I have started > > working > > > > on > > > > > > > > > this, but I would like to have [4812] merged first, since > > that's > > > > a > > > > > > > > > prerequisite. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > [4812]: https://github.com/apache/polaris/pull/4812 > > > > > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > I believe the original idea of TreeMapMetastore was that it > > > > looked > > > > > > > very > > > > > > > > > > similar to FoundationDB from an API perspective, so it > > served > > > > well > > > > > > > as > > > > > > > > a > > > > > > > > > > test system for the original backend being developed at SF. > > > > > > > > > > > > > > > > > > > > I agree that H2 + JDBC makes sense to me now for the > > project. > > > > > > > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > Thank you for your feedback. > > > > > > > > > > > > > > > > > > > > > > There seems to be a general agreement on the idea of > > > > deprecating > > > > > > > > > > > TreeMapMetaStore, coupled with the JDBC + H2 solution for > > > > tests. > > > > > > > > > > > > > > > > > > > > > > In most modules, the tests migration won't pose any > > serious > > > > > > > > > > > challenges, as replacing the metastore is generally just > > a > > > > matter > > > > > > > of > > > > > > > > > > > changing the configuration, and making sure the realm is > > > > properly > > > > > > > > > > > bootstrapped. > > > > > > > > > > > > > > > > > > > > > > There will be, however, a few tricky situations in > > > > polaris-core: a > > > > > > > > few > > > > > > > > > > > tests rely on the TreeMapMetaStore, mostly as a test > > > > convenience; > > > > > > > but > > > > > > > > > > > there is no obvious replacement for it in that module. I > > am > > > > however > > > > > > > > > > > confident that we can find a solution for that, either > > based > > > > on > > > > > > > > mocks, > > > > > > > > > > > or by bringing in a real metastore. > > > > > > > > > > > > > > > > > > > > > > If no objections are raised, I am going to prepare a PR > > for > > > > this. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri Bourlatchkov < > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > > > > > > > > > > > > > Thanks for starting this thread. > > > > > > > > > > > > > > > > > > > > > > > > I also find the TransactionalMetaStoreManagerImpl and > > > > related > > > > > > > call > > > > > > > > paths > > > > > > > > > > > > conuising in Apache Polaris code. > > > > > > > > > > > > > > > > > > > > > > > > I support promoting H2 to the default (in memory) > > > > Persistence > > > > > > > > backend for > > > > > > > > > > > > getting started cases. This should also resolve the > > old H2 > > > > > > > > evolution > > > > > > > > > > > thread > > > > > > > > > > > > [1] by ensuring it is used regularly on the same code > > > > paths as > > > > > > > > > > > PostgreSQL. > > > > > > > > > > > > > > > > > > > > > > > > Deprecating TransactionalMetaStoreManagerImpl for > > removal > > > > also > > > > > > > > sounds > > > > > > > > > > > > reasonable to me. Existing downstream users > > > > > > > > > > > > of TransactionalMetaStoreManagerImpl will have time to > > > > migrate, > > > > > > > or > > > > > > > > even > > > > > > > > > > > > embed that code (per ASF license) into local builds, > > > > during the > > > > > > > > > > > deprecation > > > > > > > > > > > > phase. > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39 > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre Dutra < > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > I am writing to ask the community whether it is OK to > > > > deprecate > > > > > > > > > > > > > TreeMapMetaStore, as well as all the in-memory > > metastore > > > > > > > manager, > > > > > > > > > > > > > metastore manager factory, and persistence types that > > > > rely > > > > > > > > solely on > > > > > > > > > > > > > TreeMapMetaStore. > > > > > > > > > > > > > > > > > > > > > > > > > > As we know, these components are test-grade only, > > and not > > > > > > > > suitable for > > > > > > > > > > > > > production. They trigger a production readiness > > alert on > > > > > > > Polaris > > > > > > > > > > > > > startup. It's a considerable amount of code that is > > > > virtually > > > > > > > > dead in > > > > > > > > > > > > > production. > > > > > > > > > > > > > > > > > > > > > > > > > > It's also confusing for developers. E.g. the > > > > "transactional" > > > > > > > > metastore > > > > > > > > > > > > > is not transactional in the JDBC sense of the term, > > and > > > > thus > > > > > > > not > > > > > > > > used > > > > > > > > > > > > > by JDBC persistence. It also has its quirks: some > > return > > > > > > > > statuses are > > > > > > > > > > > > > only returned by that manager. > > > > > > > > > > > > > > > > > > > > > > > > > > However, these components are used in tests, and I > > agree > > > > that > > > > > > > > it's > > > > > > > > > > > > > useful to have an in-memory version of the > > persistence > > > > layer > > > > > > > for > > > > > > > > > > > > > tests. But we have today two alternatives that are > > imho > > > > > > > superior > > > > > > > > for > > > > > > > > > > > > > tests in polaris-runtime-service: > > > > > > > > > > > > > > > > > > > > > > > > > > - JDBC persistence with H2 backend. There are > > already a > > > > few > > > > > > > tests > > > > > > > > > > > > > using this setup. > > > > > > > > > > > > > - NoSQL persistence with InMemory backend. > > > > > > > > > > > > > > > > > > > > > > > > > > Both alternatives test real production-grade > > persistence > > > > code. > > > > > > > > > > > > > > > > > > > > > > > > > > And finally, TreeMapMetaStore is currently the > > default > > > > runtime > > > > > > > > > > > > > persistence in application.properties; and the Helm > > > > chart also > > > > > > > > > > > > > advertises it as the default. These are not sane > > > > defaults, > > > > > > > imho. > > > > > > > > It's > > > > > > > > > > > > > always tricky to provide a good default for > > datastores, > > > > but > > > > > > > > since JDBC > > > > > > > > > > > > > persistence is included by default in the server > > image, > > > > I think > > > > > > > > that > > > > > > > > > > > > > including the H2 driver by default could give us a > > saner > > > > > > > default > > > > > > > > while > > > > > > > > > > > > > keeping the out-of-the-box experience intact (the > > > > license is > > > > > > > > Category > > > > > > > > > > > > > A). > > > > > > > > > > > > > > > > > > > > > > > > > > Concretely: > > > > > > > > > > > > > > > > > > > > > > > > > > On the MetaStoreManagerFactory hierarchy, the > > following > > > > > > > > > > > > > implementations are completely in-memory: > > > > > > > > > > > > > > > > > > > > > > > > > > - InMemoryPolarisMetaStoreManagerFactory: could be > > > > removed > > > > > > > > > > > > > > > > > > > > > > > > > > - InMemoryAtomicOperationMetaStoreManagerFactory: > > could > > > > be > > > > > > > > removed > > > > > > > > > > > > > > > > > > > > > > > > > > - LocalPolarisMetaStoreManagerFactory: is the base > > class > > > > of the > > > > > > > > two > > > > > > > > > > > > > above; imho it can be removed, but since it's an > > abstract > > > > > > > class, > > > > > > > > it > > > > > > > > > > > > > may have been extended outside Polaris. But neither > > JDBC > > > > nor > > > > > > > > NoSQL use > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > On the PolarisMetaStoreManager hierarchy: > > > > > > > > > > > > > > > > > > > > > > > > > > - TransactionalMetaStoreManagerImpl: is only used by > > > > > > > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC and NoSQL > > do > > > > not use > > > > > > > > it. > > > > > > > > > > > > > Could be removed. > > > > > > > > > > > > > > > > > > > > > > > > > > - TransactionWorkspaceMetaStoreManager however is a > > > > different > > > > > > > > beast, > > > > > > > > > > > > > in spite of the similar name. It is in use today on > > the > > > > commit > > > > > > > > path, > > > > > > > > > > > > > so should not be removed. > > > > > > > > > > > > > > > > > > > > > > > > > > On the BasePersistence hierarchy: > > > > > > > > > > > > > > > > > > > > > > > > > > - TreeMapTransactionalPersistenceImpl and its > > > > TreeMapMetaStore > > > > > > > > are > > > > > > > > > > > > > only used by InMemoryPolarisMetaStoreManagerFactory, > > and > > > > could > > > > > > > be > > > > > > > > > > > > > removed; > > > > > > > > > > > > > > > > > > > > > > > > > > - TransactionalPersistence and > > > > > > > AbstractTransactionalPersistence: > > > > > > > > these > > > > > > > > > > > > > are supertypes of TreeMapTransactionalPersistenceImpl > > > > and thus > > > > > > > > only > > > > > > > > > > > > > used for in-memory. They imo can be removed, but they > > > > might > > > > > > > have > > > > > > > > been > > > > > > > > > > > > > extended or implemented outside Polaris. > > > > > > > > > > > > > > > > > > > > > > > > > > I think it's important to keep Polaris code tidy by > > > > removing > > > > > > > > unused, > > > > > > > > > > > > > unimplementable, or test-grade only components. > > > > > > > > > > > > > > > > > > > > > > > > > > What are your thoughts? > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
