Re: NoSQL database agnostic persistence

I'm starting to open some small, isolated PRs. The first ones are https://github.com/apache/polaris/pull/1229 and https://github.com/apache/polaris/pull/1230 - those don't change anything to the existing code base, but help reducing the size of the big https://github.com/apache/polaris/pull/11

Re: NoSQL database agnostic persistence

IMO we should test the s..t out of the implementation to ensure that it's stable and reliable and correct. Once that's done, we can certainly have some "realistic reference benchmark" with defined, different and concurrent user stories. That's definitely possible with Gatling. On 20.03.25 14

Re: NoSQL database agnostic persistence

We could just keep the exact same total number of entities, so 65 catalogs, 10 namespaces each with 100 Tables and Views. Or you can scale it up if you want. The insights I'm particularly interested in are the dynamics of listing and accessing tables in a namespace with multiple entries. The areas

Re: NoSQL database agnostic persistence

I think what you are referring to is the "number of tables per namespace" property. See the binary tree example in the docs where, after a binary tree of namespaces, 5 tables are created in each nam

Re: NoSQL database agnostic persistence

I think I saw in the other document you had some benchmarks with a less 1N to 1T ratio? Could we run some of those as well? It would be great to have something with closer to a 1 Namspace to 100 tables sort of layout. On Wed, Mar 19, 2025 at 3:06 PM Pierre Laporte wrote: > Just a heads up, I upd

Re: NoSQL database agnostic persistence

Just a heads up, I updated the report with the latest results from the persistence work, as well as the tarball with raw results. -- Pierre Laporte @pingtimeout pie...@pingtimeout.fr On Wed, Mar 19, 2025 at 3:20 PM Pierre Laporte wrote: > Hi, > > I have been

Re: NoSQL database agnostic persistence

Hi, I have been working on a set of benchmarks for Polaris [1]. I have run them against the current main branch (Eclipselink+Postgresql) implementation as well as the NoSQL persistence layer implementation [2]. The complete report for these performance tests is available at this address: https://

Re: NoSQL database agnostic persistence

Hi Robert, Thanks for the update and the draft PR ! I would like to use this thread to thank Dennis. Big kudos to Dennis for the changes he made: without these changes, it would have been impossible to add new backends like MongoDB. I propose we review and comment on Robert's PR. I would also l

NoSQL database agnostic persistence

Hi, I’ve made quite some progress on building the integration for NoSQL databases. The initial code supports MongoDB [A], but is not limited to that database. A working implementation has been pushed as a draft-PR [1] for illustration purposes how it can look like when it is fully integrated.