Re: [PROPOSAL] Add Data Lake operational metrics to Polaris

2025-09-05 Thread Eric Maynard
> I am not against the idea of collecting telemetry (I think we would require an auxiliary compute for doing this, though), +1, I think collecting such telemetry is actually a great idea and agree that auxiliary compute is probably the right design. This was one of the initial motivations for the

Re: [DISCUSS] Limiting namespace locations

2025-08-26 Thread Eric Maynard
catalog, they should update the StorageConfig to add that > ALLOWED_LOCATION. > > > On Mon, Aug 25, 2025 at 11:18 PM Jean-Baptiste Onofré > wrote: > > > Hi Eric > > > > Thanks for the heads up and the fix on namespaces. > > > > I agree that it should be

[DISCUSS] Limiting namespace locations

2025-08-25 Thread Eric Maynard
Hi team, I recently filed PR#2422 and there was an ask there about creating a mailing list thread for discussion. The tl;dr of this change is that Polaris currently allows the creation of namespaces & tables outside of a catalog allowed-locations. The

1.0.1 and versioned docs

2025-08-21 Thread Eric Maynard
Hi all, I was looking at the site today and noticed we have a section in the docs for 1.0.1. On one hand, this is great to see as I'm sure we're all excited about the release. On the other, I could imagine this list getting pretty crowded as patch releases continue to happen. I've been wondering a

Re: [DISCUSS] Labels for GitHub issues + PRs

2025-08-20 Thread Eric Maynard
> Generally, IIRC, we agreed on not using a "blocker label". Where was this? I thought the 1.0-blocker label was quite useful when preparing for the 1.0 release, and I imagined that we'd continue to use something similar for future releases. The closest thing I see is this email

Re: [DISCUSS] Add feature flag: PURGE_VIEWS_ON_DROP

2025-08-18 Thread Eric Maynard
I'm a +1 on the PR and thought the existing naming was ok. >From the user POV, the information stored in the metastore is also metadata, so "delete metadata" is a bit misleading in that metadata is still deleted if you have the flag set to false. PURGE feels accurate enough to me -- I believe the

Re: Removal of CallContext.copy()

2025-08-13 Thread Eric Maynard
I agree that we shouldn't need the entire CallContext in each & every task -- tasks should include whatever information is necessary to execute them, and in the end CallContext.copy() shouldn't be needed. However, at least in Will's proposal, we would keep the existing framework around (with its u

Re: Schema setup in admin tool and server

2025-08-01 Thread Eric Maynard
+1 to everything Dmitri said, well put. Ideally, I would say that "schema" ought not to be a first-class concept within Polaris administration, just whether a realm has been bootstrapped or not. In practice though it is possible that different realms get bootstrapped with different database schema

Re: [DISCUSS] Adding endpointInternal to AwsStorageConfigInfo

2025-08-01 Thread Eric Maynard
> It is only relevant for on-prem S3-compatible storage. I imagine, "endpointInternal" will never be needed for AWS storage. That's not true, is it? If we are imagining scenarios where the client and the server are on totally different networks, the regular endpoint could indeed need to be address

Re: [DISCUSS] Adding endpointInternal to AwsStorageConfigInfo

2025-08-01 Thread Eric Maynard
> Two endpoints are necessary in cases when the server's view of the network is different from the engine's view. That could be true -- but at present it seems like a bit of a contrived scenario. It's also not exclusive to any particular cloud right? On Fri, Aug 1, 2025 at 7:47 AM Dmitri Bourlatc

Re: [PROPOSAL] Asynchronous & Reliable Tasks

2025-08-01 Thread Eric Maynard
I agree with Robert that the current implementation is not good and should be ripped out ASAP. However, I see this effort as complementary to Will's refactor, not as a dependency. We should first add a layer of abstraction between the business logic in Polaris and the task execution -- once that's

Re: [DISCUSS] S3 Credential vending without STS

2025-07-31 Thread Eric Maynard
The existence of SKIP_CREDENTIAL_SUBCOPING_INDIRECTION may be helpful context for this thread. With that config set, we already support loading tables without subscoped vended credentials. If there are certain storage configurations that only work with this config set, that doesn't seem like it wou

Re: [DISCUSS] Merge polaris-service-common into polaris-runtime-service

2025-07-31 Thread Eric Maynard
The project already *has* adopted Quarkus for good, so I think making the module structure reflect that is fine. In addition to the configuration issue, which is significant, I think cutting down on the number of modules we publish is a noble goal. —EM On Fri, Aug 1, 2025 at 00:43 Alexandre Dutra

Re: [DISCUSS] Apache Polaris (incubating) 1.1.0 release mid August ?

2025-07-29 Thread Eric Maynard
as the discussion happens and experimental > flag is removed it will be included in the next release (monthly). > > > > Wdyt ? > > > > Regards > > JB > > > > Le mar. 29 juil. 2025 à 11:40, Eric Maynard > a écrit : > >> > >> I agr

Re: [DISCUSS] Apache Polaris (incubating) 1.1.0 release mid August ?

2025-07-29 Thread Eric Maynard
I agree with everything you said JB, but I’m not that it addresses the question around generic tables and the experimental label. In the case of generic tables the feature is already in 1.0. It will be in 1.1.0. However we are labeling it “experimental” and the question here is about under what co

Re: [Discussion] Decoupling Helm chart releases (+ nightly build proposal)

2025-07-21 Thread Eric Maynard
Sorry, I missed this point about 1.1.0… so Helm Chart 1.1.0 will point at what commit from Polaris? Not the 1.1.0 release, right? Is the alternative that we just cut a 1.0.1 for Polaris and the Helm chart? On Mon, Jul 21, 2025 at 10:06 PM Jean-Baptiste Onofré wrote: > Hi, > > I would propose Po

Re: [PR] Move page "External Identity Providers" out of "unreleased" [polaris]

2025-07-15 Thread Eric Maynard
, Jul 15, 2025 at 7:47 AM Mark Hoerth wrote: > Thanks, Eric, for this step. When I build the page in my sandbox, I still > don't see the issue, and I guess that's because of the new addition of the > 1.0 directory. How should I avoid this issue next time > > On Tue, Jul 15

Re: Status Renovate

2025-07-15 Thread Eric Maynard
No problem, please see PR #2085.

Re: Status Renovate

2025-07-15 Thread Eric Maynard
ordingly I think renovatebot should not be automatically opening PRs against that branch. --EM On Tue, Jul 15, 2025 at 6:18 AM Robert Stupp wrote: > Not sure what you mean Eric. > IIRC you merged the PR 1891, so I assumed you're also okay with that? > > On Tue, Jul 15, 2

Re: Status Renovate

2025-07-15 Thread Eric Maynard
Do we actually want it to run on release/1.0.x? Per "[DISCUSS] Disable renovatebot on release branches", it looks like we do not. On Tue, Jul 15, 2025 at 5:18 AM Robert Stupp wrote: > Oh! > According to the dependency dashboard [1] it works on both main and > release/1.0.x :) > > [1] https://git

Re: [Discuss] Default commit message during merge/squash

2025-07-10 Thread Eric Maynard
tupp wrote: > That would remove the useful information like links > release-notes/changelog, effective Renovate config for the PR, ability > to re-create the PR, link to Renovate logs, etc > > On Wed, Jul 9, 2025 at 6:25 PM Eric Maynard > wrote: > > > > Is it

Re: [PROPOSAL] Commit Deconfliction

2025-07-09 Thread Eric Maynard
; resolution > > may not want to allow preemptive resolution. WDYT? > > > > I'm +1 to evolving Polartis to be able to properly support real conflict > > resolution (starting with append/append conflicts). > > > > Cheers, > > Dmitri. > > >

Re: [Community Meeting] NoSQL Support in Apache Polaris

2025-07-09 Thread Eric Maynard
I know that there was a spanner implementation mentioned in a recent community sync, is that also part of the same discussion? On Wed, Jul 9, 2025 at 11:12 AM Jean-Baptiste Onofré wrote: > Actually the invite was not correct: the meeting is on 7/17 as > originally in my first email. > > Sorry fo

Re: [Discuss] Default commit message during merge/squash

2025-07-09 Thread Eric Maynard
ok really really awful with a > ton of not useful information and links - leaking into (automatically > generated) changelogs for releases. > > All "human originated" PRs need manual attention anyway. > > > On Wed, Jul 9, 2025 at 5:54 PM Eric Maynard > wrote: >

Re: [Discuss] Default commit message during merge/squash

2025-07-09 Thread Eric Maynard
Hi Robert, > Having the list of commits in the merge commit message proposed by GitHub helps there. Can you elaborate on this? How exactly does it help? You can always view the commits on the original PR. Beyond that, I actually don't see these commit messages being added currently when I look a

Re: [PROPOSAL] Commit Deconfliction

2025-07-08 Thread Eric Maynard
; changes from two conflicting commit requests or are you proposing the > > clients to do that? > > > > In other words, who will make the final metadata JSON? Clients (and the > > server merely commits it) or the Polaris Server itself (modifying what > the >

Re: [PROPOSAL] Commit Deconfliction

2025-07-07 Thread Eric Maynard
uot; ID checks > or will Polaris validate actual table changes? > > I added some comments to the doc too. > > Thanks, > Dmitri. > > On Mon, Jul 7, 2025 at 6:33 PM Eric Maynard > wrote: > > > Hi all, > > > > Wanted to share this short design

[PROPOSAL] Commit Deconfliction

2025-07-07 Thread Eric Maynard
Hi all, Wanted to share this short design doc for a simple method of allowing conflicting commits to both be committed. If implemented, this would allow e.g. two writers doing append-only operations to a table in Pol

Re: [DISCUSS] Hive Catalog federation in Polaris

2025-07-07 Thread Eric Maynard
1. We (Polaris) can provide end users a way to migrate off of these catalogs that the Iceberg project no longer wants to invest into. Implementing HMS federation is in service to the goal of removing non-Iceberg catalogs, not in contradiction to it. 2. This does not seem like a user-centered conce

Re: Discussion: Adding NO_AUTH Support for Catalog Federation

2025-07-02 Thread Eric Maynard
hen requests > should be denied on the Polaris side. > > As an example, IMPLICIT with AWS SDK is always allowed because the SDK has > well-known file-based configuration / profiling mechanisms. > > I do not know enough about Hadoop, though. > > WDYT? > > Cheers, &g

Re: Discussion: Adding NO_AUTH Support for Catalog Federation

2025-07-02 Thread Eric Maynard
e: > How about using the enum name IMPLICIT in this case? > > YAML comments will briefly mention runtime env. implications. Documentation > will (later) explain how it works in detail. > > From my POV, "NONE" means strictly no auth. > > Cheers, > Dmitri. > >

Re: [DISCUSS] Hive Catalog federation in Polaris

2025-07-02 Thread Eric Maynard
mechanism to determine all the files that were read to create > the final configuration used for a connection. > > Thanks, > Pooja > > > On 2025/07/02 18:05:54 Eric Maynard wrote: > > IMO 2a should be off the table; the person creating an EXTERNAL catalog > > do

Re: Discussion: Adding NO_AUTH Support for Catalog Federation

2025-07-02 Thread Eric Maynard
> When the new NONE (or any proposed alternative name) is used as the authentication type in an External Catalog, what kind of auth flow will actually happen in runtime? This question really gets to the core of what we are discussing. From my perspective in implementing HADOOP, we can interpret NO

Re: Discussion: Adding NO_AUTH Support for Catalog Federation

2025-07-02 Thread Eric Maynard
> I believe we need to consider both the server-side impact and user-side impact > NONE [...] feels like the federated catalog will not perform any authentication. I see multiple users here. There's an admin configuring the Polaris service, there's a person creating the EXTERNAL catalog, and there

Re: [DISCUSS] Hive Catalog federation in Polaris

2025-07-02 Thread Eric Maynard
IMO 2a should be off the table; the person creating an EXTERNAL catalog does not necessarily have permission to access the path that a hive-site.xml is as (whether local to the Polaris catalog service or in object storage) or to even know what paths the catalog has access to. It's the same problem

Re: Discussion: Adding NO_AUTH Support for Catalog Federation

2025-06-30 Thread Eric Maynard
1 & 4 both sound good to me! From the perspective of the EXTERNAL catalog connection, we're talking about a situation where the connection itself does not manage or describe any authentication mechanism. ENVIRONMENT seems simultaneously both too vague (what environment? The client's? The metastore

Re: Discussion Regarding Events Instrumentation - GH PR #1904

2025-06-25 Thread Eric Maynard
e > currently in Polaris. This is why they offer "at most once" guarantees > and are only used for optimization purposes. (The advantage of Nesse > though, is that it has a commit log, and thus can also fulfil audit > use cases.) > > Thanks, > Alex > > On Tue, Jun 24

Re: Discussion Regarding Events Instrumentation - GH PR #1904

2025-06-25 Thread Eric Maynard
I think we may be making too many assumptions about the semantics of an event being emitted and letting that paralyze us. If you're looking for an event to be emitted *iff* a table change is committed to the metastore, the current system simply does not support that use case: - BeforeTableCommi

Re: [DISCUSS] Adding endpoint to AwsStorageConfigInfo

2025-06-25 Thread Eric Maynard
+1, I think this is a good idea. If there is a similar concept for the other storage types, could we consider adding that too? * https://learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints --EM On Tue, Jun 24, 2025 at 9:23 PM Dmitri Bourlatchkov wrote: > Hi All, > > I pro

Re: [DISCUSS] Polaris Delegation Service for Long-Running Tasks

2025-06-23 Thread Eric Maynard
Hey Dmitri, There's a section in the email above and the linked doc that talks about the linked proposal. See "Relationship to the "Asynchronous & Reliable Tasks" Proposal". As for pulling away from a REST API in favor of driving things directly from persistence, there's a lot to discuss here. Be

Re: Adding Support for Apache Hudi within Polaris

2025-06-13 Thread Eric Maynard
Hey Laurent, For general concerns about generic tables, it may be best to revisit some of the threads about that design. There’s a proposal doc from February that discusses some of the questions you raise here. I’m not sure if it makes sense to relitigate the generic tables design every time an ex

Re: [DISCUSS] CODEOWNERS file

2025-06-13 Thread Eric Maynard
I agree that the automatic tagging is getting out of hand, thanks for raising this. I think just deleting CODEOWNERS is a pretty good idea. Once that's done however, I worry that a potential contributor might not know who to tag on a PR and might either just pick someone randomly or be stuck waiti

Re: Polaris Event Persistence Schema

2025-06-12 Thread Eric Maynard
The Polaris event listener framework is not necessarily 1:1 to the Iceberg Events API -- IIRC, it actually predated the Iceberg Events API proposal. I think the pieces of code that just involve persisting Polaris events somewhere may not need to block on the Iceberg API changes to retrieve Iceberg

Re: [Discuss] Add `location` to generic table spec

2025-06-11 Thread Eric Maynard
> I don't think there's a lot of value where the specification of a table format is left to the client Considering that you currently can use non-Iceberg tables in Polaris with the Spark client and it works end-to-end, I'd have a hard time agreeing that there is no value. But I think this discussi

Re: [Discuss] Add `location` to generic table spec

2025-06-10 Thread Eric Maynard
> unifying? > > Delta for example is working on some catalog protocol for instance: > https://github.com/delta-io/delta/issues/4381 > > Laurent > > On Tue, Jun 10, 2025 at 5:01 PM Eric Maynard > wrote: > > > After this latest round of changes, it looks good to me too!

Re: [Discuss] Add `location` to generic table spec

2025-06-10 Thread Eric Maynard
ions a table can have, > > > because every snapshot can potentially write into a different location, > > > and those are not tracked anywhere by anyone today. > > > Furthermore it might require information more than just a location, for > > > example, it mi

Re: [DISCUSS] Skip 0.10.0-beta-incubating release to directly release 1.0.0-incubating ?

2025-06-06 Thread Eric Maynard
+1, as I understand it the role of 0.10 was to get us ready for a release with binaries and to that end it has served its purpose. We're now 1.0 ready thanks in large part to the work done to make 0.10 happen even if there winds up being no actual 0.10 release. On Fri, Jun 6, 2025 at 4:00 PM Prash

Re: Context-Aware Functions for Apache Polaris

2025-06-06 Thread Eric Maynard
It seems to me that the *easiest* to start with would be role-based column filtering. There are no functions to grapple with, no dialect differences. You simply grab the list of columns that a given principal role has access to according to the FGAC policy attached to a given table. --EM

Re: [DISCUSS] Prepare for 1.0 Release

2025-06-03 Thread Eric Maynard
+1 to making 801 a blocker. Based on Alex's comments in 1799, it looks like the rotation is only happening in JdbcMetastoreManagerFactory? If so, I think we have a very simple fix in PR#1804 . --EM

Re: [DISCUSS] Disable renovatebot on release branches

2025-06-02 Thread Eric Maynard
+1, great idea. This process change will reduce noise in the repo as we add more releases, and it also aligns more closely with the idea of a release as a stable snapshot of the code at some point in time -- including the dependencies as they were at that time. --EM On Mon, Jun 2, 2025 at 9:14 A

Re: [DISCUSS] Standardize nullness annotations

2025-06-02 Thread Eric Maynard
following, I > > > get a bug during the build: > > > > > > String foo = getTheValue(); > > > return foo.length(); > > > > > > Now the tool is actually doing its job of reminding me that I'm using a > > > value without having che

Re: [DISCUSS] Gradle and Maven runner plugins

2025-05-29 Thread Eric Maynard
Just one small note -- we already have tests that require docker to pass.

Re: [DISCUSS] Standardize nullness annotations

2025-05-27 Thread Eric Maynard
using > > > > > > Guava's Preconditions. This is especially useful if we can't > > > guarantee > > > > > > that a method parameter, for instance, will never be null, > because > > > > > > it's being provided by some ext

Re: [DISCUSS] Standardize nullness annotations

2025-05-27 Thread Eric Maynard
t; > Cheers, > Dmitri. > > On Tue, May 27, 2025 at 5:07 PM Eric Maynard > wrote: > > > If we've checked something is non-null or received a non-null value from > a > > library, we can continue annotating it as @Nonnull within Polaris. We can > > als

Re: [DISCUSS] ContainerSpecHelper and the upcoming NoSQL persistence (1563)

2025-05-27 Thread Eric Maynard
Could we start with it scoped to the NoSQL module and promote it out when another use case for it shows up? On Tue, May 27, 2025 at 9:41 AM Dmitri Bourlatchkov wrote: > 1. If ContainerSpecHelper will be used by NoSql implementation, it should > go to the future NoSQL module. We should make it pr

Re: [DISCUSS] Storing Table Metadata in the Metastore

2025-05-26 Thread Eric Maynard
s would be different from the Entity Cache > - Entity Cache is optional in the Resolver > - Entity Cache is tightly coupled with the Resolver (if present)... as > discussed wrt this cache's consistency concerns. > > Thanks, > Dmitri. > > On Fri, May 23, 2025 at

Re: [DISCUSS] Storing Table Metadata in the Metastore

2025-05-23 Thread Eric Maynard
with at the Persistence layer. The proposed NoSQL impl. > can do that efficiently. As for the current JDBC impl. it might be > reasonable to store a BLOB (at least initially). > > WDYT? > > Thanks, > Dmitri. > > [1] https://github.com/apache/polaris/pull/1285 > > On

[DISCUSS] Storing Table Metadata in the Metastore

2025-05-23 Thread Eric Maynard
Hi all, Some time ago I opened this PR which proposes to store/cache TableMetadata in the Polaris metastore, avoiding a trip to object storage in many cases. Based on this recent comment

Re: [Discuss] Add `location` to generic table spec

2025-05-22 Thread Eric Maynard
> i meant no two tables under the same catalog can have the same location This is a stricter requirement than we have for Iceberg tables. Are we really going to enforce this? How will we do it efficiently? If not, let's not put it in the spec. > we do not have any update support It would be tri

Re: [Discuss] Add `location` to generic table spec

2025-05-21 Thread Eric Maynard
; >> the > >> > > > > location > >> > > > > > > is > >> > > > > > > > > still controlled by the engine and users today like > table > >> > > names. > >> > > > > > > > > P

Re: Proposal for Implementation of the Proposed Iceberg REST Spec - Events API

2025-05-21 Thread Eric Maynard
-devlist If we design by interface properly, it should be relatively easy to offer both a disk buffering and an always-write implementation right? On Thu, May 22, 2025 at 12:12 AM Adnan Hemani wrote: > Hi all, > > Thanks for sharing these thoughts. I’m also not completely sure about how > much

Re: [DISCUSS] Standardize nullness annotations

2025-05-21 Thread Eric Maynard
Thanks for bringing this up as I’ve been confused by this a few times. Before Polaris I hadn’t really encountered these annotations and I was surprised to learn they don’t “do anything” — that is, there is no additional safety you get at runtime when a null value is passed into a parameter marked

Re: Context-Aware Functions for Apache Polaris

2025-05-20 Thread Eric Maynard
or. > > FGAC is a very complex topic. The right way would be to have a holistic > design and agreed-on approach. That does take time. > > On 20.05.25 16:47, Eric Maynard wrote: > > I wouldn’t say that Polaris is changing a view definition, but per my > > understanding Polari

Re: Context-Aware Functions for Apache Polaris

2025-05-20 Thread Eric Maynard
I wouldn’t say that Polaris is changing a view definition, but per my understanding Polaris is actually generating a view based on a Policy. We will need a way for Polaris to embed some information into these views. I don’t think this is a P0 to make FGAC work, and I don’t think this necessarily n

Re: [DISCUSS] On-going CHANGELOG with breaking changes

2025-05-16 Thread Eric Maynard
There is some manual overhead, but I think the alternative is worse (breaking changes go untracked). It's a good idea to have an ongoing changelog like this, and I don't see a way to manage it other than manually. Your point around consistency is an important one. The example in the linked Nessie

Re: [Discuss] Add `location` to generic table spec

2025-05-07 Thread Eric Maynard
t Regards, > > > Yun > > > > > > > > > On Wed, May 7, 2025 at 3:54 PM Dmitri Bourlatchkov > > > wrote: > > > > > >> Another point: I'm pretty sure sooner or later users will want to move > > >> their data to some oth

Re: [Discuss] Add `location` to generic table spec

2025-05-07 Thread Eric Maynard
All good questions Dmitri — I’m especially interested in the first one as from what I understand Iceberg tables can have metadata and data at two different paths that we need to vend credentials for. For iceberg tables, we just use special properties to track these locations. I wonder if we couldn

Re: [DISCUSS] Deprecate Eclipse Link and make Relational JDBC as default

2025-05-05 Thread Eric Maynard
So long as EclipseLink is not completely yanked from the repo I would prefer to keep the tests running, even if it means the tests take longer to run. The risk of a regression seems too great. On Mon, May 5, 2025 at 3:18 PM Dmitri Bourlatchkov wrote: > Good point about tests! > > However, I beli

Re: Merge module polaris-quarkus-admin and polaris-quarkus-server

2025-05-05 Thread Eric Maynard
ckages. > > > I do not really see how unifying code simplifies LICENCE/NOTICE. We still > have to keep track of the same set of dependencies and update them for > every release, and this is the hard part. Where we put the mentions (after > we figure out what needs to be mentioned)

Re: Merge module polaris-quarkus-admin and polaris-quarkus-server

2025-05-05 Thread Eric Maynard
I don’t really understand the question. Are you asking how a single gradle module can contain a CLI and a service? The naive way is just to have two different main classes but perhaps you mean something else? In Dropwizard this was seemingly the standard way of doing things, as you could register

Re: [VOTE] Release Apache Polaris 0.10.0-beta-incubating (rc2)

2025-05-05 Thread Eric Maynard
+1 (non-binding) Let’s cherry pick doc & other changes back into 0.10.x as needed rather than block the release. On Mon, May 5, 2025 at 2:40 PM Adnan Hemani wrote: > This makes sense to me - but the 0.10 release tag would not line up with > the shell scripts included in that release if we only

Re: [DISCUSS] Adding an API to allow service admins to reset any principal's credentials.

2025-04-18 Thread Eric Maynard
>From what I understand, there is not a historical reason for this not having been implemented. It was discussed, but never prioritized. The doc looks great Mansehaj, thanks for putting this together. On Thu, Apr 17, 2025 at 3:14 PM Dmitri Bourlatchkov wrote: > Thanks, Mansehaj! > > Very nice p

Re: [DISCUSS] Polaris Federated Principals and Roles

2025-04-17 Thread Eric Maynard
+1 on the spec change On Wed, Apr 16, 2025 at 3:44 PM Michael Collado wrote: > Hey folks > > Some of you already know that I posted an initial PR to get federated > principals/roles added. One thing that came out of the feedback was a spec > change to make it clear when federated identities can

Re: [DISCUSS] Add In-Dev Polaris Migrator/Synchronizer Tool to apache/polaris-tools

2025-04-11 Thread Eric Maynard
For keeping two Polaris instances in sync, I agree that replicating at the persistence layer probably makes the most sense. However there are cases when you want to copy data from one Polaris instance to another but you may not have direct access to the metastore. For example, migrating from a sel

Re: [DISCUSS] Add Polaris Observe API

2025-04-10 Thread Eric Maynard
I think the concept is really useful. The only thing I think which would require some more investigation is how exactly we implement this API -- where the data is stored, how long it's retained, etc. We might need to consider pushing this data out into another service or at least supporting such an

Re: [Discuss] Change EntityTable properties and internal properties from TEXT to JSONB

2025-03-29 Thread Eric Maynard
I'm generally a +1 on using JSONB within Postgres. However I am in agreement with Dennis that we should avoid his item (2) above. If the application will need to load entities by some attributes A, B, and C then we should create methods loadEntityByA(a: A), loadEntityByB(b: B), and loadEntityByC(c

Re: Polaris-XTable Proposal

2025-03-27 Thread Eric Maynard
Hey Rahil! Thanks for bringing this up. My understanding is that we plan to do this through generic tables. I have this old design doc, but I haven't done anything with it yet because we're working our way through the initial generic tables implementation: https://docs.google.com/document/d/1eZQbw

Re: [PROPOSAL] Improve our "collaboration schema"

2025-03-27 Thread Eric Maynard
record, we can bypass request > change flag if a given number of reviewers approve the PR anyway. > > Thoughts ? > > Regards > JB > > Le mar. 25 mars 2025 à 19:54, Eric Maynard a > écrit : > > > In this case I think rather than a mistake, there could simply

Re: [ANNOUNCE] Welcome Dmitri Bourlatchkov, Dennis Huo and Yufei Gu to the Apache Polaris PPMC

2025-03-26 Thread Eric Maynard
Congrats all! The podling is in good hands. On Wed, Mar 26, 2025 at 9:23 PM Keith Chapman wrote: > Congratulations all, well deserved. It's nice to see the community > expanding. > > Regards, > Keith. > > http://keith-chapman.com > > > On Wed, Mar 26, 2025 at 7:40 PM Ajantha Bhat > wrote: > > >

Re: Clarification on Unique Entity Identifier in Polaris

2025-03-26 Thread Eric Maynard
ill require catalogId + entityId at the > persistence API level to perform an entity lookup now and in the future, > correct? > > On Wed, Mar 26, 2025 at 1:45 PM Dennis Huo wrote: > > > Correct, the entityId must be unique across all types. > > > > On Wed, Mar 26, 2025 at 1

Re: Clarification on Unique Entity Identifier in Polaris

2025-03-26 Thread Eric Maynard
I believe that entity ID by itself is meant to be a unique identifier. On Wed, Mar 26, 2025 at 12:27 PM Honah J. wrote: > Hi folks, > > I have a question about what constitutes a unique identifier for an entity > in Polaris in the future. > > Right now, `lookupEntity` takes catalogId, entityId,

Re: [PROPOSAL] Improve our "collaboration schema"

2025-03-25 Thread Eric Maynard
gt; So, I propose to continue with our soft rules and good intentions, and > > if it doesn't work, then we can discuss about a stricter approach. > > > > Thoughts ? > > > > Regards > > JB > > > > On Sun, Mar 23, 2025 at 8:05 AM Eric Maynard &

Re: Polaris benchmarks proposal

2025-03-24 Thread Eric Maynard
+1 to what JB said. My concern with Scala has mostly been that it can alienate new contributors and add ambiguity about when we should use Scala vs. Java. If we’re putting this in polaris-tools for now and the philosophy for polaris-tools is to more or less use whatever language you prefer, there

Re: [PROPOSAL] Improve our "collaboration schema"

2025-03-23 Thread Eric Maynard
Revisiting this thread, I wonder if the "soft rules" are working. I have noticed quite a few PRs merged recently with outstanding comments. The most recent of these that I personally reviewed are #1220 , #1226

Re: Clear separation of REST APIs

2025-03-21 Thread Eric Maynard
> I do not see a reason to change public facing Polaris APIs just because Iceberg's APIs changed/evolved. Really? If our goal is compatibility with the Iceberg REST spec, wouldn't Polaris APIs necessarily need to change if the Iceberg ones do? On Fri, Mar 21, 2025 at 4:14 AM Robert Stupp wrote:

Re: [DISCUSS] Preparing 0.10.0 release including binary distributions

2025-03-17 Thread Eric Maynard
Wasn’t that the intent of naming the first release 0.9.0? It seems wrong to cut a new version not from main On Mon, Mar 17, 2025 at 12:16 AM Jean-Baptiste Onofré wrote: > Hi Yufei > > Yeah, pre-1.0 or 1.0-alpha is OK for me. Good idea. > > Regards > JB > > On Mon, Mar 17, 2025 at 7:59 AM Yufei

Re: [DISCUSS] Voting on REST API changes

2025-03-13 Thread Eric Maynard
+1, this is a great idea. —EM On Thu, Mar 13, 2025 at 5:58 PM Dmitri Bourlatchkov wrote: > Hi All, > > A lot of REST API changes have been happening lately in GitHub. > > Client-facing APIs changes are relatively a lot more important than > refactorings and other code fixes, but can easily be h

Re: [VOTE] Use renovatebot weekly schedule

2025-03-03 Thread Eric Maynard
Is this vote closed? On Tue, Feb 25, 2025 at 10:53 AM Eric Maynard wrote: > > Weekly updates will _increase_ the amount of "noise" ... If all people > are fine with more noise, let's go for it > > That does seem to be what the votes reflect. > > On

Re: Donate Nessie Iceberg Catalog migrator

2025-02-25 Thread Eric Maynard
log-migrator" is also fine to me, given that the package name > already implies its association with Polaris. > > Yufei > > > On Tue, Feb 25, 2025 at 5:25 PM Eric Maynard > wrote: > > > iceberg-catalog-migrator sounds good to me! > > > > On Tue, Feb

Re: Donate Nessie Iceberg Catalog migrator

2025-02-25 Thread Eric Maynard
migrator` > > Suggestions are welcome! > > - Ajantha > > On Wed, Feb 26, 2025 at 1:20 AM Dmitri Bourlatchkov > wrote: > > > Good point re: grammar, and I'll defer to native speakers on that :) > > > > On Tue, Feb 25, 2025 at 2:45 PM Eric Maynard > &

Re: Donate Nessie Iceberg Catalog migrator

2025-02-25 Thread Eric Maynard
The Nessie migrator was `iceberg-catalog-migrator`, and "catalogs" does not seem grammatical to me. If I'm grading papers, I'm not a papers grader, I'm a paper grader. The fact that it works with different catalog implementations doesn't change that, but if we think polaris-catalog-migrator sounds

Re: [VOTE] Use renovatebot weekly schedule

2025-02-25 Thread Eric Maynard
> Weekly updates will _increase_ the amount of "noise" ... If all people are fine with more noise, let's go for it That does seem to be what the votes reflect. On Tue, Feb 25, 2025 at 7:11 AM Dmitri Bourlatchkov wrote: > What was probably not considered at all: Weekly updates will _increase_ >

Re: Donate Nessie Iceberg Catalog migrator

2025-02-25 Thread Eric Maynard
Small question, why catalog*s*? On Tue, Feb 25, 2025 at 5:51 AM Jean-Baptiste Onofré wrote: > Hi folks, > > I created the https://github.com/apache/polaris-catalogs-migrator > repository. > > I will work with Ajantha to populate it. > > Regards > JB > > On Thu, Feb 20, 2025 at 12:33 PM Jean-Bapt

Re: [VOTE] Use renovatebot weekly schedule

2025-02-21 Thread Eric Maynard
Obligatory +1 as it's my issue -- I'm also open to less frequent. My pitch: This is an upper bound on how often we update dependencies, not a lower bound. If an actual person sees an actual reason to update a dependency, they can and should still do that :) --EM On Fri, Feb 21, 2025 at 9:27 AM A

Re: Apache Polaris 1.0 Release – Next Milestones and Release Manager Volunteer

2025-02-17 Thread Eric Maynard
There are still many issues labelled as "bug" that are not bugs but feature requests or requests for some refactoring, e.g. (1 , 2 , 3 ). I do not think that we

Re: Polaris persistence refactor POC

2025-02-17 Thread Eric Maynard
Generating code based on the spec is completely different than generating code based on the annotations introduced by Immutables. More importantly, we shouldn’t allow a discussion about pushing the project forward to devolve into a debate around some third-party library that’s not necessary to the

Re: [DISCUSS] Separate Helm charts release

2025-01-30 Thread Eric Maynard
That seems like it could confuse users to me. The docs will refer to feature X being in version Y of the application — how do I connect that to a helm chart? Or if I want to go read the source code that’s connected to the helm chart I’m running, where do I find that mapping? Couldn’t we just cut a

Re: Polaris has two websites

2025-01-30 Thread Eric Maynard
; On Thu, Jan 23, 2025 at 7:26 PM Yufei Gu wrote: > > > > The polaris.io is outdated. I recommend shutting it down and migrating > > relevant pages. > > > > Yufei > > > > > > On Thu, Jan 23, 2025 at 10:20 AM Eric Maynard > > wrote: > > > > > Both polaris.io and polaris.apache.org are live. Should we keep both > up? > > > > > > --EM > > > >

Re: Proposal: Use of Realm Instead of RealmId

2025-01-29 Thread Eric Maynard
> Renaming `RealmId` to `Realm` and then also add another `RealmXyz` type makes it difficult to distinguish, causing unnecessary more confusion. I didn't see another such type in Yufei's PR , unless you're talking about the ImmutableRealmId type you add

Re: Polaris has two websites

2025-01-23 Thread Eric Maynard
> > > relevant pages. > > > > > > Yufei > > > > > > > > > On Thu, Jan 23, 2025 at 10:20 AM Eric Maynard < > eric.w.mayn...@gmail.com> > > > wrote: > > > > > > > Both polaris.io and polaris.apache.org are live. Should we keep both > > up? > > > > > > > > --EM > > > > > > > > > >

  1   2   >