REST catalog proposal

2021-12-13 Thread Ryan Murray
Hi all, For those of you who haven't been following there has been some interesting discussion around the proposal for a REST based catalog[1]. One of the primary questions I had while reading it was 'what is the overall goal of the API?'. Given the size of this question I thought it might be b

Re: REST catalog proposal

2021-12-13 Thread Jack Ye
Hi Ryan, Thanks for starting the thread! Just want to share some of my thoughts related to this topic. I think the AWS Glue, DynamoDB and JDBC catalogs will continue to live, I don't see a unification through REST as we are not going to build a REST server between Iceberg and the related AWS serv

Re: Schedule for 0.13 release

2021-12-13 Thread Puneet Zaroo
Hi Jack, Does this milestone : https://github.com/apache/iceberg/milestone/16 track all the differences between 0.12 and 0.13 or is there another document capturing the major changes in 0.13. Thanks, - Puneet On Fri, Dec 3, 2021 at 12:47 PM Jack Ye wrote: > Hi Mayur, > > The milestone is tracked

Re: Schedule for 0.13 release

2021-12-13 Thread Jack Ye
https://docs.google.com/document/d/18yc8_Q6Hpc_r7JSoQO4oswQSHgHxJFDnr6Zif9_tceA/edit The document is not up-to-date, it collects all the updates up to mid November. I will update the doc again once all blockers in the milestone are in. btw, in the milestone, there are a few PRs also for documenta

Re: Schedule for 0.13 release

2021-12-13 Thread Puneet Zaroo
Thanks Jack! On Mon, Dec 13, 2021 at 4:49 PM Jack Ye wrote: > > https://docs.google.com/document/d/18yc8_Q6Hpc_r7JSoQO4oswQSHgHxJFDnr6Zif9_tceA/edit > > The document is not up-to-date, it collects all the updates up to mid > November. I will update the doc again once all blockers in the mileston

Re: Vendor integration strategy

2021-12-13 Thread Jack Ye
Thank you Openinx for preparing all these PRs and the vote options! In the community sync, we also talked about not including any new vendor integration modules in engine runtimes. In this approach, vendor dependencies do not need to be listed in the provided (compile only) scope. Vendors will pub

Re: Vendor integration strategy

2021-12-13 Thread OpenInx
Thanks Jack for the feedback ! Yes, I agree that the ideal approach is option#3 as you said. The core issue for the vendor bundled runtimes jar is : More vendors currently provide maven dependency configuration, but may not provide bundled runtime jar. Because maven dependency configuration can

Re: REST catalog proposal

2021-12-13 Thread Ryan Blue
I think Jack does a great job of pointing out a lot of the advantages. I agree with him, but I’ll add my perspective as well. I suggested the REST catalog a couple (few?) months ago when we were talking about the DynamoDB catalog and it stuck with me as the solution to quite a few problems. First,