Re: Meeting Minutes from 10/20 Iceberg Sync

My impression came from when Anton proposed the following in the earlier thread on Spark version support strategy: *Option 3 (separate modules for Spark 3.1/3.2)* > Introduce more modules in the same project. > Can consume unreleased changes but it will required at least 5 modules to > support

Re: Meeting Minutes from 10/20 Iceberg Sync

Thanks Ryan, I've added more details to the meeting minutes in the sync doc. Adding v3.2 to Spark Build Refactoring - Russell and Anton will coordinate on dropping in a Spark 3.2 module - We currently have 3.1 in the `spark3` module. We’ll move that out to its own module and

Re: Meeting Minutes from 10/20 Iceberg Sync

I don't recall there being a consensus to deprecate all but 2 versions of Spark. I think the confusion may be because that's what Flink versions are supported in that community. For Spark, I think we will need to support older versions until most people are able to move off of them, which can take

Re: Meeting Minutes from 10/20 Iceberg Sync

Thanks Sam. Was there also agreement to deprecate Spark 3.0 support and go with supporting the latest 2 versions of Spark 3? On Tue, Oct 26, 2021 at 11:36 AM Sam Redai wrote: > If I remember correctly, we landed on option 1, creating a v3.1 without > the extra reflection logic and then just

Re: Meeting Minutes from 10/20 Iceberg Sync

If I remember correctly, we landed on option 1, creating a v3.1 without the extra reflection logic and then just deprecating 3.0 when the time comes. If everyone agrees with that I can amend the notes to describe that more explicitly. -Sam On Mon, Oct 25, 2021 at 11:30 AM Wing Yew Poon wrote:

Re: Meeting Minutes from 10/20 Iceberg Sync

> > Adding v3.2 to Spark Build Refactoring > >- > >Russell and Anton will coordinate on dropping in a Spark 3.2 module >- > >We currently have 3.1 in the `spark3` module. We’ll move that out to >its own module and mirror what we do with the 3.2 module. (This will enable >

Re: Meeting Minutes from 10/20 Iceberg Sync

Added as an item to the sync doc, thank you! -Sam On Thu, Oct 21, 2021 at 11:35 PM OpenInx wrote: > Thanks for the detailed report ! > > One more thing: We now have made a lot of progress in integrating Alibaba > Cloud (https://www.aliyun.com/), Please see >

Re: Meeting Minutes from 10/20 Iceberg Sync

Thanks for the detailed report ! One more thing: We now have made a lot of progress in integrating Alibaba Cloud (https://www.aliyun.com/), Please see https://github.com/apache/iceberg/projects/21 (Thanks @xingbowu - https://github.com/xingbowu). On Thu, Oct 21, 2021 at 11:30 PM Sam Redai

Meeting Minutes from 10/20 Iceberg Sync

Good Morning Everyone, Here are the minutes from our Iceberg Sync that took place on October 20th, 9am-10am PT. Please remember that anyone can join the discussion so feel free to share the Iceberg-Sync google group with anyone who is seeking an invite.