Re: [Discuss] Merge spark-3 branch into master

Thanks Matt, If branching is the only choice, then we would potentially have two *master* branches until spark-3 is vastly adopted. That will somehow increase the maintenance burden and lead to inconsistency. IMO I'm OK with the branching way, just think that we should have a clear way to keep

Re: [Discuss] Merge spark-3 branch into master

I think it’s generally dangerous and error-prone to try to support two versions of the same library in the same build, in the same published artifacts. This is the stance that Baseline + Gradle Consistent

Re: [Discuss] Merge spark-3 branch into master

I didn't realized that Gradle cannot support two different versions in one build. I think I did such things for Livy to build scala 2.10 and 2.11 jars simultaneously with Maven. I'm not so familiar with Gradle thing, I can take a shot to see if there's some hacky ways to make it work. Besides,

Re: [Discuss] Merge spark-3 branch into master

+1 for a 0.8.0 release with Spark 2.4 and then move on for Spark 3.0 when it's ready. On Tue, 3 Mar 2020 at 16:32, Ryan Blue wrote: > Thanks for bringing this up, Saisai. I tried to do this a couple of months > ago, but ran into a problem with dependency locks. I couldn't get two > different

Re: [Discuss] Merge spark-3 branch into master

Gradle Consistent Versions makes it such that it’s not allowed to have two different Spark versions in the module. I might be missing some context here, but why should Iceberg ever depend on two different versions of Spark at a time? -Matt Cheah From: Ryan Blue Reply-To:

Re: [Discuss] Merge spark-3 branch into master

Thanks for bringing this up, Saisai. I tried to do this a couple of months ago, but ran into a problem with dependency locks. I couldn't get two different versions of Spark packages in the build with baseline, but maybe I was missing something. If you can get it working, I think it's a great idea

[Discuss] Merge spark-3 branch into master

Hi team, I was thinking of merging spark-3 branch into master, also per the discussion before we could make spark-2 and spark-3 coexisted into 2 different sub-modules. With this, one build could generate both spark-2 and spark-3 runtime jars, user could pick either at preference. One concern is

Re: upsert base on copy on write mode

Typo: Maybe the discussion is very clear before.-> Maybe the discussion is NOT very clear before. Thanks OpenInx. I was thinking to add an interface DeleteRows and an API to table so that one could do things like: DeleteRows deleteRows = table.newRowLevelDelete();

Re: upsert base on copy on write mode

I think we should abstract the API firstly, then implement the MOR. COW is also a necessary implementation, but it's easy to implement and no so urgent. On Tue, Mar 3, 2020 at 3:45 PM Junjie Chen wrote: > Thanks, Ryan > > Maybe the discussion is very clear before. Actually, we have built an >