Re: Spark 3.1 branch cut 4th Dec?

2020-11-20 Thread Hyukjin Kwon
Just for the record, I'll stick to the date we documented at https://spark.apache.org/versioning-policy.html Should be best to stick to what we wrote there given they we delayed once already. On Sat, 21 Nov 2020, 02:28 Xiao Li, wrote: > Thank you, Ryan! > > Xiao > > Dongjoon Hyun

[SS] full outer stream-stream join

2020-11-20 Thread real-cheng-su
Hi, Stream-stream join in spark structured streaming right now supports INNER, LEFT OUTER, RIGHT OUTER and LEFT SEMI join type. But it does not support FULL OUTER join and we are working on to add it in https://github.com/apache/spark/pull/30395 . Given LEFT OUTER and RIGHT OUTER stream-stream

Re: Spark 3.1 branch cut 4th Dec?

2020-11-20 Thread Xiao Li
Thank you, Ryan! Xiao Dongjoon Hyun 于2020年11月20日周五 上午9:20写道: > It sounds great! :) > > Thanks, Ryan. > > On Fri, Nov 20, 2020 at 9:19 AM Ryan Blue wrote: > >> I think we should be able to get the CREATE TABLE changes in. Now that >> the main blocker (EXTERNAL) has been decided, it's just a

Re: Spark 3.1 branch cut 4th Dec?

2020-11-20 Thread Dongjoon Hyun
It sounds great! :) Thanks, Ryan. On Fri, Nov 20, 2020 at 9:19 AM Ryan Blue wrote: > I think we should be able to get the CREATE TABLE changes in. Now that the > main blocker (EXTERNAL) has been decided, it's just a matter of normal > review comments. > > On Fri, Nov 20, 2020 at 9:05 AM

Re: Spark 3.1 branch cut 4th Dec?

2020-11-20 Thread Ryan Blue
I think we should be able to get the CREATE TABLE changes in. Now that the main blocker (EXTERNAL) has been decided, it's just a matter of normal review comments. On Fri, Nov 20, 2020 at 9:05 AM Dongjoon Hyun wrote: > Thank you for sharing, Xiao. > > I hope we are able to make some agreement

Re: Spark 3.1 branch cut 4th Dec?

2020-11-20 Thread Dongjoon Hyun
Thank you for sharing, Xiao. I hope we are able to make some agreement for CREATE TABLE DDLs, too. Bests, Dongjoon. On Fri, Nov 20, 2020 at 9:01 AM Xiao Li wrote: > https://github.com/apache/spark/pull/28026 is the major feature I am > tracking. It is painful to keep two sets of CREATE TABLE

Re: Spark 3.1 branch cut 4th Dec?

2020-11-20 Thread Xiao Li
https://github.com/apache/spark/pull/28026 is the major feature I am tracking. It is painful to keep two sets of CREATE TABLE DDLs with different behaviors. This hurts the usability of our SQL users, based on what I heard. Unfortunately, this PR missed Spark 3.0 release. Now, I think we should try

Re: Spark 3.1 branch cut 4th Dec?

2020-11-20 Thread Xiao Li
Hi, Dongjoon, Thank you for your feedback. I think *Early December* does not mean we will cut the branch on Dec 1st. I do not think Dec 1st and Dec 4th are a big deal. Normally, it would be nice to give enough buffer. Based on my understanding, this email is just a *proposal* and a *reminder*. In

Re: Spark 3.1 branch cut 4th Dec?

2020-11-20 Thread Dongjoon Hyun
Hi, Xiao. I agree. > Merging the feature work after the branch cut should not be encouraged in general, although some committers did make some exceptions based on their own judgement. We should try to avoid merging the feature work after the branch cut. So, the Apache Spark community