Re: Apache Spark 2.4.8 (and EOL of 2.4)

2021-03-04 Thread Dongjoon Hyun
Thank you, Liang-Chi! Next Monday sounds good. To All. Please ping Liang-Chi if you have a missed backport. Bests, Dongjoon. On Thu, Mar 4, 2021 at 7:00 PM Xiao Li wrote: > Thank you, Liang-Chi! > > Xiao > > On Thu, Mar 4, 2021 at 6:25 PM Hyukjin Kwon wrote: > >> Thanks @Liang-Chi Hsieh

Re: Apache Spark 2.4.8 (and EOL of 2.4)

2021-03-04 Thread Xiao Li
Thank you, Liang-Chi! Xiao On Thu, Mar 4, 2021 at 6:25 PM Hyukjin Kwon wrote: > Thanks @Liang-Chi Hsieh for driving this. > > 2021년 3월 5일 (금) 오전 5:21, Liang-Chi Hsieh 님이 작성: > >> >> Thanks all for the input. >> >> If there is no objection, I am going to cut the branch next Monday. >> >>

Re: Apache Spark 2.4.8 (and EOL of 2.4)

2021-03-04 Thread Hyukjin Kwon
Thanks @Liang-Chi Hsieh for driving this. 2021년 3월 5일 (금) 오전 5:21, Liang-Chi Hsieh 님이 작성: > > Thanks all for the input. > > If there is no objection, I am going to cut the branch next Monday. > > Thanks. > Liang-Chi > > > Takeshi Yamamuro wrote > > +1 for releasing 2.4.8 and thanks, Liang-chi,

Re: Apache Spark 2.4.8 (and EOL of 2.4)

2021-03-04 Thread Liang-Chi Hsieh
Thanks all for the input. If there is no objection, I am going to cut the branch next Monday. Thanks. Liang-Chi Takeshi Yamamuro wrote > +1 for releasing 2.4.8 and thanks, Liang-chi, for volunteering. > Btw, anyone roughly know how many v2.4 users still are based on some stats > (e.g., # of

Re: [DISCUSS] SPIP: FunctionCatalog

2021-03-04 Thread Liang-Chi Hsieh
Yeah, in short this is a great compromise approach and I do like to see this proposal move forward to next step. This discussion is valuable. Chao Sun wrote > +1 on Dongjoon's proposal. Great to see this is getting moved forward and > thanks everyone for the insightful discussion! > > > >

Re: [DISCUSS] SPIP: FunctionCatalog

2021-03-04 Thread Chao Sun
+1 on Dongjoon's proposal. Great to see this is getting moved forward and thanks everyone for the insightful discussion! On Thu, Mar 4, 2021 at 8:58 AM Ryan Blue wrote: > Okay, great. I'll update the SPIP doc and call a vote in the next day or > two. > > On Thu, Mar 4, 2021 at 8:26 AM Erik

Re: minikube and kubernetes cluster versions for integration testing

2021-03-04 Thread shane knapp ☠
fwiw, upgrading minikube and the associated VM drivers is potentially a PITA. your PR will absolutely be tested before merging. :) On Thu, Mar 4, 2021 at 10:13 AM attilapiros wrote: > Thanks Shane! > > I can do the documentation task and the Minikube version check can be > incorporated into

Re: minikube and kubernetes cluster versions for integration testing

2021-03-04 Thread attilapiros
Thanks Shane! I can do the documentation task and the Minikube version check can be incorporated into my PR. When my PR is finalized (probably next week) I will create a jira for you and you can set up the test systems and you can even test my PR before merging it. Is this possible / fine for

Re: [DISCUSS] SPIP: FunctionCatalog

2021-03-04 Thread Ryan Blue
Okay, great. I'll update the SPIP doc and call a vote in the next day or two. On Thu, Mar 4, 2021 at 8:26 AM Erik Krogen wrote: > +1 on Dongjoon's proposal. This is a very nice compromise between the > reflective/magic-method approach and the InternalRow approach, providing > a lot of

Re: [DISCUSS] SPIP: FunctionCatalog

2021-03-04 Thread Erik Krogen
+1 on Dongjoon's proposal. This is a very nice compromise between the reflective/magic-method approach and the InternalRow approach, providing a lot of flexibility for our users, and allowing for the more complicated reflection-based approach to evolve at its own pace, since you can always fall

using accumulators in (MicroBatch) InputPartitionReader

2021-03-04 Thread kordex
I tried to create a data source, however our use case is bit hard as we do only know the available offsets within the tasks, not on the driver. I therefore planned to use accumulators in the InputPartitionReader but they seem not to work. Example accumulation is done here