Re: [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

+1, looks interesting. On Wed, Sep 16, 2020 at 11:29 AM 郑瑞峰 wrote: > +1 > > > -- 原始邮件 -- > *发件人:* "叶先进" ; > *发送时间:* 2020年9月15日(星期二) 晚上7:09 > *收件人:* "Yi Wu"; > *抄送:* "Wenchen Fan";"Dongjoon Hyun"< > dongjoon.h...@gmail.com>;"kalyan";"Joseph Torres"< >

回复： [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

+1 --原始邮件-- 发件人: "叶先进"

Re: [DISCUSS] Time to evaluate "continuous mode" in SS?

Yeah I realized there's a proposal for push-based shuffle, and I agree that may unblock the architectural issue on true-streaming. (The root concern of the continuous mode has been that it doesn't fit with the architecture of Spark, and probably push-based shuffle could persuade me.) I guess

Re: [DISCUSS] Time to evaluate "continuous mode" in SS?

Hi Joseph, Would be interested in discussing your thoughts for how push-based shuffle could help with continuous mode in SS. We have discussed internally at LinkedIn with our Samza peers as well as with Alibaba Flink team for applicability of push-based shuffle on streaming engines, especially

Re: [DISCUSS] Time to evaluate "continuous mode" in SS?

It's worth noting that the push-based shuffle SPIP currently in progress addresses a substantial blocker in the area. If you remember when we removed the half-finished stateful query support, the lack of that functionality and the challenge of implementing it is basically why it was half-finished.

Re: [DISCUSS] Time to evaluate "continuous mode" in SS?

I think we certainly can't remove it without deprecation and a few releases. If there were big problems with it that weren't getting fixed, sure maybe, but lack of interest in reviewing minor changes isn't necessarily a bad sign. By the same logic you'd delete graphx long ago. Anecdotally, yes

Re: [DISCUSS] Time to evaluate "continuous mode" in SS?

Probably it would depend on the meaning of "experimental". My understanding of "experimental" is more likely "incubation", which may be graduated finally, or may be retired. To be clear, I'm evaluating the continuous mode as "candidate to retire", unless there are actual use cases in production

Re: [DISCUSS] Time to evaluate "continuous mode" in SS?

If you're suggesting making it un-Experimental, probably yes, as it is de facto not going to change much I expect. If you're saying remove it, probably not? I don't see that it's anywhere near deprecated, and not sure it's unmaintained - obviously tests etc still have to keep passing. On Mon, Sep

Re: [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

+1. > On Sep 15, 2020, at 5:10 PM, Yi Wu wrote: > > +1. Thank you for the effort! > > Best regards, > Yi > > On Tue, Sep 15, 2020 at 3:44 PM Wenchen Fan > wrote: > +1 > > On Tue, Sep 15, 2020 at 2:42 PM Dongjoon Hyun > wrote: > +1

Re: [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

+1. Thank you for the effort! Best regards, Yi On Tue, Sep 15, 2020 at 3:44 PM Wenchen Fan wrote: > +1 > > On Tue, Sep 15, 2020 at 2:42 PM Dongjoon Hyun > wrote: > >> +1 >> >> Bests, >> Dongjoon. >> >> On Mon, Sep 14, 2020 at 9:19 PM kalyan wrote: >> >>> +1 >>> >>> Will positively improve

Re: [DISCUSS] Time to evaluate "continuous mode" in SS?

Hi Jungtaek, All I see at the moment is that most of the users choose Flink over Spark when continues processing is needed. Unless there is a revolution in this area there is no point to keep maintenance. 2.5 years is lot in bigdata industry. If there will be efforts in this area then happy to

Re: [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

+1 On Tue, Sep 15, 2020 at 2:42 PM Dongjoon Hyun wrote: > +1 > > Bests, > Dongjoon. > > On Mon, Sep 14, 2020 at 9:19 PM kalyan wrote: > >> +1 >> >> Will positively improve the performance and reliability of spark... >> Looking fwd to this.. >> >> Regards >> Kalyan. >> >> On Tue, Sep 15, 2020,

Re: [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

+1 Bests, Dongjoon. On Mon, Sep 14, 2020 at 9:19 PM kalyan wrote: > +1 > > Will positively improve the performance and reliability of spark... > Looking fwd to this.. > > Regards > Kalyan. > > On Tue, Sep 15, 2020, 9:26 AM Joseph Torres > wrote: > >> +1 >> >> On Mon, Sep 14, 2020 at 6:39 PM