Thanks all for the support! Great to see we drive the discussion for
Structured Streaming and have sufficient support.
We would like to move forward with the vote thread. Please also participate
in the vote. Thanks again!
On Thu, Dec 1, 2022 at 10:04 AM Wenchen Fan wrote:
> +1 to improve the
+1 to improve the widely used micro-batch mode first.
On Thu, Dec 1, 2022 at 8:49 AM Hyukjin Kwon wrote:
> +1
>
> On Thu, 1 Dec 2022 at 08:10, Shixiong Zhu wrote:
>
>> +1
>>
>> This is exciting. I agree with Jerry that this SPIP and continuous
>> processing are orthogonal. This SPIP itself
+1
On Thu, 1 Dec 2022 at 08:10, Shixiong Zhu wrote:
> +1
>
> This is exciting. I agree with Jerry that this SPIP and continuous
> processing are orthogonal. This SPIP itself would be a great improvement
> and impact most Structured Streaming users.
>
> Best Regards,
> Shixiong
>
>
> On Wed, Nov
+1
This is exciting. I agree with Jerry that this SPIP and continuous
processing are orthogonal. This SPIP itself would be a great improvement
and impact most Structured Streaming users.
Best Regards,
Shixiong
On Wed, Nov 30, 2022 at 6:57 AM Mridul Muralidharan
wrote:
>
> Thanks for all the
Thanks for all the clarifications and details Jerry, Jungtaek :-)
This looks like an exciting improvement to Structured Streaming - looking
forward to it becoming part of Apache Spark !
Regards,
Mridul
On Mon, Nov 28, 2022 at 8:40 PM Jerry Peng
wrote:
> Hi all,
>
> I will add my two cents.
Hi all,
I will add my two cents. Improving the Microbatch execution engine does
not prevent us from working/improving on the continuous execution engine in
the future. These are orthogonal issues. This new mode I am proposing in
the microbatch execution engine intends to lower latency of this
Hi Mridul,
I'd like to make clear to avoid any misunderstanding - the decision was not
led by me. (I'm just a one of engineers in the team. Not even TL.) As you
see the direction, there was an internal consensus to not revisit the
continuous mode. There are various reasons, which I think we know
Hi Jungtaek,
Given the goal of the SPIP is reducing latency for stateless apps, and
should reasonably fit continuous mode design goals, it feels odd to not
support it fin the proposal.
I know you have raised concerns about continuous mode in past as well in
dev@ list, and we are further
Jungtaek,
Thanks for taking up the role to shepard this SPIP! Thank you for also
chiming in on your thoughts concerning the continuous mode!
Best,
Jerry
On Tue, Nov 22, 2022 at 5:57 PM Jungtaek Lim
wrote:
> Just FYI, I'm shepherding this SPIP project.
>
> I think the major meta question
Just FYI, I'm shepherding this SPIP project.
I think the major meta question would be, "why don't we spend effort on
continuous mode rather than initiating another feature aiming for the
same workload?". Jerry already updated the doc to answer the question, but
I can also share my thoughts about
Hi all,
I would like to start the discussion for a SPIP, Asynchronous Offset
Management in Structured Streaming. The high level summary of the SPIP is
that currently in Structured Streaming we perform a couple of offset
management operations for progress tracking purposes synchronously on the
11 matches
Mail list logo
