Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

2022-11-23 Thread Jungtaek Lim
Hi Mridul, I'd like to make clear to avoid any misunderstanding - the decision was not led by me. (I'm just a one of engineers in the team. Not even TL.) As you see the direction, there was an internal consensus to not revisit the continuous mode. There are various reasons, which I think we know

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

2022-11-23 Thread Mridul Muralidharan
Hi Jungtaek, Given the goal of the SPIP is reducing latency for stateless apps, and should reasonably fit continuous mode design goals, it feels odd to not support it fin the proposal. I know you have raised concerns about continuous mode in past as well in dev@ list, and we are further