Re: Re: [DISCUSS] FLIP-380: Support Full Partition Processing On Non-keyed DataStream

2023-12-12 Thread weijie guo
Thanks Wencong for the quick response, the updated FLIP LGTM. Best regards, Weijie Wencong Liu 于2023年12月12日周二 19:02写道: > Dear Weijie, > > Thanks for your feedback and the +1 on this feature. > > In response to your query, you are indeed correct. The > PartitionWindowedStream's > > upstream op

Re:Re: [DISCUSS] FLIP-380: Support Full Partition Processing On Non-keyed DataStream

2023-12-12 Thread Wencong Liu
Dear Weijie, Thanks for your feedback and the +1 on this feature. In response to your query, you are indeed correct. The PartitionWindowedStream's upstream operator must utilize a Forward type partitioner. Consequently, the framework will apply a POINTWISE edge connection pattern between the u

Re:Re: [DISCUSS] FLIP-380: Support Full Partition Processing On Non-keyed DataStream

2023-12-12 Thread Wencong Liu
Dear Yuxin, Thank you for your support and for raising those insightful queries. Indeed, the full window processing aligns more naturally with batch scenarios. The primary reason we've limited it to batch processing is due to state management considerations in a streaming context. If we were

Re: [DISCUSS] FLIP-380: Support Full Partition Processing On Non-keyed DataStream

2023-12-11 Thread Yuxin Tan
Thanks Wencong for driving this FLIP. +1 from my side. It appears to significantly improve the handling of full-window data within the DataStream API. However, I do have a small question regarding the current limitation to batch processing: does this stem from performance-related considerations? A

Re: [DISCUSS] FLIP-380: Support Full Partition Processing On Non-keyed DataStream

2023-12-11 Thread weijie guo
Thanks Wencong for driving this! I believe this is a useful feature, so +1 from my side. I only have one minor question about the exchange mode of `xxxPartition` method. Does this means the window operator must be connected to the upstream operator in forward edge (otherwise the concept of mapPar

[DISCUSS] FLIP-380: Support Full Partition Processing On Non-keyed DataStream

2023-11-30 Thread Wencong Liu
Hi devs, I'm excited to propose a new FLIP[1] aimed at enhancing the DataStream API to support full window processing on non-keyed streams. This feature addresses the current limitation where non-keyed DataStreams cannot accumulate records per subtask for collective processing at the end of input