Dear Weijie, Thanks for your feedback and the +1 on this feature.
In response to your query, you are indeed correct. The PartitionWindowedStream's upstream operator must utilize a Forward type partitioner. Consequently, the framework will apply a POINTWISE edge connection pattern between the upstream operator and the operators within the PartitionWindowedStream. As the parallelism of upstream operator and operators in the PartitionWindowedStream are same, the semantic integrity of the 'xxxPartition' can be guaranteed. Additionally, I'll ensure that this explanation is updated in the FLIP [1]. I hope this clarification addresses your concern, and I'm glad to assist if you have any more questions or insights to share. Best regards, Wencong [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream At 2023-12-12 15:08:31, "weijie guo" <guoweijieres...@gmail.com> wrote: >Thanks Wencong for driving this! > >I believe this is a useful feature, so +1 from my side. > >I only have one minor question about the exchange mode of `xxxPartition` >method. Does this means the window operator must be connected to the >upstream operator in forward edge (otherwise the concept of mapPartition is >a bit far-fetched). > >Best regards, > >Weijie > > >Wencong Liu <liuwencle...@163.com> 于2023年12月1日周五 14:04写道： > >> Hi devs, >> >> I'm excited to propose a new FLIP[1] aimed at enhancing the DataStream API >> >> to support full window processing on non-keyed streams. This feature >> addresses >> the current limitation where non-keyed DataStreams cannot accumulate >> records >> per subtask for collective processing at the end of input. >> >> Key proposals include: >> >> >> 1. Introduction of PartitionWindowedStream allowing non-keyed DataStreams >> to >> be transformed for full window processing per subtask. >> >> 2. Addition of four new APIs - mapPartition, sortPartition, aggregate, and >> reduce >> - to enable powerful operations on PartitionWindowedStream. >> >> This initiative seeks to fill the gap left by the deprecation of the >> DataSet API, >> marrying its partition processing strengths with the dynamic capabilities >> of the DataStream API. >> >> Looking forward to your feedback on this FLIP. >> >> [1] >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream >> >> Best regards, >> Wencong Liu
