Thanks Wencong for the quick response, the updated FLIP LGTM. Best regards,
Weijie Wencong Liu <liuwencle...@163.com> 于2023年12月12日周二 19:02写道： > Dear Weijie, > > Thanks for your feedback and the +1 on this feature. > > In response to your query, you are indeed correct. The > PartitionWindowedStream's > > upstream operator must utilize a Forward type partitioner. Consequently, > the framework will apply a POINTWISE edge connection pattern between > the upstream operator and the operators within the > PartitionWindowedStream. > As the parallelism of upstream operator and operators in the > PartitionWindowedStream are same, the semantic integrity > of the 'xxxPartition' can be guaranteed. > > Additionally, I'll ensure that this explanation is updated in the FLIP [1]. > > I hope this clarification addresses your concern, and I'm glad to assist > if you have any more questions or insights to share. > > Best regards, > > Wencong > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream > > > > > > > > > > > > > > > > > At 2023-12-12 15:08:31, "weijie guo" <guoweijieres...@gmail.com> wrote: > >Thanks Wencong for driving this! > > > >I believe this is a useful feature, so +1 from my side. > > > >I only have one minor question about the exchange mode of `xxxPartition` > >method. Does this means the window operator must be connected to the > >upstream operator in forward edge (otherwise the concept of mapPartition > is > >a bit far-fetched). > > > >Best regards, > > > >Weijie > > > > > >Wencong Liu <liuwencle...@163.com> 于2023年12月1日周五 14:04写道： > > > >> Hi devs, > >> > >> I'm excited to propose a new FLIP[1] aimed at enhancing the DataStream > API > >> > >> to support full window processing on non-keyed streams. This feature > >> addresses > >> the current limitation where non-keyed DataStreams cannot accumulate > >> records > >> per subtask for collective processing at the end of input. > >> > >> Key proposals include: > >> > >> > >> 1. Introduction of PartitionWindowedStream allowing non-keyed > DataStreams > >> to > >> be transformed for full window processing per subtask. > >> > >> 2. Addition of four new APIs - mapPartition, sortPartition, aggregate, > and > >> reduce > >> - to enable powerful operations on PartitionWindowedStream. > >> > >> This initiative seeks to fill the gap left by the deprecation of the > >> DataSet API, > >> marrying its partition processing strengths with the dynamic > capabilities > >> of the DataStream API. > >> > >> Looking forward to your feedback on this FLIP. > >> > >> [1] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream > >> > >> Best regards, > >> Wencong Liu >
