Re: Sorting bounded data stream

2020-01-30 Thread Jingsong Li
Hi Łukasz, First, we are planning to design and implement the BoundedStream story, which will be discussed further in 1.11 or 1.12. SortedMapState was discussed in FLINK-6219 [1], But there are some problems that can not be solved well, so they have not been introduced. If it is a pure BoundedSt

Re: Sorting bounded data stream

2020-01-30 Thread Łukasz Jędrzejewski
Hi, Thank you very much for suggestions. I will check out the UnilateralSortMerge. However in our case we are using checkpoints. Kind regards, Łukasz W dniu 31.01.2020 o 07:54, Jingsong Li pisze: Hi Łukasz, First, we are planning to design and implement the BoundedStream story, which will

Re: Sorting bounded data stream

2020-01-30 Thread Jingsong Li
Hi, If you are using checkpoints, I think a simple way is using a ListState to store all coming records. And in endInput(), drain all records from ListState to a sorter to sort all records. Best, Jingsong Lee On Fri, Jan 31, 2020 at 3:10 PM Łukasz Jędrzejewski wrote: > Hi, > > Thank you very m