Hello Fabian, Thanks ! According to your answers on this post https://stackoverflow.com/questions/50340107/order-of-events-with-chained-keyby-calls-on-same-key, if I'm right I can use my sort function followed by a keyby and use a Window for aggregate these events. And the order will be preserved if I use the same Key and the same partionning. I'm right ?
SingleOutputStreamOperator<XXX> sortStream = conversionStreamKeyed.process(new SortEventFunction()).setParallelism(1).name("Sort events"); // use the same key KeyedStream<XX, String> sortStreamKeyed = sortStream.keyBy((XXX event) -> event.getPartitionKey()); sortStreamKeyed.window(TumblingProcessingTimeWindow....setParallelism(1).name("Aggregate events"); Thanks David On 2019/02/04 13:54:14, Fabian Hueske <fhue...@gmail.com> wrote: > Hi, > > A WindowAll is executed in a single task. If you sort the data before the > window, the sorting must also happen in a single task, i.e., with > parallelism 1. > The reasons is that an operator somewhat randomly merges multiple input > partitions. So even if each input partition is sorted, the merging will > result in out-of-order data. > > Best, > Fabian > > Am Sa., 2. Feb. 2019 um 17:11 Uhr schrieb morin.david....@gmail.com < > morin.david....@gmail.com>: > > > Hello, > > > > I use Watermarks and a function to sort events at the end of my pipeline. > > I've used this tutorial to sort my data: > > https://training.da-platform.com/exercises/carSort.html > > SingleOutputStreamOperator<XXXX> sortStream = streamKeyed.process(new > > SortEventFunction()).. > > > > Then I want to apply a Window and use AggregateFunction to obtain a group > > of data. Thus when a trigger is launched, I can push all these data to my > > backend at the same time (with puts method for Hbase for example) > > But the order here must be guaranteed. > > Can I use a windowAll on that stream ? > > sortStream.windowAll(... > > > > Thanks in advance > > David > > > > > > >