Hello Fabian,
Thanks !
According to your answers on this post
https://stackoverflow.com/questions/50340107/order-of-events-with-chained-keyby-calls-on-same-key,
if I'm right I can use my sort function followed by a keyby and use a Window
for aggregate these events. And the order will be preserved if I use the same
Key and the same partionning. I'm right ?
SingleOutputStreamOperator<XXX> sortStream =
conversionStreamKeyed.process(new
SortEventFunction()).setParallelism(1).name("Sort events");
// use the same key
KeyedStream<XX, String> sortStreamKeyed = sortStream.keyBy((XXX event)
-> event.getPartitionKey());
sortStreamKeyed.window(TumblingProcessingTimeWindow....setParallelism(1).name("Aggregate
events");
Thanks
David
On 2019/02/04 13:54:14, Fabian Hueske <fhue...@gmail.com> wrote:
> Hi,
>
> A WindowAll is executed in a single task. If you sort the data before the
> window, the sorting must also happen in a single task, i.e., with
> parallelism 1.
> The reasons is that an operator somewhat randomly merges multiple input
> partitions. So even if each input partition is sorted, the merging will
> result in out-of-order data.
>
> Best,
> Fabian
>
> Am Sa., 2. Feb. 2019 um 17:11 Uhr schrieb morin.david....@gmail.com <
> morin.david....@gmail.com>:
>
> > Hello,
> >
> > I use Watermarks and a function to sort events at the end of my pipeline.
> > I've used this tutorial to sort my data:
> > https://training.da-platform.com/exercises/carSort.html
> > SingleOutputStreamOperator<XXXX> sortStream = streamKeyed.process(new
> > SortEventFunction())..
> >
> > Then I want to apply a Window and use AggregateFunction to obtain a group
> > of data. Thus when a trigger is launched, I can push all these data to my
> > backend at the same time (with puts method for Hbase for example)
> > But the order here must be guaranteed.
> > Can I use a windowAll on that stream ?
> > sortStream.windowAll(...
> >
> > Thanks in advance
> > David
> >
> >
> >
>
