Re: Kafka partition alignment for event time

2016-02-25 Thread Erdem Agaoglu
-6 is not. This generally causes >>>> multiple >>>> (late-)windows to be fired after restart, because we now generate >>>> watermarks >>>> off partition-3 which says the windows of the unseen data in >>>> partition-6 are >>>> a

Re: Kafka partition alignment for event time

2016-02-25 Thread Robert Metzger
; already complete. >>> >>> This also has a side effect of windows not triggering unless some >>> rebalancing is done beforehand. Since only 1 of the 4 threads will source >>> data and generate watermarks, window triggers won't get watermarks from >>> other

Re: Kafka partition alignment for event time

2016-02-18 Thread Stephan Ewen
. I > think it may also create some edge cases even if things were not as extreme > as ours. If checkpoints could contain offsets of all of the partitions > regardless of their contents, probably storing start offsets in first run, > i > guess that would solve the problems around restar

Re: Kafka partition alignment for event time

2016-02-18 Thread agaoglu
that would solve the problems around restarting. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4998.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Kafka partition alignment for event time

2016-02-09 Thread Stephan Ewen
. allowOversubscription() > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4788.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >

Re: Kafka partition alignment for event time

2016-02-09 Thread Fabian Hueske
> > > > -- > > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4788.html > > Sent from the Apache Flink User Mailing List archive. mailing list > archive at Nabble.com. > >

Re: Kafka partition alignment for event time

2016-02-09 Thread shikhar
mins and starting it back up (the offsets it's resuming from are approx the same number of messages behind). -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4819.html Sent from the Apache Flink User

Re: Kafka partition alignment for event time

2016-02-09 Thread shikhar
-alignment-for-event-time-tp4782p4817.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Kafka partition alignment for event time

2016-02-09 Thread Stephan Ewen
perhaps allowing for more than 1 Kafka > >> partition > >> on a source should require opt-in, e.g. allowOversubscription() > >> > >> > >> > >> -- > >> View this message in context: > >> > http://apache-flink-user-mailing

Re: Kafka partition alignment for event time

2016-02-08 Thread Aljoscha Krettek
or automatically > enforced. > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4786.html > Sent from the Apache Flink User Mailing List archive. mailing list archive at > Nabble.com.

Re: Kafka partition alignment for event time

2016-02-08 Thread shikhar
documented or automatically enforced. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4786.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Kafka partition alignment for event time

2016-02-08 Thread shikhar
s used. Re: automatic enforcement, perhaps allowing for more than 1 Kafka partition on a source should require opt-in, e.g. allowOversubscription() -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp478

Kafka partition alignment for event time

2016-02-08 Thread shikhar
-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.