I see, what I consider highly likely here is that the lookup to HBase is the bottleneck. If the lookup takes to long events “sit in a queue” between the map and flatMap operations. If you replace the HBase lookup by some dummy code you should see the latency go away.
The reason you don’t see latency when you don’t have a custom partitioner is that here the map and flatMap are chained together: sending an event from one operator to the next is basically just a function call and there is therefore no queue that can be filled that makes events “wait”. Best, Aljoscha > On 28. Jun 2017, at 15:17, sohimankotia <sohimanko...@gmail.com> wrote: > > Source is KafKa . > FlatMap has HBase Lookup > Sink is Kafka . > > I tried to get stats over the days . I see that almost 40 % were having > latency of 0 seconds , 10 % 0-30 sec, approx 10% 30-60 sec and 10 % around > 60 - 120 sec and 30 % around 120 - 210 secs . > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Partitioner-is-spending-around-2-to-4-minutes-while-pushing-data-to-next-operator-tp13913p14036.html > Sent from the Apache Flink User Mailing List archive. mailing list archive at > Nabble.com.
