I have a execution flow (Streaming Job) with parallelism 1. source -> map -> partitioner -> flatmap -> sink
Since adding partitioner will start new thread but partitioner is spending average of 2 to 4 minutes while moving data from map to flat map . For more details about this : http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Using-Custom-Partitioner-in-Streaming-with-parallelism-1-adding-latency-td13766.html In some link here : https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks they have mentioned that the PipelinedSubpartition is a pipelined implementation to support streaming data exchange. The SpillableSubpartition is a blocking implementation to support batch data exchange. I am not sure how would i use these or reduce latency from map -> partitioner -> flatmap . -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Partitioner-is-spending-around-2-to-4-minutes-while-pushing-data-to-next-operator-tp13913.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
