Hi, Thanks a lot for the reply. I added some more metrics to the pipeline to understand bottleneck. Seems like avro deserialization introduces some delay. With use of histogram I found processing of a single message takes ~300us(p99). ~180(p50). Which means a single slot can output at most 3000 messages per second. This essentially means, to support QPS of 3mil/s I will need parallelism of 1000. Is my understanding correct ? Can I do anything else apart from having so many slots in my job cluster ? Also do you have any guides or pointers how to do such setups. eg, large number of taskmanagers with smaller slots or bigger TMs with many slots and bigger jvms, larger network buffers etc ?
-- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
