Dear Flink Users, I'm running the Yahoo streaming benchmarks (the original version) [1] on Flink 1.8.3 and got 60K tuples per second. Because I got 282K tuples per second with Flink 1.1.3, I would like to ask your opinions where I should look at.
I have been using one node for a JobManager and 10 nodes for a TaskManager per each. Below is my current setting for the benchmark and Flink 1.8.3: * 16 vCPUs and 24 GiB for the JobManager node * 32 vCPUs and 32 GiB for each TaskManager node # localConf.yaml kafka.partitions: 5 process.hosts: 1 process.cores: 32 # flink-conf.yaml jobmanager.heap.size: 5120m taskmanager.heap.size: 20480m taskmanager.numberOfTaskSlots: 16 parallelism.default: 1 And the following is the previous settings for the benchmark and Flink 1.1.3: * 16 vCPUs and 24 GiB for the JobManager node and 10 TaskManager nodes #localConf.yaml kafka.partitions: 5 process.hosts: 1 process.cores: 16 # flink-conf.yaml jobmanager.heap.mb: 1024 taskmanager.heap.mb: 15360 taskmanager.numberOfTaskSlots: 16 taskmanager.memory.preallocate: false parallelism.default: 1 taskmanager.network.numberOfBuffers: 6432 Thank you and with best regards, Shinhyung Yang [1]: https://github.com/yahoo/streaming-benchmarks
