Hi Shinhyung,
Can you compare the performance of the different Flink versions based on
the same environment (Or at least the same configuration of the node and
framework)?
I see there are some different configurations of both clusters and
frameworks. It would be better to comparison in the same environment so
that we can figure out why there are more than 4x performance differences.
WDYT?
Best,
Vino
Shinhyung Yang 于2019年12月30日周一 下午1:45写道：
> Dear Flink Users,
>
> I'm running the Yahoo streaming benchmarks (the original version) [1]
> on Flink 1.8.3 and got 60K tuples per second. Because I got 282K
> tuples per second with Flink 1.1.3, I would like to ask your opinions
> where I should look at.
>
> I have been using one node for a JobManager and 10 nodes for a
> TaskManager per each.
>
> Below is my current setting for the benchmark and Flink 1.8.3:
>
> * 16 vCPUs and 24 GiB for the JobManager node
> * 32 vCPUs and 32 GiB for each TaskManager node
>
> # localConf.yaml
> kafka.partitions: 5
> process.hosts: 1
> process.cores: 32
>
> # flink-conf.yaml
> jobmanager.heap.size: 5120m
> taskmanager.heap.size: 20480m
> taskmanager.numberOfTaskSlots: 16
> parallelism.default: 1
>
> And the following is the previous settings for the benchmark and Flink
> 1.1.3:
>
> * 16 vCPUs and 24 GiB for the JobManager node and 10 TaskManager nodes
>
> #localConf.yaml
> kafka.partitions: 5
> process.hosts: 1
> process.cores: 16
>
> # flink-conf.yaml
> jobmanager.heap.mb: 1024
> taskmanager.heap.mb: 15360
> taskmanager.numberOfTaskSlots: 16
> taskmanager.memory.preallocate: false
> parallelism.default: 1
> taskmanager.network.numberOfBuffers: 6432
>
>
> Thank you and with best regards,
> Shinhyung Yang
>
> [1]: https://github.com/yahoo/streaming-benchmarks
>
