[ https://issues.apache.org/jira/browse/FLINK-20612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17283676#comment-17283676 ]
Piotr Nowojski commented on FLINK-20612: ---------------------------------------- [~Thesharing] some benchmarks are a bit unstable. I'm talking about those two: http://codespeed.dak8s.net:8000/timeline/?ben=initSchedulingStrategy.STREAMING&env=2 http://codespeed.dak8s.net:8000/timeline/?ben=partitionRelease.BATCH&env=2 do you think we could do something to improve theirs stability? > Add benchmarks for scheduler > ---------------------------- > > Key: FLINK-20612 > URL: https://issues.apache.org/jira/browse/FLINK-20612 > Project: Flink > Issue Type: Improvement > Components: Benchmarks, Runtime / Coordination > Affects Versions: 1.13.0 > Reporter: Zhilong Hong > Assignee: Zhilong Hong > Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > With Flink 1.12, we failed to run large-scale jobs on our cluster. When we > were trying to run the jobs, we met the exceptions like out of heap memory, > taskmanager heartbeat timeout, and etc. We increased the size of heap memory > and extended the heartbeat timeout, the job still failed. After the > troubleshooting, we found that there are some performance bottlenecks in the > jobmaster. These bottlenecks are highly related to the complexity of the > topology. > We implemented several benchmarks on these bottlenecks based on > flink-benchmark. The topology of the benchmarks is a simple graph, which > consists of only two vertices: one source vertex and one sink vertex. They > are both connected with all-to-all blocking edges. The parallelisms of the > vertices are both 8000. The execution mode is batch. The results of the > benchmarks are illustrated below: > Table 1: The result of benchmarks on bottlenecks in the jobmaster > | |*Time spent*| > |Build topology|45725.466 ms| > |Init scheduling strategy|38960.602 ms| > |Deploy tasks|17472.884 ms| > |Calculate failover region to restart|12960.912 ms| > We'd like to propose these benchmarks for procedures related to the > scheduler. There are three main benefits: > # They help us to understand the current status of task deployment > performance and locate where the bottleneck is. > # We can use the benchmarks to evaluate the optimization in the future. > # As we run the benchmarks daily, they will help us to trace how the > performance changes and locate the commit that introduces the performance > regression if there is any. > In the first version of the benchmarks, we mainly focus on the procedures we > mentioned above. The methods corresponding to the procedures are: > # Building topology: {{ExecutionGraph#attachJobGraph}} > # Initializing scheduling strategies: > {{PipelinedRegionSchedulingStrategy#init}} > # Deploying tasks: {{Execution#deploy}} > # Calculating failover regions: > {{RestartPipelinedRegionFailoverStrategy#getTasksNeedingRestart}} > In the benchmarks, the topology consists of two vertices: source -> sink. > They are connected with all-to-all edges. The result partition type > ({{PIPELINED}} and {{BLOCKING}}) should be considered separately. -- This message was sent by Atlassian Jira (v8.3.4#803005)