Hi Jing, I will answer current questions.
> 1. will it replace the current flame graph, i.e. the current flame graph will be deprecated and removed? Although I think the new java profiler introduced in FLIP-375 is more powerful, just as Rui has replied, I don't think it could replace current flame graph totally. > 2.does it make sense to provide the performance difference between enable and disable it? The new java profiler would not introduce any performance impact after we enable it, it will only start work when we trigger the profiling. And from our experiences, the overhead of profiling is extremely light. For Rui's question: > Are all process-level flamegraphs stored at BlobStore? Are they maintained by > JobManager after sampling? Is there cleanup strategy? Or max-save-count > strategy? Yes, we use blobstore to store the process-level flamegraph-files and maintained on taskmanager side. They flamegraph-files will be cleanup automatically once reached to rest.profiling.history-size. Best Yun Tang ________________________________ From: Rui Fan <1996fan...@gmail.com> Sent: Tuesday, October 10, 2023 10:10 To: dev@flink.apache.org <dev@flink.apache.org> Subject: Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers Hi Jing, > 1. will it replace the current flame graph, i.e. the current flame graph will be deprecated and removed? I think the current flame graph cannot be removed. As a core contributor to the current flame graph, and I use it almost every week. I would like to clarify the difference between the current flame graph and the flame graph proposed by FLIP-375. @The current flame graph The current flame graph is the operator level or task level, when one operator is the bottleneck of current job. We can see the current flamegraph to check what the operator is doing. It includes three types: On-CPU, Off-CPU and Mixed-Type. The Mixed-Type is very useful, it can detect why operator is slow even if the operator doesn't use CPU. For example, the operator is blocked on querying hbase. It just support the task thread, it means it cannot detect the cpu usage of other threads, such as: RocksDB Flush or compaction. This's the limitation of current flamegraph. @The flame graph proposed by FLIP-375. The flamegraph proposed by FLIP-375 works on process level, such as JobManager or TaskManager, so it can monitor all threads. Such as: rocksdb background threads. When the CPU usage of one TM is high, and all tasks are not busy. The new flamegraph will be useful. Back to the question: It includes task or operator thread, why the current flamegraph is still needed? 1. The flamegraph of process level cannot easily distinguish tasks. Especially if there are multiple slots in a TM, and different subtasks of the same task running in multiple slots, their stacks are very similar. 2. The Mixed-Type of current flamegraph may not be replaced by the process-level flame graph. Please correct me if anything is wrong, thanks~ Hi Yu, > Jobmanager allows the user to download the results of the corresponding files on taskmanager with the blob service. Are all process-level flamegraphs stored at BlobStore? Are they maintained by JobManager after sampling? Is there cleanup strategy? Or max-save-count strategy? Best, Rui On Tue, Oct 10, 2023 at 1:24 AM Jing Ge <j...@ververica.com.invalid> wrote: > Hi Yu, Hi Yun, > > Brilliant idea! People are keen to use it. Thanks for your proposal! I was > wondering: > > 1. will it replace the current flame graph, i.e. the current flame graph > will be deprecated and removed? > 2. does it make sense to provide the performance difference between enable > and disable it? > > Best regards, > Jing > > On Mon, Oct 9, 2023 at 1:50 PM Yu Chen <yuchen.e...@gmail.com> wrote: > > > Hi zhanghao, > > > > Yes, agree with you. We'll take Jobmanager into consideration and update > > the FLIP later! > > > > Best, > > Yu Chen > > > > Zhanghao Chen <zhanghao.c...@outlook.com> 于2023年10月9日周一 19:22写道： > > > > > Hi Yun and Yu, > > > > > > Thanks for driving this. This would definitely help users identify > > > performance bottlenecks, especially for the cases where the bottleneck > > lies > > > in the system stack (e.g. GC), and big +1 for the downloadable > flamegraph > > > to ease sharing. I'm wondering if we could add this for the job manager > > as > > > well. In the OLAP scenario and sometimes in the streaming scenario > (when > > > there're some heavy operations during execution plan generation or in > > > operator coordinators), the JM can have bottleneck as well. > > > > > > Best, > > > Zhanghao Chen > > > ________________________________ > > > From: Yu Chen <yuchen.e...@gmail.com> > > > Sent: Monday, October 9, 2023 17:24 > > > To: dev@flink.apache.org <dev@flink.apache.org> > > > Subject: [DISCUSS] FLIP-375: Built-in cross-platform powerful java > > > profiler on taskmanagers > > > > > > Hi all, > > > > > > Yun Tang and I are opening this thread to discuss our proposal to > > integrate > > > async-profiler's capabilities for profiling taskmananger (e.g., > > generating > > > flame graphs) in the Flink Web [1]. > > > > > > > > > Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by > > > sampling task threads. The results generated in such way missing the > > > relevant stack of java threads and system calls. The async-profiler[2] > > is a > > > low-overhead sampling profiler for Java, but the steps to use it in the > > > production environment are cumbersome and suffer from permissions and > > > security risks. > > > > > > Therefore, we propose adding rest APIs to provide the capability to > > invoke > > > async-profiler on multiple platforms through JNI, which can be easily > > > operated on Web UI. This enhancement will improve the efficiency and > > > experience of Flink users in identifying performance bottlenecks. > > > > > > > > > > > > Please refer to the FLIP document for more details about the proposed > > > design > > > and implementation. We welcome any feedback and opinions on this > > proposal. > > > > > > > > > > > > [1] FLIP-375: Built-in cross-platform powerful java profiler on > > > taskmanagers - Apache Flink - Apache Software Foundation > > > < > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers > > > > > > > > > > [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP > > profiler > > > for Java featuring AsyncGetCallTrace + perf_events > > > <https://github.com/async-profiler/async-profiler> > > > > > > > > > > > > Best regards, > > > > > > Yun Tang and Yu Chen > > > > > >
