On my ARM64 platform, I observed that certain tracing module initializations run for up to 200ms???for example, init_kprobe_trace(). Analysis reveals the root cause: the execution flow eval_map_work_func() ???trace_event_update_with_eval_map()???trace_event_update_all() is highly time-consuming. Although this flow is placed in eval_map_wq for asynchronous execution, it holds the trace_event_sem lock, causing other modules to be blocked either directly or indirectly. Also in init_blk_tracer(), this functions require trace_event_sem device_initcall.
To resolve this issue, I rename `eval_map_wq` and make it global and moved init_blk_tracer that are related to this lock to run asynchronously on this workqueue. Also check for kprobe_event= grub parameter; if not provided, init_kprobe_trace() returns directly. After optimization, boot time is reduced by approximately 200ms. Based on my analysis and testing, I've identified that only these two locations significantly impact timing. Other initcall_* functions do not exhibit relevant lock contention. A brief summary of the test results is as follows: Before this PATCHS: [ 0.224933] calling init_kprobe_trace+0x0/0xe0 @ 1 [ 0.455016] initcall init_kprobe_trace+0x0/0xe0 returned 0 after 230080 usecs Only opt setup_boot_kprobe_events() can see: [ 0.258609] calling init_blk_tracer+0x0/0x68 @ 1 [ 0.454991] initcall init_blk_tracer+0x0/0x68 returned 0 after 196377 usecs After this PATCHS: [ 0.224940] calling init_kprobe_trace+0x0/0xe0 @ 1 [ 0.224946] initcall init_kprobe_trace+0x0/0xe0 returned 0 after 3 usecs skip -------- [ 0.264835] calling init_blk_tracer+0x0/0x68 @ 1 [ 0.264841] initcall init_blk_tracer+0x0/0x68 returned 0 after 2 usecs --- Changes in v2: - Rename eval_map_wq to trace_init_wq. Changes in v3: - Opt PATCH 1/3 commit Changes in v4: - add trace_async_init boot parameter in patch2 - add init_kprobe_trace's skip logic in patch3 - add Suggested-by tag - Other synchronous optimizations related to trace_async_init https://lore.kernel.org/all/[email protected]/ Changes in v5: - remove trace_async_init boot parameter (patch2 v4) - remove make Make setup_boot_kprobe_events() asynchronous (patch4 v4) - Adjusted the patch sequence. Yaxiong Tian (3): tracing: Rename `eval_map_wq` and allow other parts of tracing use it blktrace: Make init_blk_tracer() asynchronous tracing/kprobes: Skip setup_boot_kprobe_events() when no cmdline event kernel/trace/blktrace.c | 23 ++++++++++++++++++++++- kernel/trace/trace.c | 18 +++++++++--------- kernel/trace/trace.h | 1 + kernel/trace/trace_kprobe.c | 4 ++++ 4 files changed, 36 insertions(+), 10 deletions(-) -- 2.25.1
