This is an automated email from the ASF dual-hosted git repository.
feiwang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 3ee3a2622 [CELEBORN-2046] Specify extractionDir of AsyncProfilerLoader
with celeborn.worker.jvmProfiler.localDir
3ee3a2622 is described below
commit 3ee3a26220f3e5364c0a8bef5799b227f98fe8f9
Author: SteNicholas <[email protected]>
AuthorDate: Wed Jun 25 10:15:38 2025 -0700
[CELEBORN-2046] Specify extractionDir of AsyncProfilerLoader with
celeborn.worker.jvmProfiler.localDir
### What changes were proposed in this pull request?
Specify `extractionDir` of `AsyncProfilerLoader` with
`celeborn.worker.jvmProfiler.localDir`.
### Why are the changes needed?
`AsyncProfilerLoader` uses `user.home` directory to store the extracted
libraries by default . When `user.home` directory is not initialized, it will
cause `AsyncProfilerLoader#load` to fail. `extractionDir` of
`AsyncProfilerLoader` could be specified with
`celeborn.worker.jvmProfiler.localDir` to avoid failure of loading.
Backport https://github.com/apache/spark/pull/51229.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test.
Closes #3345 from SteNicholas/CELEBORN-2046.
Lead-authored-by: SteNicholas <[email protected]>
Co-authored-by: 子懿 <[email protected]>
Signed-off-by: Wang, Fei <[email protected]>
---
.../celeborn/service/deploy/worker/profiler/JVMProfiler.scala | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git
a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/profiler/JVMProfiler.scala
b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/profiler/JVMProfiler.scala
index 3aa275388..41c9c7ea1 100644
---
a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/profiler/JVMProfiler.scala
+++
b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/profiler/JVMProfiler.scala
@@ -23,13 +23,14 @@ import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
import org.apache.celeborn.common.CelebornConf
import org.apache.celeborn.common.internal.Logging
+import org.apache.celeborn.common.util.Utils
/**
* The JVM profiler provides code profiling of worker based on the the async
profiler, a low overhead sampling profiler for Java.
* This allows a worker to capture CPU and memory profiles for worker which
can later be analyzed for performance issues.
* The profiler captures Java Flight Recorder (jfr) files for each worker read
by tools including Java Mission Control and Intellij.
*
- * <p> Note: The profiler writes the jfr files to the worker's working
directory in the worker's local file system and the files can grow to be large
so it is advisable
+ * <p>Note: The profiler writes the jfr files to the worker's working
directory in the worker's local file system and the files can grow to be large
so it is advisable
* that the worker machines have adequate storage.
*
* <p>Note: code copied from Apache Spark.
@@ -46,9 +47,15 @@ class JVMProfiler(conf: CelebornConf) extends Logging {
private val startcmd =
s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
private val stopcmd =
s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private lazy val extractionDir = Utils.createTempDir(profilerLocalDir,
"profiler").toPath
+
val profiler: Option[AsyncProfiler] = {
Option(
- if (enableProfiler && AsyncProfilerLoader.isSupported)
AsyncProfilerLoader.load() else null)
+ if (enableProfiler && AsyncProfilerLoader.isSupported) {
+ logInfo(s"Profiler extraction directory: ${extractionDir.toString}.")
+ AsyncProfilerLoader.setExtractionDirectory(extractionDir)
+ AsyncProfilerLoader.load()
+ } else { null })
}
def start(): Unit = {
