[
https://issues.apache.org/jira/browse/SPARK-52185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Roczei updated SPARK-52185:
---------------------------------
Target Version/s: 4.1.0
> Automate the thread dump collection for Spark applications
> ----------------------------------------------------------
>
> Key: SPARK-52185
> URL: https://issues.apache.org/jira/browse/SPARK-52185
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 4.1.0
> Reporter: Gabor Roczei
> Priority: Major
> Labels: pull-request-available
>
> When a Java program runs for a long time without giving you any
> feedback/output, how do you determine what the program might be doing and
> whether it’s stuck? Thread dumps can help in such case. It shows the status
> of the threads (if it's running/waiting/blocked) and which part of the code
> is being executed by each thread, being important to detect deadlocks and
> which part of the program is running.
> Obtain a series of thread dumps at regular intervals. Why? Getting a single
> thread dump only shows a snapshot of threads, getting several allows us to
> see if threads are progressing by comparing states.
> Collecting thread dump samples from slow Spark executors or drivers can be
> challenging, especially in YARN or Kubernetes environments.
> Actual solutions which are available for debugging:
> 1)
> We need to find out where the Java Virtual Machine (JVM) is running then run
> the jstack command manually.
> 2)
> Download the thread dumps from the Spark UI. For example:
> [http://localhost:4040/executors/threadDump/?executorId=driver]
> 3)
> Download the thread dumps via Spark API. For example:
> {code:java}
> curl
> "http://localhost:4040/api/v1/applications/local-1747400853731/executors/driver/threads"{code}
>
> The purpose of this feature request is to automate the thread dump collection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
