[
https://issues.apache.org/jira/browse/SPARK-12803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097091#comment-15097091
]
Rajesh Balamohan commented on SPARK-12803:
------------------------------------------
It is for connecting to profiler. Adding profiler options with
"spark.executor.extraJavaOptions" ends up adding profiler options on all
executors. However, that might not be useful if the cluster has 100 executors.
There are scenarios where one wants to profile 1 or 2 executors in the cluster.
Ideal would be to provide an option to enable profiling only on specific tasks
in different stages (e.g enable profiling on task 10 in stage 5 which is
performing badly. One need not enable profiling on all executors to do this). I
am not sure if this can be supported at this time.
> Consider adding ability to profile specific instances of executors in spark
> ---------------------------------------------------------------------------
>
> Key: SPARK-12803
> URL: https://issues.apache.org/jira/browse/SPARK-12803
> Project: Spark
> Issue Type: Improvement
> Components: Java API
> Reporter: Rajesh Balamohan
>
> It would be useful to profile specific instances of executors as opposed to
> adding profiler details to all executors via
> "spark.executor.extraJavaOptions".
> Setting the number of executors to just 1 and profiling wouldn't be much
> useful (in some cases, most of the time with single executor mode would be
> spent in terms of reading data from remote node). At the same time, setting
> profiling option to all executors could just create too many number of
> snapshots; making it harder to analyze.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
