[
https://issues.apache.org/jira/browse/SPARK-12803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097836#comment-15097836
]
Rajesh Balamohan commented on SPARK-12803:
------------------------------------------
Letting the profiler agent run on all executors and connecting to it works out
for very long running jobs. If trying to profile a task in a job (e.g 30
seconds job), it would take sometime to locate the node and start profiling. By
that time job itself would be over. Another option is to have offline profiling
enabled (wherein it dumps the entire profiler snapshots on exit, or on periodic
basis). But that would also generate too many snapshots. Having the option
mentioned in the ticket would help in enabling profiling on specific places on
need basis.
> Consider adding ability to profile specific instances of executors in spark
> ---------------------------------------------------------------------------
>
> Key: SPARK-12803
> URL: https://issues.apache.org/jira/browse/SPARK-12803
> Project: Spark
> Issue Type: Improvement
> Components: Java API
> Reporter: Rajesh Balamohan
>
> It would be useful to profile specific instances of executors as opposed to
> adding profiler details to all executors via
> "spark.executor.extraJavaOptions".
> Setting the number of executors to just 1 and profiling wouldn't be much
> useful (in some cases, most of the time with single executor mode would be
> spent in terms of reading data from remote node). At the same time, setting
> profiling option to all executors could just create too many number of
> snapshots; making it harder to analyze.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
