[jira] [Commented] (SPARK-12803) Consider adding ability to profile specific instances of executors in spark
[ https://issues.apache.org/jira/browse/SPARK-12803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097933#comment-15097933 ] Sean Owen commented on SPARK-12803: --- You wouldn't know which task executes on which executor anyway ahead of time, so being able to pick one executor doesn't do a lot. You would connect a profile to any executor and watch it execute tasks, and presumably one is about as good as the next for that purpose. > Consider adding ability to profile specific instances of executors in spark > --- > > Key: SPARK-12803 > URL: https://issues.apache.org/jira/browse/SPARK-12803 > Project: Spark > Issue Type: Improvement > Components: Java API >Reporter: Rajesh Balamohan > > It would be useful to profile specific instances of executors as opposed to > adding profiler details to all executors via > "spark.executor.extraJavaOptions". > Setting the number of executors to just 1 and profiling wouldn't be much > useful (in some cases, most of the time with single executor mode would be > spent in terms of reading data from remote node). At the same time, setting > profiling option to all executors could just create too many number of > snapshots; making it harder to analyze. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12803) Consider adding ability to profile specific instances of executors in spark
[ https://issues.apache.org/jira/browse/SPARK-12803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097836#comment-15097836 ] Rajesh Balamohan commented on SPARK-12803: -- Letting the profiler agent run on all executors and connecting to it works out for very long running jobs. If trying to profile a task in a job (e.g 30 seconds job), it would take sometime to locate the node and start profiling. By that time job itself would be over. Another option is to have offline profiling enabled (wherein it dumps the entire profiler snapshots on exit, or on periodic basis). But that would also generate too many snapshots. Having the option mentioned in the ticket would help in enabling profiling on specific places on need basis. > Consider adding ability to profile specific instances of executors in spark > --- > > Key: SPARK-12803 > URL: https://issues.apache.org/jira/browse/SPARK-12803 > Project: Spark > Issue Type: Improvement > Components: Java API >Reporter: Rajesh Balamohan > > It would be useful to profile specific instances of executors as opposed to > adding profiler details to all executors via > "spark.executor.extraJavaOptions". > Setting the number of executors to just 1 and profiling wouldn't be much > useful (in some cases, most of the time with single executor mode would be > spent in terms of reading data from remote node). At the same time, setting > profiling option to all executors could just create too many number of > snapshots; making it harder to analyze. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12803) Consider adding ability to profile specific instances of executors in spark
[ https://issues.apache.org/jira/browse/SPARK-12803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097097#comment-15097097 ] Sean Owen commented on SPARK-12803: --- What does it hurt though? enabling a profiling agent in the JVM doesn't do anything if nothing connects. > Consider adding ability to profile specific instances of executors in spark > --- > > Key: SPARK-12803 > URL: https://issues.apache.org/jira/browse/SPARK-12803 > Project: Spark > Issue Type: Improvement > Components: Java API >Reporter: Rajesh Balamohan > > It would be useful to profile specific instances of executors as opposed to > adding profiler details to all executors via > "spark.executor.extraJavaOptions". > Setting the number of executors to just 1 and profiling wouldn't be much > useful (in some cases, most of the time with single executor mode would be > spent in terms of reading data from remote node). At the same time, setting > profiling option to all executors could just create too many number of > snapshots; making it harder to analyze. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12803) Consider adding ability to profile specific instances of executors in spark
[ https://issues.apache.org/jira/browse/SPARK-12803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097091#comment-15097091 ] Rajesh Balamohan commented on SPARK-12803: -- It is for connecting to profiler. Adding profiler options with "spark.executor.extraJavaOptions" ends up adding profiler options on all executors. However, that might not be useful if the cluster has 100 executors. There are scenarios where one wants to profile 1 or 2 executors in the cluster. Ideal would be to provide an option to enable profiling only on specific tasks in different stages (e.g enable profiling on task 10 in stage 5 which is performing badly. One need not enable profiling on all executors to do this). I am not sure if this can be supported at this time. > Consider adding ability to profile specific instances of executors in spark > --- > > Key: SPARK-12803 > URL: https://issues.apache.org/jira/browse/SPARK-12803 > Project: Spark > Issue Type: Improvement > Components: Java API >Reporter: Rajesh Balamohan > > It would be useful to profile specific instances of executors as opposed to > adding profiler details to all executors via > "spark.executor.extraJavaOptions". > Setting the number of executors to just 1 and profiling wouldn't be much > useful (in some cases, most of the time with single executor mode would be > spent in terms of reading data from remote node). At the same time, setting > profiling option to all executors could just create too many number of > snapshots; making it harder to analyze. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12803) Consider adding ability to profile specific instances of executors in spark
[ https://issues.apache.org/jira/browse/SPARK-12803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096058#comment-15096058 ] Sean Owen commented on SPARK-12803: --- What do you mean here by profiling -- generating heap dumps or hprof output? or connecting with a profiler? I don't see a problem with configuring all executors to allow profiling at the time you need to do it. > Consider adding ability to profile specific instances of executors in spark > --- > > Key: SPARK-12803 > URL: https://issues.apache.org/jira/browse/SPARK-12803 > Project: Spark > Issue Type: Improvement > Components: Java API >Reporter: Rajesh Balamohan > > It would be useful to profile specific instances of executors as opposed to > adding profiler details to all executors via > "spark.executor.extraJavaOptions". > Setting the number of executors to just 1 and profiling wouldn't be much > useful (in some cases, most of the time with single executor mode would be > spent in terms of reading data from remote node). At the same time, setting > profiling option to all executors could just create too many number of > snapshots; making it harder to analyze. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
