[ https://issues.apache.org/jira/browse/TEZ-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175078#comment-15175078 ]
Hitesh Shah edited comment on TEZ-3154 at 3/2/16 5:46 AM: ---------------------------------------------------------- bq. Creating this ticket to explore the possibility of adding thread-dump on periodic basis for specific tasks. This seems like a debugging only feature which needs to be pre-setup and would be a lot of noise if enabled by default. The current description seems to imply that the dag would need to be configured before it is executed unless there is a way to do this while a dag is running. What would be more useful is to trigger a thread dump when certain conditions are hit: - if there is a task timeout - this would trigger a thread dump before killing the container - if the user defines a policy that if certain counter thresholds are hit ( GC counters, etc ) then trigger it. - command-line tool to trigger thread dumps on an demand basis if the user wants to see what is going on within a task Also, please take a look at the support yarn is introducing or has introduced for sending a signal to a container - this would be useful to trigger a jstack dump if needed. was (Author: hitesh): bq. Creating this ticket to explore the possibility of adding thread-dump on periodic basis for specific tasks. This seems like a debugging only feature which needs to be pre-setup and would be a lot of noise if enabled by default. The current description seems to imply that the dag would need to be configured before it is executed unless there is a way to do this while a dag is running. What would be more useful is to trigger a thread dump when certain conditions are hit: - if there is a task timeout - this would trigger a thread dump before killing the container - if the user defines a policy that if certain counter thresholds are hit ( GC counters, etc ) - command-line tool to trigger thread dumps on an demand basis if the user wants to see what is going on within a task Also, please take a look at the support yarn is introducing or has introduced for sending a signal to a container - this would be useful to trigger a jstack dump if needed. > Debuggability : Add an option to take threaddump from a specific vertex/task > ---------------------------------------------------------------------------- > > Key: TEZ-3154 > URL: https://issues.apache.org/jira/browse/TEZ-3154 > Project: Apache Tez > Issue Type: Improvement > Reporter: Rajesh Balamohan > > tez.task-specific.launch.cmd-opts and tez.task-specific.launch.cmd-opts.list > (e.g "Map 1[10]", 10th task in map 1) options are available to add certain > parameters to task specific command line options. It has been useful for > launching profilers on specific tasks. > There are scenarios in which taking threaddumps on periodic basis on specific > tasks could be helpful. E.g > - In certain clusters it could be difficult to add profilers. > - There could be scenarios where the tasks are slow due apps using Tez (but > the counters might indicate no issues in Tez). (e.g Parsing using > SimpleDateFormat for every record could be time consuming) > - In certain clusters, access might not be there to take threaddumps of tasks > from NM. YARN's threadstack (in RM UI) is mainly for NM and doesn't work on > task level. > Creating this ticket to explore the possibility of adding thread-dump on > periodic basis for specific tasks. > High level e.g: "--hiveconf tez.task-specific.launch.cmd-opts=" > -DthreadDumpInterval=5 " --hiveconf > tez.task-specific.launch.cmd-opts.list="Map 1[10,15]" - This should print > thread-dumps in tasks 10, 15 in Map-1 every 5 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)