[jira] [Commented] (YARN-5401) yarn application kill does not let mapreduce jobs show up in jobhistory
[ https://issues.apache.org/jira/browse/YARN-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384529#comment-15384529 ] Jason Lowe commented on YARN-5401: -- Yes, if an application framework provides a kill command then that should be preferred over the yarn kill approach. The MapReduce framework kill will automatically fallback to the yarn kill if the application master is unresponsive or if the job fails to enter the killed state within a configurable amount of time (controlled via yarn.app.mapreduce.am.hard-kill-timeout-ms). > yarn application kill does not let mapreduce jobs show up in jobhistory > --- > > Key: YARN-5401 > URL: https://issues.apache.org/jira/browse/YARN-5401 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: centos 6.6 > apache hadoop 2.6.4 >Reporter: Nikhil Mulley > > Hi, > Its been found in our cluster running apache hadoop 2.6.4, that while the > mapreduce jobs that are killed with 'hadoop job -kill' command do end up have > the job and its counters to jobhistory server but when 'yarn application > -kill' is used on mapreduce application, job does not show up in jobhistory > server interface. > Is this intentional? If so, any particular reasons? > It would be better to have mapreduce application history reported on > jobhistory irrespective of whether kill is performed using yarn application > cli or hadoop job cli. > thanks, > Nikhil -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5401) yarn application kill does not let mapreduce jobs show up in jobhistory
[ https://issues.apache.org/jira/browse/YARN-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384516#comment-15384516 ] Nikhil Mulley commented on YARN-5401: - So, should apps always use app specific methods to kill their jobs but never use yarn kill unless really necessary. (like always use kill(TERM) unless kill -9 becomes necessary) > yarn application kill does not let mapreduce jobs show up in jobhistory > --- > > Key: YARN-5401 > URL: https://issues.apache.org/jira/browse/YARN-5401 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: centos 6.6 > apache hadoop 2.6.4 >Reporter: Nikhil Mulley > > Hi, > Its been found in our cluster running apache hadoop 2.6.4, that while the > mapreduce jobs that are killed with 'hadoop job -kill' command do end up have > the job and its counters to jobhistory server but when 'yarn application > -kill' is used on mapreduce application, job does not show up in jobhistory > server interface. > Is this intentional? If so, any particular reasons? > It would be better to have mapreduce application history reported on > jobhistory irrespective of whether kill is performed using yarn application > cli or hadoop job cli. > thanks, > Nikhil -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5401) yarn application kill does not let mapreduce jobs show up in jobhistory
[ https://issues.apache.org/jira/browse/YARN-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384330#comment-15384330 ] Jason Lowe commented on YARN-5401: -- This is effectively a duplicate of YARN-2261. MapReduce history requires the MapReduce ApplicationMaster to generate the history when it completes. hadoop job -kill or mapred job -kill accomplishes the kill by having the client connect to the MapReduce ApplicationMaster for the job and asks it to kill the job. Since this goes through the ApplicationMaster it allows the history to be generated properly. When the kill is done via YARN then the ApplicationMaster is not involved. The ResourceManager kills the AM without the AM's knowledge. This is similar to kill vs. kill -9 (i.e.: SIGTERM vs SIGKILL) in POSIX. The former allows the application to perform cleanup tasks on the way down, while the latter mercilessly kills the process without any chance for cleanup. Since YARN does not allow the application to specify a cleanup task to be performed when the app dies the MapReduce framework doesn't get a chance to finish generating the history for the job. > yarn application kill does not let mapreduce jobs show up in jobhistory > --- > > Key: YARN-5401 > URL: https://issues.apache.org/jira/browse/YARN-5401 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: centos 6.6 > apache hadoop 2.6.4 >Reporter: Nikhil Mulley > > Hi, > Its been found in our cluster running apache hadoop 2.6.4, that while the > mapreduce jobs that are killed with 'hadoop job -kill' command do end up have > the job and its counters to jobhistory server but when 'yarn application > -kill' is used on mapreduce application, job does not show up in jobhistory > server interface. > Is this intentional? If so, any particular reasons? > It would be better to have mapreduce application history reported on > jobhistory irrespective of whether kill is performed using yarn application > cli or hadoop job cli. > thanks, > Nikhil -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
