[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208477#comment-13208477
]
Hudson commented on MAPREDUCE-3802:
---
Integrated in Hadoop-Mapreduce-trunk #991 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/991/])
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
still can recover successfully after MAPREDUCE-3846. (vinodkv) (Revision
1244178)
Result = SUCCESS
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244178
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt,
> MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208457#comment-13208457
]
Hudson commented on MAPREDUCE-3802:
---
Integrated in Hadoop-Mapreduce-0.23-Build #197 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/197/])
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
still can recover successfully after MAPREDUCE-3846. (vinodkv)
svn merge --ignore-ancestry -c 1244178 ../../trunk/ (Revision 1244180)
Result = FAILURE
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244180
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt,
> MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208433#comment-13208433
]
Hudson commented on MAPREDUCE-3802:
---
Integrated in Hadoop-Hdfs-0.23-Build #169 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/169/])
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
still can recover successfully after MAPREDUCE-3846. (vinodkv)
svn merge --ignore-ancestry -c 1244178 ../../trunk/ (Revision 1244180)
Result = FAILURE
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244180
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt,
> MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208427#comment-13208427
]
Hudson commented on MAPREDUCE-3802:
---
Integrated in Hadoop-Hdfs-trunk #956 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/956/])
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
still can recover successfully after MAPREDUCE-3846. (vinodkv) (Revision
1244178)
Result = FAILURE
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244178
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt,
> MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207992#comment-13207992
]
Hudson commented on MAPREDUCE-3802:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #1737 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1737/])
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
still can recover successfully after MAPREDUCE-3846. (vinodkv) (Revision
1244178)
Result = ABORTED
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244178
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt,
> MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207989#comment-13207989
]
Hudson commented on MAPREDUCE-3802:
---
Integrated in Hadoop-Mapreduce-0.23-Commit #553 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/553/])
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
still can recover successfully after MAPREDUCE-3846. (vinodkv)
svn merge --ignore-ancestry -c 1244178 ../../trunk/ (Revision 1244180)
Result = ABORTED
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244180
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt,
> MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207974#comment-13207974
]
Hudson commented on MAPREDUCE-3802:
---
Integrated in Hadoop-Common-trunk-Commit #1726 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1726/])
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
still can recover successfully after MAPREDUCE-3846. (vinodkv) (Revision
1244178)
Result = SUCCESS
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244178
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt,
> MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207966#comment-13207966
]
Hudson commented on MAPREDUCE-3802:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1800 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1800/])
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
still can recover successfully after MAPREDUCE-3846. (vinodkv) (Revision
1244178)
Result = SUCCESS
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244178
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt,
> MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207959#comment-13207959
]
Hudson commented on MAPREDUCE-3802:
---
Integrated in Hadoop-Common-0.23-Commit #549 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/549/])
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
still can recover successfully after MAPREDUCE-3846. (vinodkv)
svn merge --ignore-ancestry -c 1244178 ../../trunk/ (Revision 1244180)
Result = SUCCESS
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244180
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt,
> MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207954#comment-13207954
]
Hudson commented on MAPREDUCE-3802:
---
Integrated in Hadoop-Hdfs-0.23-Commit #537 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/537/])
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
still can recover successfully after MAPREDUCE-3846. (vinodkv)
svn merge --ignore-ancestry -c 1244178 ../../trunk/ (Revision 1244180)
Result = SUCCESS
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244180
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt,
> MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207811#comment-13207811
]
Robert Joseph Evans commented on MAPREDUCE-3802:
+1 you are correct I manually verified that 0.23.2 does not show this problem
any more, and so I assume that it is MAPREDUCE-3846 that fixed it.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt,
> MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207431#comment-13207431
]
Hadoop QA commented on MAPREDUCE-3802:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12514424/MAPREDUCE-3802-20120213.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1849//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1849//console
This message is automatically generated.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt,
> MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by J
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207308#comment-13207308
]
Hadoop QA commented on MAPREDUCE-3802:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12514406/MAPREDUCE-3802-20120213.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapreduce.v2.app.TestRecovery
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1848//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1848//console
This message is automatically generated.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3802-20120213.txt, syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This mes
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205845#comment-13205845
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3802:
MAPREDUCE-3846 is a related ticket with same symptom but a different cause.
I uploaded a patch there, where instead of generating AttemptIDs sequentially
when recovering, I am using the previous generation attemptIDs first before
moving onto generating new ones for this generation. I thought for a while and
now it seems to me that that patch will automatically fix this issue also,
except I don't have a test validating this.
Robert/Sharad, can you please look at my patch there and see if that fixes
this? Thanks.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Critical
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203332#comment-13203332
]
Sharad Agarwal commented on MAPREDUCE-3802:
---
The bug is in TaskImpl over here:
{code}
//attempt ids are generated based on MR app startCount so that attempts
//from previous lives don't overstep the current one.
//this assumes that a task won't have more than 1000 attempts in its single
//life
nextAttemptNumber = (startCount - 1) * 1000;
{code}
The completed task could be from any earlier AM generation not just from the
previous one. I am looking into the way to fix this.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Critical
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203325#comment-13203325
]
Sharad Agarwal commented on MAPREDUCE-3802:
---
bq. need to understand a little bit better how these names are determined
The task attemptIds are unique across all the generations of AM. This is to
avoid any remote task attempt from previous generation of AM joining the
current AM. The assumption is there won't be more than 1000 attempts of a task
in AM run. The suffix part of task attemptId is determined as follows:
_(AMGeneration-1)*1000. For first AM it will start from 0. For second it will
start from 1000, for third from 2000 ..
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Critical
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202585#comment-13202585
]
Robert Joseph Evans commented on MAPREDUCE-3802:
OK I found the issue, sort of, and it has nothing to do with order. The issue
is with the name of the task attempt. If the task was completed by the first
AM and recovered by the second AM the name of the task attempt in the jhist
file will look like attempt_1328637230353_0001_m_00_0, but the Recovery
Service is trying to recover a task with attempt id
attempt_1328637230353_0001_m_00_1000, which appears to be the format for
attempts that completed successfully with the second AM.
I need to understand a little bit better how these names are determined, and
where they are set, so I can determine how to fix the issue. I don't see how
this could only be a problem for a single node cluster.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Critical
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201414#comment-13201414
]
Robert Joseph Evans commented on MAPREDUCE-3802:
I have been looking for differences in the jhist files between the initial
kill, and the second one, and I cannot find much as far as the content is
concerned. So I think it might have something to do with the order of the
events.
In the second jhist file there are two AMStarted events. I assume that is to
allow for the history server to show how many have been started.
The mapFinishTime of all the new MAP_ATTEMPT_FINISH_EVENTS is 0. The finishTime
for these events are all showing the same finish time which I think is when the
attempt was recovered, not the original event finish time. The state of the
event changed from "map" (which seems like a bug) to "SUCCEEDED" which looks
more correct to me. And finally the clockSplits counters are all 0 in the new
one as well.
Similarly the TASK_FINISH_EVENTS have the finish time of the recovery, not the
actual finish time.
I could not find anything else that is significantly different.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Priority: Critical
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201329#comment-13201329
]
Robert Joseph Evans commented on MAPREDUCE-3802:
That appears to be the case. We are getting an NPE which is caused by calling
RecoverService.getTaskAttemptInfo() and getting a null back.
RecoveryService.getTaskAttemptInfo() first gets a task info, and then gets a
task attempt info from inside that task. It looks like the task info is parsed
and populated just fine, but the task attempt info is not. That seems to be
caused by no TaskAttemptStarted events being put in the history log at all
during the recovery process. This also seems like no MapAttemptFinishedEvents,
ReduceAttemp0tFinishedEvents, TaskAttemptFailedEvents nor
TaskAttemptFinishedEvents are in the log either, or we would get null pointer
exceptions while parsing them too.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Priority: Critical
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200302#comment-13200302
]
Mahadev konar commented on MAPREDUCE-3802:
--
@Bobby,
So this is an issue with the events being logged in the job history file?
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Priority: Critical
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200140#comment-13200140
]
Robert Joseph Evans commented on MAPREDUCE-3802:
Sorry I traced that wrong. It actually looks like the Task does not have an
attempt info for that task attempt.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Priority: Critical
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200097#comment-13200097
]
Robert Joseph Evans commented on MAPREDUCE-3802:
I just found the following in the log
{noformat}
2012-02-03 21:08:24,401 FATAL [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread.
Exiting..
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$InterceptingEventHandler.sendAssignedEvent(RecoveryService.java:437)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$InterceptingEventHandler.handle(RecoveryService.java:336)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$RequestContainerTransition.transition(TaskAttemptImpl.java:1082)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$RequestContainerTransition.transition(TaskAttemptImpl.java:1)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:919)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:871)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.realDispatch(RecoveryService.java:291)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.dispatch(RecoveryService.java:287)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
at java.lang.Thread.run(Thread.java:662)
{noformat}
It looks like for some reason someone is sending a CONTAINER_REQ event without
an attempt ID.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% redu
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200061#comment-13200061
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3802:
This is inline with I always used to see while working on sub-tickets at
MAPREDUCE-2692 but on my localhost. Contrary to this, Karam never saw this on a
cluster. So hopefully and before any further testing reveals otherwise, I treat
this as a non-blocker.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200060#comment-13200060
]
Robert Joseph Evans commented on MAPREDUCE-3802:
>From talking with others it looks like this may only happen on a single node
>cluster as well. The fact that the mapper % dropped a lot seems a bit odd,
>almost like it lost some mapper tasks as part of recovery.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
