[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539744#comment-14539744
]
Hudson commented on YARN-3493:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #194 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/194/])
Move YARN-3493 in CHANGES.txt from 2.8 to 2.7.1 (wangda: rev
3d28611cc6850de129b831158c420f9487103213)
* hadoop-yarn-project/CHANGES.txt
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0, 2.7.1
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539713#comment-14539713
]
Hudson commented on YARN-3493:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #925 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/925/])
Move YARN-3493 in CHANGES.txt from 2.8 to 2.7.1 (wangda: rev
3d28611cc6850de129b831158c420f9487103213)
* hadoop-yarn-project/CHANGES.txt
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0, 2.7.1
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539900#comment-14539900
]
Hudson commented on YARN-3493:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2141 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2141/])
Move YARN-3493 in CHANGES.txt from 2.8 to 2.7.1 (wangda: rev
3d28611cc6850de129b831158c420f9487103213)
* hadoop-yarn-project/CHANGES.txt
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0, 2.7.1
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539931#comment-14539931
]
Hudson commented on YARN-3493:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2123 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2123/])
Move YARN-3493 in CHANGES.txt from 2.8 to 2.7.1 (wangda: rev
3d28611cc6850de129b831158c420f9487103213)
* hadoop-yarn-project/CHANGES.txt
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0, 2.7.1
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14540005#comment-14540005
]
Hudson commented on YARN-3493:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #193 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/193/])
Move YARN-3493 in CHANGES.txt from 2.8 to 2.7.1 (wangda: rev
3d28611cc6850de129b831158c420f9487103213)
* hadoop-yarn-project/CHANGES.txt
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0, 2.7.1
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539953#comment-14539953
]
Hudson commented on YARN-3493:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #183 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/183/])
Move YARN-3493 in CHANGES.txt from 2.8 to 2.7.1 (wangda: rev
3d28611cc6850de129b831158c420f9487103213)
* hadoop-yarn-project/CHANGES.txt
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0, 2.7.1
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539057#comment-14539057
]
Hudson commented on YARN-3493:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7801 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7801/])
Move YARN-3493 in CHANGES.txt from 2.8 to 2.7.1 (wangda: rev
3d28611cc6850de129b831158c420f9487103213)
* hadoop-yarn-project/CHANGES.txt
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0, 2.7.1
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14501296#comment-14501296
]
Hudson commented on YARN-3493:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #901 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/901/])
YARN-3493. RM fails to come up with error Failed to load/recover state when
mem settings are changed. (Jian He via wangda) (wangda: rev
f65eeb412d140a3808bcf99344a9f3a965918f70)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* hadoop-yarn-project/CHANGES.txt
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14501288#comment-14501288
]
Hudson commented on YARN-3493:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #158 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/158/])
YARN-3493. RM fails to come up with error Failed to load/recover state when
mem settings are changed. (Jian He via wangda) (wangda: rev
f65eeb412d140a3808bcf99344a9f3a965918f70)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14501282#comment-14501282
]
Hudson commented on YARN-3493:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2099 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2099/])
YARN-3493. RM fails to come up with error Failed to load/recover state when
mem settings are changed. (Jian He via wangda) (wangda: rev
f65eeb412d140a3808bcf99344a9f3a965918f70)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14501417#comment-14501417
]
Hudson commented on YARN-3493:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2117 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2117/])
YARN-3493. RM fails to come up with error Failed to load/recover state when
mem settings are changed. (Jian He via wangda) (wangda: rev
f65eeb412d140a3808bcf99344a9f3a965918f70)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14501239#comment-14501239
]
Hudson commented on YARN-3493:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #167 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/167/])
YARN-3493. RM fails to come up with error Failed to load/recover state when
mem settings are changed. (Jian He via wangda) (wangda: rev
f65eeb412d140a3808bcf99344a9f3a965918f70)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14501402#comment-14501402
]
Hudson commented on YARN-3493:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #168 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/168/])
YARN-3493. RM fails to come up with error Failed to load/recover state when
mem settings are changed. (Jian He via wangda) (wangda: rev
f65eeb412d140a3808bcf99344a9f3a965918f70)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499288#comment-14499288
]
Hadoop QA commented on YARN-3493:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726049/YARN-3493.3.patch
against trunk revision bb6dde6.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7373//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7373//console
This message is automatically generated.
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500787#comment-14500787
]
Wangda Tan commented on YARN-3493:
--
Some comments:
- validateResourceRequest - normalizeAndValidateRequest and make the
isRecovery to be false, and it should be package-visible
- normalizeNodeLabelForRequest - normalizeNodeLabelExpressionInRequest, and it
should be private
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500940#comment-14500940
]
Jian He commented on YARN-3493:
---
TestAMRestart failure unrelated, passing locally.
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500946#comment-14500946
]
Hudson commented on YARN-3493:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #7609 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7609/])
YARN-3493. RM fails to come up with error Failed to load/recover state when
mem settings are changed. (Jian He via wangda) (wangda: rev
f65eeb412d140a3808bcf99344a9f3a965918f70)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* hadoop-yarn-project/CHANGES.txt
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Fix For: 2.8.0
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
YARN-3493.4.patch, YARN-3493.5.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498754#comment-14498754
]
Hadoop QA commented on YARN-3493:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725743/YARN-3493.2.patch
against trunk revision 80a2a12.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7365//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7365//console
This message is automatically generated.
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Attachments: YARN-3493.1.patch, YARN-3493.2.patch,
yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499142#comment-14499142
]
Jian He commented on YARN-3493:
---
bq. This causes throw InvalidResourceRequestException to AM which AM do not
expect it.
[~rohithsharma], thanks for your review !
I agree. I think this can also happen in general if the men settings are
changed while AM is running.
Upload a new patch to fix the test failures. The failure is due to the the node
label expression in the AM resource request is not normalized on recovery which
causes NPE.
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Attachments: YARN-3493.1.patch, YARN-3493.2.patch,
yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499200#comment-14499200
]
Hadoop QA commented on YARN-3493:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726049/YARN-3493.3.patch
against trunk revision 9595cc0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7371//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7371//console
This message is automatically generated.
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Attachments: YARN-3493.1.patch, YARN-3493.2.patch, YARN-3493.3.patch,
yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497129#comment-14497129
]
Jian He commented on YARN-3493:
---
[~kasha], I think this happened on a different code path.
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Attachments: yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497122#comment-14497122
]
Karthik Kambatla commented on YARN-3493:
[~jianhe] - YARN-2010 should have fixed this right?
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Priority: Critical
Fix For: 2.7.0
Attachments: yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497317#comment-14497317
]
Jian He commented on YARN-3493:
---
cancel the patch, uploading a newer version.
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Attachments: YARN-3493.1.patch, yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497378#comment-14497378
]
Rohith commented on YARN-3493:
--
+1(non-binding)
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Attachments: YARN-3493.1.patch, YARN-3493.2.patch,
yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497413#comment-14497413
]
Rohith commented on YARN-3493:
--
The same problem would occur enabling RM work preserving restart where Running
AM updates its ResourceRequest on RESYNC command from RM. This causes throw
InvalidResourceRequestException to AM which AM do not expect it.
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Attachments: YARN-3493.1.patch, YARN-3493.2.patch,
yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497439#comment-14497439
]
Hadoop QA commented on YARN-3493:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725743/YARN-3493.2.patch
against trunk revision 1b89a3e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7352//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7352//console
This message is automatically generated.
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Attachments: YARN-3493.1.patch, YARN-3493.2.patch,
yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
[jira] [Commented] (YARN-3493) RM fails to come up with error Failed to load/recover state when mem settings are changed
[
https://issues.apache.org/jira/browse/YARN-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497416#comment-14497416
]
Rohith commented on YARN-3493:
--
bq. The same problem would occur
Just to clarify I am referring to InvalidResourceRequestException, not the RM
start failure
RM fails to come up with error Failed to load/recover state when mem
settings are changed
Key: YARN-3493
URL: https://issues.apache.org/jira/browse/YARN-3493
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.0
Reporter: Sumana Sathish
Assignee: Jian He
Priority: Critical
Attachments: YARN-3493.1.patch, YARN-3493.2.patch,
yarn-yarn-resourcemanager.log.zip
RM fails to come up for the following case:
1. Change yarn.nodemanager.resource.memory-mb and
yarn.scheduler.maximum-allocation-mb to 4000 in yarn-site.xml
2. Start a randomtextwriter job with mapreduce.map.memory.mb=4000 in
background and wait for the job to reach running state
3. Restore yarn-site.xml to have yarn.scheduler.maximum-allocation-mb to 2048
before the above job completes
4. Restart RM
5. RM fails to come up with the below error
{code:title= RM error for Mem settings changed}
- RM app submission failed in validating AM resource request for application
application_1429094976272_0008
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:994)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1035)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1031)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1031)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1071)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
2015-04-15 13:19:18,623 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(579)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested memory 0, or requested memory max configured,
requestedMemory=3072, maxMemory=2048
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:328)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:422)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1187)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:574)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
