[jira] [Comment Edited] (YARN-10705) Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler
[
https://issues.apache.org/jira/browse/YARN-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309038#comment-17309038
]
Siddharth Ahuja edited comment on YARN-10705 at 3/25/21, 11:36 PM:
---
Added a patch to ensure that logging only happens in case of an actual
container assignment/allocation, not reservation.
Tested this on a single node cluster from generated distribution after
compilation of the patch on trunk using the below steps:
* Set {{yarn.resourcemanager.scheduler.class}} to
{{org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler}},
* Started YARN on the single node cluster, it has 1 NodeManager with 8GB to run
containers,
* Enabled DEBUG logging for the FSLeafQueue class to check for debug logs:
{code}
bin/yarn daemonlog -setlevel localhost:8088
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue DEBUG
{code}
* Check for the DEBUG allocation message in the RM logs :
{code}
tail -f rmlogs.log | grep "DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.somequeue"
{code}
* Ran the first job requiring 1 AM + 1 non-AM worth 2GB each, so 4GB out of 8GB
are used up:
{code}
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar
sleep -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.map.memory.mb=2048
-m 1 -mt 60
{code}
* Ran a second job requiring 1 AM + 1 non-AM worth 2GB and 4GB respectively.
2nd application starts i.e. AM starts but there is no room for the 4GB
container yet so reservation for the 4GB non-AM happens.
{code}
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar
sleep -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.map.memory.mb=4096
-m 1 -mt 60
{code}
* With the patch only following 3 lines are present when reservation occurs
which is expected after the patch is applied:
{code}
2021-03-25 17:54:13,475 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.somequeue container:
2021-03-25 17:54:20,507 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.somequeue container:
2021-03-25 17:54:35,558 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.somequeue container:
{code}
however, in the case of no patch, this was getting added before:
{code}
2021-03-25 17:54:43,589 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.somequeue container:
{code}
No JUnits required, as the change is about a "lack" of log, no change to
functionality, as such, re-running existing JUnits should suffice.
was (Author: sahuja):
Added a patch to ensure that logging only happens in case of an actual
container assignment/allocation, not reservation.
Tested this on a single node cluster from generated distribution after
compilation of the patch on trunk using the below steps:
* Set {{yarn.resourcemanager.scheduler.class}} to
{{org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler}},
* Started YARN on the single node cluster, it has 1 NodeManager with 8GB to run
containers,
* Enabled DEBUG logging for the FSLeafQueue class to check for debug logs:
{code}
bin/yarn daemonlog -setlevel localhost:8088
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue DEBUG
{code}
* Check for the DEBUG allocation message in the RM logs :
{code}
tail -f rmlogs.log | grep "DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.somequeue"
{code}
* Ran the first job requiring 1 AM + 1 non-AM worth 2GB each, so 4GB out of 8GB
are used up:
{code}
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar
sleep -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.map.memory.mb=2048
-m 1 -mt 60
{code}
* Ran a second job requiring 1 AM + 1 non-AM worth 2GB and 4GB respectively.
2nd application starts i.e. AM starts but there is no room for the 4GB
container yet so reservation for the 4GB non-AM happens.
{code}
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar
sleep -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.map.memory.mb=4096
-m 1 -mt 60
{code}
* With the patch only following 3 lines are present when reservation occurs
which is expected after the patch is applied:
{code}
2021-03-25 17:54:13,475 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.sidtheadmin container:
2021-03-25 17:54:20,507 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned
[jira] [Comment Edited] (YARN-10705) Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler
[
https://issues.apache.org/jira/browse/YARN-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309038#comment-17309038
]
Siddharth Ahuja edited comment on YARN-10705 at 3/25/21, 11:36 PM:
---
Added a patch to ensure that logging only happens in case of an actual
container assignment/allocation, not reservation.
Tested this on a single node cluster from generated distribution after
compilation of the patch on trunk using the below steps:
* Set {{yarn.resourcemanager.scheduler.class}} to
{{org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler}},
* Started YARN on the single node cluster, it has 1 NodeManager with 8GB to run
containers,
* Enabled DEBUG logging for the FSLeafQueue class to check for debug logs:
{code}
bin/yarn daemonlog -setlevel localhost:8088
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue DEBUG
{code}
* Check for the DEBUG allocation message in the RM logs :
{code}
tail -f rmlogs.log | grep "DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.somequeue"
{code}
* Ran the first job requiring 1 AM + 1 non-AM worth 2GB each, so 4GB out of 8GB
are used up:
{code}
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar
sleep -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.map.memory.mb=2048
-m 1 -mt 60
{code}
* Ran a second job requiring 1 AM + 1 non-AM worth 2GB and 4GB respectively.
2nd application starts i.e. AM starts but there is no room for the 4GB
container yet so reservation for the 4GB non-AM happens.
{code}
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar
sleep -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.map.memory.mb=4096
-m 1 -mt 60
{code}
* With the patch only following 3 lines are present when reservation occurs
which is expected after the patch is applied:
{code}
2021-03-25 17:54:13,475 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.sidtheadmin container:
2021-03-25 17:54:20,507 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.sidtheadmin container:
2021-03-25 17:54:35,558 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.somequeue container:
{code}
however, in the case of no patch, this was getting added before:
{code}
2021-03-25 17:54:43,589 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.somequeue container:
{code}
No JUnits required, as the change is about a "lack" of log, no change to
functionality, as such, re-running existing JUnits should suffice.
was (Author: sahuja):
Added a patch to ensure that logging only happens in case of an actual
container assignment/allocation, not reservation.
Tested this on a single node cluster from generated distribution after
compilation of the patch on trunk using the below steps:
* Set {{yarn.resourcemanager.scheduler.class}} to
{{org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler}},
* Started YARN on the single node cluster, it has 1 NodeManager with 8GB to run
containers,
* Enabled DEBUG logging for the FSLeafQueue class to check for debug logs:
{code}
bin/yarn daemonlog -setlevel localhost:8088
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue DEBUG
{code}
* Check for the DEBUG allocation message in the RM logs :
{code}
tail -f rmlogs.log | grep "DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.somequeue"
{code}
* Ran the first job requiring 1 AM + 1 non-AM worth 2GB each, so 4GB out of 8GB
are used up:
{code}
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar
sleep -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.map.memory.mb=2048
-m 1 -mt 60
{code}
* Ran a second job requiring 1 AM + 1 non-AM worth 2GB and 4GB respectively.
2nd application starts i.e. AM starts but there is no room for the 4GB
container yet so reservation for the 4GB non-AM happens.
{code}
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar
sleep -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.map.memory.mb=4096
-m 1 -mt 60
{code}
* With the patch only following 3 lines are present when reservation occurs
which is expected after the patch is applied:
{code}
2021-03-25 17:54:13,475 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.sidtheadmin container:
2021-03-25 17:54:20,507 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
[jira] [Comment Edited] (YARN-10705) Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler
[
https://issues.apache.org/jira/browse/YARN-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309038#comment-17309038
]
Siddharth Ahuja edited comment on YARN-10705 at 3/25/21, 11:18 PM:
---
Added a patch to ensure that logging only happens in case of an actual
container assignment/allocation, not reservation.
Tested this on a single node cluster from generated distribution after
compilation of the patch on trunk using the below steps:
* Set {{yarn.resourcemanager.scheduler.class}} to
{{org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler}},
* Started YARN on the single node cluster, it has 1 NodeManager with 8GB to run
containers,
* Enabled DEBUG logging for the FSLeafQueue class to check for debug logs:
{code}
bin/yarn daemonlog -setlevel localhost:8088
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue DEBUG
{code}
* Check for the DEBUG allocation message in the RM logs :
{code}
tail -f rmlogs.log | grep "DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.somequeue"
{code}
* Ran the first job requiring 1 AM + 1 non-AM worth 2GB each, so 4GB out of 8GB
are used up:
{code}
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar
sleep -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.map.memory.mb=2048
-m 1 -mt 60
{code}
* Ran a second job requiring 1 AM + 1 non-AM worth 2GB and 4GB respectively.
2nd application starts i.e. AM starts but there is no room for the 4GB
container yet so reservation for the 4GB non-AM happens.
{code}
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar
sleep -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.map.memory.mb=4096
-m 1 -mt 60
{code}
* With the patch only following 3 lines are present when reservation occurs
which is expected after the patch is applied:
{code}
2021-03-25 17:54:13,475 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.sidtheadmin container:
2021-03-25 17:54:20,507 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.sidtheadmin container:
2021-03-25 17:54:35,558 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.sidtheadmin container:
{code}
however, in the case of no patch, this was getting added before:
{code}
2021-03-25 17:54:43,589 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.sidtheadmin container:
{code}
No JUnits required, as the change is about a "lack" of log, no change to
functionality, as such, re-running existing JUnits should suffice.
was (Author: sahuja):
Added a patch to ensure that logging only happens in case of an actual
container assignment/allocation, not reservation.
Tested this on a single node cluster from generated distribution after
compilation of the patch on trunk using the below steps:
* Set {{yarn.resourcemanager.scheduler.class}} to
{{org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler}},
* Started YARN on the single node cluster, it has 1 NodeManager with 8GB to run
containers,
* Enabled DEBUG logging for the FSLeafQueue class to check for debug logs:
{code}
bin/yarn daemonlog -setlevel localhost:8088
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue DEBUG
{code}
* Check for the DEBUG allocation message in the RM logs :
{code}
tail -f rmlogs.log | grep "DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.somequeue"
{code}
* Ran the first job requiring 1 AM + 1 non-AM worth 2GB each, so 4GB out of 8GB
are used up:
{code}
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar
sleep -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.map.memory.mb=2048
-m 1 -mt 60
{code}
* Ran a second job requiring 1 AM + 1 non-AM worth 2GB and 4GB respectively.
2nd application starts i.e. AM starts but there is no room for the 4GB
container yet so reservation for the 4GB non-AM happens.
{code}
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar
sleep -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.map.memory.mb=4096
-m 1 -mt 60
{code}
* With the patch only following 3 lines are pretty when reservation occurs:
{code}
2021-03-25 17:54:13,475 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.sidtheadmin container:
2021-03-25 17:54:20,507 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
Assigned container in queue:root.sidtheadmin
