[jira] [Created] (YARN-11670) Add CallerContext in NodeManager
Jiandan Yang created YARN-11670: Summary: Add CallerContext in NodeManager Key: YARN-11670 URL: https://issues.apache.org/jira/browse/YARN-11670 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: Jiandan Yang Currently, MR and Spark have added caller context, enabling tracing of HDFS/ResourceManager operators from Spark apps and MapReduce apps. However, operators from NodeManagers cannot be identified in the audit log. For example, HDFS operations issued from NodeManagers during resource localization cannot be identified. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11699) Diagnostics
Jiandan Yang created YARN-11699: Summary: Diagnostics Key: YARN-11699 URL: https://issues.apache.org/jira/browse/YARN-11699 Project: Hadoop YARN Issue Type: Improvement Reporter: Jiandan Yang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11699) Diagnostics lacks userlimit info when user capacity has reached its maximum limit
[ https://issues.apache.org/jira/browse/YARN-11699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-11699: - Summary: Diagnostics lacks userlimit info when user capacity has reached its maximum limit (was: Diagnostics) > Diagnostics lacks userlimit info when user capacity has reached its maximum > limit > - > > Key: YARN-11699 > URL: https://issues.apache.org/jira/browse/YARN-11699 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jiandan Yang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11699) Diagnostics lacks userlimit info when user capacity has reached its maximum limit
[ https://issues.apache.org/jira/browse/YARN-11699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-11699: - Attachment: image-2024-05-29-15-47-53-217.png > Diagnostics lacks userlimit info when user capacity has reached its maximum > limit > - > > Key: YARN-11699 > URL: https://issues.apache.org/jira/browse/YARN-11699 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jiandan Yang >Priority: Major > Attachments: image-2024-05-29-15-47-53-217.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11699) Diagnostics lacks userlimit info when user capacity has reached its maximum limit
[ https://issues.apache.org/jira/browse/YARN-11699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-11699: - Description: Capacity scheduler supports user limit to avoid a single user use the whole queue resource. but when resource used by a user reached its user limit, it lacks related info in diagnostics in web page, just as shown in the figure below. We may need add user limit and resource used by the user to help debug. !image-2024-05-29-15-47-53-217.png|width=831,height=145! > Diagnostics lacks userlimit info when user capacity has reached its maximum > limit > - > > Key: YARN-11699 > URL: https://issues.apache.org/jira/browse/YARN-11699 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jiandan Yang >Priority: Major > Attachments: image-2024-05-29-15-47-53-217.png > > > Capacity scheduler supports user limit to avoid a single user use the whole > queue resource. but when resource used by a user reached its user limit, it > lacks related info in diagnostics in web page, just as shown in the figure > below. We may need add user limit and resource used by the user to help debug. > !image-2024-05-29-15-47-53-217.png|width=831,height=145! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11538) CS UI: queue filter do not work as expected when submitting apps with leaf queue‘s name
Jiandan Yang created YARN-11538: Summary: CS UI: queue filter do not work as expected when submitting apps with leaf queue‘s name Key: YARN-11538 URL: https://issues.apache.org/jira/browse/YARN-11538 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Jiandan Yang Assignee: Jiandan Yang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11538) CS UI: queue filter do not work as expected when submitting apps with leaf queue‘s name
[
https://issues.apache.org/jira/browse/YARN-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11538:
-
Description:
reproduction steps:
1. create a mr app with leaf queue's name without *root.*, for example:
{code:java}
hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
-Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
{code}
2. open the schedule web page of CapacityScheduler
3. click the queue of the app submited in step 1
4. check the app list showed in web page, and can not find the app submited in
step 1
> CS UI: queue filter do not work as expected when submitting apps with leaf
> queue‘s name
>
>
> Key: YARN-11538
> URL: https://issues.apache.org/jira/browse/YARN-11538
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
>
> reproduction steps:
> 1. create a mr app with leaf queue's name without *root.*, for example:
> {code:java}
> hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
> -Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
> {code}
> 2. open the schedule web page of CapacityScheduler
> 3. click the queue of the app submited in step 1
> 4. check the app list showed in web page, and can not find the app submited
> in step 1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11538) CS UI: queue filter do not work as expected when submitting apps with leaf queue‘s name
[
https://issues.apache.org/jira/browse/YARN-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11538:
-
Description:
I have noticed that the application can not be filtered correctly in the
CapacityScheduer page when I submit apps
*Steps to Reproduce:*
1. create a mr app with leaf queue's name without *root.*, for example:
{code:java}
hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
-Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
{code}
2. open the schedule web page of CapacityScheduler
3. click the queue of the app submited in step 1
4. check the app list showed in web page, and can not find the app submited in
step 1
was:
reproduction steps:
1. create a mr app with leaf queue's name without *root.*, for example:
{code:java}
hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
-Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
{code}
2. open the schedule web page of CapacityScheduler
3. click the queue of the app submited in step 1
4. check the app list showed in web page, and can not find the app submited in
step 1
> CS UI: queue filter do not work as expected when submitting apps with leaf
> queue‘s name
>
>
> Key: YARN-11538
> URL: https://issues.apache.org/jira/browse/YARN-11538
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
>
> I have noticed that the application can not be filtered correctly in the
> CapacityScheduer page when I submit apps
> *Steps to Reproduce:*
> 1. create a mr app with leaf queue's name without *root.*, for example:
> {code:java}
> hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
> -Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
> {code}
> 2. open the schedule web page of CapacityScheduler
> 3. click the queue of the app submited in step 1
> 4. check the app list showed in web page, and can not find the app submited
> in step 1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11538) CS UI: queue filter do not work as expected when submitting apps with leaf queue‘s name
[
https://issues.apache.org/jira/browse/YARN-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11538:
-
Description:
I have noticed that the application can not be filtered correctly in the
CapacityScheduer page when I submit apps with leaf queue‘s name.
*Steps to Reproduce:*
1. create a mr app with leaf queue's name without *root.*, for example:
{code:java}
hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
-Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
{code}
2. open the schedule web page of CapacityScheduler
3. click the queue of the app submited in step 1
4. check the apps showed in web page, and can not find the app submited in step
1
*Problem Analysis:*
1.
was:
I have noticed that the application can not be filtered correctly in the
CapacityScheduer page when I submit apps
*Steps to Reproduce:*
1. create a mr app with leaf queue's name without *root.*, for example:
{code:java}
hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
-Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
{code}
2. open the schedule web page of CapacityScheduler
3. click the queue of the app submited in step 1
4. check the app list showed in web page, and can not find the app submited in
step 1
> CS UI: queue filter do not work as expected when submitting apps with leaf
> queue‘s name
>
>
> Key: YARN-11538
> URL: https://issues.apache.org/jira/browse/YARN-11538
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
>
> I have noticed that the application can not be filtered correctly in the
> CapacityScheduer page when I submit apps with leaf queue‘s name.
> *Steps to Reproduce:*
> 1. create a mr app with leaf queue's name without *root.*, for example:
> {code:java}
> hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
> -Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
> {code}
> 2. open the schedule web page of CapacityScheduler
> 3. click the queue of the app submited in step 1
> 4. check the apps showed in web page, and can not find the app submited in
> step 1
> *Problem Analysis:*
> 1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11538) CS UI: queue filter do not work as expected when submitting apps with leaf queue‘s name
[
https://issues.apache.org/jira/browse/YARN-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11538:
-
Description:
I have noticed that the application can not be filtered correctly in the
CapacityScheduer page when I submit apps with leaf queue‘s name.
*Steps to Reproduce:*
1. create a mr app with leaf queue's name without *root.*, for example:
{code:java}
hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
-Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
{code}
2. open the schedule web page of CapacityScheduler
3. click the queue of the app submited in step 1
4. check the apps showed in web page, and can not find the app submited in step
1
*Problem Analysis:*
when clicking a queue in scheduler page, it filters by full path of Queue in
web page. Beause the queues of apps returned by *getApplications* is short
path, the result of filter is empty
*Solution:*
*getApplications* returns full queue path of apps.
was:
I have noticed that the application can not be filtered correctly in the
CapacityScheduer page when I submit apps with leaf queue‘s name.
*Steps to Reproduce:*
1. create a mr app with leaf queue's name without *root.*, for example:
{code:java}
hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
-Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
{code}
2. open the schedule web page of CapacityScheduler
3. click the queue of the app submited in step 1
4. check the apps showed in web page, and can not find the app submited in step
1
*Problem Analysis:*
1.
> CS UI: queue filter do not work as expected when submitting apps with leaf
> queue‘s name
>
>
> Key: YARN-11538
> URL: https://issues.apache.org/jira/browse/YARN-11538
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
>
> I have noticed that the application can not be filtered correctly in the
> CapacityScheduer page when I submit apps with leaf queue‘s name.
> *Steps to Reproduce:*
> 1. create a mr app with leaf queue's name without *root.*, for example:
> {code:java}
> hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
> -Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
> {code}
> 2. open the schedule web page of CapacityScheduler
> 3. click the queue of the app submited in step 1
> 4. check the apps showed in web page, and can not find the app submited in
> step 1
> *Problem Analysis:*
> when clicking a queue in scheduler page, it filters by full path of Queue in
> web page. Beause the queues of apps returned by *getApplications* is short
> path, the result of filter is empty
> *Solution:*
> *getApplications* returns full queue path of apps.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11538) CS UI: queue filter do not work as expected when submitting apps with leaf queue‘s name
[
https://issues.apache.org/jira/browse/YARN-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11538:
-
Description:
I have noticed that the application can not be filtered correctly in the
CapacityScheduer page when I submit apps with leaf queue‘s name.
*Steps to Reproduce:*
1. create a mr app with leaf queue's name without *root.*, for example:
{code:java}
hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
-Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
{code}
2. open the schedule web page of CapacityScheduler
3. click the queue of the app submited in step 1
4. check the apps showed in web page, and can not find the app submited in step
1
*Problem Analysis:*
when clicking a queue in scheduler page, it filters by full path of Queue in
web page. Beause the queues of apps returned by *getApplications* is short
path, the result of filter is empty
*Solution:*
getApplications returns full queue path of apps.
was:
I have noticed that the application can not be filtered correctly in the
CapacityScheduer page when I submit apps with leaf queue‘s name.
*Steps to Reproduce:*
1. create a mr app with leaf queue's name without *root.*, for example:
{code:java}
hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
-Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
{code}
2. open the schedule web page of CapacityScheduler
3. click the queue of the app submited in step 1
4. check the apps showed in web page, and can not find the app submited in step
1
*Problem Analysis:*
when clicking a queue in scheduler page, it filters by full path of Queue in
web page. Beause the queues of apps returned by *getApplications* is short
path, the result of filter is empty
*Solution:*
*getApplications* returns full queue path of apps.
> CS UI: queue filter do not work as expected when submitting apps with leaf
> queue‘s name
>
>
> Key: YARN-11538
> URL: https://issues.apache.org/jira/browse/YARN-11538
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
>
> I have noticed that the application can not be filtered correctly in the
> CapacityScheduer page when I submit apps with leaf queue‘s name.
> *Steps to Reproduce:*
> 1. create a mr app with leaf queue's name without *root.*, for example:
> {code:java}
> hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
> -Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
> {code}
> 2. open the schedule web page of CapacityScheduler
> 3. click the queue of the app submited in step 1
> 4. check the apps showed in web page, and can not find the app submited in
> step 1
> *Problem Analysis:*
> when clicking a queue in scheduler page, it filters by full path of Queue in
> web page. Beause the queues of apps returned by *getApplications* is short
> path, the result of filter is empty
> *Solution:*
> getApplications returns full queue path of apps.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11538) CS UI: queue filter do not work as expected when submitting apps with leaf queue‘s name
[
https://issues.apache.org/jira/browse/YARN-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11538:
-
Attachment: YARN-11538.patch
> CS UI: queue filter do not work as expected when submitting apps with leaf
> queue‘s name
>
>
> Key: YARN-11538
> URL: https://issues.apache.org/jira/browse/YARN-11538
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-11538.patch
>
>
> I have noticed that the application can not be filtered correctly in the
> CapacityScheduer page when I submit apps with leaf queue‘s name.
> *Steps to Reproduce:*
> 1. create a mr app with leaf queue's name without *root.*, for example:
> {code:java}
> hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
> -Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
> {code}
> 2. open the schedule web page of CapacityScheduler
> 3. click the queue of the app submited in step 1
> 4. check the apps showed in web page, and can not find the app submited in
> step 1
> *Problem Analysis:*
> when clicking a queue in scheduler page, it filters by full path of Queue in
> web page. Beause the queues of apps returned by *getApplications* is short
> path, the result of filter is empty
> *Solution:*
> getApplications returns full queue path of apps.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11538) CS UI: queue filter do not work as expected when submitting apps with leaf queue‘s name
[
https://issues.apache.org/jira/browse/YARN-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11538:
-
Attachment: (was: YARN-11538.patch)
> CS UI: queue filter do not work as expected when submitting apps with leaf
> queue‘s name
>
>
> Key: YARN-11538
> URL: https://issues.apache.org/jira/browse/YARN-11538
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-11538.patch
>
>
> I have noticed that the application can not be filtered correctly in the
> CapacityScheduer page when I submit apps with leaf queue‘s name.
> *Steps to Reproduce:*
> 1. create a mr app with leaf queue's name without *root.*, for example:
> {code:java}
> hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
> -Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
> {code}
> 2. open the schedule web page of CapacityScheduler
> 3. click the queue of the app submited in step 1
> 4. check the apps showed in web page, and can not find the app submited in
> step 1
> *Problem Analysis:*
> when clicking a queue in scheduler page, it filters by full path of Queue in
> web page. Beause the queues of apps returned by *getApplications* is short
> path, the result of filter is empty
> *Solution:*
> getApplications returns full queue path of apps.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11538) CS UI: queue filter do not work as expected when submitting apps with leaf queue‘s name
[
https://issues.apache.org/jira/browse/YARN-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11538:
-
Attachment: YARN-11538-001.patch
> CS UI: queue filter do not work as expected when submitting apps with leaf
> queue‘s name
>
>
> Key: YARN-11538
> URL: https://issues.apache.org/jira/browse/YARN-11538
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-11538-001.patch
>
>
> I have noticed that the application can not be filtered correctly in the
> CapacityScheduer page when I submit apps with leaf queue‘s name.
> *Steps to Reproduce:*
> 1. create a mr app with leaf queue's name without *root.*, for example:
> {code:java}
> hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
> -Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
> {code}
> 2. open the schedule web page of CapacityScheduler
> 3. click the queue of the app submited in step 1
> 4. check the apps showed in web page, and can not find the app submited in
> step 1
> *Problem Analysis:*
> when clicking a queue in scheduler page, it filters by full path of Queue in
> web page. Beause the queues of apps returned by *getApplications* is short
> path, the result of filter is empty
> *Solution:*
> getApplications returns full queue path of apps.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11538) CS UI: queue filter do not work as expected when submitting apps with leaf queue‘s name
[
https://issues.apache.org/jira/browse/YARN-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11538:
-
Attachment: (was: YARN-11538.patch)
> CS UI: queue filter do not work as expected when submitting apps with leaf
> queue‘s name
>
>
> Key: YARN-11538
> URL: https://issues.apache.org/jira/browse/YARN-11538
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-11538-001.patch
>
>
> I have noticed that the application can not be filtered correctly in the
> CapacityScheduer page when I submit apps with leaf queue‘s name.
> *Steps to Reproduce:*
> 1. create a mr app with leaf queue's name without *root.*, for example:
> {code:java}
> hadoop jar hadoop-mapreduce-examples-3.3.4.jar teragen
> -Dmapreduce.job.queuename=ia_ana 100 /user/yjd/teragen
> {code}
> 2. open the schedule web page of CapacityScheduler
> 3. click the queue of the app submited in step 1
> 4. check the apps showed in web page, and can not find the app submited in
> step 1
> *Problem Analysis:*
> when clicking a queue in scheduler page, it filters by full path of Queue in
> web page. Beause the queues of apps returned by *getApplications* is short
> path, the result of filter is empty
> *Solution:*
> getApplications returns full queue path of apps.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5357) Timeline service v2 integration with Federation
[
https://issues.apache.org/jira/browse/YARN-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17755773#comment-17755773
]
Jiandan Yang commented on YARN-5357:
-
Hello, [~abmodi] [~vrushalic]
I've noticed that this JIRA hasn't seen any updates in over four years. I'm
interested in continuing the development on this issue. I was wondering if you
are still following or working on this topic?
I'd like to build upon the discussions that have taken place here. If anyone
involved in the previous discussions could provide or point me to the proposed
solutions or strategies discussed earlier, that would be immensely helpful.
Looking forward to your insights and feedback as I venture into progressing
this issue further.
Thank you for your time and consideration.
> Timeline service v2 integration with Federation
>
>
> Key: YARN-5357
> URL: https://issues.apache.org/jira/browse/YARN-5357
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Abhishek Modi
>Priority: Major
>
> Jira to note the discussion points from an initial chat about integrating
> Timeline Service v2 with Federation (YARN-2915).
> cc [~subru] [~curino]
> For Federation:
> - all entities that belong to the same flow run should have the same cluster
> name
> - app id in the same flow run strongly ordered in time
> - need a logical cluster name and physical cluster name
> - a possibility to implement the Application TimelineCollector as an
> interceptor in the AMRMProxyService.
> For Timeline Service:
> - need to store physical cluster id and logical cluster id so that we don't
> lose information at any level (flow/app/entity etc)
> - add a new table app id to cluster mapping table
> - need a different entity table/some table to store node level metrics for
> physical cluster stats. Once we get to node-level rollup, we probably have to
> store something in a dc, cluster, rack, node hierarchy. In that case a
> physical cluster makes sense, but we'd still need some way to tie physical
> and logical together in order to make automatic error detection etc that
> we're envisioning feasible within a federated setup.
> For the Cluster Naming convention:
> - three situations for cluster name:
> > app submitted to router should take federated (aka logical) cluster name
> > app submitted directly to RM should take physical cluster name
> > Info about the physical cluster in entities?
> - suggestion to set the cluster name as yarn tag at the router level (in the
> app submission context)
> Other points to note:
> - for federation to work smoothly in environments that use HDFS some
> additional considerations are needed, and possibly some solution like what is
> being used at Twitter with the nFly approach.
> Email thread context:
> {code}
> -- Forwarded message --
> From: Joep Rottinghuis
> Date: Fri, Jul 8, 2016 at 1:22 PM
> Subject: Re: Federation -Timeline Service meeting notes
> To: Subramaniam Venkatraman Krishnan
> Cc: Sangjin Lee, Vrushali Channapattan , Carlo Curino
> Thanks for the notes.
> I think that for federation to work smoothly in environments that use HDFS
> some additional considerations are needed, and possibly some solution like
> what we're using at Twitter with our nFly approach.
> bq. - need a different entity table/some table to store node level metrics
> for physical cluster stats
> Once we get to node-level rollup, we probably have to store something in a
> dc, cluster, rack, node hierarchy. In that case a physical cluster makes
> sense, but we'd still need some way to tie physical and logical together in
> order to make automatic error detection etc that we're envisioning feasible
> within a federated setup.
> Cheers,
> Joep
> On Fri, Jul 8, 2016 at 1:00 PM, Subramaniam Venkatraman Krishnan wrote:
> Thanks Vrushali for crisply capturing the essential from our rambling
> discussion J.
>
> Sangjin, I just want to add one comment to yours – we want to retain the
> physical cluster name (possibly as a new entity type) so that we don’t lose
> information & we can cluster level rollups even if they are not efficient.
>
> Additionally, based on the walkthrough of Federation design:
> · There was general agreement with the proposed approach.
> · There is a possibility to implement the Application
> TimelineCollector as an interceptor in the AMRMProxyService.
> · Joep raised the concern that it would be better if the RMs
> obtain the epoch from FederationStateStore. This is not currently in the
> roadmap of our MVP but we definitely plan to address this in future.
>
> Regards,
> Subru
>
> From: Sangjin Lee
[jira] [Created] (YARN-11552) /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
Jiandan Yang created YARN-11552:
Summary: /clusters/{clusterid}/apps/{appid}/entity-types Error
when using hdfs store
Key: YARN-11552
URL: https://issues.apache.org/jira/browse/YARN-11552
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jiandan Yang
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11552) /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11552:
-
Description:
*Problem description*
I start hadoop cluster and timeservice based on hdfs store，it throw execption
when accessing
> /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
> ---
>
> Key: YARN-11552
> URL: https://issues.apache.org/jira/browse/YARN-11552
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jiandan Yang
>Priority: Major
>
> *Problem description*
> I start hadoop cluster and timeservice based on hdfs store，it throw execption
> when accessing
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11552) /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11552:
-
Description:
*Problem description*
I start hadoop cluster and timeservice based on hdfs store ，it throw execption
when accessing
http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1,
and detailed logs includes exception:
{code:java}
2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372 -
Processed URL
/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001
(Took 56 ms.)
2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
INTERNAL_SERVER_ERROR
2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
java.io.FileNotFoundException: File
/tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
does not exist.
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
2023-08-18 17:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
2023-08-18 17:40:51 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-18 17:40:51 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
2023-08-18 17:40:51 at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:650)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
o
[jira] [Updated] (YARN-11552) /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11552:
-
Description:
*Problem description*
I start hadoop cluster and timeservice based on hdfs store ，it throw execption
when accessing
http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1,
and detailed logs includes exception:
{code:java}
2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372 -
Processed URL
/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001
(Took 56 ms.)
2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
INTERNAL_SERVER_ERROR
2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
java.io.FileNotFoundException: File
/tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
does not exist.
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
2023-08-18 17:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
2023-08-18 17:40:51 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-18 17:40:51 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
2023-08-18 17:40:51 at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:650)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
o
[jira] [Updated] (YARN-11552) /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11552:
-
Description:
*Problem description*
I start hadoop cluster and timeservice based on hdfs store ，it throw execption
when accessing
http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1,
and detailed logs includes exception:
{code:java}
2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372 -
Processed URL
/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001
(Took 56 ms.)
2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
INTERNAL_SERVER_ERROR
2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
java.io.FileNotFoundException: File
/tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
does not exist.
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
2023-08-18 17:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
2023-08-18 17:40:51 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-18 17:40:51 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
2023-08-18 17:40:51 at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:650)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
o
[jira] [Updated] (YARN-11552) /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11552:
-
Description:
*Problem description*
I start hadoop cluster and timeservice based on hdfs store ，it throw execption
when accessing
http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1,
and detailed logs includes exception:
{code:java}
2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372 -
Processed URL
/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001
(Took 56 ms.)
2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
INTERNAL_SERVER_ERROR
2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
java.io.FileNotFoundException: File
/tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
does not exist.
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
2023-08-18 17:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
2023-08-18 17:40:51 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-18 17:40:51 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
2023-08-18 17:40:51 at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:650)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
o
[jira] [Updated] (YARN-11552) /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11552:
-
Description:
*Problem description*
I start hadoop cluster and timeservice based on hdfs store ，it throw execption
when accessing
http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1,
and detailed logs includes exception:
{code:java}
2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372 -
Processed URL
/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001
(Took 56 ms.)
2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
INTERNAL_SERVER_ERROR
2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
java.io.FileNotFoundException: File
/tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
does not exist.
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
2023-08-18 17:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
2023-08-18 17:40:51 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-18 17:40:51 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
2023-08-18 17:40:51 at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:650)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
o
[jira] [Updated] (YARN-11552) /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11552:
-
Description:
*Problem description*
I start hadoop cluster and timeservice based on hdfs store ，it throw execption
when accessing
[http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1],
and detailed logs includes exception:
{code:java}
2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372 -
Processed URL
/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001
(Took 56 ms.)
2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
INTERNAL_SERVER_ERROR
2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
java.io.FileNotFoundException: File
/tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
does not exist.
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
2023-08-18 17:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
2023-08-18 17:40:51 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-18 17:40:51 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
2023-08-18 17:40:51 at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:650)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
[jira] [Updated] (YARN-11552) /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11552:
-
Description:
*Problem description*
I start hadoop cluster and timeservice based on hdfs store ，it throw execption
when accessing
[http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1],
and detailed logs includes exception:
{code:java}
2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372 -
Processed URL
/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001
(Took 56 ms.)
2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
INTERNAL_SERVER_ERROR
2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
java.io.FileNotFoundException: File
/tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
does not exist.
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
2023-08-18 17:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
2023-08-18 17:40:51 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-18 17:40:51 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
2023-08-18 17:40:51 at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:650)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
[jira] [Updated] (YARN-11552) /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11552:
-
Description:
*Problem description*
I start hadoop cluster and timeservice based on hdfs store ，it throw execption
when accessing
[http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1],
and detailed logs includes exception:
{code:java}
2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372 -
Processed URL
/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001
(Took 56 ms.)
2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
INTERNAL_SERVER_ERROR
2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
java.io.FileNotFoundException: File
/tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
does not exist.
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
2023-08-18 17:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
2023-08-18 17:40:51 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-18 17:40:51 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
2023-08-18 17:40:51 at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:650)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
[jira] [Updated] (YARN-11552) /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11552:
-
Description:
*Problem description*
I start hadoop cluster and timeservice based on hdfs store ，it throw execption
when accessing
[http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1],
and detailed logs includes exception:
{code:java}
2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372 -
Processed URL
/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001
(Took 56 ms.)
2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
INTERNAL_SERVER_ERROR
2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
java.io.FileNotFoundException: File
/tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
does not exist.
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
2023-08-18 17:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
2023-08-18 17:40:51 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-18 17:40:51 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
2023-08-18 17:40:51 at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:650)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
[jira] [Updated] (YARN-11552) timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11552:
-
Summary: timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types
Error when using hdfs store (was:
/clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store)
> timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types Error when
> using hdfs store
> --
>
> Key: YARN-11552
> URL: https://issues.apache.org/jira/browse/YARN-11552
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jiandan Yang
>Priority: Major
>
> *Problem description*
> I start hadoop cluster and timeservice based on hdfs store ，it throw
> execption when accessing
> [http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1],
> and detailed logs includes exception:
> {code:java}
> 2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372
> - Processed URL
> /ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001
> (Took 56 ms.)
> 2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
> INTERNAL_SERVER_ERROR
> 2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
> java.io.FileNotFoundException: File
> /tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
> does not exist.
> 2023-08-18 17:40:51 at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
> 2023-08-18 17:40:51 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2023-08-18 17:40:51 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2023-08-18 17:40:51 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
> 2023-08-18 17:40:51 at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> 2023-08-18 17:40:51 at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> 2023-08-18 17:40:51 at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> 2023-08-18 17:40:51 at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> 2023-08-18 17:40:51 at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthoriza
[jira] [Updated] (YARN-11552) timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11552:
-
Description:
*Problem description*
I start hadoop cluster and timeservice based on hdfs store ，it throw execption
when accessing
[http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1692343022285],
and detailed logs includes exception:
{code:java}
2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372 -
Processed URL
/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001
(Took 56 ms.)
2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
INTERNAL_SERVER_ERROR
2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
java.io.FileNotFoundException: File
/tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
does not exist.
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
2023-08-18 17:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
2023-08-18 17:40:51 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-18 17:40:51 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
2023-08-18 17:40:51 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
2023-08-18 17:40:51 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
2023-08-18 17:40:51 at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
2023-08-18 17:40:51 at
org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:650)
2023-08-18 17:40:51 at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
2023-08-18 17:40:51 at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
2023-08-18 17:40
[jira] [Created] (YARN-11650) refactor policyName to policyClassName
Jiandan Yang created YARN-11650: Summary: refactor policyName to policyClassName Key: YARN-11650 URL: https://issues.apache.org/jira/browse/YARN-11650 Project: Hadoop YARN Issue Type: Improvement Components: RM Reporter: Jiandan Yang In classes related to MultiNodePolicy support, some variable names do not accurately reflect their true meanings. For instance, a variable named *queue* is actually representing the class name of a policy, and a variable named *policyName* denotes the class name of the policy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11650) refactor policyName to policyClassName
[ https://issues.apache.org/jira/browse/YARN-11650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-11650: - Description: In classes related to MultiNodePolicy support, some variable names do not accurately reflect their true meanings. For instance, a variable named *queue* is actually representing the class name of a policy, and a variable named *policyName* denotes the class name of the policy. This may cause confusion for the readers of the code. (was: In classes related to MultiNodePolicy support, some variable names do not accurately reflect their true meanings. For instance, a variable named *queue* is actually representing the class name of a policy, and a variable named *policyName* denotes the class name of the policy.) > refactor policyName to policyClassName > -- > > Key: YARN-11650 > URL: https://issues.apache.org/jira/browse/YARN-11650 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Reporter: Jiandan Yang >Priority: Major > > In classes related to MultiNodePolicy support, some variable names do not > accurately reflect their true meanings. For instance, a variable named > *queue* is actually representing the class name of a policy, and a variable > named *policyName* denotes the class name of the policy. This may cause > confusion for the readers of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11650) refactor policyName to policyClassName
[ https://issues.apache.org/jira/browse/YARN-11650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang reassigned YARN-11650: Assignee: Jiandan Yang > refactor policyName to policyClassName > -- > > Key: YARN-11650 > URL: https://issues.apache.org/jira/browse/YARN-11650 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > > In classes related to MultiNodePolicy support, some variable names do not > accurately reflect their true meanings. For instance, a variable named > *queue* is actually representing the class name of a policy, and a variable > named *policyName* denotes the class name of the policy. This may cause > confusion for the readers of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11650) Refactoring variable names in Class MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue
[ https://issues.apache.org/jira/browse/YARN-11650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-11650: - Summary: Refactoring variable names in Class MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue (was: refactor policyName to policyClassName) > Refactoring variable names in Class MultiNodePolicySpec, FiCaSchedulerApp and > AbstractCSQueue > - > > Key: YARN-11650 > URL: https://issues.apache.org/jira/browse/YARN-11650 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > > In classes related to MultiNodePolicy support, some variable names do not > accurately reflect their true meanings. For instance, a variable named > *queue* is actually representing the class name of a policy, and a variable > named *policyName* denotes the class name of the policy. This may cause > confusion for the readers of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11650) Refactoring variable names related in Class MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue
[ https://issues.apache.org/jira/browse/YARN-11650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-11650: - Summary: Refactoring variable names related in Class MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue (was: Refactoring variable names in Class MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue) > Refactoring variable names related in Class MultiNodePolicySpec, > FiCaSchedulerApp and AbstractCSQueue > - > > Key: YARN-11650 > URL: https://issues.apache.org/jira/browse/YARN-11650 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > > In classes related to MultiNodePolicy support, some variable names do not > accurately reflect their true meanings. For instance, a variable named > *queue* is actually representing the class name of a policy, and a variable > named *policyName* denotes the class name of the policy. This may cause > confusion for the readers of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11650) Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue
[ https://issues.apache.org/jira/browse/YARN-11650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-11650: - Summary: Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue (was: Refactoring variable names related in Class MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue) > Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, > FiCaSchedulerApp and AbstractCSQueue > --- > > Key: YARN-11650 > URL: https://issues.apache.org/jira/browse/YARN-11650 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > > In classes related to MultiNodePolicy support, some variable names do not > accurately reflect their true meanings. For instance, a variable named > *queue* is actually representing the class name of a policy, and a variable > named *policyName* denotes the class name of the policy. This may cause > confusion for the readers of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11651) Fix UT TestQueueCapacityConfigParser
Jiandan Yang created YARN-11651: Summary: Fix UT TestQueueCapacityConfigParser Key: YARN-11651 URL: https://issues.apache.org/jira/browse/YARN-11651 Project: Hadoop YARN Issue Type: Bug Reporter: Jiandan Yang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11651) Fix UT TestQueueCapacityConfigParser compile error
[ https://issues.apache.org/jira/browse/YARN-11651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-11651: - Summary: Fix UT TestQueueCapacityConfigParser compile error (was: Fix UT TestQueueCapacityConfigParser) > Fix UT TestQueueCapacityConfigParser compile error > -- > > Key: YARN-11651 > URL: https://issues.apache.org/jira/browse/YARN-11651 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jiandan Yang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11651) Fix UT TestQueueCapacityConfigParser compile error
[
https://issues.apache.org/jira/browse/YARN-11651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11651:
-
Description:
The following error is reported during compilation:
{code:java}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile
(default-testCompile) on project hadoop-yarn-server-resourcemanager:
Compilation failure
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-6490/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestQueueCapacityConfigParser.java:[224,80]
incompatible types: java.lang.String cannot be converted to
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueuePath
[ERROR] -> [Help 1]
{code}
> Fix UT TestQueueCapacityConfigParser compile error
> --
>
> Key: YARN-11651
> URL: https://issues.apache.org/jira/browse/YARN-11651
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jiandan Yang
>Priority: Major
> Labels: pull-request-available
>
> The following error is reported during compilation:
> {code:java}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile
> (default-testCompile) on project hadoop-yarn-server-resourcemanager:
> Compilation failure
> [ERROR]
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-6490/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestQueueCapacityConfigParser.java:[224,80]
> incompatible types: java.lang.String cannot be converted to
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueuePath
> [ERROR] -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11651) Fix UT TestQueueCapacityConfigParser compile error
[
https://issues.apache.org/jira/browse/YARN-11651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang reassigned YARN-11651:
Assignee: Jiandan Yang
> Fix UT TestQueueCapacityConfigParser compile error
> --
>
> Key: YARN-11651
> URL: https://issues.apache.org/jira/browse/YARN-11651
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Labels: pull-request-available
>
> The following error is reported during compilation:
> {code:java}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile
> (default-testCompile) on project hadoop-yarn-server-resourcemanager:
> Compilation failure
> [ERROR]
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-6490/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestQueueCapacityConfigParser.java:[224,80]
> incompatible types: java.lang.String cannot be converted to
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueuePath
> [ERROR] -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11041) Replace all occurences of queuePath with the new QueuePath class - followup
[ https://issues.apache.org/jira/browse/YARN-11041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810643#comment-17810643 ] Jiandan Yang commented on YARN-11041: -- This results in a compilation error for TestQueueCapacityConfigParser on the trunk branch. I file a issue https://issues.apache.org/jira/browse/YARN-11651 to fix. > Replace all occurences of queuePath with the new QueuePath class - followup > --- > > Key: YARN-11041 > URL: https://issues.apache.org/jira/browse/YARN-11041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Tibor Kovács >Assignee: Peter Szucs >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > The QueuePath class was introduced in YARN-10897, however, its current > adoption happened only for code changes after this JIRA. We need to adopt it > retrospectively. > > A lot of changes are introduced via ticket YARN-10982. The replacing should > be continued by touching the next comments: > > [...g/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java|https://github.com/apache/hadoop/pull/3660/files/f956918bc154d0e35fce07c5dd8be804eb007acc#diff-fde6885144b59bb06b2c3358780388d958829b13f68aceee7bb6d394bb5e0548] > |[~snemeth] [https://github.com/apache/hadoop/pull/3660#discussion_r765012937] > I think this could be also refactored in a follow-up jira so the string magic > could probably be replaced with some more elegant solution. Though, I think > this would be too much in this patch, hence I do suggest the follow-up jira.| > |[~snemeth] [https://github.com/apache/hadoop/pull/3660#discussion_r765013096] > [~bteke] [ |https://github.com/9uapaw] [~gandras] [ > \|https://github.com/9uapaw] Thoughts?| > |[~bteke] [https://github.com/apache/hadoop/pull/3660#discussion_r765110750] > +1, even the QueuePath object could have some kind of support for this.| > |[~gandras] [https://github.com/apache/hadoop/pull/3660#discussion_r765131244] > Agreed, let's handle it in a followup!| > > > > [...he/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java|https://github.com/apache/hadoop/pull/3660/files/f956918bc154d0e35fce07c5dd8be804eb007acc#diff-c4b0c5e70208f1e3cfbd5a86ffa2393e5c996cc8b45605d9d41abcb7e0bd382a] > |[~snemeth] [https://github.com/apache/hadoop/pull/3660#discussion_r765023717] > There are many string operations in this class: > E.g. * getQueuePrefix that works with the full queue path > * getNodeLabelPrefix that also works with the full queue path| > I suggest to create a static class, called "QueuePrefixes" or something like > that and add some static methods there to convert the QueuePath object to > those various queue prefix strings that are ultimately keys in the > Configuration object. > > > > [...he/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java|https://github.com/apache/hadoop/pull/3660/files/f956918bc154d0e35fce07c5dd8be804eb007acc#diff-c4b0c5e70208f1e3cfbd5a86ffa2393e5c996cc8b45605d9d41abcb7e0bd382a] > |[~snemeth] [https://github.com/apache/hadoop/pull/3660#discussion_r765026119] > This seems hacky, just based on the constructor parameter names of QueuePath: > parent, leaf. > The AQC Template prefix is not the leaf, obviously. > Could we somehow circumvent this?| > |[~bteke] [https://github.com/apache/hadoop/pull/3660#discussion_r765126207] > Maybe a factory method could be created, which returns a new QueuePath with > the parent set as the original queuePath. I.e > rootQueuePath.createChild(String childName) -> this could return a new > QueuePath object with root.childName path, and rootQueuePath as parent.| > |[~snemeth] [https://github.com/apache/hadoop/pull/3660#discussion_r765039033] > Looking at this getQueues method, I realized almost all the callers are using > some kind of string magic that should be addressed with this patch. > For example, take a look at: > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.MutableCSConfigurationProvider#addQueue > I think getQueues should also receive the QueuePath object instead of > Strings.| > > > > [.../src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java|https://github.com/apache/hadoop/pull/3660/files/0c3dd17c936260fc9c386dcabc6368b54b27aa82..39f4ec203377244f840e4593aa02386ff51cc3c4#diff-0adf8192c51cbe4671324f06f7f8cbd48898df0376bbcc516451a3bdb2b48d3b] > |[~bteke] [https://github.com/apache/hadoop/pull/3660#discussion_r765912967] > Nit: Gets the queue path object. > The object of the queue suggests a CSQueue object.| > |[~snemeth] [https://github.com/apache/hadoop/pull/3
[jira] [Updated] (YARN-11651) Fix UT TestQueueCapacityConfigParser compile error
[
https://issues.apache.org/jira/browse/YARN-11651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-11651:
-
Description:
The following error is reported during compilation:
{code:java}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile
(default-testCompile) on project hadoop-yarn-server-resourcemanager:
Compilation failure
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-6490/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestQueueCapacityConfigParser.java:[224,80]
incompatible types: java.lang.String cannot be converted to
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueuePath
[ERROR] -> [Help 1]
{code}
This caused by YARN-11041
was:
The following error is reported during compilation:
{code:java}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile
(default-testCompile) on project hadoop-yarn-server-resourcemanager:
Compilation failure
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-6490/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestQueueCapacityConfigParser.java:[224,80]
incompatible types: java.lang.String cannot be converted to
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueuePath
[ERROR] -> [Help 1]
{code}
> Fix UT TestQueueCapacityConfigParser compile error
> --
>
> Key: YARN-11651
> URL: https://issues.apache.org/jira/browse/YARN-11651
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Labels: pull-request-available
>
> The following error is reported during compilation:
> {code:java}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile
> (default-testCompile) on project hadoop-yarn-server-resourcemanager:
> Compilation failure
> [ERROR]
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-6490/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestQueueCapacityConfigParser.java:[224,80]
> incompatible types: java.lang.String cannot be converted to
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueuePath
> [ERROR] -> [Help 1]
> {code}
> This caused by YARN-11041
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11651) Fix UT TestQueueCapacityConfigParser compile error
[
https://issues.apache.org/jira/browse/YARN-11651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang resolved YARN-11651.
--
Resolution: Invalid
> Fix UT TestQueueCapacityConfigParser compile error
> --
>
> Key: YARN-11651
> URL: https://issues.apache.org/jira/browse/YARN-11651
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Labels: pull-request-available
>
> The following error is reported during compilation:
> {code:java}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile
> (default-testCompile) on project hadoop-yarn-server-resourcemanager:
> Compilation failure
> [ERROR]
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-6490/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestQueueCapacityConfigParser.java:[224,80]
> incompatible types: java.lang.String cannot be converted to
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueuePath
> [ERROR] -> [Help 1]
> {code}
> This caused by YARN-11041
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11653) Add Totoal_Memory and Total_Vcores columns in Nodes page
Jiandan Yang created YARN-11653: Summary: Add Totoal_Memory and Total_Vcores columns in Nodes page Key: YARN-11653 URL: https://issues.apache.org/jira/browse/YARN-11653 Project: Hadoop YARN Issue Type: New Feature Reporter: Jiandan Yang Assignee: Jiandan Yang Currently, the RM nodes page includes used and available memory/vcore , but it lacks a total column, which is not intuitive enough for users. When the resource capacities of nodes in the cluster vary widely, we may need to sort the nodes to facilitate the comparison of metrics among same types of nodes. Therefore, it is necessary to add columns for total CPU/memory on the nodes page. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9199) Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
Jiandan Yang created YARN-9199:
---
Summary: Compatible issue: AM throws NoSuchMethodError when 2.7.2
client submits mr job to 3.1.1 RM
Key: YARN-9199
URL: https://issues.apache.org/jira/browse/YARN-9199
Project: Hadoop YARN
Issue Type: Improvement
Components: yarn
Affects Versions: 3.1.1
Reporter: Jiandan Yang
Assignee: Jiandan Yang
AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
{code:java}
2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
at
org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
at org.apache.hadoop.ipc.Client.call(Client.java:1503)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy80.allocate(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:253)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy81.allocate(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:684)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9199) Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
[
https://issues.apache.org/jira/browse/YARN-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9199:
Description:
*Background*
Rolling upgrade Yarn from 2.7.2 to 3.1.1; the version of RM and some NM is
3.1.1, but the version of some NM is 2.7.2, some AM containers may start in the
NM of old version.
AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
{code:java}
2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
at
org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
at org.apache.hadoop.ipc.Client.call(Client.java:1503)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy80.allocate(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:253)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy81.allocate(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:684)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
at java.lang.Thread.run(Thread.java:745)
{code}
was:
AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
{code:java}
2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
at
org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBSe
[jira] [Commented] (YARN-9199) Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
[
https://issues.apache.org/jira/browse/YARN-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16741893#comment-16741893
]
Jiandan Yang commented on YARN-9199:
-
this problem is introduced by YARN-7974, I think RM should check existence of
method *getTrackingUrl*
> Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr
> job to 3.1.1 RM
> --
>
> Key: YARN-9199
> URL: https://issues.apache.org/jira/browse/YARN-9199
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
>
> AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
> {code:java}
> 2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
> org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
> org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
> at org.apache.hadoop.ipc.Client.call(Client.java:1503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy80.allocate(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:253)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy81.allocate(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:684)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9199) Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
[
https://issues.apache.org/jira/browse/YARN-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9199:
Description:
*Background*
Rolling upgrade Yarn from 2.7.2 to 3.1.1; the version of RM is 3.1.1, but the
version of NM is 2.7.2
AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
{code:java}
2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
at
org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
at org.apache.hadoop.ipc.Client.call(Client.java:1503)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy80.allocate(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:253)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy81.allocate(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:684)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
at java.lang.Thread.run(Thread.java:745)
{code}
was:
*Background*
Rolling upgrade Yarn from 2.7.2 to 3.1.1; the version of RM is 3.1.1, but the
version of NM is 2.7.2
AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
{code:java}
2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
at
org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
at
org.apache.hadoop.yarn.api.im
[jira] [Updated] (YARN-9199) Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
[
https://issues.apache.org/jira/browse/YARN-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9199:
Description:
*Background*
Rolling upgrade Yarn from 2.7.2 to 3.1.1; the version of RM is 3.1.1, but the
version of NM is 2.7.2
AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
{code:java}
2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
at
org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
at org.apache.hadoop.ipc.Client.call(Client.java:1503)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy80.allocate(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:253)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy81.allocate(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:684)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
at java.lang.Thread.run(Thread.java:745)
{code}
was:
*Background*
Rolling upgrade Yarn from 2.7.2 to 3.1.1; the version of RM and some NM is
3.1.1, but the version of some NM is 2.7.2, some AM containers may start in the
NM of old version.
AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
{code:java}
2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
at
org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(App
[jira] [Updated] (YARN-9199) Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
[
https://issues.apache.org/jira/browse/YARN-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9199:
Attachment: YARN-9199.001.patch
> Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr
> job to 3.1.1 RM
> --
>
> Key: YARN-9199
> URL: https://issues.apache.org/jira/browse/YARN-9199
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9199.001.patch
>
>
> *Background*
> Rolling upgrade Yarn from 2.7.2 to 3.1.1; the version of RM is 3.1.1, but the
> version of NM is 2.7.2
> AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
> {code:java}
> 2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
> org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
> org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
> at org.apache.hadoop.ipc.Client.call(Client.java:1503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy80.allocate(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:253)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy81.allocate(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:684)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9199) Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
[
https://issues.apache.org/jira/browse/YARN-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9199:
Affects Version/s: (was: 3.1.1)
3.1.3
> Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr
> job to 3.1.1 RM
> --
>
> Key: YARN-9199
> URL: https://issues.apache.org/jira/browse/YARN-9199
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9199.001.patch
>
>
> *Background*
> Rolling upgrade Yarn from 2.7.2 to 3.1.1; the version of RM is 3.1.1, but the
> version of NM is 2.7.2
> AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
> {code:java}
> 2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
> org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
> org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
> at org.apache.hadoop.ipc.Client.call(Client.java:1503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy80.allocate(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:253)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy81.allocate(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:684)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9199) Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
[
https://issues.apache.org/jira/browse/YARN-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16742711#comment-16742711
]
Jiandan Yang commented on YARN-9199:
-
Thanks [~bibinchundatt] for your comment, The new version of hadoop I installed
is hadoop-3.1.3-snapshot pulled from branch-3.1, and I will change the title of
this jira.
[~bibinchundatt], Could you help me review this patch?
> Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr
> job to 3.1.1 RM
> --
>
> Key: YARN-9199
> URL: https://issues.apache.org/jira/browse/YARN-9199
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9199.001.patch
>
>
> *Background*
> Rolling upgrade Yarn from 2.7.2 to 3.1.1; the version of RM is 3.1.1, but the
> version of NM is 2.7.2
> AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
> {code:java}
> 2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
> org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
> org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
> at org.apache.hadoop.ipc.Client.call(Client.java:1503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy80.allocate(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:253)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy81.allocate(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:684)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9199) Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.3 RM
[
https://issues.apache.org/jira/browse/YARN-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9199:
Summary: Compatible issue: AM throws NoSuchMethodError when 2.7.2 client
submits mr job to 3.1.3 RM (was: Compatible issue: AM throws NoSuchMethodError
when 2.7.2 client submits mr job to 3.1.1 RM)
> Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr
> job to 3.1.3 RM
> --
>
> Key: YARN-9199
> URL: https://issues.apache.org/jira/browse/YARN-9199
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.2.0, 3.1.2, 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9199.001.patch
>
>
> *Background*
> Rolling upgrade Yarn from 2.7.2 to 3.1.1; the version of RM is 3.1.1, but the
> version of NM is 2.7.2
> AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
> {code:java}
> 2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
> org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
> org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
> at org.apache.hadoop.ipc.Client.call(Client.java:1503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy80.allocate(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:253)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy81.allocate(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:684)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9199) Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.3 RM
[
https://issues.apache.org/jira/browse/YARN-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9199:
Description:
*Background*
Rolling upgrade Yarn from 2.7.2 to 3.1.3-SNAPSHOT; the version of RM is
3.1.3-SNAPSHOT, but the version of NM is 2.7.2
AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.3 RM
{code:java}
2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
at
org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
at org.apache.hadoop.ipc.Client.call(Client.java:1503)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy80.allocate(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:253)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy81.allocate(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:684)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
at java.lang.Thread.run(Thread.java:745)
{code}
was:
*Background*
Rolling upgrade Yarn from 2.7.2 to 3.1.1; the version of RM is 3.1.1, but the
version of NM is 2.7.2
AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
{code:java}
2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
at
org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
at
org.apache
[jira] [Updated] (YARN-9199) Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
[
https://issues.apache.org/jira/browse/YARN-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9199:
Affects Version/s: 3.1.2
3.2.0
> Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr
> job to 3.1.1 RM
> --
>
> Key: YARN-9199
> URL: https://issues.apache.org/jira/browse/YARN-9199
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.2.0, 3.1.2, 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9199.001.patch
>
>
> *Background*
> Rolling upgrade Yarn from 2.7.2 to 3.1.1; the version of RM is 3.1.1, but the
> version of NM is 2.7.2
> AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.1 RM
> {code:java}
> 2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
> org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
> org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
> at org.apache.hadoop.ipc.Client.call(Client.java:1503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy80.allocate(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:253)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy81.allocate(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:684)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9199) Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.3 RM
[
https://issues.apache.org/jira/browse/YARN-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743560#comment-16743560
]
Jiandan Yang commented on YARN-9199:
-
Thank [~leftnoteasy] very much.
You are right, I first deploy RM with hadoop-3.1.1, and override
resourcemanager jar using 3.1.3-snapshot.
The problem is resolved when using the whole version of hadoop-3.1.3-snapshot
to start RM.
I realize that PB can cover adding fields after reading your comment.
> Compatible issue: AM throws NoSuchMethodError when 2.7.2 client submits mr
> job to 3.1.3 RM
> --
>
> Key: YARN-9199
> URL: https://issues.apache.org/jira/browse/YARN-9199
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.2.0, 3.1.2, 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9199.001.patch
>
>
> *Background*
> Rolling upgrade Yarn from 2.7.2 to 3.1.3-SNAPSHOT; the version of RM is
> 3.1.3-SNAPSHOT, but the version of NM is 2.7.2
> AM throws NoSuchMethodError when 2.7.2 client submits mr job to 3.1.3 RM
> {code:java}
> 2019-01-14 17:20:36,131 ERROR [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
> org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
> org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.getTrackingUrl()Ljava/lang/String;
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:386)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:201)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
> at org.apache.hadoop.ipc.Client.call(Client.java:1503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy80.allocate(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:253)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy81.allocate(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:684)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop
[jira] [Created] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
Jiandan Yang created YARN-9204:
---
Summary:
yarn.scheduler.capacity..accessible-node-labels..capacity
can not support absolute resource value
Key: YARN-9204
URL: https://issues.apache.org/jira/browse/YARN-9204
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 3.1.3
Reporter: Jiandan Yang
Assignee: Jiandan Yang
When I set *yarn.scheduler.capacity..capacity* and
*yarn.scheduler.capacity..accessible-node-labels..capacity*
to absolute resource value, staring RM fails, and throw following exception,
and after diving into relate code, I found the logic of checking absolute
resource value maybe wrong.
{code:java}
2019-01-17 20:25:45,716 FATAL
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
ResourceManager
java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
at java.lang.Float.parseFloat(Float.java:451)
at
org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
Capacity(CapacitySchedulerConfiguration.java:655)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
(CapacitySchedulerConfiguration.java:670)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
ls.java:135)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
.java:110)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
Queue.java:179)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
:356)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
:323)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
ulerQueueManager.java:275)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
ySchedulerQueueManager.java:158)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
ava:715)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
:360)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
25)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
2019-01-17 20:25:45,719 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9204:
Attachment: YARN-9204.001.patch
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9204.001.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional comma
[jira] [Commented] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745025#comment-16745025
]
Jiandan Yang commented on YARN-9204:
-
I apply patch in my test env and RM can start normally.
[~leftnoteasy] Please help me review this patch.
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9204.001.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
[jira] [Comment Edited] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745025#comment-16745025
]
Jiandan Yang edited comment on YARN-9204 at 1/17/19 1:12 PM:
--
I apply patch in my test env and RM can start normally.
Hi, [~leftnoteasy] Please help me review this patch.
was (Author: yangjiandan):
I apply patch in my test env and RM can start normally.
[~leftnoteasy] Please help me review this patch.
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9204.001.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,71
[jira] [Commented] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745806#comment-16745806
]
Jiandan Yang commented on YARN-9204:
-
Thanks [~leftnoteasy]
I will add a UT and upload v2 patch.
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9204.001.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To un
[jira] [Updated] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9204:
Attachment: YARN-9204.002.patch
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9204.001.patch, YARN-9204.002.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
[jira] [Created] (YARN-9210) YARN UI can not display node info
Jiandan Yang created YARN-9210: --- Summary: YARN UI can not display node info Key: YARN-9210 URL: https://issues.apache.org/jira/browse/YARN-9210 Project: Hadoop YARN Issue Type: Bug Components: yarn Reporter: Jiandan Yang Assignee: Jiandan Yang visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" in the area of "Cluster Nodes Metrics" , but detailed info of node does not display. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9210) YARN UI can not display node info
[ https://issues.apache.org/jira/browse/YARN-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-9210: Attachment: screenshot-1.png > YARN UI can not display node info > - > > Key: YARN-9210 > URL: https://issues.apache.org/jira/browse/YARN-9210 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Attachments: screenshot-1.png > > > visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" > in the area of "Cluster Nodes Metrics" , but detailed info of node does not > display. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9210) YARN UI can not display node info
[ https://issues.apache.org/jira/browse/YARN-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-9210: Description: visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" in the area of "Cluster Nodes Metrics" , but detailed info of node does not display. Just as showed in [#anchor] was: visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" in the area of "Cluster Nodes Metrics" , but detailed info of node does not display. Just as showed in > YARN UI can not display node info > - > > Key: YARN-9210 > URL: https://issues.apache.org/jira/browse/YARN-9210 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Attachments: screenshot-1.png > > > visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" > in the area of "Cluster Nodes Metrics" , but detailed info of node does not > display. > Just as showed in [#anchor] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9210) YARN UI can not display node info
[ https://issues.apache.org/jira/browse/YARN-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-9210: Description: visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" in the area of "Cluster Nodes Metrics" , but detailed info of node does not display. Just as was:visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" in the area of "Cluster Nodes Metrics" , but detailed info of node does not display. > YARN UI can not display node info > - > > Key: YARN-9210 > URL: https://issues.apache.org/jira/browse/YARN-9210 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Attachments: screenshot-1.png > > > visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" > in the area of "Cluster Nodes Metrics" , but detailed info of node does not > display. > Just as -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9210) YARN UI can not display node info
[ https://issues.apache.org/jira/browse/YARN-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-9210: Description: visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" in the area of "Cluster Nodes Metrics" , but detailed info of node does not display. Just as showed in was: visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" in the area of "Cluster Nodes Metrics" , but detailed info of node does not display. Just as > YARN UI can not display node info > - > > Key: YARN-9210 > URL: https://issues.apache.org/jira/browse/YARN-9210 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Attachments: screenshot-1.png > > > visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" > in the area of "Cluster Nodes Metrics" , but detailed info of node does not > display. > Just as showed in -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9210) YARN UI can not display node info
[ https://issues.apache.org/jira/browse/YARN-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-9210: Description: visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" in the area of "Cluster Nodes Metrics" , but detailed info of node does not display. Just as showed in [#screenshot-1.png] was: visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" in the area of "Cluster Nodes Metrics" , but detailed info of node does not display. Just as showed in [#anchor] > YARN UI can not display node info > - > > Key: YARN-9210 > URL: https://issues.apache.org/jira/browse/YARN-9210 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Attachments: screenshot-1.png > > > visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" > in the area of "Cluster Nodes Metrics" , but detailed info of node does not > display. > Just as showed in [#screenshot-1.png] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9210) YARN UI can not display node info
[ https://issues.apache.org/jira/browse/YARN-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-9210: Description: visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" in the area of "Cluster Nodes Metrics" , but detailed info of node does not display. Just as showed in [screenshot-1.png|https://issues.apache.org/jira/secure/attachment/12955358/screenshot-1.png] was: visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" in the area of "Cluster Nodes Metrics" , but detailed info of node does not display. Just as showed in [#screenshot-1.png] > YARN UI can not display node info > - > > Key: YARN-9210 > URL: https://issues.apache.org/jira/browse/YARN-9210 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Attachments: screenshot-1.png > > > visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" > in the area of "Cluster Nodes Metrics" , but detailed info of node does not > display. > Just as showed in > [screenshot-1.png|https://issues.apache.org/jira/secure/attachment/12955358/screenshot-1.png] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9210) YARN UI can not display node info
[
https://issues.apache.org/jira/browse/YARN-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745922#comment-16745922
]
Jiandan Yang edited comment on YARN-9210 at 1/18/19 8:55 AM:
--
Viewing source code of webpage I found nodeTableData was abnormal
{code:java}
var
nodeTableData=[\n[\"yarn_test\",\"/default-rack\",\"RUNNING\",\"nm_hostname:8041\",\"nm_hostname:8042\",\"
Fri Jan 18 16:33:09 +0800 2019\",\"\",\"0\",\"\",\"
0 B\",\"
40 GB\",\"0\",\"48\",\"3.1.3-SNAPSHOT\"]\n]
{code}
was (Author: yangjiandan):
Viewing source code of webpage I found nodeTableData was abnormal
{code:java}
var
nodeTableData=[\n[\"yarn_test\",\"/default-rack\",\"RUNNING\",\"bigdata-nmg-hdfstest10.nmg01.diditaxi.com:8041\",\"bigdata-nmg-hdfstest10.nmg01.diditaxi.com:8042\",\"
Fri Jan 18 16:33:09 +0800 2019\",\"\",\"0\",\"\",\"
0 B\",\"
40 GB\",\"0\",\"48\",\"3.1.3-SNAPSHOT\"]\n]
{code}
> YARN UI can not display node info
> -
>
> Key: YARN-9210
> URL: https://issues.apache.org/jira/browse/YARN-9210
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: screenshot-1.png
>
>
> visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes"
> in the area of "Cluster Nodes Metrics" , but detailed info of node does not
> display.
> Just as showed in
> [screenshot-1.png|https://issues.apache.org/jira/secure/attachment/12955358/screenshot-1.png]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9210) YARN UI can not display node info
[ https://issues.apache.org/jira/browse/YARN-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-9210: Attachment: YARN-9210.001.patch > YARN UI can not display node info > - > > Key: YARN-9210 > URL: https://issues.apache.org/jira/browse/YARN-9210 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Attachments: YARN-9210.001.patch, screenshot-1.png > > > visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" > in the area of "Cluster Nodes Metrics" , but detailed info of node does not > display. > Just as showed in > [screenshot-1.png|https://issues.apache.org/jira/secure/attachment/12955358/screenshot-1.png] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9210) YARN UI can not display node info
[
https://issues.apache.org/jira/browse/YARN-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745922#comment-16745922
]
Jiandan Yang commented on YARN-9210:
-
Viewing source code of webpage I found nodeTableData was abnormal
{code:java}
var
nodeTableData=[\n[\"yarn_test\",\"/default-rack\",\"RUNNING\",\"bigdata-nmg-hdfstest10.nmg01.diditaxi.com:8041\",\"bigdata-nmg-hdfstest10.nmg01.diditaxi.com:8042\",\"
Fri Jan 18 16:33:09 +0800 2019\",\"\",\"0\",\"\",\"
0 B\",\"
40 GB\",\"0\",\"48\",\"3.1.3-SNAPSHOT\"]\n]
{code}
> YARN UI can not display node info
> -
>
> Key: YARN-9210
> URL: https://issues.apache.org/jira/browse/YARN-9210
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: screenshot-1.png
>
>
> visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes"
> in the area of "Cluster Nodes Metrics" , but detailed info of node does not
> display.
> Just as showed in
> [screenshot-1.png|https://issues.apache.org/jira/secure/attachment/12955358/screenshot-1.png]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9204:
Attachment: YARN-9204.003.patch
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9204.001.patch, YARN-9204.002.patch,
> YARN-9204.003.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsub
[jira] [Commented] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16746062#comment-16746062
]
Jiandan Yang commented on YARN-9204:
-
rebase trunk and upload patch [YARN-9204.003.patch]
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9204.001.patch, YARN-9204.002.patch,
> YARN-9204.003.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
[jira] [Comment Edited] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16746062#comment-16746062
]
Jiandan Yang edited comment on YARN-9204 at 1/18/19 9:21 AM:
--
rebase trunk and upload patch
[YARN-9204.003.patch|https://issues.apache.org/jira/secure/attachment/12955373/YARN-9204.003.patch]
was (Author: yangjiandan):
rebase trunk and upload patch [YARN-9204.003.patch]
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9204.001.patch, YARN-9204.002.patch,
> YARN-9204.003.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-
[jira] [Commented] (YARN-9142) UI cluster nodes page is broken
[ https://issues.apache.org/jira/browse/YARN-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16746089#comment-16746089 ] Jiandan Yang commented on YARN-9142: - I encounter the same problem in hadoop-3.1.3-SNAPSHOT, and trunk also have problem after read through related code. Sorry for delaying to know this jira, and I have file YARN-9210 to fix this problem. > UI cluster nodes page is broken > --- > > Key: YARN-9142 > URL: https://issues.apache.org/jira/browse/YARN-9142 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Akhil PB >Priority: Critical > Attachments: ClusterNodePage.png, > cluster-nodes-page-hadoop-3.3.0-SNAPSHOT.png > > > It is observed in trunk build YARN cluster node pages is broken even though > data exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16746837#comment-16746837
]
Jiandan Yang commented on YARN-9204:
-
fix checkstyle error and upload v4 patch
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9204.001.patch, YARN-9204.002.patch,
> YARN-9204.003.patch, YARN-9204.004.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---
[jira] [Updated] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9204:
Attachment: YARN-9204.004.patch
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9204.001.patch, YARN-9204.002.patch,
> YARN-9204.003.patch, YARN-9204.004.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-ma
[jira] [Updated] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9204:
Attachment: YARN-9204.005.patch
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9204.001.patch, YARN-9204.002.patch,
> YARN-9204.003.patch, YARN-9204.004.patch, YARN-9204.005.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
[jira] [Commented] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16747046#comment-16747046
]
Jiandan Yang commented on YARN-9204:
-
unit is -1, but I can not find failed ut from testReport.
[~leftnoteasy] Do you know how to find it?
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9204.001.patch, YARN-9204.002.patch,
> YARN-9204.003.patch, YARN-9204.004.patch, YARN-9204.005.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
Th
[jira] [Commented] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16747682#comment-16747682
]
Jiandan Yang commented on YARN-9204:
-
Thank [~cheersyang] for your review comment, I'll update code in v6 patch
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Blocker
> Attachments: YARN-9204.001.patch, YARN-9204.002.patch,
> YARN-9204.003.patch, YARN-9204.004.patch, YARN-9204.005.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by At
[jira] [Updated] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9204:
Attachment: YARN-9204.006.patch
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Blocker
> Attachments: YARN-9204.001.patch, YARN-9204.002.patch,
> YARN-9204.003.patch, YARN-9204.004.patch, YARN-9204.005.patch,
> YARN-9204.006.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---
[jira] [Updated] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9204:
Attachment: (was: YARN-9204.006.patch)
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Blocker
> Attachments: YARN-9204.001.patch, YARN-9204.002.patch,
> YARN-9204.003.patch, YARN-9204.004.patch, YARN-9204.005.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (YARN-9204) yarn.scheduler.capacity..accessible-node-labels..capacity can not support absolute resource value
[
https://issues.apache.org/jira/browse/YARN-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9204:
Attachment: YARN-9204.006.patch
>
> yarn.scheduler.capacity..accessible-node-labels..capacity
> can not support absolute resource value
> --
>
> Key: YARN-9204
> URL: https://issues.apache.org/jira/browse/YARN-9204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.3
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Blocker
> Attachments: YARN-9204.001.patch, YARN-9204.002.patch,
> YARN-9204.003.patch, YARN-9204.004.patch, YARN-9204.005.patch,
> YARN-9204.006.patch
>
>
> When I set *yarn.scheduler.capacity..capacity* and
> *yarn.scheduler.capacity..accessible-node-labels..capacity*
> to absolute resource value, staring RM fails, and throw following
> exception, and after diving into relate code, I found the logic of checking
> absolute resource value maybe wrong.
> {code:java}
> 2019-01-17 20:25:45,716 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
> Capacity(CapacitySchedulerConfiguration.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
> (CapacitySchedulerConfiguration.java:670)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
> ls.java:135)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
> .java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
> Queue.java:179)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :356)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
> :323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
> ulerQueueManager.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
> ySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
> ava:715)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
> :360)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
> 25)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
> 2019-01-17 20:25:45,719 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---
[jira] [Updated] (YARN-9210) RM nodes web page can not display node info
[ https://issues.apache.org/jira/browse/YARN-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-9210: Attachment: YARN-9210-branch-2.9.patch > RM nodes web page can not display node info > --- > > Key: YARN-9210 > URL: https://issues.apache.org/jira/browse/YARN-9210 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.1.2, 3.3.0 >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Blocker > Fix For: 3.0.4, 3.1.2, 3.3.0, 3.2.1, 3.1.3 > > Attachments: YARN-9210-branch-2.9.patch, YARN-9210.001.patch, > screenshot-1.png > > > visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" > in the area of "Cluster Nodes Metrics" , but detailed info of node does not > display. > Just as showed in > [screenshot-1.png|https://issues.apache.org/jira/secure/attachment/12955358/screenshot-1.png] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9210) RM nodes web page can not display node info
[ https://issues.apache.org/jira/browse/YARN-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-9210: Attachment: YARN-9210-branch-2.patch > RM nodes web page can not display node info > --- > > Key: YARN-9210 > URL: https://issues.apache.org/jira/browse/YARN-9210 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.1.2, 3.3.0 >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Blocker > Fix For: 3.0.4, 3.1.2, 3.3.0, 3.2.1, 3.1.3 > > Attachments: YARN-9210-branch-2.9.patch, YARN-9210-branch-2.patch, > YARN-9210.001.patch, screenshot-1.png > > > visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" > in the area of "Cluster Nodes Metrics" , but detailed info of node does not > display. > Just as showed in > [screenshot-1.png|https://issues.apache.org/jira/secure/attachment/12955358/screenshot-1.png] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9210) RM nodes web page can not display node info
[ https://issues.apache.org/jira/browse/YARN-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748444#comment-16748444 ] Jiandan Yang commented on YARN-9210: - Thanks [~cheersyang] for your commit, I've upload patch for branch-2 and branch-2.9. > RM nodes web page can not display node info > --- > > Key: YARN-9210 > URL: https://issues.apache.org/jira/browse/YARN-9210 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.1.2, 3.3.0 >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Blocker > Fix For: 3.0.4, 3.1.2, 3.3.0, 3.2.1, 3.1.3 > > Attachments: YARN-9210-branch-2.9.patch, YARN-9210-branch-2.patch, > YARN-9210.001.patch, screenshot-1.png > > > visting http://rm_hostname:8088/cluster/nodes, there are one "Active Nodes" > in the area of "Cluster Nodes Metrics" , but detailed info of node does not > display. > Just as showed in > [screenshot-1.png|https://issues.apache.org/jira/secure/attachment/12955358/screenshot-1.png] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9225) ACL checking invalidates when setting yarn.acl.enable=true
Jiandan Yang created YARN-9225:
---
Summary: ACL checking invalidates when setting yarn.acl.enable=true
Key: YARN-9225
URL: https://issues.apache.org/jira/browse/YARN-9225
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Reporter: Jiandan Yang
Assignee: Jiandan Yang
my configuration is:
yarn-site.xml: set scheduler is CapacityScheduler and enable acl
{code:java}
yarn.acl.enable
true
yarn.admin.acl
yarn.resourcemanager.scheduler.class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
{code}
capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
{code:java}
yarn.scheduler.capacity.root.queues
default,test
yarn.scheduler.capacity.root.default.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.default.maximum-capacity
[memory=409600,vcores=480]
yarn.scheduler.capacity.root.default.acl_submit_applications
yarn
yarn.scheduler.capacity.root.default.acl_administer_queue
yarn
yarn.scheduler.capacity.root.test.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.test.maximum-capacity
[memory=409600,vcores=480]
*yarn.scheduler.capacity.root.test.acl_submit_applications*
yarn
yarn.scheduler.capacity.root.test.acl_administer_queue
yarn
{code}
I submit MR job into test queue using username of yangjiandan successfully.
I check related code and found the root cause is
ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
queue when acl checking of leaf queue fails, but acl of root queue is *ALL_ACL*
in CapacitySchedulerConfiguration#getAcl, so acl checking can always pass.
{code:java}
private boolean checkPermissionInternal(AccessType accessType,
PrivilegedEntity target, UserGroupInformation user) {
boolean ret = false;
Map acls = allAcls.get(target);
if (acls != null) {
AccessControlList list = acls.get(accessType);
if (list != null) {
ret = list.isUserAllowed(user);
}
}
// recursively look up the queue to see if parent queue has the permission.
if (target.getType() == EntityType.QUEUE && !ret) {
String queueName = target.getName();
if (!queueName.contains(".")) {
return ret;
}
String parentQueueName =
queueName.substring(0, queueName.lastIndexOf("."));
return checkPermissionInternal(accessType,
new PrivilegedEntity(target.getType(), parentQueueName), user);
}
return ret;
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9225) ACL checking invalidates when setting yarn.acl.enable=true
[
https://issues.apache.org/jira/browse/YARN-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749833#comment-16749833
]
Jiandan Yang commented on YARN-9225:
-
I think the default acl of queue should ALL_ACL, and does not need to check
parent queue in checkPermissionInternal
> ACL checking invalidates when setting yarn.acl.enable=true
> --
>
> Key: YARN-9225
> URL: https://issues.apache.org/jira/browse/YARN-9225
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
>
> my configuration is:
> yarn-site.xml: set scheduler is CapacityScheduler and enable acl
> {code:java}
>
> yarn.acl.enable
> true
>
>
> yarn.admin.acl
>
>
>
> yarn.resourcemanager.scheduler.class
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
>
> {code}
> capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
> {code:java}
>
> yarn.scheduler.capacity.root.queues
> default,test
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=409600,vcores=480]
>
>
> yarn.scheduler.capacity.root.default.acl_submit_applications
> yarn
>
>
> yarn.scheduler.capacity.root.default.acl_administer_queue
> yarn
>
>
> yarn.scheduler.capacity.root.test.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.test.maximum-capacity
> [memory=409600,vcores=480]
>
>
> *yarn.scheduler.capacity.root.test.acl_submit_applications*
> yarn
>
>
> yarn.scheduler.capacity.root.test.acl_administer_queue
> yarn
>
> {code}
> I submit MR job into test queue using username of yangjiandan successfully.
> I check related code and found the root cause is
> ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
> queue when acl checking of leaf queue fails, but acl of root queue is
> *ALL_ACL* in CapacitySchedulerConfiguration#getAcl, so acl checking can
> always pass.
> {code:java}
> private boolean checkPermissionInternal(AccessType accessType,
> PrivilegedEntity target, UserGroupInformation user) {
> boolean ret = false;
> Map acls = allAcls.get(target);
> if (acls != null) {
> AccessControlList list = acls.get(accessType);
> if (list != null) {
> ret = list.isUserAllowed(user);
> }
> }
> // recursively look up the queue to see if parent queue has the
> permission.
> if (target.getType() == EntityType.QUEUE && !ret) {
> String queueName = target.getName();
> if (!queueName.contains(".")) {
> return ret;
> }
> String parentQueueName =
> queueName.substring(0, queueName.lastIndexOf("."));
> return checkPermissionInternal(accessType,
> new PrivilegedEntity(target.getType(), parentQueueName), user);
> }
> return ret;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9225) ACL checking fails when setting yarn.acl.enable=true
[
https://issues.apache.org/jira/browse/YARN-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9225:
Summary: ACL checking fails when setting yarn.acl.enable=true (was: ACL
checking invalidates when setting yarn.acl.enable=true)
> ACL checking fails when setting yarn.acl.enable=true
>
>
> Key: YARN-9225
> URL: https://issues.apache.org/jira/browse/YARN-9225
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
>
> my configuration is:
> yarn-site.xml: set scheduler is CapacityScheduler and enable acl
> {code:java}
>
> yarn.acl.enable
> true
>
>
> yarn.admin.acl
>
>
>
> yarn.resourcemanager.scheduler.class
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
>
> {code}
> capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
> {code:java}
>
> yarn.scheduler.capacity.root.queues
> default,test
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=409600,vcores=480]
>
>
> yarn.scheduler.capacity.root.default.acl_submit_applications
> yarn
>
>
> yarn.scheduler.capacity.root.default.acl_administer_queue
> yarn
>
>
> yarn.scheduler.capacity.root.test.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.test.maximum-capacity
> [memory=409600,vcores=480]
>
>
> *yarn.scheduler.capacity.root.test.acl_submit_applications*
> yarn
>
>
> yarn.scheduler.capacity.root.test.acl_administer_queue
> yarn
>
> {code}
> I submit MR job into test queue using username of yangjiandan successfully.
> I check related code and found the root cause is
> ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
> queue when acl checking of leaf queue fails, but acl of root queue is
> *ALL_ACL* in CapacitySchedulerConfiguration#getAcl, so acl checking can
> always pass.
> {code:java}
> private boolean checkPermissionInternal(AccessType accessType,
> PrivilegedEntity target, UserGroupInformation user) {
> boolean ret = false;
> Map acls = allAcls.get(target);
> if (acls != null) {
> AccessControlList list = acls.get(accessType);
> if (list != null) {
> ret = list.isUserAllowed(user);
> }
> }
> // recursively look up the queue to see if parent queue has the
> permission.
> if (target.getType() == EntityType.QUEUE && !ret) {
> String queueName = target.getName();
> if (!queueName.contains(".")) {
> return ret;
> }
> String parentQueueName =
> queueName.substring(0, queueName.lastIndexOf("."));
> return checkPermissionInternal(accessType,
> new PrivilegedEntity(target.getType(), parentQueueName), user);
> }
> return ret;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9225) ACL checking fails when setting yarn.acl.enable=true
[
https://issues.apache.org/jira/browse/YARN-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9225:
Attachment: YARN-9225.001.patch
> ACL checking fails when setting yarn.acl.enable=true
>
>
> Key: YARN-9225
> URL: https://issues.apache.org/jira/browse/YARN-9225
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9225.001.patch
>
>
> my configuration is:
> yarn-site.xml: set scheduler is CapacityScheduler and enable acl
> {code:java}
>
> yarn.acl.enable
> true
>
>
> yarn.admin.acl
>
>
>
> yarn.resourcemanager.scheduler.class
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
>
> {code}
> capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
> {code:java}
>
> yarn.scheduler.capacity.root.queues
> default,test
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=409600,vcores=480]
>
>
> yarn.scheduler.capacity.root.default.acl_submit_applications
> yarn
>
>
> yarn.scheduler.capacity.root.default.acl_administer_queue
> yarn
>
>
> yarn.scheduler.capacity.root.test.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.test.maximum-capacity
> [memory=409600,vcores=480]
>
>
> *yarn.scheduler.capacity.root.test.acl_submit_applications*
> yarn
>
>
> yarn.scheduler.capacity.root.test.acl_administer_queue
> yarn
>
> {code}
> I submit MR job into test queue using username of yangjiandan successfully.
> I check related code and found the root cause is
> ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
> queue when acl checking of leaf queue fails, but acl of root queue is
> *ALL_ACL* in CapacitySchedulerConfiguration#getAcl, so acl checking can
> always pass.
> {code:java}
> private boolean checkPermissionInternal(AccessType accessType,
> PrivilegedEntity target, UserGroupInformation user) {
> boolean ret = false;
> Map acls = allAcls.get(target);
> if (acls != null) {
> AccessControlList list = acls.get(accessType);
> if (list != null) {
> ret = list.isUserAllowed(user);
> }
> }
> // recursively look up the queue to see if parent queue has the
> permission.
> if (target.getType() == EntityType.QUEUE && !ret) {
> String queueName = target.getName();
> if (!queueName.contains(".")) {
> return ret;
> }
> String parentQueueName =
> queueName.substring(0, queueName.lastIndexOf("."));
> return checkPermissionInternal(accessType,
> new PrivilegedEntity(target.getType(), parentQueueName), user);
> }
> return ret;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9225) user can submit application even though they are not in the submit&admin acl
[
https://issues.apache.org/jira/browse/YARN-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9225:
Summary: user can submit application even though they are not in the
submit&admin acl (was: ACL checking fails when setting yarn.acl.enable=true)
> user can submit application even though they are not in the submit&admin acl
>
>
> Key: YARN-9225
> URL: https://issues.apache.org/jira/browse/YARN-9225
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9225.001.patch
>
>
> my configuration is:
> yarn-site.xml: set scheduler is CapacityScheduler and enable acl
> {code:java}
>
> yarn.acl.enable
> true
>
>
> yarn.admin.acl
>
>
>
> yarn.resourcemanager.scheduler.class
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
>
> {code}
> capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
> {code:java}
>
> yarn.scheduler.capacity.root.queues
> default,test
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=409600,vcores=480]
>
>
> yarn.scheduler.capacity.root.default.acl_submit_applications
> yarn
>
>
> yarn.scheduler.capacity.root.default.acl_administer_queue
> yarn
>
>
> yarn.scheduler.capacity.root.test.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.test.maximum-capacity
> [memory=409600,vcores=480]
>
>
> *yarn.scheduler.capacity.root.test.acl_submit_applications*
> yarn
>
>
> yarn.scheduler.capacity.root.test.acl_administer_queue
> yarn
>
> {code}
> I submit MR job into test queue using username of yangjiandan successfully.
> I check related code and found the root cause is
> ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
> queue when acl checking of leaf queue fails, but acl of root queue is
> *ALL_ACL* in CapacitySchedulerConfiguration#getAcl, so acl checking can
> always pass.
> {code:java}
> private boolean checkPermissionInternal(AccessType accessType,
> PrivilegedEntity target, UserGroupInformation user) {
> boolean ret = false;
> Map acls = allAcls.get(target);
> if (acls != null) {
> AccessControlList list = acls.get(accessType);
> if (list != null) {
> ret = list.isUserAllowed(user);
> }
> }
> // recursively look up the queue to see if parent queue has the
> permission.
> if (target.getType() == EntityType.QUEUE && !ret) {
> String queueName = target.getName();
> if (!queueName.contains(".")) {
> return ret;
> }
> String parentQueueName =
> queueName.substring(0, queueName.lastIndexOf("."));
> return checkPermissionInternal(accessType,
> new PrivilegedEntity(target.getType(), parentQueueName), user);
> }
> return ret;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9225) user can submit applications even though they are not in the submit&admin acl
[
https://issues.apache.org/jira/browse/YARN-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9225:
Description:
I submit MR job even though username is not in the submit&admin acl.
I check related code and found the root cause is
ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
queue when acl checking of leaf queue fails, but acl of root queue is *ALL_ACL*
in CapacitySchedulerConfiguration#getAcl, so acl checking can always pass.
{code:java}
private boolean checkPermissionInternal(AccessType accessType,
PrivilegedEntity target, UserGroupInformation user) {
boolean ret = false;
Map acls = allAcls.get(target);
if (acls != null) {
AccessControlList list = acls.get(accessType);
if (list != null) {
ret = list.isUserAllowed(user);
}
}
// recursively look up the queue to see if parent queue has the permission.
if (target.getType() == EntityType.QUEUE && !ret) {
String queueName = target.getName();
if (!queueName.contains(".")) {
return ret;
}
String parentQueueName =
queueName.substring(0, queueName.lastIndexOf("."));
return checkPermissionInternal(accessType,
new PrivilegedEntity(target.getType(), parentQueueName), user);
}
return ret;
}
{code}
my configuration is:
yarn-site.xml: set scheduler is CapacityScheduler and enable acl
{code:java}
yarn.acl.enable
true
yarn.admin.acl
yarn.resourcemanager.scheduler.class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
{code}
capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
{code:java}
yarn.scheduler.capacity.root.queues
default,test
yarn.scheduler.capacity.root.default.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.default.maximum-capacity
[memory=409600,vcores=480]
yarn.scheduler.capacity.root.default.acl_submit_applications
yarn
yarn.scheduler.capacity.root.default.acl_administer_queue
yarn
yarn.scheduler.capacity.root.test.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.test.maximum-capacity
[memory=409600,vcores=480]
*yarn.scheduler.capacity.root.test.acl_submit_applications*
yarn
yarn.scheduler.capacity.root.test.acl_administer_queue
yarn
{code}
was:
my configuration is:
yarn-site.xml: set scheduler is CapacityScheduler and enable acl
{code:java}
yarn.acl.enable
true
yarn.admin.acl
yarn.resourcemanager.scheduler.class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
{code}
capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
{code:java}
yarn.scheduler.capacity.root.queues
default,test
yarn.scheduler.capacity.root.default.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.default.maximum-capacity
[memory=409600,vcores=480]
yarn.scheduler.capacity.root.default.acl_submit_applications
yarn
yarn.scheduler.capacity.root.default.acl_administer_queue
yarn
yarn.scheduler.capacity.root.test.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.test.maximum-capacity
[memory=409600,vcores=480]
*yarn.scheduler.capacity.root.test.acl_submit_applications*
yarn
yarn.scheduler.capacity.root.test.acl_administer_queue
yarn
{code}
I submit MR job into test queue using username of yangjiandan successfully.
I check related code and found the root cause is
ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
queue when acl checking of leaf queue fails, but acl of root queue is *ALL_ACL*
in CapacitySchedulerConfiguration#getAcl, so acl checking can always pass.
{code:java}
private boolean checkPermissionInternal(AccessType accessType,
PrivilegedEntity target, UserGroupInformation user) {
boolean ret = false;
Map acls = allAcls.get(target);
if (acls != null) {
AccessControlList list = acls.get(accessType);
if (list != null) {
ret = list.isUserAllowed(user);
}
}
// recursively look up the queue to see if parent queue has the permission.
if (target.getType() == EntityType.QUEUE && !ret) {
String queueName = target.getName();
if (!queueName.contains(".")) {
return ret;
}
String parentQueueName =
queueName.substring(0, queueName.lastIndexOf("."));
return checkPermissionInternal(accessType,
new PrivilegedEntity(target.getType(), parentQueueName), user);
}
return ret;
}
{code}
> user can submit applications even though they are not in the submit&admin acl
> ---
[jira] [Commented] (YARN-9225) user can submit applications even though they are not in the submit&admin acl
[
https://issues.apache.org/jira/browse/YARN-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750614#comment-16750614
]
Jiandan Yang commented on YARN-9225:
-
[~cheersyang], what do you think about my opinion?
> user can submit applications even though they are not in the submit&admin acl
> -
>
> Key: YARN-9225
> URL: https://issues.apache.org/jira/browse/YARN-9225
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9225.001.patch
>
>
> I submit MR job even though username is not in the submit&admin acl.
> I check related code and found the root cause is
> ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
> queue when acl checking of leaf queue fails, but acl of root queue is
> *ALL_ACL* in CapacitySchedulerConfiguration#getAcl, so acl checking can
> always pass.
> {code:java}
> private boolean checkPermissionInternal(AccessType accessType,
> PrivilegedEntity target, UserGroupInformation user) {
> boolean ret = false;
> Map acls = allAcls.get(target);
> if (acls != null) {
> AccessControlList list = acls.get(accessType);
> if (list != null) {
> ret = list.isUserAllowed(user);
> }
> }
> // does it need to check parent queue?
> // recursively look up the queue to see if parent queue has the
> permission.
> if (target.getType() == EntityType.QUEUE && !ret) {
> String queueName = target.getName();
> if (!queueName.contains(".")) {
> return ret;
> }
> String parentQueueName =
> queueName.substring(0, queueName.lastIndexOf("."));
> return checkPermissionInternal(accessType,
> new PrivilegedEntity(target.getType(), parentQueueName), user);
> }
> return ret;
> }
> {code}
> my configuration is:
> yarn-site.xml: set scheduler is CapacityScheduler and enable acl
> {code:java}
>
> yarn.acl.enable
> true
>
>
> yarn.admin.acl
>
>
>
> yarn.resourcemanager.scheduler.class
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
>
> {code}
> capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
> {code:java}
>
> yarn.scheduler.capacity.root.queues
> default,test
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=409600,vcores=480]
>
>
> yarn.scheduler.capacity.root.default.acl_submit_applications
> yarn
>
>
> yarn.scheduler.capacity.root.default.acl_administer_queue
> yarn
>
>
> yarn.scheduler.capacity.root.test.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.test.maximum-capacity
> [memory=409600,vcores=480]
>
>
> *yarn.scheduler.capacity.root.test.acl_submit_applications*
> yarn
>
>
> yarn.scheduler.capacity.root.test.acl_administer_queue
> yarn
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9225) user can submit applications even though they are not in the submit&admin acl
[
https://issues.apache.org/jira/browse/YARN-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9225:
Summary: user can submit applications even though they are not in the
submit&admin acl (was: user can submit application even though they are not in
the submit&admin acl)
> user can submit applications even though they are not in the submit&admin acl
> -
>
> Key: YARN-9225
> URL: https://issues.apache.org/jira/browse/YARN-9225
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9225.001.patch
>
>
> my configuration is:
> yarn-site.xml: set scheduler is CapacityScheduler and enable acl
> {code:java}
>
> yarn.acl.enable
> true
>
>
> yarn.admin.acl
>
>
>
> yarn.resourcemanager.scheduler.class
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
>
> {code}
> capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
> {code:java}
>
> yarn.scheduler.capacity.root.queues
> default,test
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=409600,vcores=480]
>
>
> yarn.scheduler.capacity.root.default.acl_submit_applications
> yarn
>
>
> yarn.scheduler.capacity.root.default.acl_administer_queue
> yarn
>
>
> yarn.scheduler.capacity.root.test.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.test.maximum-capacity
> [memory=409600,vcores=480]
>
>
> *yarn.scheduler.capacity.root.test.acl_submit_applications*
> yarn
>
>
> yarn.scheduler.capacity.root.test.acl_administer_queue
> yarn
>
> {code}
> I submit MR job into test queue using username of yangjiandan successfully.
> I check related code and found the root cause is
> ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
> queue when acl checking of leaf queue fails, but acl of root queue is
> *ALL_ACL* in CapacitySchedulerConfiguration#getAcl, so acl checking can
> always pass.
> {code:java}
> private boolean checkPermissionInternal(AccessType accessType,
> PrivilegedEntity target, UserGroupInformation user) {
> boolean ret = false;
> Map acls = allAcls.get(target);
> if (acls != null) {
> AccessControlList list = acls.get(accessType);
> if (list != null) {
> ret = list.isUserAllowed(user);
> }
> }
> // recursively look up the queue to see if parent queue has the
> permission.
> if (target.getType() == EntityType.QUEUE && !ret) {
> String queueName = target.getName();
> if (!queueName.contains(".")) {
> return ret;
> }
> String parentQueueName =
> queueName.substring(0, queueName.lastIndexOf("."));
> return checkPermissionInternal(accessType,
> new PrivilegedEntity(target.getType(), parentQueueName), user);
> }
> return ret;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9225) user can submit applications even though they are not in the submit&admin acl
[
https://issues.apache.org/jira/browse/YARN-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9225:
Description:
I submit MR job even though username is not in the submit&admin acl.
the admin&submit acl of test queue is yarn, and I submit app using username of
yangjiandan which is not in the acl.
I check related code and found the root cause is
ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
queue when acl checking of leaf queue fails, but acl of root queue is *ALL_ACL*
in CapacitySchedulerConfiguration#getAcl, so acl checking can always pass.
{code:java}
private boolean checkPermissionInternal(AccessType accessType,
PrivilegedEntity target, UserGroupInformation user) {
boolean ret = false;
Map acls = allAcls.get(target);
if (acls != null) {
AccessControlList list = acls.get(accessType);
if (list != null) {
ret = list.isUserAllowed(user);
}
}
// does it need to check parent queue?
// recursively look up the queue to see if parent queue has the permission.
if (target.getType() == EntityType.QUEUE && !ret) {
String queueName = target.getName();
if (!queueName.contains(".")) {
return ret;
}
String parentQueueName =
queueName.substring(0, queueName.lastIndexOf("."));
return checkPermissionInternal(accessType,
new PrivilegedEntity(target.getType(), parentQueueName), user);
}
return ret;
}
{code}
my configuration is:
yarn-site.xml: set scheduler is CapacityScheduler and enable acl
{code:java}
yarn.acl.enable
true
yarn.admin.acl
yarn.resourcemanager.scheduler.class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
{code}
capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
{code:java}
yarn.scheduler.capacity.root.queues
default,test
yarn.scheduler.capacity.root.default.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.default.maximum-capacity
[memory=409600,vcores=480]
yarn.scheduler.capacity.root.default.acl_submit_applications
yarn
yarn.scheduler.capacity.root.default.acl_administer_queue
yarn
yarn.scheduler.capacity.root.test.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.test.maximum-capacity
[memory=409600,vcores=480]
*yarn.scheduler.capacity.root.test.acl_submit_applications*
yarn
yarn.scheduler.capacity.root.test.acl_administer_queue
yarn
{code}
was:
I submit MR job even though username is not in the submit&admin acl.
I check related code and found the root cause is
ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
queue when acl checking of leaf queue fails, but acl of root queue is *ALL_ACL*
in CapacitySchedulerConfiguration#getAcl, so acl checking can always pass.
{code:java}
private boolean checkPermissionInternal(AccessType accessType,
PrivilegedEntity target, UserGroupInformation user) {
boolean ret = false;
Map acls = allAcls.get(target);
if (acls != null) {
AccessControlList list = acls.get(accessType);
if (list != null) {
ret = list.isUserAllowed(user);
}
}
// does it need to check parent queue?
// recursively look up the queue to see if parent queue has the permission.
if (target.getType() == EntityType.QUEUE && !ret) {
String queueName = target.getName();
if (!queueName.contains(".")) {
return ret;
}
String parentQueueName =
queueName.substring(0, queueName.lastIndexOf("."));
return checkPermissionInternal(accessType,
new PrivilegedEntity(target.getType(), parentQueueName), user);
}
return ret;
}
{code}
my configuration is:
yarn-site.xml: set scheduler is CapacityScheduler and enable acl
{code:java}
yarn.acl.enable
true
yarn.admin.acl
yarn.resourcemanager.scheduler.class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
{code}
capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
{code:java}
yarn.scheduler.capacity.root.queues
default,test
yarn.scheduler.capacity.root.default.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.default.maximum-capacity
[memory=409600,vcores=480]
yarn.scheduler.capacity.root.default.acl_submit_applications
yarn
yarn.scheduler.capacity.root.default.acl_administer_queue
yarn
yarn.scheduler.capacity.root.test.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.test.maximum-capacity
[memory=409600,vcores=480]
*yarn.scheduler.capacity.root.test.acl_submit_applications*
yarn
yarn.s
[jira] [Updated] (YARN-9225) user can submit applications even though they are not in the submit&admin acl
[
https://issues.apache.org/jira/browse/YARN-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9225:
Description:
I submit MR job even though username is not in the submit&admin acl.
I check related code and found the root cause is
ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
queue when acl checking of leaf queue fails, but acl of root queue is *ALL_ACL*
in CapacitySchedulerConfiguration#getAcl, so acl checking can always pass.
{code:java}
private boolean checkPermissionInternal(AccessType accessType,
PrivilegedEntity target, UserGroupInformation user) {
boolean ret = false;
Map acls = allAcls.get(target);
if (acls != null) {
AccessControlList list = acls.get(accessType);
if (list != null) {
ret = list.isUserAllowed(user);
}
}
// does it need to check parent queue?
// recursively look up the queue to see if parent queue has the permission.
if (target.getType() == EntityType.QUEUE && !ret) {
String queueName = target.getName();
if (!queueName.contains(".")) {
return ret;
}
String parentQueueName =
queueName.substring(0, queueName.lastIndexOf("."));
return checkPermissionInternal(accessType,
new PrivilegedEntity(target.getType(), parentQueueName), user);
}
return ret;
}
{code}
my configuration is:
yarn-site.xml: set scheduler is CapacityScheduler and enable acl
{code:java}
yarn.acl.enable
true
yarn.admin.acl
yarn.resourcemanager.scheduler.class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
{code}
capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
{code:java}
yarn.scheduler.capacity.root.queues
default,test
yarn.scheduler.capacity.root.default.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.default.maximum-capacity
[memory=409600,vcores=480]
yarn.scheduler.capacity.root.default.acl_submit_applications
yarn
yarn.scheduler.capacity.root.default.acl_administer_queue
yarn
yarn.scheduler.capacity.root.test.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.test.maximum-capacity
[memory=409600,vcores=480]
*yarn.scheduler.capacity.root.test.acl_submit_applications*
yarn
yarn.scheduler.capacity.root.test.acl_administer_queue
yarn
{code}
was:
I submit MR job even though username is not in the submit&admin acl.
I check related code and found the root cause is
ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
queue when acl checking of leaf queue fails, but acl of root queue is *ALL_ACL*
in CapacitySchedulerConfiguration#getAcl, so acl checking can always pass.
{code:java}
private boolean checkPermissionInternal(AccessType accessType,
PrivilegedEntity target, UserGroupInformation user) {
boolean ret = false;
Map acls = allAcls.get(target);
if (acls != null) {
AccessControlList list = acls.get(accessType);
if (list != null) {
ret = list.isUserAllowed(user);
}
}
// recursively look up the queue to see if parent queue has the permission.
if (target.getType() == EntityType.QUEUE && !ret) {
String queueName = target.getName();
if (!queueName.contains(".")) {
return ret;
}
String parentQueueName =
queueName.substring(0, queueName.lastIndexOf("."));
return checkPermissionInternal(accessType,
new PrivilegedEntity(target.getType(), parentQueueName), user);
}
return ret;
}
{code}
my configuration is:
yarn-site.xml: set scheduler is CapacityScheduler and enable acl
{code:java}
yarn.acl.enable
true
yarn.admin.acl
yarn.resourcemanager.scheduler.class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
{code}
capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
{code:java}
yarn.scheduler.capacity.root.queues
default,test
yarn.scheduler.capacity.root.default.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.default.maximum-capacity
[memory=409600,vcores=480]
yarn.scheduler.capacity.root.default.acl_submit_applications
yarn
yarn.scheduler.capacity.root.default.acl_administer_queue
yarn
yarn.scheduler.capacity.root.test.capacity
[memory=40960,vcores=100]
yarn.scheduler.capacity.root.test.maximum-capacity
[memory=409600,vcores=480]
*yarn.scheduler.capacity.root.test.acl_submit_applications*
yarn
yarn.scheduler.capacity.root.test.acl_administer_queue
yarn
{code}
> user can submit applications even though they are not in the submit&admin acl
> ---
[jira] [Commented] (YARN-9225) user can submit applications even though they are not in the submit&admin acl
[
https://issues.apache.org/jira/browse/YARN-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750779#comment-16750779
]
Jiandan Yang commented on YARN-9225:
-
Setting root.acl_submit_applications and root.acl_administer_queue not empty
also can resolve my problem.
I suggest to add some example about configuring acl of queue.
> user can submit applications even though they are not in the submit&admin acl
> -
>
> Key: YARN-9225
> URL: https://issues.apache.org/jira/browse/YARN-9225
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9225.001.patch
>
>
> I submit MR job even though username is not in the submit&admin acl.
> the admin&submit acl of test queue is yarn, and I submit app using username
> of yangjiandan which is not in the acl.
> I check related code and found the root cause is
> ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
> queue when acl checking of leaf queue fails, but acl of root queue is
> *ALL_ACL* in CapacitySchedulerConfiguration#getAcl, so acl checking can
> always pass.
> {code:java}
> private boolean checkPermissionInternal(AccessType accessType,
> PrivilegedEntity target, UserGroupInformation user) {
> boolean ret = false;
> Map acls = allAcls.get(target);
> if (acls != null) {
> AccessControlList list = acls.get(accessType);
> if (list != null) {
> ret = list.isUserAllowed(user);
> }
> }
> // does it need to check parent queue?
> // recursively look up the queue to see if parent queue has the
> permission.
> if (target.getType() == EntityType.QUEUE && !ret) {
> String queueName = target.getName();
> if (!queueName.contains(".")) {
> return ret;
> }
> String parentQueueName =
> queueName.substring(0, queueName.lastIndexOf("."));
> return checkPermissionInternal(accessType,
> new PrivilegedEntity(target.getType(), parentQueueName), user);
> }
> return ret;
> }
> {code}
> my configuration is:
> yarn-site.xml: set scheduler is CapacityScheduler and enable acl
> {code:java}
>
> yarn.acl.enable
> true
>
>
> yarn.admin.acl
>
>
>
> yarn.resourcemanager.scheduler.class
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
>
> {code}
> capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
> {code:java}
>
> yarn.scheduler.capacity.root.queues
> default,test
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=409600,vcores=480]
>
>
> yarn.scheduler.capacity.root.default.acl_submit_applications
> yarn
>
>
> yarn.scheduler.capacity.root.default.acl_administer_queue
> yarn
>
>
> yarn.scheduler.capacity.root.test.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.test.maximum-capacity
> [memory=409600,vcores=480]
>
>
> *yarn.scheduler.capacity.root.test.acl_submit_applications*
> yarn
>
>
> yarn.scheduler.capacity.root.test.acl_administer_queue
> yarn
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9225) user can submit applications even though they are not in the submit&admin acl
[
https://issues.apache.org/jira/browse/YARN-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750779#comment-16750779
]
Jiandan Yang edited comment on YARN-9225 at 1/24/19 6:44 AM:
--
Setting root.acl_submit_applications and root.acl_administer_queue a empty
value also can resolve my problem.
I suggest to add some example about configuring acl of queue.
was (Author: yangjiandan):
Setting root.acl_submit_applications and root.acl_administer_queue not empty
also can resolve my problem.
I suggest to add some example about configuring acl of queue.
> user can submit applications even though they are not in the submit&admin acl
> -
>
> Key: YARN-9225
> URL: https://issues.apache.org/jira/browse/YARN-9225
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9225.001.patch
>
>
> I submit MR job even though username is not in the submit&admin acl.
> the admin&submit acl of test queue is yarn, and I submit app using username
> of yangjiandan which is not in the acl.
> I check related code and found the root cause is
> ConfiguredYarnAuthorizer#checkPermissionInternal, it will look through parent
> queue when acl checking of leaf queue fails, but acl of root queue is
> *ALL_ACL* in CapacitySchedulerConfiguration#getAcl, so acl checking can
> always pass.
> {code:java}
> private boolean checkPermissionInternal(AccessType accessType,
> PrivilegedEntity target, UserGroupInformation user) {
> boolean ret = false;
> Map acls = allAcls.get(target);
> if (acls != null) {
> AccessControlList list = acls.get(accessType);
> if (list != null) {
> ret = list.isUserAllowed(user);
> }
> }
> // does it need to check parent queue?
> // recursively look up the queue to see if parent queue has the
> permission.
> if (target.getType() == EntityType.QUEUE && !ret) {
> String queueName = target.getName();
> if (!queueName.contains(".")) {
> return ret;
> }
> String parentQueueName =
> queueName.substring(0, queueName.lastIndexOf("."));
> return checkPermissionInternal(accessType,
> new PrivilegedEntity(target.getType(), parentQueueName), user);
> }
> return ret;
> }
> {code}
> my configuration is:
> yarn-site.xml: set scheduler is CapacityScheduler and enable acl
> {code:java}
>
> yarn.acl.enable
> true
>
>
> yarn.admin.acl
>
>
>
> yarn.resourcemanager.scheduler.class
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
>
> {code}
> capacity-scheduler.xml set submitAcl and adminAcl of test queue to yarn
> {code:java}
>
> yarn.scheduler.capacity.root.queues
> default,test
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=409600,vcores=480]
>
>
> yarn.scheduler.capacity.root.default.acl_submit_applications
> yarn
>
>
> yarn.scheduler.capacity.root.default.acl_administer_queue
> yarn
>
>
> yarn.scheduler.capacity.root.test.capacity
> [memory=40960,vcores=100]
>
>
> yarn.scheduler.capacity.root.test.maximum-capacity
> [memory=409600,vcores=480]
>
>
> *yarn.scheduler.capacity.root.test.acl_submit_applications*
> yarn
>
>
> yarn.scheduler.capacity.root.test.acl_administer_queue
> yarn
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9237) RM prints a lot of "Cannot get RMApp by appId" log when RM failover
Jiandan Yang created YARN-9237:
---
Summary: RM prints a lot of "Cannot get RMApp by appId" log when
RM failover
Key: YARN-9237
URL: https://issues.apache.org/jira/browse/YARN-9237
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Reporter: Jiandan Yang
I found a lot of following log in active RM log file after doing failover RM
{code:java}
2019-01-24 15:43:58,999 WARN
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Cannot get
RMApp by appId=application_1542178952162_34746156, just added it to
finishedApplications list for cleanup
.
{code}
I looked forward RM logs and find this app had finished before hours
{code:java}
2019-01-23 21:49:55,683 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1542178952162_34746156_01 State change from FINAL_SAVING to
FINISHING
{code}
The reason of RM prints " Cannot get RMApp by appId" is as follows:
1. RM failover
2. NM reports all running apps to RM in register request
3. The running apps are from NMContext, some apps may already finished
4. In my cluster, yarn.log-aggregation-enable=false,
yarn.nodemanager.log.retain-seconds=86400(1day), so app is kept in NMContext
before app has finished for 24 hours
5. My Yarn cluster runs 50k apps per day and 7k nodes, and NM will report many
finished apps to RM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9237) RM prints a lot of "Cannot get RMApp by appId" log when RM failover
[
https://issues.apache.org/jira/browse/YARN-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang reassigned YARN-9237:
---
Assignee: Jiandan Yang
> RM prints a lot of "Cannot get RMApp by appId" log when RM failover
> ---
>
> Key: YARN-9237
> URL: https://issues.apache.org/jira/browse/YARN-9237
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9237.001.patch
>
>
> I found a lot of following log in active RM log file after doing failover RM
> {code:java}
> 2019-01-24 15:43:58,999 WARN
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Cannot get
> RMApp by appId=application_1542178952162_34746156, just added it to
> finishedApplications list for cleanup
> .
> {code}
> I looked forward RM logs and find this app had finished before hours
> {code:java}
> 2019-01-23 21:49:55,683 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> appattempt_1542178952162_34746156_01 State change from FINAL_SAVING to
> FINISHING
> {code}
> The reason of RM prints " Cannot get RMApp by appId" is as follows:
> 1. RM failover
> 2. NM reports all running apps to RM in register request
> 3. The running apps are from NMContext, some apps may already finished
> 4. In my cluster, yarn.log-aggregation-enable=false,
> yarn.nodemanager.log.retain-seconds=86400(1day), so app is kept in NMContext
> before app has finished for 24 hours
> 5. My Yarn cluster runs 50k apps per day and 7k nodes, and NM will report
> many finished apps to RM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
