[jira] [Commented] (YARN-10833) RM logs endpoint vulnerable to clickjacking
[
https://issues.apache.org/jira/browse/YARN-10833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371888#comment-17371888
]
Andras Gyori commented on YARN-10833:
-
Thank you [~bteke] for the patch! Altogether it seems good to me, but I have
the following additions:
* xfsConfigPrefix is null in TestWebApp, hence the NPE.
* WebApps#L337 is already calculated once and named xFrameOptions
* This part should also be more resilient to NPEs.
* In TestRMWithXFSFilter I can see that the default behaviour for
X-FRAME-OPTIONS was DENY. Is it an intentional design decision to change it to
SAME-ORIGIN?
> RM logs endpoint vulnerable to clickjacking
> ---
>
> Key: YARN-10833
> URL: https://issues.apache.org/jira/browse/YARN-10833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10833.001.patch
>
>
> The /logs endpoint is missing the X-FRAME-OPTIONS in the response header,
> even though YARN is configured to do include it. This makes it vulnerable to
> clickjacking.
> {code:java}
> Request URL: http://{{rm_host}}:8088/logs/
> Request Method: GET
> Status Code: 200 OK
> Remote Address: [::1]:8088
> Referrer Policy: strict-origin-when-cross-origin
> HTTP/1.1 200 OK
> Date: Fri, 25 Jun 2021 17:38:38 GMT
> Cache-Control: no-cache
> Expires: Fri, 25 Jun 2021 17:38:38 GMT
> Date: Fri, 25 Jun 2021 17:38:38 GMT
> Pragma: no-cache
> Content-Type: text/html;charset=utf-8
> X-Content-Type-Options: nosniff
> X-XSS-Protection: 1; mode=block
> Content-Length: 469
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10836) Clean up queue config mode methods
Andras Gyori created YARN-10836: --- Summary: Clean up queue config mode methods Key: YARN-10836 URL: https://issues.apache.org/jira/browse/YARN-10836 Project: Hadoop YARN Issue Type: Improvement Components: capacity scheduler Reporter: Andras Gyori Assignee: Szilard Nemeth After YARN-10759 is merged, it would be advisable to refactor long methods inside the different classes. Also the error messages could be improved. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10837) Break down effectiveMinRatio calculation in ResourceConfigMode
Andras Gyori created YARN-10837: --- Summary: Break down effectiveMinRatio calculation in ResourceConfigMode Key: YARN-10837 URL: https://issues.apache.org/jira/browse/YARN-10837 Project: Hadoop YARN Issue Type: Improvement Reporter: Andras Gyori Assignee: Szilard Nemeth In ResourceConfigMode, the effectiveMinRatio resource calculation is hard to understand, not documented and involves long methods. It must be refactored and cleaned up in order to eliminate the future code debt. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10759) Encapsulate queue config modes
[ https://issues.apache.org/jira/browse/YARN-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371935#comment-17371935 ] Andras Gyori commented on YARN-10759: - Thank you [~snemeth] for the thorough review. I have addressed these issues with the following comments: # Fixed # Fixed # Fixed # I have realised that we will need to introduce the RelativeResourceMode as well, which will inherit from ResourceMode. I think that a more generic naming is appropriate here. # I have created a followup Jira to refactor this effectiveMinRatio, because I also think that its convoluted: YARN-10837 ## I would refrain from introducing such a collection as I have tried it and realised that a HashMap is generic enough for this use case, and a custom collection provided no real benefit (not to mention that in a lot of case a ConcurrentHashMap is more appropriate, therefore we would need to choose invariant implementations as well if we were to use a custom collection) The remaining java-doc issues are also fixed. For the rest, a followup Jira was also added YARN-10836. > Encapsulate queue config modes > -- > > Key: YARN-10759 > URL: https://issues.apache.org/jira/browse/YARN-10759 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10759.001.patch, YARN-10759.002.patch, > YARN-10759.003.patch, YARN-10759.004.patch, YARN-10759.005.patch > > > Capacity Scheduler queues have three modes: > * relative/percentage > * weight > * absolute > Most of them have their own: > * validation logic > * config setting logic > * effective capacity calculation logic > These logics can be easily extracted and encapsulated in separate config mode > classes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10833) RM logs endpoint vulnerable to clickjacking
[
https://issues.apache.org/jira/browse/YARN-10833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10833:
-
Attachment: YARN-10833.002.patch
> RM logs endpoint vulnerable to clickjacking
> ---
>
> Key: YARN-10833
> URL: https://issues.apache.org/jira/browse/YARN-10833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10833.001.patch, YARN-10833.002.patch
>
>
> The /logs endpoint is missing the X-FRAME-OPTIONS in the response header,
> even though YARN is configured to do include it. This makes it vulnerable to
> clickjacking.
> {code:java}
> Request URL: http://{{rm_host}}:8088/logs/
> Request Method: GET
> Status Code: 200 OK
> Remote Address: [::1]:8088
> Referrer Policy: strict-origin-when-cross-origin
> HTTP/1.1 200 OK
> Date: Fri, 25 Jun 2021 17:38:38 GMT
> Cache-Control: no-cache
> Expires: Fri, 25 Jun 2021 17:38:38 GMT
> Date: Fri, 25 Jun 2021 17:38:38 GMT
> Pragma: no-cache
> Content-Type: text/html;charset=utf-8
> X-Content-Type-Options: nosniff
> X-XSS-Protection: 1; mode=block
> Content-Length: 469
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10833) RM logs endpoint vulnerable to clickjacking
[
https://issues.apache.org/jira/browse/YARN-10833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371951#comment-17371951
]
Benjamin Teke commented on YARN-10833:
--
[~gandras], thanks for the review. Fixed the test issues, refactored the config
collection, it should be more resilient to NPEs now. As for why the default
changed: the DENY came from the XFrameOptionsFilter but HttpServer2 uses
SAMEORIGIN. This however only impacts the UTs, as yarn-default.xml already
overrode the DENY with SAMEORIGIN.
> RM logs endpoint vulnerable to clickjacking
> ---
>
> Key: YARN-10833
> URL: https://issues.apache.org/jira/browse/YARN-10833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10833.001.patch, YARN-10833.002.patch
>
>
> The /logs endpoint is missing the X-FRAME-OPTIONS in the response header,
> even though YARN is configured to do include it. This makes it vulnerable to
> clickjacking.
> {code:java}
> Request URL: http://{{rm_host}}:8088/logs/
> Request Method: GET
> Status Code: 200 OK
> Remote Address: [::1]:8088
> Referrer Policy: strict-origin-when-cross-origin
> HTTP/1.1 200 OK
> Date: Fri, 25 Jun 2021 17:38:38 GMT
> Cache-Control: no-cache
> Expires: Fri, 25 Jun 2021 17:38:38 GMT
> Date: Fri, 25 Jun 2021 17:38:38 GMT
> Pragma: no-cache
> Content-Type: text/html;charset=utf-8
> X-Content-Type-Options: nosniff
> X-XSS-Protection: 1; mode=block
> Content-Length: 469
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10759) Encapsulate queue config modes
[ https://issues.apache.org/jira/browse/YARN-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Gyori updated YARN-10759: Attachment: YARN-10759.006.patch > Encapsulate queue config modes > -- > > Key: YARN-10759 > URL: https://issues.apache.org/jira/browse/YARN-10759 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10759.001.patch, YARN-10759.002.patch, > YARN-10759.003.patch, YARN-10759.004.patch, YARN-10759.005.patch, > YARN-10759.006.patch > > > Capacity Scheduler queues have three modes: > * relative/percentage > * weight > * absolute > Most of them have their own: > * validation logic > * config setting logic > * effective capacity calculation logic > These logics can be easily extracted and encapsulated in separate config mode > classes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10838) Improve auto queue template parsing performance
Andras Gyori created YARN-10838: --- Summary: Improve auto queue template parsing performance Key: YARN-10838 URL: https://issues.apache.org/jira/browse/YARN-10838 Project: Hadoop YARN Issue Type: Sub-task Reporter: Andras Gyori Assignee: Andras Gyori AutoCreatedQueueTemplate also has multiple call to Configuration#getPropsWithPrefix. It must be eliminated in order to improve the performance on reinitialisation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10839) queueMaxAppsDefault when set blindly caps the root queue's maxRunningApps setting to this value ignoring any individually overriden maxRunningApps setting for child queue
Siddharth Ahuja created YARN-10839:
--
Summary: queueMaxAppsDefault when set blindly caps the root
queue's maxRunningApps setting to this value ignoring any individually
overriden maxRunningApps setting for child queues in FairScheduler
Key: YARN-10839
URL: https://issues.apache.org/jira/browse/YARN-10839
Project: Hadoop YARN
Issue Type: Bug
Reporter: Siddharth Ahuja
[queueMaxAppsDefault|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Allocation_file_format]
sets the default running app limit for queues (including the root queue) which
can be overridden by individual child queues through the maxRunningApps setting.
Consider a simple FairScheduler XML as follows:
{code}
1.0
drf
*
*
1.0
drf
1024000 mb, 1000 vcores
15
2.0
drf
512000 mb, 500 vcores
10
1.0
drf
3
drf
{code}
Here:
* {{queueMaxAppsDefault}} is set to 3 {{maxRunningApps}} by default.
* root queue does not have any maxRunningApps limit set,
* maxRunningApps for child queues - root.A is 15 and for root.B is 10.
>From above, if users wants to submit jobs to root.B, they are (incorrectly)
>capped to 3, not 15 because the root queue (parent) itself is capped to 3
>because of the queueMaxAppsDefault setting.
Users' observations are thus seeing their apps stuck in ACCEPTED state.
Either the above FairScheduler XML should have been rejected by the
ResourceManager, or, the root queue should have been capped to the maximum
maxRunningApps setting defined for a leaf queue.
Possible solution -> If root queue has no maxRunningApps set and
queueMaxAppsDefault is set to a lower value than maxRunningApps for an
individual leaf queue, then, the root queue should implicitly be capped to the
latter, instead of queueMaxAppsDefault.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10839) queueMaxAppsDefault when set blindly caps the root queue's maxRunningApps setting to this value ignoring any individually overriden maxRunningApps setting for child queu
[
https://issues.apache.org/jira/browse/YARN-10839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja reassigned YARN-10839:
--
Assignee: Siddharth Ahuja
> queueMaxAppsDefault when set blindly caps the root queue's maxRunningApps
> setting to this value ignoring any individually overriden maxRunningApps
> setting for child queues in FairScheduler
>
>
> Key: YARN-10839
> URL: https://issues.apache.org/jira/browse/YARN-10839
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
>
> [queueMaxAppsDefault|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Allocation_file_format]
> sets the default running app limit for queues (including the root queue)
> which can be overridden by individual child queues through the maxRunningApps
> setting.
> Consider a simple FairScheduler XML as follows:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1024000 mb, 1000 vcores
> 15
> 2.0
> drf
>
>
> 512000 mb, 500 vcores
> 10
> 1.0
> drf
>
>
> 3
> drf
>
>
>
>
>
> {code}
> Here:
> * {{queueMaxAppsDefault}} is set to 3 {{maxRunningApps}} by default.
> * root queue does not have any maxRunningApps limit set,
> * maxRunningApps for child queues - root.A is 15 and for root.B is 10.
> From above, if users wants to submit jobs to root.B, they are (incorrectly)
> capped to 3, not 15 because the root queue (parent) itself is capped to 3
> because of the queueMaxAppsDefault setting.
> Users' observations are thus seeing their apps stuck in ACCEPTED state.
> Either the above FairScheduler XML should have been rejected by the
> ResourceManager, or, the root queue should have been capped to the maximum
> maxRunningApps setting defined for a leaf queue.
> Possible solution -> If root queue has no maxRunningApps set and
> queueMaxAppsDefault is set to a lower value than maxRunningApps for an
> individual leaf queue, then, the root queue should implicitly be capped to
> the latter, instead of queueMaxAppsDefault.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10839) queueMaxAppsDefault when set blindly caps the root queue's maxRunningApps setting to this value ignoring any individually overriden maxRunningApps setting for child queue
[
https://issues.apache.org/jira/browse/YARN-10839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10839:
---
Component/s: yarn
> queueMaxAppsDefault when set blindly caps the root queue's maxRunningApps
> setting to this value ignoring any individually overriden maxRunningApps
> setting for child queues in FairScheduler
>
>
> Key: YARN-10839
> URL: https://issues.apache.org/jira/browse/YARN-10839
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.7.5, 3.3.1
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
>
> [queueMaxAppsDefault|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Allocation_file_format]
> sets the default running app limit for queues (including the root queue)
> which can be overridden by individual child queues through the maxRunningApps
> setting.
> Consider a simple FairScheduler XML as follows:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1024000 mb, 1000 vcores
> 15
> 2.0
> drf
>
>
> 512000 mb, 500 vcores
> 10
> 1.0
> drf
>
>
> 3
> drf
>
>
>
>
>
> {code}
> Here:
> * {{queueMaxAppsDefault}} is set to 3 {{maxRunningApps}} by default.
> * root queue does not have any maxRunningApps limit set,
> * maxRunningApps for child queues - root.A is 15 and for root.B is 10.
> From above, if users wants to submit jobs to root.B, they are (incorrectly)
> capped to 3, not 15 because the root queue (parent) itself is capped to 3
> because of the queueMaxAppsDefault setting.
> Users' observations are thus seeing their apps stuck in ACCEPTED state.
> Either the above FairScheduler XML should have been rejected by the
> ResourceManager, or, the root queue should have been capped to the maximum
> maxRunningApps setting defined for a leaf queue.
> Possible solution -> If root queue has no maxRunningApps set and
> queueMaxAppsDefault is set to a lower value than maxRunningApps for an
> individual leaf queue, then, the root queue should implicitly be capped to
> the latter, instead of queueMaxAppsDefault.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10839) queueMaxAppsDefault when set blindly caps the root queue's maxRunningApps setting to this value ignoring any individually overriden maxRunningApps setting for child queue
[
https://issues.apache.org/jira/browse/YARN-10839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10839:
---
Labels: scheduler (was: )
> queueMaxAppsDefault when set blindly caps the root queue's maxRunningApps
> setting to this value ignoring any individually overriden maxRunningApps
> setting for child queues in FairScheduler
>
>
> Key: YARN-10839
> URL: https://issues.apache.org/jira/browse/YARN-10839
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.7.5, 3.3.1
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Labels: scheduler
>
> [queueMaxAppsDefault|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Allocation_file_format]
> sets the default running app limit for queues (including the root queue)
> which can be overridden by individual child queues through the maxRunningApps
> setting.
> Consider a simple FairScheduler XML as follows:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1024000 mb, 1000 vcores
> 15
> 2.0
> drf
>
>
> 512000 mb, 500 vcores
> 10
> 1.0
> drf
>
>
> 3
> drf
>
>
>
>
>
> {code}
> Here:
> * {{queueMaxAppsDefault}} is set to 3 {{maxRunningApps}} by default.
> * root queue does not have any maxRunningApps limit set,
> * maxRunningApps for child queues - root.A is 15 and for root.B is 10.
> From above, if users wants to submit jobs to root.B, they are (incorrectly)
> capped to 3, not 15 because the root queue (parent) itself is capped to 3
> because of the queueMaxAppsDefault setting.
> Users' observations are thus seeing their apps stuck in ACCEPTED state.
> Either the above FairScheduler XML should have been rejected by the
> ResourceManager, or, the root queue should have been capped to the maximum
> maxRunningApps setting defined for a leaf queue.
> Possible solution -> If root queue has no maxRunningApps set and
> queueMaxAppsDefault is set to a lower value than maxRunningApps for an
> individual leaf queue, then, the root queue should implicitly be capped to
> the latter, instead of queueMaxAppsDefault.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10839) queueMaxAppsDefault when set blindly caps the root queue's maxRunningApps setting to this value ignoring any individually overriden maxRunningApps setting for child queue
[
https://issues.apache.org/jira/browse/YARN-10839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10839:
---
Affects Version/s: 2.7.5
3.3.1
> queueMaxAppsDefault when set blindly caps the root queue's maxRunningApps
> setting to this value ignoring any individually overriden maxRunningApps
> setting for child queues in FairScheduler
>
>
> Key: YARN-10839
> URL: https://issues.apache.org/jira/browse/YARN-10839
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.5, 3.3.1
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
>
> [queueMaxAppsDefault|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Allocation_file_format]
> sets the default running app limit for queues (including the root queue)
> which can be overridden by individual child queues through the maxRunningApps
> setting.
> Consider a simple FairScheduler XML as follows:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1024000 mb, 1000 vcores
> 15
> 2.0
> drf
>
>
> 512000 mb, 500 vcores
> 10
> 1.0
> drf
>
>
> 3
> drf
>
>
>
>
>
> {code}
> Here:
> * {{queueMaxAppsDefault}} is set to 3 {{maxRunningApps}} by default.
> * root queue does not have any maxRunningApps limit set,
> * maxRunningApps for child queues - root.A is 15 and for root.B is 10.
> From above, if users wants to submit jobs to root.B, they are (incorrectly)
> capped to 3, not 15 because the root queue (parent) itself is capped to 3
> because of the queueMaxAppsDefault setting.
> Users' observations are thus seeing their apps stuck in ACCEPTED state.
> Either the above FairScheduler XML should have been rejected by the
> ResourceManager, or, the root queue should have been capped to the maximum
> maxRunningApps setting defined for a leaf queue.
> Possible solution -> If root queue has no maxRunningApps set and
> queueMaxAppsDefault is set to a lower value than maxRunningApps for an
> individual leaf queue, then, the root queue should implicitly be capped to
> the latter, instead of queueMaxAppsDefault.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
