[jira] [Commented] (YARN-548) Add tests for YarnUncaughtExceptionHandler
[ https://issues.apache.org/jira/browse/YARN-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665736#comment-13665736 ] Jonathan Eagles commented on YARN-548: -- +1. Thanks for the patch, Vadim. Coverage for for YarnUncaughtExceptionHandler is now 88%. Add tests for YarnUncaughtExceptionHandler -- Key: YARN-548 URL: https://issues.apache.org/jira/browse/YARN-548 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Vadim Bondarev Assignee: Vadim Bondarev Attachments: HADOOP-4528-branch-0.23-a.patch, HADOOP-4528-branch-2-a.patch, HADOOP-4528-trunk-a.patch, YARN-548-branch-0.23-a.patch, YARN-548-trunk-a.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-427) Coverage fix for org.apache.hadoop.yarn.server.api.*
[ https://issues.apache.org/jira/browse/YARN-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13668721#comment-13668721 ] Jonathan Eagles commented on YARN-427: -- Thanks for the tests, Aleksey. Test coverage is improved in the following areas *org.apache.hadoop.yarn.server.api.impl.pb.client* (10.3%) - 37.9% *org.apache.hadoop.yarn.server.api.impl.pb.service* (14.3%) - 85.7% *org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb* (0.7%) - 78.4% *org.apache.hadoop.yarn.server.api.records.impl.pb* (0.4%) - 76% Couple of minor comments *TestResourceTrackerPBClientImpl.java* - remove unneeded junit.framework.Assert junit 3 style import - remove junit @Test timeout parameter *TestYarnServerApiClasses.java* - remove junit @Test timeout parameter - copy/paste error update the javadoc for each test function to be specific to the class being tested - verify getVirtualCores in testRegisterNodeManagerRequestPBImpl (trunk and branch-2) - testNodeHeartbeatResponsePBImpl is missing (trunk and branch-2) - testRegisterNodeManagerResponsePBImpl does not test RegistrationResponse (trunk and branch 2) - testHeartbeatResponsePBImpl is missing (trunk and branch-2) Coverage fix for org.apache.hadoop.yarn.server.api.* Key: YARN-427 URL: https://issues.apache.org/jira/browse/YARN-427 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 3.0.0, 0.23.7, 2.0.5-beta Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Attachments: YARN-427-branch-0.23-b.patch, YARN-427-branch-2-a.patch, YARN-427-branch-2-b.patch, YARN-427-branch-2.patch, YARN-427-trunk-a.patch, YARN-427-trunk-b.patch, YARN-427-trunk.patch Coverage fix for org.apache.hadoop.yarn.server.api.* patch YARN-427-trunk.patch for trunk patch YARN-427-branch-2.patch for branch-2 and branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-169) Update log4j.appender.EventCounter to use org.apache.hadoop.log.metrics.EventCounter
[
https://issues.apache.org/jira/browse/YARN-169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated YARN-169:
-
Fix Version/s: 0.23.9
Update log4j.appender.EventCounter to use
org.apache.hadoop.log.metrics.EventCounter
Key: YARN-169
URL: https://issues.apache.org/jira/browse/YARN-169
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Anthony Rojas
Assignee: Anthony Rojas
Priority: Minor
Fix For: 2.0.3-alpha, 0.23.9
Attachments: YARN-169.patch
We should update the log4j.appender.EventCounter in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties
to use *org.apache.hadoop.log.metrics.EventCounter* rather than
*org.apache.hadoop.metrics.jvm.EventCounter* to avoid triggering the
following warning:
{code}WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated.
Please use org.apache.hadoop.log.metrics.EventCounter in all the
log4j.properties files{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-427) Coverage fix for org.apache.hadoop.yarn.server.api.*
[ https://issues.apache.org/jira/browse/YARN-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13681374#comment-13681374 ] Jonathan Eagles commented on YARN-427: -- +1. Thanks, Aleksey. Looks really good now. Coverage fix for org.apache.hadoop.yarn.server.api.* Key: YARN-427 URL: https://issues.apache.org/jira/browse/YARN-427 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 3.0.0, 0.23.7, 2.1.0-beta Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Attachments: YARN-427-branch-0.23-b.patch, YARN-427-branch-0.23-c.patch, YARN-427-branch-2-a.patch, YARN-427-branch-2-b.patch, YARN-427-branch-2.patch, YARN-427-trunk-a.patch, YARN-427-trunk-b.patch, YARN-427-trunk-c.patch, YARN-427-trunk.patch Coverage fix for org.apache.hadoop.yarn.server.api.* patch YARN-427-trunk.patch for trunk patch YARN-427-branch-2.patch for branch-2 and branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-427) Coverage fix for org.apache.hadoop.yarn.server.api.*
[ https://issues.apache.org/jira/browse/YARN-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13681465#comment-13681465 ] Jonathan Eagles commented on YARN-427: -- Good catch, Sid. The intention was just for 2.3. I have corrected fix versions to reflect that. Coverage fix for org.apache.hadoop.yarn.server.api.* Key: YARN-427 URL: https://issues.apache.org/jira/browse/YARN-427 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 3.0.0, 0.23.7, 2.1.0-beta Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Fix For: 3.0.0, 0.23.9, 2.3.0 Attachments: YARN-427-branch-0.23-b.patch, YARN-427-branch-0.23-c.patch, YARN-427-branch-2-a.patch, YARN-427-branch-2-b.patch, YARN-427-branch-2.patch, YARN-427-trunk-a.patch, YARN-427-trunk-b.patch, YARN-427-trunk-c.patch, YARN-427-trunk.patch Coverage fix for org.apache.hadoop.yarn.server.api.* patch YARN-427-trunk.patch for trunk patch YARN-427-branch-2.patch for branch-2 and branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-425) coverage fix for yarn api
[ https://issues.apache.org/jira/browse/YARN-425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685990#comment-13685990 ] Jonathan Eagles commented on YARN-425: -- Hi, Aleksey. I'll be happy to take a look at this patch. However, it looks like the patch for trunk and branch-2 and not up-to-date after YARN-387 got checked in. coverage fix for yarn api - Key: YARN-425 URL: https://issues.apache.org/jira/browse/YARN-425 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 3.0.0, 0.23.7, 2.1.0-beta Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Attachments: YARN-425-branch-0.23-d.patch, YARN-425-branch-0.23.patch, YARN-425-branch-2-b.patch, YARN-425-branch-2-c.patch, YARN-425-branch-2.patch, YARN-425-trunk-a.patch, YARN-425-trunk-b.patch, YARN-425-trunk-c.patch, YARN-425-trunk-d.patch, YARN-425-trunk.patch coverage fix for yarn api patch YARN-425-trunk-a.patch for trunk patch YARN-425-branch-2.patch for branch-2 patch YARN-425-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-478) fix coverage org.apache.hadoop.yarn.webapp.log
[ https://issues.apache.org/jira/browse/YARN-478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689484#comment-13689484 ] Jonathan Eagles commented on YARN-478: -- +1. Looks like some more good additions to coverage, Aleksey. *org.apache.hadoop.yarn.server.security* ApplicationACLsManager 0% - 66.7% *org.apache.hadoop.yarn.webapp.log* AggregatedLogsBlock 0% - 66.7% fix coverage org.apache.hadoop.yarn.webapp.log -- Key: YARN-478 URL: https://issues.apache.org/jira/browse/YARN-478 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Attachments: YARN-478-branch-0.23-v1.patch, YARN-478-trunk.patch, YARN-478-trunk-v1.patch fix coverage org.apache.hadoop.yarn.webapp.log one patch for trunk, branch-2, branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-985) Nodemanager should log where a resource was localized
[ https://issues.apache.org/jira/browse/YARN-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724507#comment-13724507 ] Jonathan Eagles commented on YARN-985: -- +1. I like the simplicity of this patch and agree with the tradeoff of logging in the state transition handler versus the transition itself. Omkar if you feel very strongly, i can be ok with that too. Let me know since I will check this in tomorrow. Jon Nodemanager should log where a resource was localized - Key: YARN-985 URL: https://issues.apache.org/jira/browse/YARN-985 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.9 Reporter: Ravi Prakash Assignee: Ravi Prakash Attachments: YARN-985.patch When a resource is localized, we should log WHERE on the local disk it was localized. This helps in debugging afterwards (e.g. if the disk was to go bad). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1031) JQuery UI components reference external css in branch-23
Jonathan Eagles created YARN-1031: - Summary: JQuery UI components reference external css in branch-23 Key: YARN-1031 URL: https://issues.apache.org/jira/browse/YARN-1031 Project: Hadoop YARN Issue Type: Bug Affects Versions: 0.23.9 Reporter: Jonathan Eagles Assignee: Jonathan Eagles -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1031) JQuery UI components reference external css in branch-23
[ https://issues.apache.org/jira/browse/YARN-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729865#comment-13729865 ] Jonathan Eagles commented on YARN-1031: --- This issue in not present in branch-2 or trunk. Thinking that this will be just a minor bug instead of bringing back entire jquery themes into the source base in branch-0.23 JQuery UI components reference external css in branch-23 Key: YARN-1031 URL: https://issues.apache.org/jira/browse/YARN-1031 Project: Hadoop YARN Issue Type: Bug Affects Versions: 0.23.9 Reporter: Jonathan Eagles Assignee: Jonathan Eagles -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1031) JQuery UI components reference external css in branch-23
[ https://issues.apache.org/jira/browse/YARN-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-1031: -- Attachment: YARN-1031-branch-0.23.patch JQuery UI components reference external css in branch-23 Key: YARN-1031 URL: https://issues.apache.org/jira/browse/YARN-1031 Project: Hadoop YARN Issue Type: Bug Affects Versions: 0.23.9 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-1031-branch-0.23.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-1031) JQuery UI components reference external css in branch-23
[ https://issues.apache.org/jira/browse/YARN-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved YARN-1031. --- Resolution: Fixed Fix Version/s: 0.23.10 Thanks for the review, Jason. JQuery UI components reference external css in branch-23 Key: YARN-1031 URL: https://issues.apache.org/jira/browse/YARN-1031 Project: Hadoop YARN Issue Type: Bug Affects Versions: 0.23.9 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.10 Attachments: YARN-1031-branch-0.23.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-985) Nodemanager should log where a resource was localized
[ https://issues.apache.org/jira/browse/YARN-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731166#comment-13731166 ] Jonathan Eagles commented on YARN-985: -- Looks like we are all happy. Putting this in. Thanks, everybody. Nodemanager should log where a resource was localized - Key: YARN-985 URL: https://issues.apache.org/jira/browse/YARN-985 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.9 Reporter: Ravi Prakash Assignee: Ravi Prakash Attachments: YARN-985.branch-0.23.patch, YARN-985.patch, YARN-985.patch When a resource is localized, we should log WHERE on the local disk it was localized. This helps in debugging afterwards (e.g. if the disk was to go bad). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-649) Make container logs available over HTTP in plain text
[ https://issues.apache.org/jira/browse/YARN-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13762411#comment-13762411 ] Jonathan Eagles commented on YARN-649: -- [~sandyr] TestContainerLogsPage#testContainerLogPageAccess (native-only test) is failing after this check in. I have created YARN-1178 to address this test failure mvn clean test -Pnative -Dtest=TestContainerLogsPage Make container logs available over HTTP in plain text - Key: YARN-649 URL: https://issues.apache.org/jira/browse/YARN-649 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: YARN-649-2.patch, YARN-649-3.patch, YARN-649-4.patch, YARN-649-5.patch, YARN-649-6.patch, YARN-649-7.patch, YARN-649.patch, YARN-752-1.patch It would be good to make container logs available over the REST API for MAPREDUCE-4362 and so that they can be accessed programatically in general. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1178) TestContainerLogsPage#testContainerLogPageAccess is failing
Jonathan Eagles created YARN-1178: - Summary: TestContainerLogsPage#testContainerLogPageAccess is failing Key: YARN-1178 URL: https://issues.apache.org/jira/browse/YARN-1178 Project: Hadoop YARN Issue Type: Bug Reporter: Jonathan Eagles Test is failing after YARN-649. This test is only run in native mode mvn clean test -Pnative -Dtest=TestContainerLogsPage -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1119) Add ClusterMetrics checks to tho TestRMNodeTransitions tests
[ https://issues.apache.org/jira/browse/YARN-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13763404#comment-13763404 ] Jonathan Eagles commented on YARN-1119: --- +1 lgtm. Thanks for the patches, Mit. Add ClusterMetrics checks to tho TestRMNodeTransitions tests Key: YARN-1119 URL: https://issues.apache.org/jira/browse/YARN-1119 Project: Hadoop YARN Issue Type: Test Components: resourcemanager Affects Versions: 3.0.0, 0.23.9, 2.0.6-alpha Reporter: Robert Parker Assignee: Mit Desai Attachments: YARN-1119.patch, YARN-1119-v1-b23.patch YARN-1101 identified an issue where UNHEALTHY nodes could double decrement the active nodes. We should add checks for RUNNING node transitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1176) RM web services ClusterMetricsInfo total nodes doesn't include unhealthy nodes
[ https://issues.apache.org/jira/browse/YARN-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-1176: -- Attachment: YARN-1176.patch RM web services ClusterMetricsInfo total nodes doesn't include unhealthy nodes -- Key: YARN-1176 URL: https://issues.apache.org/jira/browse/YARN-1176 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.0.0, 0.23.9, 2.1.1-beta Reporter: Thomas Graves Assignee: Jonathan Eagles Priority: Critical Attachments: YARN-1176.patch In the web services api for the cluster/metrics, the totalNodes reported doesn't include the unhealthy nodes. this.totalNodes = activeNodes + lostNodes + decommissionedNodes + rebootedNodes; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1199) Make NM/RM Versions Available
[ https://issues.apache.org/jira/browse/YARN-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-1199: -- Attachment: YARN-1199.patch Make NM/RM Versions Available - Key: YARN-1199 URL: https://issues.apache.org/jira/browse/YARN-1199 Project: Hadoop YARN Issue Type: Improvement Reporter: Mit Desai Assignee: Mit Desai Attachments: YARN-1199.patch Now as we have the NM and RM Versions available, we can display the YARN version of nodes running in the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2277: -- Description: As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. was: As the Application Timeline Server is provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063518#comment-14063518 ] Jonathan Eagles commented on YARN-2277: --- Exactly. Cross-origin (perhaps 3rd party) UI could be more easily written. Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072704#comment-14072704 ] Jonathan Eagles commented on YARN-2277: --- [~zjshen], [~vinodkv], do you have any comments or concerns with the approach above? Would like to get some feed back soon since TEZ-8 is basing work off of the CORS patch above. Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-2375) Allow enabling/disabling timeline server per framework
Jonathan Eagles created YARN-2375: - Summary: Allow enabling/disabling timeline server per framework Key: YARN-2375 URL: https://issues.apache.org/jira/browse/YARN-2375 Project: Hadoop YARN Issue Type: Improvement Reporter: Jonathan Eagles -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2277: -- Attachment: YARN-2277-v2.patch Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch, YARN-2277-v2.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082904#comment-14082904 ] Jonathan Eagles commented on YARN-2277: --- Thanks for the thorough feedback, [~zjshen]. I have updated the patch to reflect the updated understand of CORS. In this I have further read on that page that Origin is need to be specifically set to a host in order to use cookie based authentication. Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch, YARN-2277-v2.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2277: -- Attachment: YARN-2277-v3.patch Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch, YARN-2277-v2.patch, YARN-2277-v3.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2277: -- Attachment: YARN-2277-v3.patch Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch, YARN-2277-v2.patch, YARN-2277-v3.patch, YARN-2277-v3.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091304#comment-14091304 ] Jonathan Eagles commented on YARN-2277: --- [~zjshen], Thank you for you feedback as this is going to have a much bigger impact on Hadoop as a whole. I have provided a minimal CORS filter that will give us an idea if this is the direction to go. Based on the direction of this patch, the scope has widened to create a general CrossOriginFilter for use within all Hadoop REST APIs. Probably, we will want to split the different pieces us across JIRAs, umbrella, Filter and FilterInitializer, additional configuration, and individual REST servers. This way we can focus on the end goal of getting Tez UI done in a timely manner without forgetting completeness of CORS support. Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch, YARN-2277-v2.patch, YARN-2277-v3.patch, YARN-2277-v3.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2277: -- Attachment: YARN-2277-v4.patch Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch, YARN-2277-v2.patch, YARN-2277-v3.patch, YARN-2277-v3.patch, YARN-2277-v4.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2277: -- Attachment: YARN-2277-v5.patch Addressing findbugs warning Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch, YARN-2277-v2.patch, YARN-2277-v3.patch, YARN-2277-v3.patch, YARN-2277-v4.patch, YARN-2277-v5.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2277: -- Attachment: YARN-2277-v6.patch Addressed test failure with v6 of the patch. Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch, YARN-2277-v2.patch, YARN-2277-v3.patch, YARN-2277-v3.patch, YARN-2277-v4.patch, YARN-2277-v5.patch, YARN-2277-v6.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2277: -- Attachment: YARN-2277-v7.patch Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch, YARN-2277-v2.patch, YARN-2277-v3.patch, YARN-2277-v3.patch, YARN-2277-v4.patch, YARN-2277-v5.patch, YARN-2277-v6.patch, YARN-2277-v7.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14094772#comment-14094772 ] Jonathan Eagles commented on YARN-2277: --- [~zjshen], addressed you issues. As for chain.doFilter comment, it is the expected order for Filters, including the the order jetty CORS filter I believe. Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch, YARN-2277-v2.patch, YARN-2277-v3.patch, YARN-2277-v3.patch, YARN-2277-v4.patch, YARN-2277-v5.patch, YARN-2277-v6.patch, YARN-2277-v7.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2277) Add Cross-Origin support to the ATS REST API
[ https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14096127#comment-14096127 ] Jonathan Eagles commented on YARN-2277: --- Thanks for the great reviews, [~zjshen]! Add Cross-Origin support to the ATS REST API Key: YARN-2277 URL: https://issues.apache.org/jira/browse/YARN-2277 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 2.6.0 Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch, YARN-2277-v2.patch, YARN-2277-v3.patch, YARN-2277-v3.patch, YARN-2277-v4.patch, YARN-2277-v5.patch, YARN-2277-v6.patch, YARN-2277-v7.patch, YARN-2277-v8.patch As the Application Timeline Server is not provided with built-in UI, it may make sense to enable JSONP or CORS Rest API capabilities to allow for remote UI to access the data directly via javascript without cross side server browser blocks coming into play. Example client may be like http://api.jquery.com/jQuery.getJSON/ This can alleviate the need to create a local proxy cache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2035) FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode
[ https://issues.apache.org/jira/browse/YARN-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104391#comment-14104391 ] Jonathan Eagles commented on YARN-2035: --- [~zjshen], can you take a quick look at this? This has been a little bit of a pain for testing since it can't come up when the namenode is in safemode. FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode --- Key: YARN-2035 URL: https://issues.apache.org/jira/browse/YARN-2035 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2035.patch Small bug that prevents ResourceManager and ApplicationHistoryService from coming up while Namenode is in safemode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2035) FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode
[ https://issues.apache.org/jira/browse/YARN-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104608#comment-14104608 ] Jonathan Eagles commented on YARN-2035: --- In my scenario, the dir already exists and so I don't want to crash trying to create an existing dir. The code you mentioned could be helpful for the first time startup but it's a slightly different scenario I care about. Let me know if you if we should handle that as part of this jira or separately. FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode --- Key: YARN-2035 URL: https://issues.apache.org/jira/browse/YARN-2035 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2035.patch Small bug that prevents ResourceManager and ApplicationHistoryService from coming up while Namenode is in safemode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2035) FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode
[ https://issues.apache.org/jira/browse/YARN-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2035: -- Attachment: YARN-2035-v2.patch FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode --- Key: YARN-2035 URL: https://issues.apache.org/jira/browse/YARN-2035 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2035-v2.patch, YARN-2035.patch Small bug that prevents ResourceManager and ApplicationHistoryService from coming up while Namenode is in safemode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2035) FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode
[ https://issues.apache.org/jira/browse/YARN-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105859#comment-14105859 ] Jonathan Eagles commented on YARN-2035: --- [~zjshen], added the tests you suggested by registering a test file system to inject the failures. FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode --- Key: YARN-2035 URL: https://issues.apache.org/jira/browse/YARN-2035 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2035-v2.patch, YARN-2035.patch Small bug that prevents ResourceManager and ApplicationHistoryService from coming up while Namenode is in safemode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2035) FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode
[ https://issues.apache.org/jira/browse/YARN-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106310#comment-14106310 ] Jonathan Eagles commented on YARN-2035: --- Ignore the last patch as it has bugs. FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode --- Key: YARN-2035 URL: https://issues.apache.org/jira/browse/YARN-2035 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2035-v2.patch, YARN-2035.patch Small bug that prevents ResourceManager and ApplicationHistoryService from coming up while Namenode is in safemode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2035) FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode
[ https://issues.apache.org/jira/browse/YARN-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2035: -- Attachment: YARN-2035-v3.patch Addressed failing tests with last patch. FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode --- Key: YARN-2035 URL: https://issues.apache.org/jira/browse/YARN-2035 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2035-v2.patch, YARN-2035-v3.patch, YARN-2035.patch Small bug that prevents ResourceManager and ApplicationHistoryService from coming up while Namenode is in safemode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2035) FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode
[ https://issues.apache.org/jira/browse/YARN-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107859#comment-14107859 ] Jonathan Eagles commented on YARN-2035: --- [~zjshen], can you please review this new version of the patch? FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode --- Key: YARN-2035 URL: https://issues.apache.org/jira/browse/YARN-2035 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2035-v2.patch, YARN-2035-v3.patch, YARN-2035.patch Small bug that prevents ResourceManager and ApplicationHistoryService from coming up while Namenode is in safemode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-2507) Document CORS configuration for ATS
Jonathan Eagles created YARN-2507: - Summary: Document CORS configuration for ATS Key: YARN-2507 URL: https://issues.apache.org/jira/browse/YARN-2507 Project: Hadoop YARN Issue Type: Sub-task Components: documentation Affects Versions: 2.6.0 Reporter: Jonathan Eagles Assignee: Mit Desai CORS support was added for ATS as part of YARN-2277. This jira is to document configuration for ATS CORS support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2508) CORS configuration parameters prefix are not honored
Jonathan Eagles created YARN-2508: - Summary: CORS configuration parameters prefix are not honored Key: YARN-2508 URL: https://issues.apache.org/jira/browse/YARN-2508 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles CrossOriginFilterInitializer specifies a prefix to have http server read parameters. However, it neither trims the prefix nor passes the prefix to the CrossOriginFilter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2507) Document CORS configuration for ATS
[ https://issues.apache.org/jira/browse/YARN-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2507: -- Component/s: timelineserver Document CORS configuration for ATS --- Key: YARN-2507 URL: https://issues.apache.org/jira/browse/YARN-2507 Project: Hadoop YARN Issue Type: Sub-task Components: documentation, timelineserver Affects Versions: 2.6.0 Reporter: Jonathan Eagles Assignee: Mit Desai CORS support was added for ATS as part of YARN-2277. This jira is to document configuration for ATS CORS support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2509) Enable Cross Origin Filter for timeline server only and not all Yarn servers
Jonathan Eagles created YARN-2509: - Summary: Enable Cross Origin Filter for timeline server only and not all Yarn servers Key: YARN-2509 URL: https://issues.apache.org/jira/browse/YARN-2509 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Mit Desai -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2509) Enable Cross Origin Filter for timeline server only and not all Yarn servers
[ https://issues.apache.org/jira/browse/YARN-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2509: -- Component/s: timelineserver Enable Cross Origin Filter for timeline server only and not all Yarn servers Key: YARN-2509 URL: https://issues.apache.org/jira/browse/YARN-2509 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Mit Desai -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2508) Cross Origin configuration parameters prefix are not honored
[ https://issues.apache.org/jira/browse/YARN-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2508: -- Assignee: Mit Desai Cross Origin configuration parameters prefix are not honored Key: YARN-2508 URL: https://issues.apache.org/jira/browse/YARN-2508 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Mit Desai CrossOriginFilterInitializer specifies a prefix to have http server read parameters. However, it neither trims the prefix nor passes the prefix to the CrossOriginFilter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2508) Cross Origin configuration parameters prefix are not honored
[ https://issues.apache.org/jira/browse/YARN-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2508: -- Summary: Cross Origin configuration parameters prefix are not honored (was: CORS configuration parameters prefix are not honored) Cross Origin configuration parameters prefix are not honored Key: YARN-2508 URL: https://issues.apache.org/jira/browse/YARN-2508 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles CrossOriginFilterInitializer specifies a prefix to have http server read parameters. However, it neither trims the prefix nor passes the prefix to the CrossOriginFilter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2509) Enable Cross Origin Filter for timeline server only and not all Yarn servers
[ https://issues.apache.org/jira/browse/YARN-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14120390#comment-14120390 ] Jonathan Eagles commented on YARN-2509: --- Thanks, [~zjshen]. That's exactly the code I was planning on copying. In this case should be add a configuration parameter to enable this? If so, what is a good name? Enable Cross Origin Filter for timeline server only and not all Yarn servers Key: YARN-2509 URL: https://issues.apache.org/jira/browse/YARN-2509 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Mit Desai -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2509) Enable Cross Origin Filter for timeline server only and not all Yarn servers
[ https://issues.apache.org/jira/browse/YARN-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14120898#comment-14120898 ] Jonathan Eagles commented on YARN-2509: --- [~mitdesai], there is a bug in that actualInitializers and initializers are always equal that prevents this from working. Perhaps you can add CrossOriginFilterInitializer to actualInitalizers instead of initializers? Enable Cross Origin Filter for timeline server only and not all Yarn servers Key: YARN-2509 URL: https://issues.apache.org/jira/browse/YARN-2509 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Mit Desai Attachments: YARN-2509.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2507) Document CORS configuration for ATS
[ https://issues.apache.org/jira/browse/YARN-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2507: -- Assignee: (was: Mit Desai) Document CORS configuration for ATS --- Key: YARN-2507 URL: https://issues.apache.org/jira/browse/YARN-2507 Project: Hadoop YARN Issue Type: Sub-task Components: documentation, timelineserver Affects Versions: 2.6.0 Reporter: Jonathan Eagles CORS support was added for ATS as part of YARN-2277. This jira is to document configuration for ATS CORS support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2508) Cross Origin configuration parameters prefix are not honored
[ https://issues.apache.org/jira/browse/YARN-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2508: -- Assignee: (was: Mit Desai) Cross Origin configuration parameters prefix are not honored Key: YARN-2508 URL: https://issues.apache.org/jira/browse/YARN-2508 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles CrossOriginFilterInitializer specifies a prefix to have http server read parameters. However, it neither trims the prefix nor passes the prefix to the CrossOriginFilter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2511) Allow All Origins by default when Cross Origin Filter is enabled
[ https://issues.apache.org/jira/browse/YARN-2511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121493#comment-14121493 ] Jonathan Eagles commented on YARN-2511: --- [~zjshen], Can you give a review? This just makes the default cross-origin pattern allow all origins (same as jetty 7 Cross origin filter) making us more compatible and giving the users a better default option. Allow All Origins by default when Cross Origin Filter is enabled Key: YARN-2511 URL: https://issues.apache.org/jira/browse/YARN-2511 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2511-v1.patch This is the default for jetty 7 cross origin filter -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2509) Enable Cross Origin Filter for timeline server only and not all Yarn servers
[ https://issues.apache.org/jira/browse/YARN-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121544#comment-14121544 ] Jonathan Eagles commented on YARN-2509: --- +1. Thanks, Mit. Committing this to trunk and branch-2 Enable Cross Origin Filter for timeline server only and not all Yarn servers Key: YARN-2509 URL: https://issues.apache.org/jira/browse/YARN-2509 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Mit Desai Attachments: YARN-2509.patch, YARN-2509.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2512) Allow for origin pattern matching in cross origin filter
Jonathan Eagles created YARN-2512: - Summary: Allow for origin pattern matching in cross origin filter Key: YARN-2512 URL: https://issues.apache.org/jira/browse/YARN-2512 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2509) Enable Cross Origin Filter for timeline server only and not all Yarn servers
[ https://issues.apache.org/jira/browse/YARN-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2509: -- Fix Version/s: 2.6.0 Enable Cross Origin Filter for timeline server only and not all Yarn servers Key: YARN-2509 URL: https://issues.apache.org/jira/browse/YARN-2509 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Mit Desai Fix For: 2.6.0 Attachments: YARN-2509.patch, YARN-2509.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2513) Host framework UIs in YARN for use with the ATS
Jonathan Eagles created YARN-2513: - Summary: Host framework UIs in YARN for use with the ATS Key: YARN-2513 URL: https://issues.apache.org/jira/browse/YARN-2513 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Allow for pluggable UIs as described by TEZ-8. Yarn can provide the infrastructure to host java script and possible java UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-2513) Host framework UIs in YARN for use with the ATS
[ https://issues.apache.org/jira/browse/YARN-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles reassigned YARN-2513: - Assignee: Jonathan Eagles Host framework UIs in YARN for use with the ATS --- Key: YARN-2513 URL: https://issues.apache.org/jira/browse/YARN-2513 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Allow for pluggable UIs as described by TEZ-8. Yarn can provide the infrastructure to host java script and possible java UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2509) Enable Cross Origin Filter for timeline server only and not all Yarn servers
[ https://issues.apache.org/jira/browse/YARN-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122379#comment-14122379 ] Jonathan Eagles commented on YARN-2509: --- [~zjshen], I think this will come down to documenting this feature properly as part of YARN-2507. The current behavior is that CORS support will be added if the user does that. Let me know if you think it can be improved. Enable Cross Origin Filter for timeline server only and not all Yarn servers Key: YARN-2509 URL: https://issues.apache.org/jira/browse/YARN-2509 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Mit Desai Fix For: 2.6.0 Attachments: YARN-2509.patch, YARN-2509.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-2512) Allow for origin pattern matching in cross origin filter
[ https://issues.apache.org/jira/browse/YARN-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles reassigned YARN-2512: - Assignee: Jonathan Eagles Allow for origin pattern matching in cross origin filter Key: YARN-2512 URL: https://issues.apache.org/jira/browse/YARN-2512 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2512) Allow for origin pattern matching in cross origin filter
[ https://issues.apache.org/jira/browse/YARN-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2512: -- Attachment: YARN-2512-v1.patch Allow for origin pattern matching in cross origin filter Key: YARN-2512 URL: https://issues.apache.org/jira/browse/YARN-2512 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2512-v1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2512) Allow for origin pattern matching in cross origin filter
[ https://issues.apache.org/jira/browse/YARN-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2512: -- Description: Extending the feature set of allowed origins. Now a * in a pattern indicates this allowed origin is a pattern and will be matched including multiple sub-domains. Allow for origin pattern matching in cross origin filter Key: YARN-2512 URL: https://issues.apache.org/jira/browse/YARN-2512 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2512-v1.patch Extending the feature set of allowed origins. Now a * in a pattern indicates this allowed origin is a pattern and will be matched including multiple sub-domains. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2512) Allow for origin pattern matching in cross origin filter
[ https://issues.apache.org/jira/browse/YARN-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124248#comment-14124248 ] Jonathan Eagles commented on YARN-2512: --- [~zjshen], can you please review this cross-origin change for use with the timeline server? Allow for origin pattern matching in cross origin filter Key: YARN-2512 URL: https://issues.apache.org/jira/browse/YARN-2512 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2512-v1.patch Extending the feature set of allowed origins. Now a * in a pattern indicates this allowed origin is a pattern and will be matched including multiple sub-domains. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2507) Document Cross Origin Filter Configuration for ATS
[ https://issues.apache.org/jira/browse/YARN-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2507: -- Attachment: YARN-2507-v1.patch Document Cross Origin Filter Configuration for ATS -- Key: YARN-2507 URL: https://issues.apache.org/jira/browse/YARN-2507 Project: Hadoop YARN Issue Type: Sub-task Components: documentation, timelineserver Affects Versions: 2.6.0 Reporter: Jonathan Eagles Attachments: YARN-2507-v1.patch CORS support was added for ATS as part of YARN-2277. This jira is to document configuration for ATS CORS support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2512) Allow for origin pattern matching in cross origin filter
[ https://issues.apache.org/jira/browse/YARN-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124735#comment-14124735 ] Jonathan Eagles commented on YARN-2512: --- [~zjshen], thank you for your continued patience with me. I think there could be some confusion since URLs have a different RFC than hostnames and domains. http://en.wikipedia.org/wiki/Hostname#Restrictions_on_valid_host_names Just for further clarification of my purposes, I'm slowly bringing hadoop up to feature parity with jetty so that users will have a full feature set and make the transition to jetty seamless when we move to jetty 7+. Jetty quite thoughtfully documents the cross origin filter wildcard feature this jira is aiming to add. http://download.eclipse.org/jetty/stable-7/apidocs/org/eclipse/jetty/servlets/CrossOriginFilter.html Allow for origin pattern matching in cross origin filter Key: YARN-2512 URL: https://issues.apache.org/jira/browse/YARN-2512 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2512-v1.patch Extending the feature set of allowed origins. Now a * in a pattern indicates this allowed origin is a pattern and will be matched including multiple sub-domains. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2528) Cross Origin Filter Http response split vulnerability protection rejects valid origins
Jonathan Eagles created YARN-2528: - Summary: Cross Origin Filter Http response split vulnerability protection rejects valid origins Key: YARN-2528 URL: https://issues.apache.org/jira/browse/YARN-2528 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles URLEncoding is too strong of a protection for HTTP Response Split Vulnerability protection and major browser reject the encoded Origin. An adequate protection is simply to remove all CRs LFs as in the case of PHP's header function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2528) Cross Origin Filter Http response split vulnerability protection rejects valid origins
[ https://issues.apache.org/jira/browse/YARN-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2528: -- Attachment: YARN-2528-v1.patch Cross Origin Filter Http response split vulnerability protection rejects valid origins -- Key: YARN-2528 URL: https://issues.apache.org/jira/browse/YARN-2528 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2528-v1.patch URLEncoding is too strong of a protection for HTTP Response Split Vulnerability protection and major browser reject the encoded Origin. An adequate protection is simply to remove all CRs LFs as in the case of PHP's header function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2528) Cross Origin Filter Http response split vulnerability protection rejects valid origins
[ https://issues.apache.org/jira/browse/YARN-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127592#comment-14127592 ] Jonathan Eagles commented on YARN-2528: --- [~zjshen], sorry to bother you again. Found another bug while working on getting the Tez UI running in a hosted environment. Can you give a review? Cross Origin Filter Http response split vulnerability protection rejects valid origins -- Key: YARN-2528 URL: https://issues.apache.org/jira/browse/YARN-2528 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2528-v1.patch URLEncoding is too strong of a protection for HTTP Response Split Vulnerability protection and major browser reject the encoded Origin. An adequate protection is simply to remove all CRs LFs as in the case of PHP's header function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2547) Cross Origin Filter throws UnsupportedOperationException upon destroy
Jonathan Eagles created YARN-2547: - Summary: Cross Origin Filter throws UnsupportedOperationException upon destroy Key: YARN-2547 URL: https://issues.apache.org/jira/browse/YARN-2547 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2547) Cross Origin Filter throws UnsupportedOperationException upon destroy
[ https://issues.apache.org/jira/browse/YARN-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131711#comment-14131711 ] Jonathan Eagles commented on YARN-2547: --- [~mitdesai], thanks for the quick fix posted. Main changes look good. Couple of minor things related to the test code. Instead testing for this one exception thrown which is an implementation detail. What we really want to test is that restart works init - destroy - init. That way the test conveys the functionality of the filter we are trying to ensure. Cross Origin Filter throws UnsupportedOperationException upon destroy - Key: YARN-2547 URL: https://issues.apache.org/jira/browse/YARN-2547 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Mit Desai Attachments: YARN-2547.patch, YARN-2547.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2547) Cross Origin Filter throws UnsupportedOperationException upon destroy
[ https://issues.apache.org/jira/browse/YARN-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131848#comment-14131848 ] Jonathan Eagles commented on YARN-2547: --- +1 pending QA comment. Thanks, Mit. Cross Origin Filter throws UnsupportedOperationException upon destroy - Key: YARN-2547 URL: https://issues.apache.org/jira/browse/YARN-2547 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Mit Desai Attachments: YARN-2547.patch, YARN-2547.patch, YARN-2547.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2513) Host framework UIs in YARN for use with the ATS
[ https://issues.apache.org/jira/browse/YARN-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2513: -- Attachment: YARN-2513-v1.patch Host framework UIs in YARN for use with the ATS --- Key: YARN-2513 URL: https://issues.apache.org/jira/browse/YARN-2513 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2513-v1.patch Allow for pluggable UIs as described by TEZ-8. Yarn can provide the infrastructure to host java script and possible java UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2513) Host framework UIs in YARN for use with the ATS
[
https://issues.apache.org/jira/browse/YARN-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131946#comment-14131946
]
Jonathan Eagles commented on YARN-2513:
---
[~vinodkv], [~hitesh], [~zjshen], I have posted a patch that will simply allow
the timeline server to host generic UI that still pass through the web filters
of hadoop. Please give some feedback.
{code}
property
nameyarn.timeline-service.ui-names/name
valuetez/value
/property
property
nameyarn.timeline-service.ui-on-disk-path.tez/name
value/Users/jeagles/hadoop/tez-ui/value
/property
property
nameyarn.timeline-service.ui-web-path.tez/name
value/tez-ui-v1.0/value
/property
{code}
Host framework UIs in YARN for use with the ATS
---
Key: YARN-2513
URL: https://issues.apache.org/jira/browse/YARN-2513
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Attachments: YARN-2513-v1.patch
Allow for pluggable UIs as described by TEZ-8. Yarn can provide the
infrastructure to host java script and possible java UIs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2547) Cross Origin Filter throws UnsupportedOperationException upon destroy
[ https://issues.apache.org/jira/browse/YARN-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132075#comment-14132075 ] Jonathan Eagles commented on YARN-2547: --- Committed to trunk and branch-2. Thanks, [~mitdesai]. Cross Origin Filter throws UnsupportedOperationException upon destroy - Key: YARN-2547 URL: https://issues.apache.org/jira/browse/YARN-2547 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Mit Desai Fix For: 2.6.0 Attachments: YARN-2547.patch, YARN-2547.patch, YARN-2547.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2528) Cross Origin Filter Http response split vulnerability protection rejects valid origins
[ https://issues.apache.org/jira/browse/YARN-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2528: -- Attachment: YARN-2528-v2-split-header.patch Cross Origin Filter Http response split vulnerability protection rejects valid origins -- Key: YARN-2528 URL: https://issues.apache.org/jira/browse/YARN-2528 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2528-v1.patch, YARN-2528-v2-split-header.patch URLEncoding is too strong of a protection for HTTP Response Split Vulnerability protection and major browser reject the encoded Origin. An adequate protection is simply to remove all CRs LFs as in the case of PHP's header function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2528) Cross Origin Filter Http response split vulnerability protection rejects valid origins
[ https://issues.apache.org/jira/browse/YARN-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2528: -- Attachment: YARN-2528-v2.patch Cross Origin Filter Http response split vulnerability protection rejects valid origins -- Key: YARN-2528 URL: https://issues.apache.org/jira/browse/YARN-2528 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2528-v1.patch, YARN-2528-v2-split-header.patch, YARN-2528-v2.patch URLEncoding is too strong of a protection for HTTP Response Split Vulnerability protection and major browser reject the encoded Origin. An adequate protection is simply to remove all CRs LFs as in the case of PHP's header function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2528) Cross Origin Filter Http response split vulnerability protection rejects valid origins
[ https://issues.apache.org/jira/browse/YARN-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2528: -- Attachment: (was: YARN-2528-v2-split-header.patch) Cross Origin Filter Http response split vulnerability protection rejects valid origins -- Key: YARN-2528 URL: https://issues.apache.org/jira/browse/YARN-2528 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2528-v1.patch, YARN-2528-v2.patch URLEncoding is too strong of a protection for HTTP Response Split Vulnerability protection and major browser reject the encoded Origin. An adequate protection is simply to remove all CRs LFs as in the case of PHP's header function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2528) Cross Origin Filter Http response split vulnerability protection rejects valid origins
[
https://issues.apache.org/jira/browse/YARN-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132480#comment-14132480
]
Jonathan Eagles commented on YARN-2528:
---
{quote}
Jonathan Eagles, no problem. I compared our CrossOriginFilter with the one in
Jetty. That one seems not to do any post-process for the string obtained from
ORIGIN header. What's the reason that we need for our CrossOriginFilter?
According to test case, you want to avoid the issue that the string contains
the other header, don't you? HttpServletResponse.getHeader doesn't handle
header splitting properly?
{quote}
[~zjshen], I haven't described yet in the JIRA how I came to make that change.
To reproduce this findbugs warning, take the above v2 patch and make the
following modifications setting the origin header response directly from the
origin header request.
{code}
-res.setHeader(ACCESS_CONTROL_ALLOW_ORIGIN, origin);
+ res.setHeader(ACCESS_CONTROL_ALLOW_ORIGIN, req.getHeader(ORIGIN));
{code}
and then run findbugs
{code}
mvn clean test findbugs:findbugs -DskipTests
{code}
in the hadoop-yarn-server-applicationhistoryservice directory. Here is the
findbugs error report.
{code}
BugCollection version=1.3.9 threshold=medium effort=max
file
classname=org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter
BugInstance type=HRS_REQUEST_PARAMETER_TO_HTTP_HEADER priority=Normal
category=SECURITY message=HTTP parameter directly written to HTTP header
output in
org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter.doCrossFilter(HttpServletRequest,
HttpServletResponse) lineNumber=128/
/file
Error/
Project
SrcDir/Users/jeagles/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java
/SrcDir
/Project
/BugCollection
BugInstance category=SECURITY
instanceHash=51f813d33b6e152c981d270989331951 instanceOccurrenceNum=0
priority=2 abbrev=HRS type=HRS_REQUEST_PARAMETER_TO_HTTP_HEADER
instanceOccurrenceMax=0
ShortMessageHTTP Response splitting vulnerability/ShortMessage
LongMessageHTTP parameter directly written to HTTP header output in
org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter.doCrossFilter(HttpServletRequest,
HttpServletResponse)/LongMessage
Class
classname=org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter
primary=true
SourceLine start=43
classname=org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter
sourcepath=org/apache/hadoop/yarn/server/timeline/webapp/CrossOriginFilter.java
sourcefile=CrossOriginFilter.java end=240
MessageAt CrossOriginFilter.java:[lines 43-240]/Message
/SourceLine
MessageIn class
org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter/Message
/Class
Method isStatic=false
classname=org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter
name=doCrossFilter primary=true
signature=(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V
SourceLine endBytecode=318 startBytecode=0 start=107
classname=org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter
sourcepath=org/apache/hadoop/yarn/server/timeline/webapp/CrossOriginFilter.java
sourcefile=CrossOriginFilter.java end=133/
MessageIn method
org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter.doCrossFilter(HttpServletRequest,
HttpServletResponse)/Message
/Method
SourceLine endBytecode=80 startBytecode=80 start=128
classname=org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter
primary=true
sourcepath=org/apache/hadoop/yarn/server/timeline/webapp/CrossOriginFilter.java
sourcefile=CrossOriginFilter.java end=128
MessageAt CrossOriginFilter.java:[line 128]/Message
/SourceLine
/BugInstance
BugCategory
category=SECURITYDescriptionSecurity/Description/BugCategory
BugPattern category=SECURITY abbrev=HRS
type=HRS_REQUEST_PARAMETER_TO_HTTP_HEADER
ShortDescriptionHTTP Response splitting vulnerability/ShortDescription
Details
pThis code directly writes an HTTP parameter to an HTTP header, which
allows for a HTTP response splitting
vulnerability. See a
href=http://en.wikipedia.org/wiki/HTTP_response_splitting;http://en.wikipedia.org/wiki/HTTP_response_splitting/a
for more information./p
pFindBugs looks only for the most blatant, obvious cases of HTTP response
splitting.
If FindBugs found emany/em, you emalmost certainly/em have more
vulnerabilities that FindBugs doesn't report. If you are concerned about HTTP
response splitting, you should seriously
consider using a commercial static analysis or pen-testing tool.
/p
/Details
/BugPattern
/BugCategory
{code}
Unfortunately, It is too easy to trick findbugs in this scenario so perhaps
that is why jetty doesn't implement this security fix. It also could be a
difference with jetty 6 vs later versions of jetty. I
[jira] [Created] (YARN-2556) Tool to measure the performance of the timeline server
Jonathan Eagles created YARN-2556: - Summary: Tool to measure the performance of the timeline server Key: YARN-2556 URL: https://issues.apache.org/jira/browse/YARN-2556 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles We need to be able to understand the capacity model for the timeline server to give users the tools they need to deploy a timeline server with the correct capacity. I propose we create a mapreduce job that can measure timeline server write and read performance. Transactions per second, I/O for both read and write would be a good start. This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2556) Tool to measure the performance of the timeline server
[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134542#comment-14134542 ] Jonathan Eagles commented on YARN-2556: --- This jira is to give users realistic performance numbers running the timeline server for their setup specific to their own hardware. (HBase, leveldb, etc) FYI. LevelDB publishes their own performance statics. https://code.google.com/p/leveldb/ Tool to measure the performance of the timeline server -- Key: YARN-2556 URL: https://issues.apache.org/jira/browse/YARN-2556 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles We need to be able to understand the capacity model for the timeline server to give users the tools they need to deploy a timeline server with the correct capacity. I propose we create a mapreduce job that can measure timeline server write and read performance. Transactions per second, I/O for both read and write would be a good start. This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2363) Submitted applications occasionally lack a tracking URL
[ https://issues.apache.org/jira/browse/YARN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139269#comment-14139269 ] Jonathan Eagles commented on YARN-2363: --- +1. This will be a big help to users, [~jlowe]. Submitted applications occasionally lack a tracking URL --- Key: YARN-2363 URL: https://issues.apache.org/jira/browse/YARN-2363 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: YARN-2363.patch Sometimes when an application is submitted the client receives no tracking URL. More details in the first comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2606) Application History Server tries to access hdfs before doing secure login
[ https://issues.apache.org/jira/browse/YARN-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14148273#comment-14148273 ] Jonathan Eagles commented on YARN-2606: --- I can see this both ways. It seems to correct to login during initialization and and to wait until start to do file operations. Although a fix to either one of them does indeed fix the issue at hand. Application History Server tries to access hdfs before doing secure login - Key: YARN-2606 URL: https://issues.apache.org/jira/browse/YARN-2606 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.6.0 Reporter: Mit Desai Assignee: Mit Desai Attachments: YARN-2606.patch While testing the Application Timeline Server, the server would not come up in a secure cluster, as it would keep trying to access hdfs without having done the secure login. It would repeatedly try authenticating and finally hit stack overflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2606) Application History Server tries to access hdfs before doing secure login
[ https://issues.apache.org/jira/browse/YARN-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14148315#comment-14148315 ] Jonathan Eagles commented on YARN-2606: --- I don't have any context why login is part of start. Application History Server tries to access hdfs before doing secure login - Key: YARN-2606 URL: https://issues.apache.org/jira/browse/YARN-2606 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.6.0 Reporter: Mit Desai Assignee: Mit Desai Attachments: YARN-2606.patch While testing the Application Timeline Server, the server would not come up in a secure cluster, as it would keep trying to access hdfs without having done the secure login. It would repeatedly try authenticating and finally hit stack overflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2606) Application History Server tries to access hdfs before doing secure login
[ https://issues.apache.org/jira/browse/YARN-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152077#comment-14152077 ] Jonathan Eagles commented on YARN-2606: --- +1. Will commit at the end of the day in case any one else has comments. Application History Server tries to access hdfs before doing secure login - Key: YARN-2606 URL: https://issues.apache.org/jira/browse/YARN-2606 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.6.0 Reporter: Mit Desai Assignee: Mit Desai Attachments: YARN-2606.patch, YARN-2606.patch, YARN-2606.patch, YARN-2606.patch While testing the Application Timeline Server, the server would not come up in a secure cluster, as it would keep trying to access hdfs without having done the secure login. It would repeatedly try authenticating and finally hit stack overflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2606) Application History Server tries to access hdfs before doing secure login
[ https://issues.apache.org/jira/browse/YARN-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152348#comment-14152348 ] Jonathan Eagles commented on YARN-2606: --- Committed to trunk and branch-2 Application History Server tries to access hdfs before doing secure login - Key: YARN-2606 URL: https://issues.apache.org/jira/browse/YARN-2606 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.6.0 Reporter: Mit Desai Assignee: Mit Desai Fix For: 2.6.0 Attachments: YARN-2606.patch, YARN-2606.patch, YARN-2606.patch, YARN-2606.patch While testing the Application Timeline Server, the server would not come up in a secure cluster, as it would keep trying to access hdfs without having done the secure login. It would repeatedly try authenticating and finally hit stack overflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-206) TestApplicationCleanup.testContainerCleanup occasionally fails
[ https://issues.apache.org/jira/browse/YARN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494088#comment-13494088 ] Jonathan Eagles commented on YARN-206: -- +1 LGTM. I have verified that this test does run consistently now as compared to before. Checking this in. TestApplicationCleanup.testContainerCleanup occasionally fails -- Key: YARN-206 URL: https://issues.apache.org/jira/browse/YARN-206 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Attachments: YARN-206.patch testContainerCleanup is occasionally failing with the error: testContainerCleanup(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup): expected:2 but was:1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-214) RMContainerImpl does not handle event EXPIRE at state RUNNING
[
https://issues.apache.org/jira/browse/YARN-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles reassigned YARN-214:
Assignee: Jonathan Eagles
RMContainerImpl does not handle event EXPIRE at state RUNNING
-
Key: YARN-214
URL: https://issues.apache.org/jira/browse/YARN-214
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Jonathan Eagles
RMContainerImpl has a race condition where a container can enter the RUNNING
state just as the container expires. This results in an invalid event
transition error:
{noformat}
2012-11-11 05:31:38,954 [ResourceManager Event Processor] ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
EXPIRE at RUNNING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:205)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:44)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerCompleted(SchedulerApp.java:203)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1337)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainer(CapacityScheduler.java:739)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:659)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:80)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:340)
at java.lang.Thread.run(Thread.java:619)
{noformat}
EXPIRE needs to be handled (well at least ignored) in the RUNNING state to
account for this race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-214) RMContainerImpl does not handle event EXPIRE at state RUNNING
[
https://issues.apache.org/jira/browse/YARN-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated YARN-214:
-
Attachment: YARN-214.patch
RMContainerImpl does not handle event EXPIRE at state RUNNING
-
Key: YARN-214
URL: https://issues.apache.org/jira/browse/YARN-214
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Jonathan Eagles
Attachments: YARN-214.patch
RMContainerImpl has a race condition where a container can enter the RUNNING
state just as the container expires. This results in an invalid event
transition error:
{noformat}
2012-11-11 05:31:38,954 [ResourceManager Event Processor] ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
EXPIRE at RUNNING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:205)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:44)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerCompleted(SchedulerApp.java:203)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1337)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainer(CapacityScheduler.java:739)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:659)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:80)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:340)
at java.lang.Thread.run(Thread.java:619)
{noformat}
EXPIRE needs to be handled (well at least ignored) in the RUNNING state to
account for this race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-214) RMContainerImpl does not handle event EXPIRE at state RUNNING
[
https://issues.apache.org/jira/browse/YARN-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated YARN-214:
-
Attachment: YARN-214.patch
RMContainerImpl does not handle event EXPIRE at state RUNNING
-
Key: YARN-214
URL: https://issues.apache.org/jira/browse/YARN-214
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Jonathan Eagles
Attachments: YARN-214.patch, YARN-214.patch
RMContainerImpl has a race condition where a container can enter the RUNNING
state just as the container expires. This results in an invalid event
transition error:
{noformat}
2012-11-11 05:31:38,954 [ResourceManager Event Processor] ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
EXPIRE at RUNNING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:205)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:44)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerCompleted(SchedulerApp.java:203)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1337)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainer(CapacityScheduler.java:739)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:659)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:80)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:340)
at java.lang.Thread.run(Thread.java:619)
{noformat}
EXPIRE needs to be handled (well at least ignored) in the RUNNING state to
account for this race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-214) RMContainerImpl does not handle event EXPIRE at state RUNNING
[
https://issues.apache.org/jira/browse/YARN-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated YARN-214:
-
Attachment: YARN-214.patch
RMContainerImpl does not handle event EXPIRE at state RUNNING
-
Key: YARN-214
URL: https://issues.apache.org/jira/browse/YARN-214
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Jonathan Eagles
Attachments: YARN-214.patch, YARN-214.patch, YARN-214.patch
RMContainerImpl has a race condition where a container can enter the RUNNING
state just as the container expires. This results in an invalid event
transition error:
{noformat}
2012-11-11 05:31:38,954 [ResourceManager Event Processor] ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
EXPIRE at RUNNING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:205)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:44)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerCompleted(SchedulerApp.java:203)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1337)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainer(CapacityScheduler.java:739)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:659)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:80)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:340)
at java.lang.Thread.run(Thread.java:619)
{noformat}
EXPIRE needs to be handled (well at least ignored) in the RUNNING state to
account for this race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-214) RMContainerImpl does not handle event EXPIRE at state RUNNING
[
https://issues.apache.org/jira/browse/YARN-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated YARN-214:
-
Attachment: YARN-214.patch
RMContainerImpl does not handle event EXPIRE at state RUNNING
-
Key: YARN-214
URL: https://issues.apache.org/jira/browse/YARN-214
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Jonathan Eagles
Attachments: YARN-214.patch, YARN-214.patch, YARN-214.patch,
YARN-214.patch
RMContainerImpl has a race condition where a container can enter the RUNNING
state just as the container expires. This results in an invalid event
transition error:
{noformat}
2012-11-11 05:31:38,954 [ResourceManager Event Processor] ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
EXPIRE at RUNNING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:205)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:44)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerCompleted(SchedulerApp.java:203)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1337)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainer(CapacityScheduler.java:739)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:659)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:80)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:340)
at java.lang.Thread.run(Thread.java:619)
{noformat}
EXPIRE needs to be handled (well at least ignored) in the RUNNING state to
account for this race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-214) RMContainerImpl does not handle event EXPIRE at state RUNNING
[
https://issues.apache.org/jira/browse/YARN-214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13498580#comment-13498580
]
Jonathan Eagles commented on YARN-214:
--
I changed the transition to a no-op and removed the app attempt captor since
the app attempt isn't being modified now
RMContainerImpl does not handle event EXPIRE at state RUNNING
-
Key: YARN-214
URL: https://issues.apache.org/jira/browse/YARN-214
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Jonathan Eagles
Attachments: YARN-214.patch, YARN-214.patch, YARN-214.patch,
YARN-214.patch
RMContainerImpl has a race condition where a container can enter the RUNNING
state just as the container expires. This results in an invalid event
transition error:
{noformat}
2012-11-11 05:31:38,954 [ResourceManager Event Processor] ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
EXPIRE at RUNNING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:205)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:44)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerCompleted(SchedulerApp.java:203)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1337)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainer(CapacityScheduler.java:739)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:659)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:80)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:340)
at java.lang.Thread.run(Thread.java:619)
{noformat}
EXPIRE needs to be handled (well at least ignored) in the RUNNING state to
account for this race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-214) RMContainerImpl does not handle event EXPIRE at state RUNNING
[
https://issues.apache.org/jira/browse/YARN-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated YARN-214:
-
Attachment: YARN-214.patch
RMContainerImpl does not handle event EXPIRE at state RUNNING
-
Key: YARN-214
URL: https://issues.apache.org/jira/browse/YARN-214
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Jonathan Eagles
Attachments: YARN-214.patch, YARN-214.patch, YARN-214.patch,
YARN-214.patch, YARN-214.patch
RMContainerImpl has a race condition where a container can enter the RUNNING
state just as the container expires. This results in an invalid event
transition error:
{noformat}
2012-11-11 05:31:38,954 [ResourceManager Event Processor] ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
EXPIRE at RUNNING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:205)
at
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:44)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerCompleted(SchedulerApp.java:203)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1337)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainer(CapacityScheduler.java:739)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:659)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:80)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:340)
at java.lang.Thread.run(Thread.java:619)
{noformat}
EXPIRE needs to be handled (well at least ignored) in the RUNNING state to
account for this race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-249) Capacity Scheduler web page should show list of active users per queue like it used to (in 1.x)
[ https://issues.apache.org/jira/browse/YARN-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511069#comment-13511069 ] Jonathan Eagles commented on YARN-249: -- Ravi, can you address the test failures introduced by this patch? Capacity Scheduler web page should show list of active users per queue like it used to (in 1.x) --- Key: YARN-249 URL: https://issues.apache.org/jira/browse/YARN-249 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Affects Versions: 2.0.2-alpha, 3.0.0, 0.23.5 Reporter: Ravi Prakash Assignee: Ravi Prakash Labels: scheduler, web-ui Attachments: YARN-249.patch, YARN-249.patch On the jobtracker, the web ui showed the active users for each queue and how much resources each of those users were using. That currently isn't being displayed on the RM capacity scheduler web ui. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-236) RM should point tracking URL to RM web page when app fails to start
[ https://issues.apache.org/jira/browse/YARN-236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13582482#comment-13582482 ] Jonathan Eagles commented on YARN-236: -- +1. Verified this condition still exists and patch does redirect to the RM app webpage. Thanks, Jason. RM should point tracking URL to RM web page when app fails to start --- Key: YARN-236 URL: https://issues.apache.org/jira/browse/YARN-236 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.4 Reporter: Jason Lowe Assignee: Jason Lowe Labels: usability Attachments: YARN-236.patch Similar to YARN-165, the RM should redirect the tracking URL to the specific app page on the RM web UI when the application fails to start. For example, if the AM completely fails to start due to bad AM config or bad job config like invalid queuename, then the user gets the unhelpful The requested application exited before setting a tracking URL. Usually the diagnostic string on the RM app page has something useful, so we might as well point there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-227) Application expiration difficult to debug for end-users
[ https://issues.apache.org/jira/browse/YARN-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592558#comment-13592558 ] Jonathan Eagles commented on YARN-227: -- +1. Jason. If you can provide a 23 patch, I can check the code in there too. Application expiration difficult to debug for end-users --- Key: YARN-227 URL: https://issues.apache.org/jira/browse/YARN-227 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.3, 2.0.1-alpha Reporter: Jason Lowe Assignee: Jason Lowe Labels: usability Attachments: YARN-227.patch When an AM attempt expires the AMLivelinessMonitor in the RM will kill the job and mark it as failed. However there are no diagnostic messages set for the application indicating that the application failed because of expiration. Even if the AM logs are examined, it's often not obvious that the application was externally killed. The only evidence of what happened to the application is currently in the RM logs, and those are often not accessible by users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-227) Application expiration difficult to debug for end-users
[ https://issues.apache.org/jira/browse/YARN-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592823#comment-13592823 ] Jonathan Eagles commented on YARN-227: -- It looks like the eclipse:eclipse issue is spurious. Application expiration difficult to debug for end-users --- Key: YARN-227 URL: https://issues.apache.org/jira/browse/YARN-227 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.3, 2.0.1-alpha Reporter: Jason Lowe Assignee: Jason Lowe Labels: usability Attachments: YARN-227-branch-0.23.patch, YARN-227.patch When an AM attempt expires the AMLivelinessMonitor in the RM will kill the job and mark it as failed. However there are no diagnostic messages set for the application indicating that the application failed because of expiration. Even if the AM logs are examined, it's often not obvious that the application was externally killed. The only evidence of what happened to the application is currently in the RM logs, and those are often not accessible by users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-227) Application expiration difficult to debug for end-users
[ https://issues.apache.org/jira/browse/YARN-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13594092#comment-13594092 ] Jonathan Eagles commented on YARN-227: -- +1. Thanks so much for this patch, Jason. Application expiration difficult to debug for end-users --- Key: YARN-227 URL: https://issues.apache.org/jira/browse/YARN-227 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.3, 2.0.1-alpha Reporter: Jason Lowe Assignee: Jason Lowe Labels: usability Attachments: YARN-227-branch-0.23.patch, YARN-227-branch-0.23.patch, YARN-227.patch, YARN-227.patch When an AM attempt expires the AMLivelinessMonitor in the RM will kill the job and mark it as failed. However there are no diagnostic messages set for the application indicating that the application failed because of expiration. Even if the AM logs are examined, it's often not obvious that the application was externally killed. The only evidence of what happened to the application is currently in the RM logs, and those are often not accessible by users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-548) Add tests for YarnUncaughtExceptionHandler
[ https://issues.apache.org/jira/browse/YARN-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662213#comment-13662213 ] Jonathan Eagles commented on YARN-548: -- Thanks for the patch, Vadim. A couple of minor points. * Rename patches to reflect the new jira project under YARN *ExitUtil.java* - Fix spelling of Rintime to Runtime - Log msg in addition to status in halt(int status, String msg) - Correct javadoc to throws HaltException for halt(int status, Throwable t) and halt(int status) *YarnUncaughtExceptionHandler.java* - Remove unnecessary inline comments on new ExitUtil.halt and Exit.Util.terminate calls *TestYarnUncaughtExceptionHandler.java* -Remove use of timeout parameter in Test annotation Add tests for YarnUncaughtExceptionHandler -- Key: YARN-548 URL: https://issues.apache.org/jira/browse/YARN-548 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Vadim Bondarev Assignee: Vadim Bondarev Attachments: HADOOP-4528-branch-0.23-a.patch, HADOOP-4528-branch-2-a.patch, HADOOP-4528-trunk-a.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1769) CapacityScheduler: Improve reservations
[ https://issues.apache.org/jira/browse/YARN-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933478#comment-13933478 ] Jonathan Eagles commented on YARN-1769: --- Hi, Tom. Can you comment on the findbugs warnings that are introduced as part of this patch when you get a chance? CapacityScheduler: Improve reservations Key: YARN-1769 URL: https://issues.apache.org/jira/browse/YARN-1769 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Affects Versions: 2.3.0 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-1769.patch, YARN-1769.patch Currently the CapacityScheduler uses reservations in order to handle requests for large containers and the fact there might not currently be enough space available on a single host. The current algorithm for reservations is to reserve as many containers as currently required and then it will start to reserve more above that after a certain number of re-reservations (currently biased against larger containers). Anytime it hits the limit of number reserved it stops looking at any other nodes. This results in potentially missing nodes that have enough space to fullfill the request. The other place for improvement is currently reservations count against your queue capacity. If you have reservations you could hit the various limits which would then stop you from looking further at that node. The above 2 cases can cause an application requesting a larger container to take a long time to gets it resources. We could improve upon both of those by simply continuing to look at incoming nodes to see if we could potentially swap out a reservation for an actual allocation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1833) TestRMAdminService Fails in trunk and branch-2
[
https://issues.apache.org/jira/browse/YARN-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935546#comment-13935546
]
Jonathan Eagles commented on YARN-1833:
---
[~mdesai], instead of removing the check, could we investigate using
UserGroupInformation.createUserForTesting. This will allow us to isolate the
developer environment as a requirement to the correctness of the test.
TestRMAdminService Fails in trunk and branch-2
--
Key: YARN-1833
URL: https://issues.apache.org/jira/browse/YARN-1833
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.0.0, 2.4.0
Reporter: Mit Desai
Assignee: Mit Desai
Labels: Test
Attachments: YARN-1833.patch
In the test
testRefreshUserToGroupsMappingsWithFileSystemBasedConfigurationProvider, the
following assert is not needed.
{code}
Assert.assertTrue(groupWithInit.size() != groupBefore.size());
{code}
As the assert takes the default groups for groupWithInit (which in my case
are users, sshusers and wheel), it fails as the size of both groupWithInit
and groupBefore are same.
I do not think we need to have this assert here. Moreover we are also
checking that the groupInit does not have the userGroups that are in the
groupBefore so removing the assert may not be harmful.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
