[jira] [Assigned] (YARN-10337) TestRMHATimelineCollectors fails on hadoop trunk
[
https://issues.apache.org/jira/browse/YARN-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bilwa S T reassigned YARN-10337:
Assignee: Bilwa S T
> TestRMHATimelineCollectors fails on hadoop trunk
>
>
> Key: YARN-10337
> URL: https://issues.apache.org/jira/browse/YARN-10337
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, yarn
>Reporter: Ahmed Hussein
>Assignee: Bilwa S T
>Priority: Major
>
> {{TestRMHATimelineCollectors}} has been failing on trunk. I see it frequently
> in the qbt reports and the yetus reprts
> {code:bash}
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.95
> s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR]
> testRebuildCollectorDataOnFailover(org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors)
> Time elapsed: 5.615 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover(TestRMHATimelineCollectors.java:105)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:80)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover:105
> NullPointer
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [ERROR] There are test failures.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10337) TestRMHATimelineCollectors fails on hadoop trunk
Ahmed Hussein created YARN-10337:
Summary: TestRMHATimelineCollectors fails on hadoop trunk
Key: YARN-10337
URL: https://issues.apache.org/jira/browse/YARN-10337
Project: Hadoop YARN
Issue Type: Bug
Components: test, yarn
Reporter: Ahmed Hussein
{{TestRMHATimelineCollectors}} has been failing on trunk. I see it frequently
in the qbt reports and the yetus reprts
{code:bash}
[INFO] Running
org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.95 s
<<< FAILURE! - in
org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
[ERROR]
testRebuildCollectorDataOnFailover(org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors)
Time elapsed: 5.615 s <<< ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover(TestRMHATimelineCollectors.java:105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:80)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover:105
NullPointer
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[ERROR] There are test failures.
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10251) Show extended resources on legacy RM UI.
[ https://issues.apache.org/jira/browse/YARN-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150592#comment-17150592 ] Eric Payne commented on YARN-10251: --- Thanks [~Jim_Brennan] and [~jhung]. I see that YARN-5448 and YARN-7684 were discussing similar issues. YARN-7684 was closed as "Won't Fix." I'll open a new JIRA describing the situation. > Show extended resources on legacy RM UI. > > > Key: YARN-10251 > URL: https://issues.apache.org/jira/browse/YARN-10251 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Attachments: Legacy RM UI With Not All Resources Shown.png, Updated > NodesPage UI With GPU columns.png, Updated RM UI With All Resources > Shown.png.png, YARN-10251.003.patch, YARN-10251.004.patch, > YARN-10251.005.patch, YARN-10251.006.patch, YARN-10251.branch-2.10.001.patch, > YARN-10251.branch-2.10.002.patch, YARN-10251.branch-2.10.003.patch, > YARN-10251.branch-2.10.005.patch, YARN-10251.branch-2.10.006.patch, > YARN-10251.branch-3.2.004.patch, YARN-10251.branch-3.2.005.patch, > YARN-10251.branch-3.2.006.patch > > > It would be great to update the legacy RM UI to include GPU resources in the > overview and in the per-app sections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10251) Show extended resources on legacy RM UI.
[ https://issues.apache.org/jira/browse/YARN-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150555#comment-17150555 ] Jonathan Hung commented on YARN-10251: -- I see. Can you file a follow up jira for this? I think we should at least have it consistent. In this jira we can leave it as default partition for used, total, reserved. The follow up jira can change used, total, reserved to count for all partitions. > Show extended resources on legacy RM UI. > > > Key: YARN-10251 > URL: https://issues.apache.org/jira/browse/YARN-10251 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Attachments: Legacy RM UI With Not All Resources Shown.png, Updated > NodesPage UI With GPU columns.png, Updated RM UI With All Resources > Shown.png.png, YARN-10251.003.patch, YARN-10251.004.patch, > YARN-10251.005.patch, YARN-10251.006.patch, YARN-10251.branch-2.10.001.patch, > YARN-10251.branch-2.10.002.patch, YARN-10251.branch-2.10.003.patch, > YARN-10251.branch-2.10.005.patch, YARN-10251.branch-2.10.006.patch, > YARN-10251.branch-3.2.004.patch, YARN-10251.branch-3.2.005.patch, > YARN-10251.branch-3.2.006.patch > > > It would be great to update the legacy RM UI to include GPU resources in the > overview and in the per-app sections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10251) Show extended resources on legacy RM UI.
[ https://issues.apache.org/jira/browse/YARN-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150539#comment-17150539 ] Jim Brennan commented on YARN-10251: I agree that we ideally should be including all resources in these cluster-wide metrics, but I also agree that we should not change the existing behavior of excluding labeled partitions for these totals in this Jira. > Show extended resources on legacy RM UI. > > > Key: YARN-10251 > URL: https://issues.apache.org/jira/browse/YARN-10251 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Attachments: Legacy RM UI With Not All Resources Shown.png, Updated > NodesPage UI With GPU columns.png, Updated RM UI With All Resources > Shown.png.png, YARN-10251.003.patch, YARN-10251.004.patch, > YARN-10251.005.patch, YARN-10251.006.patch, YARN-10251.branch-2.10.001.patch, > YARN-10251.branch-2.10.002.patch, YARN-10251.branch-2.10.003.patch, > YARN-10251.branch-2.10.005.patch, YARN-10251.branch-2.10.006.patch, > YARN-10251.branch-3.2.004.patch, YARN-10251.branch-3.2.005.patch, > YARN-10251.branch-3.2.006.patch > > > It would be great to update the legacy RM UI to include GPU resources in the > overview and in the per-app sections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10336) RM page should throw exception when command injected in RM REST API to get applications
[ https://issues.apache.org/jira/browse/YARN-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150315#comment-17150315 ] Bilwa S T edited comment on YARN-10336 at 7/2/20, 7:13 PM: --- cc [~brahmareddy] [~eyang] [~inigoiri] was (Author: bilwast): cc [~brahmareddy] [~eyang] > RM page should throw exception when command injected in RM REST API to get > applications > --- > > Key: YARN-10336 > URL: https://issues.apache.org/jira/browse/YARN-10336 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rajshree Mishra >Assignee: Bilwa S T >Priority: Major > Attachments: CommandInject.jpg, RM_UI.jpg > > > Using a web application attacking, we see that injecting commands like > ACCEPTED, FAILED and FINISHED to RM REST API does not throw an exception. > Refer images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10251) Show extended resources on legacy RM UI.
[ https://issues.apache.org/jira/browse/YARN-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150533#comment-17150533 ] Eric Payne commented on YARN-10251: --- Thanks for the review [~jhung]. You bring up an interesting question. Unfortunately, it's more complicated than that. Currently, the RM UI does _not_ show used or reserved information in the top section for resources in labeled queues. Nor does it show totals for resources in labeled queues. IIRC, this is because of backwards compatibility reasons. I think these were discussed in YARN-6492. In this patch, I inadvertently modified this behavior by including all resources used and all resources total. But I failed to do so for reservations. I am reluctant to change this behavior in this JIRA (although, I think it makes sense to include all resources in these metrics). I think I should change it back to the old behavior. Thoughts? > Show extended resources on legacy RM UI. > > > Key: YARN-10251 > URL: https://issues.apache.org/jira/browse/YARN-10251 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Attachments: Legacy RM UI With Not All Resources Shown.png, Updated > NodesPage UI With GPU columns.png, Updated RM UI With All Resources > Shown.png.png, YARN-10251.003.patch, YARN-10251.004.patch, > YARN-10251.005.patch, YARN-10251.006.patch, YARN-10251.branch-2.10.001.patch, > YARN-10251.branch-2.10.002.patch, YARN-10251.branch-2.10.003.patch, > YARN-10251.branch-2.10.005.patch, YARN-10251.branch-2.10.006.patch, > YARN-10251.branch-3.2.004.patch, YARN-10251.branch-3.2.005.patch, > YARN-10251.branch-3.2.006.patch > > > It would be great to update the legacy RM UI to include GPU resources in the > overview and in the per-app sections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10336) RM page should throw exception when command injected in RM REST API to get applications
[ https://issues.apache.org/jira/browse/YARN-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150303#comment-17150303 ] Bilwa S T edited comment on YARN-10336 at 7/2/20, 4:37 PM: --- This should be handled same way as how NodesPage is handled if NodeState given is not valid. The only difference is here we catch Illegal Argument Exception thrown by "YarnApplicationState.valueOf(stateString)" . I think we shouldn't catch if exception is Illegal Argument Exception here. was (Author: bilwast): This should be handled same way as how NodesPage is handled if NodeState given is not valid. The only difference is here we catch Illegal Argument Exception thrown by "YarnApplicationState.valueOf(stateString)" . I think we shouldn't catch exception here. > RM page should throw exception when command injected in RM REST API to get > applications > --- > > Key: YARN-10336 > URL: https://issues.apache.org/jira/browse/YARN-10336 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rajshree Mishra >Assignee: Bilwa S T >Priority: Major > Attachments: CommandInject.jpg, RM_UI.jpg > > > Using a web application attacking, we see that injecting commands like > ACCEPTED, FAILED and FINISHED to RM REST API does not throw an exception. > Refer images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10297) TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently
[
https://issues.apache.org/jira/browse/YARN-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150405#comment-17150405
]
Jim Brennan commented on YARN-10297:
[~jhung] can you please review?
> TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails
> intermittently
> ---
>
> Key: YARN-10297
> URL: https://issues.apache.org/jira/browse/YARN-10297
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-10297.001.patch, YARN-10297.002.patch
>
>
> After YARN-6492, testFairSchedulerContinuousSchedulingInitTime fails
> intermittently when running {{mvn test -Dtest=TestContinuousScheduling}}
> {noformat}[INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling
> [ERROR] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.682
> s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling
> [ERROR]
> testFairSchedulerContinuousSchedulingInitTime(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling)
> Time elapsed: 0.194 s <<< ERROR!
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> PartitionQueueMetrics,partition= already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getPartitionMetrics(QueueMetrics.java:362)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.incrPendingResources(QueueMetrics.java:601)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updatePendingResources(AppSchedulingInfo.java:388)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.internalAddResourceRequests(AppSchedulingInfo.java:320)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.internalAddResourceRequests(AppSchedulingInfo.java:347)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(AppSchedulingInfo.java:183)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateResourceRequests(SchedulerApplicationAttempt.java:456)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.allocate(FairScheduler.java:898)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling.testFairSchedulerContinuousSchedulingInitTime(TestContinuousScheduling.java:375)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9459) ConfiguredYarnAuthorizer should not now any details about queues
[ https://issues.apache.org/jira/browse/YARN-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150388#comment-17150388 ] Bilwa S T commented on YARN-9459: - Hi [~snemeth] ConfiguredYarnAuthorizer#checkPermission is not just called from QueueACLsManager. its called from RMAppManager and even Scheduler. If queue related code is removed from here, then we need to make changes from all methods. > ConfiguredYarnAuthorizer should not now any details about queues > > > Key: YARN-9459 > URL: https://issues.apache.org/jira/browse/YARN-9459 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Bilwa S T >Priority: Major > > ConfiguredYarnAuthorizer should not now anything about queues, as this is not > its responsibility, see > [here|https://github.com/apache/hadoop/blob/39b4a37e02e929a698fcf9e32f1f71bb6b977635/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ConfiguredYarnAuthorizer.java#L70] > Code like this could be part of the QueueACLsManager. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10336) RM page should throw exception when command injected in RM REST API to get applications
[ https://issues.apache.org/jira/browse/YARN-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150315#comment-17150315 ] Bilwa S T edited comment on YARN-10336 at 7/2/20, 2:01 PM: --- cc [~brahmareddy] [~eyang] was (Author: bilwast): cc [~brahmareddy] > RM page should throw exception when command injected in RM REST API to get > applications > --- > > Key: YARN-10336 > URL: https://issues.apache.org/jira/browse/YARN-10336 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rajshree Mishra >Assignee: Bilwa S T >Priority: Major > Attachments: CommandInject.jpg, RM_UI.jpg > > > Using a web application attacking, we see that injecting commands like > ACCEPTED, FAILED and FINISHED to RM REST API does not throw an exception. > Refer images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10319) Record Last N Scheduler Activities from ActivitiesManager
[
https://issues.apache.org/jira/browse/YARN-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150316#comment-17150316
]
Hadoop QA commented on YARN-10319:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | {color:blue} markdownlint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
26s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no findbugs output file
(findbugsXml.xml) {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has
no data from findbugs {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 36s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
42s{color} | {color:green} hadoop-yarn-server-router in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}188m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestRMHATim
[jira] [Commented] (YARN-10336) RM page should throw exception when command injected in RM REST API to get applications
[ https://issues.apache.org/jira/browse/YARN-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150315#comment-17150315 ] Bilwa S T commented on YARN-10336: -- cc [~brahmareddy] > RM page should throw exception when command injected in RM REST API to get > applications > --- > > Key: YARN-10336 > URL: https://issues.apache.org/jira/browse/YARN-10336 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rajshree Mishra >Assignee: Bilwa S T >Priority: Major > Attachments: CommandInject.jpg, RM_UI.jpg > > > Using a web application attacking, we see that injecting commands like > ACCEPTED, FAILED and FINISHED to RM REST API does not throw an exception. > Refer images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10336) RM page should throw exception when command injected in RM REST API to get applications
[ https://issues.apache.org/jira/browse/YARN-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150303#comment-17150303 ] Bilwa S T commented on YARN-10336: -- This should be handled same way as how NodesPage is handled if NodeState given is not valid. The only difference is here we catch Illegal Argument Exception thrown by "YarnApplicationState.valueOf(stateString)" . I think we shouldn't catch exception here. > RM page should throw exception when command injected in RM REST API to get > applications > --- > > Key: YARN-10336 > URL: https://issues.apache.org/jira/browse/YARN-10336 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rajshree Mishra >Assignee: Bilwa S T >Priority: Major > Attachments: CommandInject.jpg, RM_UI.jpg > > > Using a web application attacking, we see that injecting commands like > ACCEPTED, FAILED and FINISHED to RM REST API does not throw an exception. > Refer images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10106) Yarn logs CLI filtering by application attempt
[
https://issues.apache.org/jira/browse/YARN-10106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150202#comment-17150202
]
Hadoop QA commented on YARN-10106:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} YARN-10106 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10106 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13006903/YARN-10106.004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/26247/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Yarn logs CLI filtering by application attempt
> --
>
> Key: YARN-10106
> URL: https://issues.apache.org/jira/browse/YARN-10106
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Adam Antal
>Assignee: Hudáky Márton Gyula
>Priority: Trivial
> Attachments: YARN-10106.001.patch, YARN-10106.002.patch,
> YARN-10106.003.patch, YARN-10106.004.patch
>
>
> {{ContainerLogsRequest}} got a new parameter in YARN-10101, which is the
> {{applicationAttempt}} - we can use this new parameter in Yarn logs CLI as
> well to filter by application attempt.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10106) Yarn logs CLI filtering by application attempt
[
https://issues.apache.org/jira/browse/YARN-10106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150194#comment-17150194
]
Hudáky Márton Gyula commented on YARN-10106:
Thank you for the remark, [~adam.antal]!
After fixing it every test is passing here locally.
> Yarn logs CLI filtering by application attempt
> --
>
> Key: YARN-10106
> URL: https://issues.apache.org/jira/browse/YARN-10106
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Adam Antal
>Assignee: Hudáky Márton Gyula
>Priority: Trivial
> Attachments: YARN-10106.001.patch, YARN-10106.002.patch,
> YARN-10106.003.patch, YARN-10106.004.patch
>
>
> {{ContainerLogsRequest}} got a new parameter in YARN-10101, which is the
> {{applicationAttempt}} - we can use this new parameter in Yarn logs CLI as
> well to filter by application attempt.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10106) Yarn logs CLI filtering by application attempt
[
https://issues.apache.org/jira/browse/YARN-10106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hudáky Márton Gyula updated YARN-10106:
---
Attachment: YARN-10106.004.patch
> Yarn logs CLI filtering by application attempt
> --
>
> Key: YARN-10106
> URL: https://issues.apache.org/jira/browse/YARN-10106
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Adam Antal
>Assignee: Hudáky Márton Gyula
>Priority: Trivial
> Attachments: YARN-10106.001.patch, YARN-10106.002.patch,
> YARN-10106.003.patch, YARN-10106.004.patch
>
>
> {{ContainerLogsRequest}} got a new parameter in YARN-10101, which is the
> {{applicationAttempt}} - we can use this new parameter in Yarn logs CLI as
> well to filter by application attempt.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10319) Record Last N Scheduler Activities from ActivitiesManager
[ https://issues.apache.org/jira/browse/YARN-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150173#comment-17150173 ] Prabhu Joseph commented on YARN-10319: -- Thanks [~Tao Yang] and [~adam.antal] for reviewing. Have addressed above comments in the latest patch [^YARN-10319-005.patch] . Document change can be viewed [here|https://tinyurl.com/y7mpok4q] > Record Last N Scheduler Activities from ActivitiesManager > - > > Key: YARN-10319 > URL: https://issues.apache.org/jira/browse/YARN-10319 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: activitiesmanager > Attachments: Screen Shot 2020-06-18 at 1.26.31 PM.png, > YARN-10319-001-WIP.patch, YARN-10319-002.patch, YARN-10319-003.patch, > YARN-10319-004.patch, YARN-10319-005.patch > > > ActivitiesManager records a call flow for a given nodeId or a last call flow. > This is useful when debugging the issue live where the user queries with > right nodeId. But capturing last N scheduler activities during the issue > period can help to debug the issue offline. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10319) Record Last N Scheduler Activities from ActivitiesManager
[ https://issues.apache.org/jira/browse/YARN-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-10319: - Attachment: YARN-10319-005.patch > Record Last N Scheduler Activities from ActivitiesManager > - > > Key: YARN-10319 > URL: https://issues.apache.org/jira/browse/YARN-10319 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: activitiesmanager > Attachments: Screen Shot 2020-06-18 at 1.26.31 PM.png, > YARN-10319-001-WIP.patch, YARN-10319-002.patch, YARN-10319-003.patch, > YARN-10319-004.patch, YARN-10319-005.patch > > > ActivitiesManager records a call flow for a given nodeId or a last call flow. > This is useful when debugging the issue live where the user queries with > right nodeId. But capturing last N scheduler activities during the issue > period can help to debug the issue offline. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10336) RM page should throw exception when command injected in RM REST API to get applications
[ https://issues.apache.org/jira/browse/YARN-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T reassigned YARN-10336: Assignee: Bilwa S T > RM page should throw exception when command injected in RM REST API to get > applications > --- > > Key: YARN-10336 > URL: https://issues.apache.org/jira/browse/YARN-10336 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rajshree Mishra >Assignee: Bilwa S T >Priority: Major > Attachments: CommandInject.jpg, RM_UI.jpg > > > Using a web application attacking, we see that injecting commands like > ACCEPTED, FAILED and FINISHED to RM REST API does not throw an exception. > Refer images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10336) RM page should throw exception when command injected in RM REST API to get applications
Rajshree Mishra created YARN-10336: -- Summary: RM page should throw exception when command injected in RM REST API to get applications Key: YARN-10336 URL: https://issues.apache.org/jira/browse/YARN-10336 Project: Hadoop YARN Issue Type: Bug Reporter: Rajshree Mishra Attachments: CommandInject.jpg, RM_UI.jpg Using a web application attacking, we see that injecting commands like ACCEPTED, FAILED and FINISHED to RM REST API does not throw an exception. Refer images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
