[jira] [Created] (YARN-11589) Router Policy support user as default queue
liu bin created YARN-11589: -- Summary: Router Policy support user as default queue Key: YARN-11589 URL: https://issues.apache.org/jira/browse/YARN-11589 Project: Hadoop YARN Issue Type: Improvement Components: federation Reporter: liu bin In my cluster, RM is configured to map users to queues, and users do not specify queues when submitting jobs. When using router, we do not want to change user behavior, so I think we can add config to use user as default queue when getting router policy by queue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11590) RM process stuck after confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely
Ferenc Erdelyi created YARN-11590: - Summary: RM process stuck after confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely Key: YARN-11590 URL: https://issues.apache.org/jira/browse/YARN-11590 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Ferenc Erdelyi YARN-11468 enabled Zookeeper SSL/TLS support for YARN. Curator uses ClientCnxnSocketNetty for secured connection and the thread needs to be closed with confStore.close() after calling confStore.format() to avoid the netty thread to wait indefinitely, which renders the RM unresponsive after deleting the confstore when started with the "-format-conf-store" arg. The unclosed thread, which keeps RM running: {code:java} 2023-10-10 12:13:01,000 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: The Thread[main-SendThread(ferdelyi-1.ferdelyi.root.hwx.site:2182),5,main]TIMED_WAITING is stands at [sun.misc.Unsafe.park(Native Method), java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522), java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684), org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:275), org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)] {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11590) RM process stuck after confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely
[ https://issues.apache.org/jira/browse/YARN-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdelyi reassigned YARN-11590: - Assignee: Ferenc Erdelyi > RM process stuck after confStore.format() when ZK SSL/TLS is enabled, as > netty thread waits indefinitely > - > > Key: YARN-11590 > URL: https://issues.apache.org/jira/browse/YARN-11590 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Ferenc Erdelyi >Assignee: Ferenc Erdelyi >Priority: Major > > YARN-11468 enabled Zookeeper SSL/TLS support for YARN. > Curator uses ClientCnxnSocketNetty for secured connection and the thread > needs to be closed with confStore.close() after calling confStore.format() to > avoid the netty thread to wait indefinitely, which renders the RM > unresponsive after deleting the confstore when started with the > "-format-conf-store" arg. > The unclosed thread, which keeps RM running: > {code:java} > 2023-10-10 12:13:01,000 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: The > Thread[main-SendThread(ferdelyi-1.ferdelyi.root.hwx.site:2182),5,main]TIMED_WAITING > is stands at [sun.misc.Unsafe.park(Native Method), > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), > > java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522), > java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684), > org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:275), > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11590) RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely
[ https://issues.apache.org/jira/browse/YARN-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdelyi updated YARN-11590: -- Summary: RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely (was: RM process stuck after confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely) > RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, > as netty thread waits indefinitely > - > > Key: YARN-11590 > URL: https://issues.apache.org/jira/browse/YARN-11590 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Ferenc Erdelyi >Assignee: Ferenc Erdelyi >Priority: Major > > YARN-11468 enabled Zookeeper SSL/TLS support for YARN. > Curator uses ClientCnxnSocketNetty for secured connection and the thread > needs to be closed with confStore.close() after calling confStore.format() to > avoid the netty thread to wait indefinitely, which renders the RM > unresponsive after deleting the confstore when started with the > "-format-conf-store" arg. > The unclosed thread, which keeps RM running: > {code:java} > 2023-10-10 12:13:01,000 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: The > Thread[main-SendThread(ferdelyi-1.ferdelyi.root.hwx.site:2182),5,main]TIMED_WAITING > is stands at [sun.misc.Unsafe.park(Native Method), > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), > > java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522), > java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684), > org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:275), > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11590) RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely
[ https://issues.apache.org/jira/browse/YARN-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdelyi updated YARN-11590: -- Description: YARN-11468 enabled Zookeeper SSL/TLS support for YARN. Curator uses ClientCnxnSocketNetty for secured connection and the thread needs to be closed after calling confStore.format() to avoid the netty thread waiting indefinitely, which renders the RM unresponsive after deleting the confstore when started with the "-format-conf-store" arg. The unclosed thread, which keeps RM running: {code:java} 2023-10-10 12:13:01,000 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: The Thread[main-SendThread(ferdelyi-1.ferdelyi.root.hwx.site:2182),5,main]TIMED_WAITING is stands at [sun.misc.Unsafe.park(Native Method), java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522), java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684), org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:275), org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)] {code} was: YARN-11468 enabled Zookeeper SSL/TLS support for YARN. Curator uses ClientCnxnSocketNetty for secured connection and the thread needs to be closed with confStore.close() after calling confStore.format() to avoid the netty thread to wait indefinitely, which renders the RM unresponsive after deleting the confstore when started with the "-format-conf-store" arg. The unclosed thread, which keeps RM running: {code:java} 2023-10-10 12:13:01,000 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: The Thread[main-SendThread(ferdelyi-1.ferdelyi.root.hwx.site:2182),5,main]TIMED_WAITING is stands at [sun.misc.Unsafe.park(Native Method), java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522), java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684), org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:275), org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)] {code} > RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, > as netty thread waits indefinitely > - > > Key: YARN-11590 > URL: https://issues.apache.org/jira/browse/YARN-11590 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Ferenc Erdelyi >Assignee: Ferenc Erdelyi >Priority: Major > > YARN-11468 enabled Zookeeper SSL/TLS support for YARN. > Curator uses ClientCnxnSocketNetty for secured connection and the thread > needs to be closed after calling confStore.format() to avoid the netty thread > waiting indefinitely, which renders the RM unresponsive after deleting the > confstore when started with the "-format-conf-store" arg. > The unclosed thread, which keeps RM running: > {code:java} > 2023-10-10 12:13:01,000 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: The > Thread[main-SendThread(ferdelyi-1.ferdelyi.root.hwx.site:2182),5,main]TIMED_WAITING > is stands at [sun.misc.Unsafe.park(Native Method), > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), > > java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522), > java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684), > org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:275), > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11590) RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely
[ https://issues.apache.org/jira/browse/YARN-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773709#comment-17773709 ] Ferenc Erdelyi commented on YARN-11590: --- Thanks to [~bkosztolnik] for identifying the cause of the issue and suggesting a solution. > RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, > as netty thread waits indefinitely > - > > Key: YARN-11590 > URL: https://issues.apache.org/jira/browse/YARN-11590 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Ferenc Erdelyi >Assignee: Ferenc Erdelyi >Priority: Major > > YARN-11468 enabled Zookeeper SSL/TLS support for YARN. > Curator uses ClientCnxnSocketNetty for secured connection and the thread > needs to be closed after calling confStore.format() to avoid the netty thread > waiting indefinitely, which renders the RM unresponsive after deleting the > confstore when started with the "-format-conf-store" arg. > The unclosed thread, which keeps RM running: > {code:java} > 2023-10-10 12:13:01,000 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: The > Thread[main-SendThread(ferdelyi-1.ferdelyi.root.hwx.site:2182),5,main]TIMED_WAITING > is stands at [sun.misc.Unsafe.park(Native Method), > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), > > java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522), > java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684), > org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:275), > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11590) RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely
[ https://issues.apache.org/jira/browse/YARN-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11590: -- Labels: pull-request-available (was: ) > RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, > as netty thread waits indefinitely > - > > Key: YARN-11590 > URL: https://issues.apache.org/jira/browse/YARN-11590 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Ferenc Erdelyi >Assignee: Ferenc Erdelyi >Priority: Major > Labels: pull-request-available > > YARN-11468 enabled Zookeeper SSL/TLS support for YARN. > Curator uses ClientCnxnSocketNetty for secured connection and the thread > needs to be closed after calling confStore.format() to avoid the netty thread > waiting indefinitely, which renders the RM unresponsive after deleting the > confstore when started with the "-format-conf-store" arg. > The unclosed thread, which keeps RM running: > {code:java} > 2023-10-10 12:13:01,000 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: The > Thread[main-SendThread(ferdelyi-1.ferdelyi.root.hwx.site:2182),5,main]TIMED_WAITING > is stands at [sun.misc.Unsafe.park(Native Method), > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), > > java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522), > java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684), > org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:275), > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11590) RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely
[ https://issues.apache.org/jira/browse/YARN-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773716#comment-17773716 ] ASF GitHub Bot commented on YARN-11590: --- ferdelyi opened a new pull request, #6166: URL: https://github.com/apache/hadoop/pull/6166 … SSL/TLS is enabled, as netty thread waits indefinitely ### Description of PR YarnConfigurationStore now implements AutoClosable, hence a netty thread is not lingering around when the confStore is deleted by RM. ### How was this patch tested? Tested on my local cluster. There is no unit test for the secure Curator due to CURATOR-658 "Add Support for TLS-enabled TestingZooKeeperMain" won't be fixed. ### For code changes: - [ x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, > as netty thread waits indefinitely > - > > Key: YARN-11590 > URL: https://issues.apache.org/jira/browse/YARN-11590 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Ferenc Erdelyi >Assignee: Ferenc Erdelyi >Priority: Major > > YARN-11468 enabled Zookeeper SSL/TLS support for YARN. > Curator uses ClientCnxnSocketNetty for secured connection and the thread > needs to be closed after calling confStore.format() to avoid the netty thread > waiting indefinitely, which renders the RM unresponsive after deleting the > confstore when started with the "-format-conf-store" arg. > The unclosed thread, which keeps RM running: > {code:java} > 2023-10-10 12:13:01,000 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: The > Thread[main-SendThread(ferdelyi-1.ferdelyi.root.hwx.site:2182),5,main]TIMED_WAITING > is stands at [sun.misc.Unsafe.park(Native Method), > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), > > java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522), > java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684), > org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:275), > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11590) RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely
[ https://issues.apache.org/jira/browse/YARN-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773717#comment-17773717 ] ASF GitHub Bot commented on YARN-11590: --- K0K0V0K commented on PR #6166: URL: https://github.com/apache/hadoop/pull/6166#issuecomment-174773 Thanks @ferdelyi! LGTM > RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, > as netty thread waits indefinitely > - > > Key: YARN-11590 > URL: https://issues.apache.org/jira/browse/YARN-11590 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Ferenc Erdelyi >Assignee: Ferenc Erdelyi >Priority: Major > Labels: pull-request-available > > YARN-11468 enabled Zookeeper SSL/TLS support for YARN. > Curator uses ClientCnxnSocketNetty for secured connection and the thread > needs to be closed after calling confStore.format() to avoid the netty thread > waiting indefinitely, which renders the RM unresponsive after deleting the > confstore when started with the "-format-conf-store" arg. > The unclosed thread, which keeps RM running: > {code:java} > 2023-10-10 12:13:01,000 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: The > Thread[main-SendThread(ferdelyi-1.ferdelyi.root.hwx.site:2182),5,main]TIMED_WAITING > is stands at [sun.misc.Unsafe.park(Native Method), > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), > > java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522), > java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684), > org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:275), > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10174) Add colored policies to enable manual load balancing across sub clusters
[ https://issues.apache.org/jira/browse/YARN-10174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773720#comment-17773720 ] ASF GitHub Bot commented on YARN-10174: --- slfan1989 commented on PR #6156: URL: https://github.com/apache/hadoop/pull/6156#issuecomment-1755610309 @zhengchenyu Thank you for your contribution! However, these code changes involve significant alterations, and it will take some time to validate whether they meet our expectations. Patience is crucial during this process. From a personal perspective, I hope that adding this feature will require as few modifications as possible to the code of other policies. I will organize my thoughts and continue the discussion with you as soon as possible. > Add colored policies to enable manual load balancing across sub clusters > > > Key: YARN-10174 > URL: https://issues.apache.org/jira/browse/YARN-10174 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Young Chen >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > > Add colored policies to enable manual load balancing across sub clusters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9013) [GPG] fix order of steps cleaning Registry entries in ApplicationCleaner
[ https://issues.apache.org/jira/browse/YARN-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773754#comment-17773754 ] ASF GitHub Bot commented on YARN-9013: -- hadoop-yetus commented on PR #6147: URL: https://github.com/apache/hadoop/pull/6147#issuecomment-1755859863 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 49s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 47m 36s | | trunk passed | | +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 | | +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | +1 :green_heart: | checkstyle | 0m 23s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 27s | | trunk passed | | +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 | | +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | +1 :green_heart: | spotbugs | 0m 45s | | trunk passed | | +1 :green_heart: | shadedclient | 37m 16s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 18s | | the patch passed | | +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 | | +1 :green_heart: | javac | 0m 17s | | the patch passed | | +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | +1 :green_heart: | javac | 0m 16s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 13s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 19s | | the patch passed | | +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 | | +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | +1 :green_heart: | spotbugs | 0m 44s | | the patch passed | | +1 :green_heart: | shadedclient | 37m 24s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 0m 49s | | hadoop-yarn-server-globalpolicygenerator in the patch passed. | | +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. | | | | 134m 12s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6147/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/6147 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | Linux 6c44cd20d9f1 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / ff587895a58a8263a9265f92d0a9344b60fee8ea | | Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6147/2/testReport/ | | Max. process+thread count | 602 (vs. ulimit of 5500) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6147/2/console | | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.14.0 https://yet
[jira] [Commented] (YARN-11590) RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely
[ https://issues.apache.org/jira/browse/YARN-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773762#comment-17773762 ] ASF GitHub Bot commented on YARN-11590: --- hadoop-yetus commented on PR #6166: URL: https://github.com/apache/hadoop/pull/6166#issuecomment-1755912925 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 25s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 2s | | trunk passed | | +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | +1 :green_heart: | checkstyle | 0m 39s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 41s | | trunk passed | | +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | +1 :green_heart: | spotbugs | 1m 19s | | trunk passed | | +1 :green_heart: | shadedclient | 21m 17s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 32s | | the patch passed | | +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | javac | 0m 35s | | the patch passed | | +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | +1 :green_heart: | javac | 0m 31s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 27s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 32s | | the patch passed | | +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | +1 :green_heart: | spotbugs | 1m 14s | | the patch passed | | +1 :green_heart: | shadedclient | 21m 5s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 85m 58s | | hadoop-yarn-server-resourcemanager in the patch passed. | | +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. | | | | 172m 18s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6166/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/6166 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | Linux c4c19e2f6b36 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 62f8fc3ee8b476787191db7d17b049fec923fef1 | | Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6166/1/testReport/ | | Max. process+thread count | 962 (vs. ulimit of 5500) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/P
[jira] [Commented] (YARN-11484) [Federation] Router Supports Yarn Client CLI Cmds.
[ https://issues.apache.org/jira/browse/YARN-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773829#comment-17773829 ] ASF GitHub Bot commented on YARN-11484: --- goiri commented on code in PR #6132: URL: https://github.com/apache/hadoop/pull/6132#discussion_r1353390005 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java: ## @@ -1969,6 +1971,98 @@ public void testGetQueueInfoWithEmptyNodeLabel() throws Exception { String queueInfoStr = baos.toString("UTF-8"); Assert.assertEquals(queueInfoStr, sysOutStream.toString()); } + + @Test + public void testGetQueueInfoWithFairScheduler() throws Exception { +// In this test case, we will simulate the queue information of fairScheduler +// and check the results of the queue information. +QueueCLI cli = createAndGetQueueCLI(); +RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null); +QueueInfo queueInfo = recordFactory.newRecordInstance(QueueInfo.class); +queueInfo.setQueueName("queueA"); +queueInfo.setSchedulerType("FairScheduler"); +queueInfo.setQueueState(QueueState.RUNNING); +queueInfo.setCapacity(0.3f); +queueInfo.setCurrentCapacity(0.1f); +queueInfo.setWeight(0.3f); +queueInfo.setMinResourceVCore(1); +queueInfo.setMinResourceMemory(1024); +queueInfo.setMaxResourceVCore(10); +queueInfo.setMaxResourceMemory(8192); +queueInfo.setReservedResourceVCore(0); +queueInfo.setReservedResourceMemory(0); +queueInfo.setSteadyFairShareVCore(10); +queueInfo.setSteadyFairShareMemory(8192); +queueInfo.setMaxRunningApp(10); +queueInfo.setPreemptionDisabled(true); +when(client.getQueueInfo(any(String.class))).thenReturn(queueInfo); +int result = cli.run(new String[] { "-status", "queueA" }); +assertEquals(0, result); +verify(client).getQueueInfo("queueA"); +ByteArrayOutputStream baos = new ByteArrayOutputStream(); +PrintWriter pw = new PrintWriter(baos); +pw.println("Queue Information : "); +pw.println("Scheduler Name : FairScheduler"); +pw.println("Queue Name : " + "queueA"); +pw.println("\tWeight : " + "0.30"); +pw.println("\tState : " + "RUNNING"); +pw.println("\tMinResource : " + ""); +pw.println("\tMaxResource : " + ""); Review Comment: Why not just a single one without the +? ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestFederationClientInterceptor.java: ## @@ -1174,6 +1174,22 @@ public void testGetQueueInfo() throws Exception { Assert.assertEquals(queueInfo.getAccessibleNodeLabels().size(), 1); } + @Test + public void testSubClusterGetQueueInfo() throws IOException, YarnException { +// We have set up a unit test where we access queue information for subcluster1. +GetQueueInfoResponse response = interceptor.getQueueInfo( +GetQueueInfoRequest.newInstance("root", true, true, true, "1")); +Assert.assertNotNull(response); + +QueueInfo queueInfo = response.getQueueInfo(); +Assert.assertNotNull(queueInfo); +Assert.assertEquals(queueInfo.getQueueName(), "root"); Review Comment: Usually expected is first. > [Federation] Router Supports Yarn Client CLI Cmds. > -- > > Key: YARN-11484 > URL: https://issues.apache.org/jira/browse/YARN-11484 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > > This Jira ticket aims to enhance the Router command by adding support for all > Yarn Client CLI options. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11589) Router Policy support user as default queue
[ https://issues.apache.org/jira/browse/YARN-11589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773886#comment-17773886 ] ASF GitHub Bot commented on YARN-11589: --- liubin101 opened a new pull request, #6171: URL: https://github.com/apache/hadoop/pull/6171 ### Description of PR In my cluster, RM is configured to map users to queues, and users do not specify queues when submitting jobs. When using router, we do not want to change user behavior, so I think we can add config to use user as default queue when getting router policy by queue. ### How was this patch tested? unit test > Router Policy support user as default queue > --- > > Key: YARN-11589 > URL: https://issues.apache.org/jira/browse/YARN-11589 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Reporter: liu bin >Priority: Major > > In my cluster, RM is configured to map users to queues, and users do not > specify queues when submitting jobs. > When using router, we do not want to change user behavior, so I think we can > add config to use user as default queue when getting router policy by queue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11589) Router Policy support user as default queue
[ https://issues.apache.org/jira/browse/YARN-11589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11589: -- Labels: pull-request-available (was: ) > Router Policy support user as default queue > --- > > Key: YARN-11589 > URL: https://issues.apache.org/jira/browse/YARN-11589 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Reporter: liu bin >Priority: Major > Labels: pull-request-available > > In my cluster, RM is configured to map users to queues, and users do not > specify queues when submitting jobs. > When using router, we do not want to change user behavior, so I think we can > add config to use user as default queue when getting router policy by queue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11589) Router Policy support user as default queue
[ https://issues.apache.org/jira/browse/YARN-11589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773910#comment-17773910 ] ASF GitHub Bot commented on YARN-11589: --- hadoop-yetus commented on PR #6171: URL: https://github.com/apache/hadoop/pull/6171#issuecomment-1756868634 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 27s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. | | +0 :ok: | xmllint | 0m 0s | | xmllint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +0 :ok: | mvndep | 16m 32s | | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 20m 16s | | trunk passed | | +1 :green_heart: | compile | 4m 24s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 | | +1 :green_heart: | compile | 4m 2s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | +1 :green_heart: | checkstyle | 1m 15s | | trunk passed | | +1 :green_heart: | mvnsite | 2m 59s | | trunk passed | | +1 :green_heart: | javadoc | 3m 1s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 | | +1 :green_heart: | javadoc | 2m 51s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | +1 :green_heart: | spotbugs | 4m 57s | | trunk passed | | +1 :green_heart: | shadedclient | 21m 0s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 35s | | the patch passed | | +1 :green_heart: | compile | 4m 7s | | the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 | | +1 :green_heart: | javac | 4m 7s | | the patch passed | | +1 :green_heart: | compile | 3m 52s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | | +1 :green_heart: | javac | 3m 52s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | -0 :warning: | checkstyle | 1m 6s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6171/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 165 unchanged - 0 fixed = 167 total (was 165) | | +1 :green_heart: | mvnsite | 2m 39s | | the patch passed | | -1 :x: | javadoc | 0m 39s | [/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6171/1/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -1 :x: | javadoc | 0m 38s | [/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6171/1/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05 with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | spotbugs | 4m 58s | | the patch passed | | +1 :green_heart: | shadedclient | 21m 15s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 0m 57s | | hadoop-yarn-api in the patch passed. | | +1 :green_heart: | unit | 4m 50s | | hadoop-yarn-common in the patch p