[jira] [Comment Edited] (YARN-5277) when localizers fail due to resource timestamps being out, provide more diagnostics
[
https://issues.apache.org/jira/browse/YARN-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065269#comment-17065269
]
Siddharth Ahuja edited comment on YARN-5277 at 3/24/20, 2:56 AM:
-
Hi [~aajisaka], I am working on this JIRA and have a potential
fix/implementation in terms of non-test source code. However, I did have a
question regarding the Junit code coverage tool -> _Clover_ .
I tried to run the following command:
{code}
mvn test -Pclover
{code}
but it resulted in the following error:
{code}
Failed to execute goal
com.atlassian.maven.plugins:maven-clover2-plugin:3.3.0:setup (clover-setup) on
project hadoop-main: Failed to load resource as file
[/Users//.clover.license]: Could not find resource
'/Users//.clover.license'. -> [Help 1]
that I tried to run to see if we are already covering the impacted code through
Junit testing or not. I used the following command to run it:
{code}
I could try and supply a clover license through :
{code}
mvn test -Pclover [-DcloverLicenseLocation=${user.name}/.clover.license]
{code}
as per
https://svn.apache.org/repos/asf/hadoop/common/branches/MR-4327/BUILDING.txt,
however, I need the clover.license.
I somehow found a link where I could get that potentially -
https://svn.apache.org/repos/private/committers/donated-licenses/clover/2.6.x/clover.license
but as I am not a committer, I don't have the credentials (I get asked for
username/password).
As such, can you kindly help me with a clover license? I am really interesting
in getting this so that I know if we already have an existing test method in
the test class that already covers what I am trying to modify and hence, I can
just update that method. If it is not covered yet, then, I will have to write
up a new junit test for that. I don't want to be reviewing multiple existing
test methods to understand if something is covered or not as this approach is
not robust.
Thanks in advance for your kind assistance!
was (Author: sahuja):
Hi [~aajisaka], I am working on this JIRA and have a potential
fix/implementation in terms of non-test source code. However, I did have a
question regarding the Junit code coverage tool -> _Clover_ .
I tried to run the following command:
{code}
mvn test -Pclover
{code}
but it resulted in the following error:
{code}
Failed to execute goal
com.atlassian.maven.plugins:maven-clover2-plugin:3.3.0:setup (clover-setup) on
project hadoop-main: Failed to load resource as file
[/Users//.clover.license]: Could not find resource
'/Users//.clover.license'. -> [Help 1]
that I tried to run to see if we are already covering the impacted code through
Junit testing or not. I used the following command to run it:
{code}
I could try and supply a clover license through :
{code}
mvn test -Pclover [-DcloverLicenseLocation=${user.name}/.clover.license]
{code}
as per
https://svn.apache.org/repos/asf/hadoop/common/branches/MR-4327/BUILDING.txt,
however, I need the clover.license.
I somehow found a link where I could get that potentially -
https://svn.apache.org/repos/private/committers/donated-licenses/clover/2.6.x/clover.license
but as I am not a committer, I don't have the credentials (I get asked for
username/password).
As such, can you kindly help me with a clover license? I am really interesting
in getting this so that I know if we already have an existing test method in
the test class that already covers what I am trying to modify and hence, I can
just update that method. If it is not covered yet, then, I will have to write
up a new junit test for that.
Thanks in advance for your kind assistance!
> when localizers fail due to resource timestamps being out, provide more
> diagnostics
> ---
>
> Key: YARN-5277
> URL: https://issues.apache.org/jira/browse/YARN-5277
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Siddharth Ahuja
>Priority: Major
>
> When an NM fails a resource D/L as the timestamps are wrong, there's not much
> info, just two long values.
> It would be good to also include the local time values, *and the current wall
> time*. These are the things people need to know when trying to work out what
> went wrong
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5277) when localizers fail due to resource timestamps being out, provide more diagnostics
[
https://issues.apache.org/jira/browse/YARN-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065269#comment-17065269
]
Siddharth Ahuja edited comment on YARN-5277 at 3/24/20, 2:54 AM:
-
Hi [~aajisaka], I am working on this JIRA and have a potential
fix/implementation in terms of non-test source code. However, I did have a
question regarding the Junit code coverage tool -> _Clover_ .
I tried to run the following command:
{code}
mvn test -Pclover
{code}
but it resulted in the following error:
{code}
Failed to execute goal
com.atlassian.maven.plugins:maven-clover2-plugin:3.3.0:setup (clover-setup) on
project hadoop-main: Failed to load resource as file
[/Users//.clover.license]: Could not find resource
'/Users//.clover.license'. -> [Help 1]
that I tried to run to see if we are already covering the impacted code through
Junit testing or not. I used the following command to run it:
{code}
I could try and supply a clover license through :
{code}
mvn test -Pclover [-DcloverLicenseLocation=${user.name}/.clover.license]
{code}
as per
https://svn.apache.org/repos/asf/hadoop/common/branches/MR-4327/BUILDING.txt,
however, I need the clover.license.
I somehow found a link where I could get that potentially -
https://svn.apache.org/repos/private/committers/donated-licenses/clover/2.6.x/clover.license
but as I am not a committer, I don't have the credentials (I get asked for
username/password).
As such, can you kindly help me with a clover license? I am really interesting
in getting this so that I know if we already have an existing test method in
the test class that already covers what I am trying to modify and hence, I can
just update that method. If it is not covered yet, then, I will have to write
up a new junit test for that.
Thanks in advance for your kind assistance!
was (Author: sahuja):
Hi [~aajisaka], I am working on this JIRA and have a potential
fix/implementation in terms of non-test source code. However, I did have a
question regarding the Junit code coverage tool -> _Clover_ .
I tried to run the following command:
{code}
mvn test -Pclover
{code}
but it resulted in the following error:
{code}
Failed to execute goal
com.atlassian.maven.plugins:maven-clover2-plugin:3.3.0:setup (clover-setup) on
project hadoop-main: Failed to load resource as file
[/Users//.clover.license]: Could not find resource
'/Users/sidtheadmin/.clover.license'. -> [Help 1]
that I tried to run to see if we are already covering the impacted code through
Junit testing or not. I used the following command to run it:
{code}
I could try and supply a clover license through :
{code}
mvn test -Pclover [-DcloverLicenseLocation=${user.name}/.clover.license]
{code}
as per
https://svn.apache.org/repos/asf/hadoop/common/branches/MR-4327/BUILDING.txt,
however, I need the clover.license.
I somehow found a link where I could get that potentially -
https://svn.apache.org/repos/private/committers/donated-licenses/clover/2.6.x/clover.license
but as I am not a committer, I don't have the credentials (I get asked for
username/password).
As such, can you kindly help me with a clover license? I am really interesting
in getting this so that I know if we already have an existing test method in
the test class that already covers what I am trying to modify and hence, I can
just update that method. If it is not covered yet, then, I will have to write
up a new junit test for that.
Thanks in advance for your kind assistance!
> when localizers fail due to resource timestamps being out, provide more
> diagnostics
> ---
>
> Key: YARN-5277
> URL: https://issues.apache.org/jira/browse/YARN-5277
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Siddharth Ahuja
>Priority: Major
>
> When an NM fails a resource D/L as the timestamps are wrong, there's not much
> info, just two long values.
> It would be good to also include the local time values, *and the current wall
> time*. These are the things people need to know when trying to work out what
> went wrong
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5277) when localizers fail due to resource timestamps being out, provide more diagnostics
[
https://issues.apache.org/jira/browse/YARN-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065269#comment-17065269
]
Siddharth Ahuja commented on YARN-5277:
---
Hi [~aajisaka], I am working on this JIRA and have a potential
fix/implementation in terms of non-test source code. However, I did have a
question regarding the Junit code coverage tool -> _Clover_ .
I tried to run the following command:
{code}
mvn test -Pclover
{code}
but it resulted in the following error:
{code}
Failed to execute goal
com.atlassian.maven.plugins:maven-clover2-plugin:3.3.0:setup (clover-setup) on
project hadoop-main: Failed to load resource as file
[/Users//.clover.license]: Could not find resource
'/Users/sidtheadmin/.clover.license'. -> [Help 1]
that I tried to run to see if we are already covering the impacted code through
Junit testing or not. I used the following command to run it:
{code}
I could try and supply a clover license through :
{code}
mvn test -Pclover [-DcloverLicenseLocation=${user.name}/.clover.license]
{code}
as per
https://svn.apache.org/repos/asf/hadoop/common/branches/MR-4327/BUILDING.txt,
however, I need the clover.license.
I somehow found a link where I could get that potentially -
https://svn.apache.org/repos/private/committers/donated-licenses/clover/2.6.x/clover.license
but as I am not a committer, I don't have the credentials (I get asked for
username/password).
As such, can you kindly help me with a clover license? I am really interesting
in getting this so that I know if we already have an existing test method in
the test class that already covers what I am trying to modify and hence, I can
just update that method. If it is not covered yet, then, I will have to write
up a new junit test for that.
Thanks in advance for your kind assistance!
> when localizers fail due to resource timestamps being out, provide more
> diagnostics
> ---
>
> Key: YARN-5277
> URL: https://issues.apache.org/jira/browse/YARN-5277
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Siddharth Ahuja
>Priority: Major
>
> When an NM fails a resource D/L as the timestamps are wrong, there's not much
> info, just two long values.
> It would be good to also include the local time values, *and the current wall
> time*. These are the things people need to know when trying to work out what
> went wrong
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2710) RM HA tests failed intermittently on trunk
[
https://issues.apache.org/jira/browse/YARN-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065222#comment-17065222
]
Ahmed Hussein commented on YARN-2710:
-
For branch-3.2, it has different behavior than the issue description. The test
case hangs while calling {{ResourceTrackerPBClientImpl.registerNodeManager()}}.
Should this be a separate Jira?
{code:bash}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 19.126
s <<< FAILURE! - in org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
[ERROR]
testResourceTrackerOnHA(org.apache.hadoop.yarn.client.TestResourceTrackerOnHA)
Time elapsed: 18.96 s <<< ERROR!
java.lang.Exception: test timed out after 15000 milliseconds
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:336)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:699)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
at org.apache.hadoop.ipc.Client.call(Client.java:1452)
at org.apache.hadoop.ipc.Client.call(Client.java:1405)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy92.registerNodeManager(Unknown Source)
at
org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy93.registerNodeManager(Unknown Source)
at
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA.testResourceTrackerOnHA(TestResourceTrackerOnHA.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
+*the log output is*+
{code:bash}
020-03-23 19:56:10,731 INFO server.MiniYARNCluster
(MiniYARNCluster.java:waitForNodeManagersToConnect(793)) - All Node Managers
connected in MiniYARNCluster
2020-03-23 19:56:10,842 INFO client.ConfiguredRMFailoverProxyProvider
(ConfiguredRMFailoverProxyProvider.java:performFailover(100)) - Failing over to
rm2
2020-03-23 19:56:10,847 INFO resourcemanager.RMAuditLogger
(RMAuditLogger.java:logSuccess(386)) - USER=ahusseinIP=10.0.0.110
OPERATION=Get Applications Request TARGET=ClientRMService RESULT=SUCCESS
2020-03-23 19:56:10,880 INFO zookeeper.JUnit4ZKTestRunner
(JUnit4ZKTestRunner.java:evaluate(53)) - RUNNING TEST METHOD
testResourceTrackerOnHA
2020-03-23 19:56:25,884 ERROR
[jira] [Commented] (YARN-10200) Add number of containers to RMAppManager summary
[ https://issues.apache.org/jira/browse/YARN-10200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065126#comment-17065126 ] Jonathan Hung commented on YARN-10200: -- Thanks Haibo, attached [^YARN-10200.002.patch] to fix checkstyle > Add number of containers to RMAppManager summary > > > Key: YARN-10200 > URL: https://issues.apache.org/jira/browse/YARN-10200 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-10200.001.patch, YARN-10200.002.patch > > > It would be useful to persist this so we can track containers processed by RM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10200) Add number of containers to RMAppManager summary
[ https://issues.apache.org/jira/browse/YARN-10200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-10200: - Attachment: YARN-10200.002.patch > Add number of containers to RMAppManager summary > > > Key: YARN-10200 > URL: https://issues.apache.org/jira/browse/YARN-10200 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-10200.001.patch, YARN-10200.002.patch > > > It would be useful to persist this so we can track containers processed by RM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10200) Add number of containers to RMAppManager summary
[ https://issues.apache.org/jira/browse/YARN-10200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065116#comment-17065116 ] Haibo Chen commented on YARN-10200: --- +1 on 001 patch pending the checkstyle fix. > Add number of containers to RMAppManager summary > > > Key: YARN-10200 > URL: https://issues.apache.org/jira/browse/YARN-10200 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-10200.001.patch > > > It would be useful to persist this so we can track containers processed by RM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10001) Add explanation of unimplemented methods in InMemoryConfigurationStore
[
https://issues.apache.org/jira/browse/YARN-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064845#comment-17064845
]
Siddharth Ahuja edited comment on YARN-10001 at 3/23/20, 2:40 PM:
--
This was the output from the earlier build:
{code}
-1 overall
| Vote |Subsystem | Runtime | Comment
| 0 | reexec | 0m 46s | Docker mode activated.
| | || Prechecks
| +1 | @author | 0m 0s | The patch does not contain any @author
| | || tags.
| -1 | test4tests | 0m 0s | The patch doesn't appear to include
| | || any new or modified tests. Please
| | || justify why no new tests are needed for
| | || this patch. Also please list what
| | || manual steps were performed to verify
| | || this patch.
| | || trunk Compile Tests
| +1 | mvninstall | 21m 48s | trunk passed
| +1 | compile | 0m 45s | trunk passed
| +1 | checkstyle | 0m 35s | trunk passed
| +1 | mvnsite | 0m 47s | trunk passed
| +1 |shadedclient | 15m 31s | branch has no errors when building and
| | || testing our client artifacts.
| +1 |findbugs | 1m 35s | trunk passed
| +1 | javadoc | 0m 30s | trunk passed
| | || Patch Compile Tests
| +1 | mvninstall | 0m 43s | the patch passed
| +1 | compile | 0m 38s | the patch passed
| +1 | javac | 0m 38s | the patch passed
| -0 | checkstyle | 0m 27s |
| | ||
hadoop-yarn-project/hadoop-yarn/hadoop-y
| | ||
arn-server/hadoop-yarn-server-resourcema
| | || nager: The patch generated 7 new + 1
| | || unchanged - 0 fixed = 8 total (was 1)
| +1 | mvnsite | 0m 41s | the patch passed
| +1 | whitespace | 0m 0s | The patch has no whitespace issues.
| +1 |shadedclient | 14m 22s | patch has no errors when building and
| | || testing our client artifacts.
| +1 |findbugs | 1m 40s | the patch passed
| +1 | javadoc | 0m 26s | the patch passed
| | || Other Tests
| +1 |unit | 103m 21s | hadoop-yarn-server-resourcemanager in
| | || the patch passed.
| +1 | asflicense | 0m 25s | The patch does not generate ASF
| | || License warnings.
| | | 164m 49s |
{code}
Note that the changes for this JIRA are only related to comments for methods,
therefore, no new tests were added or modified (they don't need to).
was (Author: sahuja):
This was the output from the earlier build:
{code}
-1 overall
| Vote |Subsystem | Runtime | Comment
| 0 | reexec | 0m 46s | Docker mode activated.
| | || Prechecks
| +1 | @author | 0m 0s | The patch does not contain any @author
| | || tags.
| -1 | test4tests | 0m 0s | The patch doesn't appear to include
| | || any new or modified tests. Please
| | || justify why no new tests are needed for
| | || this patch. Also please list what
| | || manual steps were performed to verify
| | || this patch.
| | || trunk Compile Tests
| +1 | mvninstall | 21m 48s | trunk passed
| +1 | compile | 0m 45s | trunk passed
| +1 | checkstyle | 0m 35s | trunk passed
| +1 | mvnsite | 0m 47s | trunk passed
| +1 |shadedclient | 15m 31s | branch has no errors when building and
| | || testing our client artifacts.
| +1 |findbugs | 1m 35s | trunk passed
| +1 | javadoc | 0m 30s | trunk passed
| | || Patch Compile Tests
| +1 | mvninstall | 0m 43s | the patch passed
| +1 | compile | 0m 38s | the patch passed
| +1 | javac | 0m 38s | the patch passed
| -0 | checkstyle | 0m 27s |
| |
[jira] [Commented] (YARN-10001) Add explanation of unimplemented methods in InMemoryConfigurationStore
[
https://issues.apache.org/jira/browse/YARN-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064845#comment-17064845
]
Siddharth Ahuja commented on YARN-10001:
This was the output from the earlier build:
{code}
-1 overall
| Vote |Subsystem | Runtime | Comment
| 0 | reexec | 0m 46s | Docker mode activated.
| | || Prechecks
| +1 | @author | 0m 0s | The patch does not contain any @author
| | || tags.
| -1 | test4tests | 0m 0s | The patch doesn't appear to include
| | || any new or modified tests. Please
| | || justify why no new tests are needed for
| | || this patch. Also please list what
| | || manual steps were performed to verify
| | || this patch.
| | || trunk Compile Tests
| +1 | mvninstall | 21m 48s | trunk passed
| +1 | compile | 0m 45s | trunk passed
| +1 | checkstyle | 0m 35s | trunk passed
| +1 | mvnsite | 0m 47s | trunk passed
| +1 |shadedclient | 15m 31s | branch has no errors when building and
| | || testing our client artifacts.
| +1 |findbugs | 1m 35s | trunk passed
| +1 | javadoc | 0m 30s | trunk passed
| | || Patch Compile Tests
| +1 | mvninstall | 0m 43s | the patch passed
| +1 | compile | 0m 38s | the patch passed
| +1 | javac | 0m 38s | the patch passed
| -0 | checkstyle | 0m 27s |
| | ||
hadoop-yarn-project/hadoop-yarn/hadoop-y
| | ||
arn-server/hadoop-yarn-server-resourcema
| | || nager: The patch generated 7 new + 1
| | || unchanged - 0 fixed = 8 total (was 1)
| +1 | mvnsite | 0m 41s | the patch passed
| +1 | whitespace | 0m 0s | The patch has no whitespace issues.
| +1 |shadedclient | 14m 22s | patch has no errors when building and
| | || testing our client artifacts.
| +1 |findbugs | 1m 40s | the patch passed
| +1 | javadoc | 0m 26s | the patch passed
| | || Other Tests
| +1 |unit | 103m 21s | hadoop-yarn-server-resourcemanager in
| | || the patch passed.
| +1 | asflicense | 0m 25s | The patch does not generate ASF
| | || License warnings.
| | | 164m 49s |
{code}
Note that the changes for this JIRA are only comments, therefore, no new tests
were added or modified (they don't need to).
> Add explanation of unimplemented methods in InMemoryConfigurationStore
> --
>
> Key: YARN-10001
> URL: https://issues.apache.org/jira/browse/YARN-10001
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: YARN-10001.001.patch, YARN-10001.002.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10001) Add explanation of unimplemented methods in InMemoryConfigurationStore
[ https://issues.apache.org/jira/browse/YARN-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064836#comment-17064836 ] Siddharth Ahuja commented on YARN-10001: Found checkstyle warnings coming from https://builds.apache.org/job/PreCommit-YARN-Build/25734/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt. Got checkstyle checks imported as per https://github.com/apache/hadoop/tree/trunk/hadoop-build-tools/src/main/resources/checkstyle/ in IntelliJ and managed to receive the same warnings there so I should be good for future patches. Fixed them all up and delivering the new patch now. > Add explanation of unimplemented methods in InMemoryConfigurationStore > -- > > Key: YARN-10001 > URL: https://issues.apache.org/jira/browse/YARN-10001 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Siddharth Ahuja >Priority: Major > Attachments: YARN-10001.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064779#comment-17064779
]
Hudson commented on YARN-10198:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18073 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18073/])
YARN-10198. Remove Validation of Queue Exist under Managed Parent Queue
(pjoseph: rev d312991ead37a5f840577a39420a18f0621b09dc)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-10198-001.patch, YARN-10198-002.patch,
> YARN-10198-003.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064768#comment-17064768
]
Prabhu Joseph commented on YARN-10198:
--
Thanks [~pbacsko], have pushed the [^YARN-10198-003.patch] to trunk.
Thanks [~maniraj...@gmail.com] for the review.
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch, YARN-10198-002.patch,
> YARN-10198-003.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10194) YARN RMWebServices /scheduler-conf/validate leaks ZK Connections
[
https://issues.apache.org/jira/browse/YARN-10194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-10194:
-
Attachment: YARN-10194-003.patch
> YARN RMWebServices /scheduler-conf/validate leaks ZK Connections
>
>
> Key: YARN-10194
> URL: https://issues.apache.org/jira/browse/YARN-10194
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.3.0
>Reporter: Akhil PB
>Assignee: Prabhu Joseph
>Priority: Critical
> Attachments: YARN-10194-001.patch, YARN-10194-002.patch,
> YARN-10194-003.patch
>
>
> YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Validation
> API creates a new CapacityScheduler and missed to close after the validation.
> Every CapacityScheduler#init opens MutableCSConfigurationProvider which opens
> ZKConfigurationStore and creates a ZK Connection.
> *ZK LOGS*
> {code}
> -03-12 16:45:51,881 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: [2
> times] Error accepting new connection: Too many connections from
> /172.27.99.64 - max is 60
> 2020-03-12 16:45:52,449 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new
> connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:52,710 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new
> connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:52,876 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: [4 times] Error accepting
> new connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:53,068 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: [2 times] Error accepting
> new connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:53,391 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: [2 times] Error accepting
> new connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:54,008 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new
> connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:54,287 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new
> connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:54,483 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: [4 times] Error accepting
> new connection: Too many connections from /172.27.99.64 - max is 60
> {code}
> And there is an another bug in ZKConfigurationStore which has not handled
> close() of ZKCuratorManager.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10160) Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo
[
https://issues.apache.org/jira/browse/YARN-10160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-10160:
-
Attachment: YARN-10160-005.patch
> Add auto queue creation related configs to
> RMWebService#CapacitySchedulerQueueInfo
> --
>
> Key: YARN-10160
> URL: https://issues.apache.org/jira/browse/YARN-10160
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: Screen Shot 2020-02-25 at 9.06.52 PM.png,
> YARN-10160-001.patch, YARN-10160-002.patch, YARN-10160-003.patch,
> YARN-10160-004.patch, YARN-10160-005.patch
>
>
> Add auto queue creation related configs to
> RMWebService#CapacitySchedulerQueueInfo.
> {code}
> yarn.scheduler.capacity..auto-create-child-queue.enabled
> yarn.scheduler.capacity..leaf-queue-template.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064750#comment-17064750
]
Peter Bacsko commented on YARN-10198:
-
Due to infra issues, we can't see the comment about the Precommit builds.
Here's a build about patch v3:
https://builds.apache.org/job/PreCommit-YARN-Build/25730/console
{noformat}
+1 overall
| Vote |Subsystem | Runtime | Comment
| 0 | reexec | 0m 48s | Docker mode activated.
| | || Prechecks
| +1 | @author | 0m 0s | The patch does not contain any @author
| | || tags.
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or
| | || modified test files.
| | || trunk Compile Tests
| +1 | mvninstall | 20m 57s | trunk passed
| +1 | compile | 0m 43s | trunk passed
| +1 | checkstyle | 0m 34s | trunk passed
| +1 | mvnsite | 0m 46s | trunk passed
| +1 |shadedclient | 15m 27s | branch has no errors when building and
| | || testing our client artifacts.
| +1 |findbugs | 1m 34s | trunk passed
| +1 | javadoc | 0m 29s | trunk passed
| | || Patch Compile Tests
| +1 | mvninstall | 0m 42s | the patch passed
| +1 | compile | 0m 37s | the patch passed
| +1 | javac | 0m 37s | the patch passed
| +1 | checkstyle | 0m 28s | the patch passed
| +1 | mvnsite | 0m 41s | the patch passed
| +1 | whitespace | 0m 0s | The patch has no whitespace issues.
| +1 |shadedclient | 14m 2s | patch has no errors when building and
| | || testing our client artifacts.
| +1 |findbugs | 1m 37s | the patch passed
| +1 | javadoc | 0m 28s | the patch passed
| | || Other Tests
| +1 |unit | 89m 8s | hadoop-yarn-server-resourcemanager in
| | || the patch passed.
| +1 | asflicense | 0m 26s | The patch does not generate ASF
| | || License warnings.
| | | 149m 17s |
{noformat}
[~sunilg] [~prabhujoseph] could you guys commit this change?
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch, YARN-10198-002.patch,
> YARN-10198-003.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10025) Various improvements in YARN log servlets
[ https://issues.apache.org/jira/browse/YARN-10025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated YARN-10025: -- Target Version/s: (was: 3.3.0) > Various improvements in YARN log servlets > - > > Key: YARN-10025 > URL: https://issues.apache.org/jira/browse/YARN-10025 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.2.1 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-10025 document.pdf > > > There are multiple ways how we can enhance the current log servlets in YARN. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
