[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=246517&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246517 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 22/May/19 01:42 Start Date: 22/May/19 01:42 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-494619638 Thank You @lokeshj1703 and @arp7 for the review. I have committed this to the trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246517) Time Spent: 12h 50m (was: 12h 40m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 12h 50m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=246515&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246515 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 22/May/19 01:41 Start Date: 22/May/19 01:41 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246515) Time Spent: 12h 40m (was: 12.5h) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 12h 40m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=246145&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246145 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 21/May/19 15:48 Start Date: 21/May/19 15:48 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-494448126 Failed test is not related to this patch. I think we can improve waitForClusterToBeReady to also check for safeMode also, so that we don't see these failures randomly. (Not completely sure we need though, if it is needed, I can open a new Jira to fix this.) Ran locally.[ https://user-images.githubusercontent.com/8586345/58110656-d3379900-7ba4-11e9-83aa-9e99f7f50951.png";> ](url) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246145) Time Spent: 12.5h (was: 12h 20m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 12.5h > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=246144&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246144 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 21/May/19 15:47 Start Date: 21/May/19 15:47 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-494448126 Failed test is not related to this patch. Ran locally.[ https://user-images.githubusercontent.com/8586345/58110656-d3379900-7ba4-11e9-83aa-9e99f7f50951.png";> ](url) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246144) Time Spent: 12h 20m (was: 12h 10m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 12h 20m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=246073&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246073 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 21/May/19 14:13 Start Date: 21/May/19 14:13 Worklog Time Spent: 10m Work Description: arp7 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-494409329 @bharatviswa504 can you check if the UT failure is related to the patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246073) Time Spent: 12h 10m (was: 12h) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 12h 10m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=246060&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246060 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 21/May/19 14:00 Start Date: 21/May/19 14:00 Worklog Time Spent: 10m Work Description: lokeshj1703 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-494404015 @bharatviswa504 Thanks for updating the PR! The changes look good to me. +1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246060) Time Spent: 12h (was: 11h 50m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 12h > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=245759&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245759 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 21/May/19 05:15 Start Date: 21/May/19 05:15 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-494242612 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 51 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 89 | Maven dependency ordering for branch | | +1 | mvninstall | 702 | trunk passed | | +1 | compile | 351 | trunk passed | | +1 | checkstyle | 98 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 1106 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 175 | trunk passed | | 0 | spotbugs | 294 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 489 | trunk passed | | -0 | patch | 349 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for patch | | +1 | mvninstall | 481 | the patch passed | | +1 | compile | 273 | the patch passed | | +1 | javac | 273 | the patch passed | | +1 | checkstyle | 84 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 738 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 146 | the patch passed | | +1 | findbugs | 499 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 161 | hadoop-hdds in the patch failed. | | -1 | unit | 1379 | hadoop-ozone in the patch failed. | | +1 | asflicense | 65 | The patch does not generate ASF License warnings. | | | | 7085 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline | | | hadoop.hdds.scm.safemode.TestSCMSafeModeWithPipelineRules | | | hadoop.ozone.container.server.TestSecureContainerServer | | | hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerHandler | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-714/13/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/714 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 9b8dd3bcb2ce 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1cb2eb0 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/13/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/13/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/13/testReport/ | | Max. process+thread count | 5418 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/13/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 245759) Time Spent: 11h 50m (was: 11h 40m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: ht
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=245722&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245722
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 21/May/19 03:22
Start Date: 21/May/19 03:22
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r285837951
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -133,7 +173,86 @@ public Pipeline create(ReplicationFactor factor,
.build();
}
+
+ @Override
+ public void shutdown() {
+forkJoinPool.shutdownNow();
+ }
+
protected void initializePipeline(Pipeline pipeline) throws IOException {
-RatisPipelineUtils.createPipeline(pipeline, conf);
+createPipeline(pipeline);
+ }
+
+ /**
+ * Sends ratis command to create pipeline on all the datanodes.
+ *
+ * @param pipeline - Pipeline to be created
+ * @throws IOException if creation fails
+ */
+ public void createPipeline(Pipeline pipeline)
+ throws IOException {
+final RaftGroup group = RatisHelper.newRaftGroup(pipeline);
+LOG.debug("creating pipeline:{} with {}", pipeline.getId(), group);
+callRatisRpc(pipeline.getNodes(),
+(raftClient, peer) -> {
+ RaftClientReply reply = raftClient.groupAdd(group, peer.getId());
+ if (reply == null || !reply.isSuccess()) {
+String msg = "Pipeline initialization failed for pipeline:"
++ pipeline.getId() + " node:" + peer.getId();
+LOG.error(msg);
+throw new IOException(msg);
+ }
+});
+ }
+
+ private void callRatisRpc(List datanodes,
+ CheckedBiConsumer< RaftClient, RaftPeer, IOException> rpc)
+ throws IOException {
+if (datanodes.isEmpty()) {
+ return;
+}
+
+final String rpcType = conf
+.get(ScmConfigKeys.DFS_CONTAINER_RATIS_RPC_TYPE_KEY,
+ScmConfigKeys.DFS_CONTAINER_RATIS_RPC_TYPE_DEFAULT);
+final RetryPolicy retryPolicy = RatisHelper.createRetryPolicy(conf);
+final List< IOException > exceptions =
+Collections.synchronizedList(new ArrayList<>());
+final int maxOutstandingRequests =
+HddsClientUtils.getMaxOutstandingRequests(conf);
+final GrpcTlsConfig tlsConfig = RatisHelper.createTlsClientConfig(new
+SecurityConfig(conf));
+final TimeDuration requestTimeout =
+RatisHelper.getClientRequestTimeout(conf);
+try {
+ forkJoinPool.submit(() -> {
Review comment:
@lokeshj1703 Sorry missed this comment earlier.
Checked this, one of the forkJoinPool thread is used for waiting and the
same is being used in one of the calls for Ratis with 3 pipeline.
**Output:**
The below line is from after Submit.
Thread name RATISCREATEPIPELINE1
` forkJoinPool.submit(() -> {`
These below log lines are inside ParallelStream
`datanodes.parallelStream().forEach(d -> {`
Internal thread name RATISCREATEPIPELINE1
Internal thread name RATISCREATEPIPELINE3
Internal thread name RATISCREATEPIPELINE2
So, I think we should be fine with parallelism set to 3. I even tried with
4, but I still see the same above output.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245722)
Time Spent: 11.5h (was: 11h 20m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11.5h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=245723&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245723 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 21/May/19 03:22 Start Date: 21/May/19 03:22 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-494224894 @lokeshj1703 Addressed the review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 245723) Time Spent: 11h 40m (was: 11.5h) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 11h 40m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=245719&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245719
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 21/May/19 03:16
Start Date: 21/May/19 03:16
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r285837310
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -133,7 +173,86 @@ public Pipeline create(ReplicationFactor factor,
.build();
}
+
+ @Override
+ public void shutdown() {
+forkJoinPool.shutdownNow();
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245719)
Time Spent: 11h 20m (was: 11h 10m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h 20m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=245718&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245718
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 21/May/19 03:16
Start Date: 21/May/19 03:16
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r285837285
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -24,35 +24,75 @@
import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeState;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.hdds.scm.client.HddsClientUtils;
import
org.apache.hadoop.hdds.scm.container.placement.algorithms.ContainerPlacementPolicy;
import
org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom;
import org.apache.hadoop.hdds.scm.node.NodeManager;
import org.apache.hadoop.hdds.scm.pipeline.Pipeline.PipelineState;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.io.MultipleIOException;
+import org.apache.ratis.RatisHelper;
+import org.apache.ratis.client.RaftClient;
+import org.apache.ratis.grpc.GrpcTlsConfig;
+import org.apache.ratis.protocol.RaftClientReply;
+import org.apache.ratis.protocol.RaftGroup;
+import org.apache.ratis.protocol.RaftPeer;
+import org.apache.ratis.retry.RetryPolicy;
+import org.apache.ratis.rpc.SupportedRpcType;
+import org.apache.ratis.util.TimeDuration;
+import org.apache.ratis.util.function.CheckedBiConsumer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
+import java.util.ArrayList;
+import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.ForkJoinWorkerThread;
+import java.util.concurrent.RejectedExecutionException;
import java.util.stream.Collectors;
/**
* Implements Api for creating ratis pipelines.
*/
public class RatisPipelineProvider implements PipelineProvider {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RatisPipelineProvider.class);
+
private final NodeManager nodeManager;
private final PipelineStateManager stateManager;
private final Configuration conf;
+ // Set parallelism at 3, as now in Ratis we create 1 and 3 node pipelines.
+ private final int parallelisimForPool = 3;
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245718)
Time Spent: 11h 10m (was: 11h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=245309&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245309
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 20/May/19 16:49
Start Date: 20/May/19 16:49
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r285682170
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -24,35 +24,75 @@
import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeState;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.hdds.scm.client.HddsClientUtils;
import
org.apache.hadoop.hdds.scm.container.placement.algorithms.ContainerPlacementPolicy;
import
org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom;
import org.apache.hadoop.hdds.scm.node.NodeManager;
import org.apache.hadoop.hdds.scm.pipeline.Pipeline.PipelineState;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.io.MultipleIOException;
+import org.apache.ratis.RatisHelper;
+import org.apache.ratis.client.RaftClient;
+import org.apache.ratis.grpc.GrpcTlsConfig;
+import org.apache.ratis.protocol.RaftClientReply;
+import org.apache.ratis.protocol.RaftGroup;
+import org.apache.ratis.protocol.RaftPeer;
+import org.apache.ratis.retry.RetryPolicy;
+import org.apache.ratis.rpc.SupportedRpcType;
+import org.apache.ratis.util.TimeDuration;
+import org.apache.ratis.util.function.CheckedBiConsumer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
+import java.util.ArrayList;
+import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.ForkJoinWorkerThread;
+import java.util.concurrent.RejectedExecutionException;
import java.util.stream.Collectors;
/**
* Implements Api for creating ratis pipelines.
*/
public class RatisPipelineProvider implements PipelineProvider {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RatisPipelineProvider.class);
+
private final NodeManager nodeManager;
private final PipelineStateManager stateManager;
private final Configuration conf;
+ // Set parallelism at 3, as now in Ratis we create 1 and 3 node pipelines.
+ private final int parallelisimForPool = 3;
Review comment:
There is a typo. parallelis'i'mForPool.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245309)
Time Spent: 11h (was: 10h 50m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=245307&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245307
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 20/May/19 16:47
Start Date: 20/May/19 16:47
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r285681320
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -133,7 +173,86 @@ public Pipeline create(ReplicationFactor factor,
.build();
}
+
+ @Override
+ public void shutdown() {
+forkJoinPool.shutdownNow();
Review comment:
@bharatviswa504 I agree. We need to use shutdownNow but we also need to use
awaitTermination. shutdownNow would interrupt the running tasks but the running
task should handle the interrupt. If the task does not exit on interrupt, it is
a better idea to wait for the task to finish.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245307)
Time Spent: 10h 50m (was: 10h 40m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=245285&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245285
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 20/May/19 16:23
Start Date: 20/May/19 16:23
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r285672796
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -133,7 +173,86 @@ public Pipeline create(ReplicationFactor factor,
.build();
}
+
+ @Override
+ public void shutdown() {
+forkJoinPool.shutdownNow();
+ }
+
protected void initializePipeline(Pipeline pipeline) throws IOException {
-RatisPipelineUtils.createPipeline(pipeline, conf);
+createPipeline(pipeline);
+ }
+
+ /**
+ * Sends ratis command to create pipeline on all the datanodes.
+ *
+ * @param pipeline - Pipeline to be created
+ * @throws IOException if creation fails
+ */
+ public void createPipeline(Pipeline pipeline)
+ throws IOException {
+final RaftGroup group = RatisHelper.newRaftGroup(pipeline);
+LOG.debug("creating pipeline:{} with {}", pipeline.getId(), group);
+callRatisRpc(pipeline.getNodes(),
+(raftClient, peer) -> {
+ RaftClientReply reply = raftClient.groupAdd(group, peer.getId());
+ if (reply == null || !reply.isSuccess()) {
+String msg = "Pipeline initialization failed for pipeline:"
++ pipeline.getId() + " node:" + peer.getId();
+LOG.error(msg);
+throw new IOException(msg);
+ }
+});
+ }
+
+ private void callRatisRpc(List datanodes,
+ CheckedBiConsumer< RaftClient, RaftPeer, IOException> rpc)
+ throws IOException {
+if (datanodes.isEmpty()) {
+ return;
+}
+
+final String rpcType = conf
+.get(ScmConfigKeys.DFS_CONTAINER_RATIS_RPC_TYPE_KEY,
+ScmConfigKeys.DFS_CONTAINER_RATIS_RPC_TYPE_DEFAULT);
+final RetryPolicy retryPolicy = RatisHelper.createRetryPolicy(conf);
+final List< IOException > exceptions =
+Collections.synchronizedList(new ArrayList<>());
+final int maxOutstandingRequests =
+HddsClientUtils.getMaxOutstandingRequests(conf);
+final GrpcTlsConfig tlsConfig = RatisHelper.createTlsClientConfig(new
+SecurityConfig(conf));
+final TimeDuration requestTimeout =
+RatisHelper.getClientRequestTimeout(conf);
+try {
+ forkJoinPool.submit(() -> {
Review comment:
@bharatviswa504 Can you please verify this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245285)
Time Spent: 10h 40m (was: 10.5h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 40m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243566&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243566
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 18:44
Start Date: 16/May/19 18:44
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284847079
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineFactory.java
##
@@ -61,4 +61,13 @@ public Pipeline create(ReplicationType type,
ReplicationFactor factor,
List nodes) {
return providers.get(type).create(factor, nodes);
}
+
+ @VisibleForTesting
+ public PipelineProvider getProvider(ReplicationType type) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243566)
Time Spent: 10.5h (was: 10h 20m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243564&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243564 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 16/May/19 18:42 Start Date: 16/May/19 18:42 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-493185963 Thank You @lokeshj1703 for the review. I have addressed the review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243564) Time Spent: 10h 20m (was: 10h 10m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10h 20m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243559&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243559
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 18:38
Start Date: 16/May/19 18:38
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284844530
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineCreateAndDestory.java
##
@@ -97,8 +97,11 @@ public void testPipelineCreationOnNodeRestart() throws
Exception {
}
// try creating another pipeline now
+RatisPipelineProvider ratisPipelineProvider = (RatisPipelineProvider)
+pipelineManager.getPipelineFactory().getProvider(
+HddsProtos.ReplicationType.RATIS);
try {
- RatisPipelineUtils.createPipeline(pipelines.get(0), conf);
+ ratisPipelineProvider.createPipeline(pipelines.get(0));
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243559)
Time Spent: 10h 10m (was: 10h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243543&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243543
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 18:23
Start Date: 16/May/19 18:23
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284838490
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##
@@ -1017,6 +1017,9 @@ public void stop() {
} catch (Exception ex) {
LOG.error("SCM Metadata store stop failed", ex);
}
+
+// shutdown pipeline provider.
+pipelineManager.getPipelineFactory().shutdown();
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243543)
Time Spent: 9h 50m (was: 9h 40m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243544&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243544
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 18:23
Start Date: 16/May/19 18:23
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284838741
##
File path:
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/MockRatisPipelineProvider.java
##
@@ -37,4 +37,9 @@ public MockRatisPipelineProvider(NodeManager nodeManager,
protected void initializePipeline(Pipeline pipeline) throws IOException {
// do nothing as the datanodes do not exists
}
+
+ @Override
+ public void shutdown() {
+
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243544)
Time Spent: 10h (was: 9h 50m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243542&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243542
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 18:23
Start Date: 16/May/19 18:23
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284838447
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SimplePipelineProvider.java
##
@@ -72,4 +72,9 @@ public Pipeline create(ReplicationFactor factor,
.setNodes(nodes)
.build();
}
+
+ @Override
+ public void shutdown() {
+
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243542)
Time Spent: 9h 40m (was: 9.5h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243539&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243539
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 18:22
Start Date: 16/May/19 18:22
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284838122
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -133,7 +173,86 @@ public Pipeline create(ReplicationFactor factor,
.build();
}
+
+ @Override
+ public void shutdown() {
+forkJoinPool.shutdownNow();
+ }
+
protected void initializePipeline(Pipeline pipeline) throws IOException {
-RatisPipelineUtils.createPipeline(pipeline, conf);
+createPipeline(pipeline);
+ }
+
+ /**
+ * Sends ratis command to create pipeline on all the datanodes.
+ *
+ * @param pipeline - Pipeline to be created
+ * @throws IOException if creation fails
+ */
+ public void createPipeline(Pipeline pipeline)
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243539)
Time Spent: 9h 20m (was: 9h 10m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243540&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243540
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 18:22
Start Date: 16/May/19 18:22
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284838303
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
##
@@ -346,6 +347,11 @@ public void triggerPipelineCreation() {
backgroundPipelineCreator.triggerPipelineCreation();
}
+ @Override
+ public PipelineFactory getPipelineFactory() {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243540)
Time Spent: 9.5h (was: 9h 20m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243536&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243536
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 18:21
Start Date: 16/May/19 18:21
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284837697
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -24,35 +24,75 @@
import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeState;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.hdds.scm.client.HddsClientUtils;
import
org.apache.hadoop.hdds.scm.container.placement.algorithms.ContainerPlacementPolicy;
import
org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom;
import org.apache.hadoop.hdds.scm.node.NodeManager;
import org.apache.hadoop.hdds.scm.pipeline.Pipeline.PipelineState;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.io.MultipleIOException;
+import org.apache.ratis.RatisHelper;
+import org.apache.ratis.client.RaftClient;
+import org.apache.ratis.grpc.GrpcTlsConfig;
+import org.apache.ratis.protocol.RaftClientReply;
+import org.apache.ratis.protocol.RaftGroup;
+import org.apache.ratis.protocol.RaftPeer;
+import org.apache.ratis.retry.RetryPolicy;
+import org.apache.ratis.rpc.SupportedRpcType;
+import org.apache.ratis.util.TimeDuration;
+import org.apache.ratis.util.function.CheckedBiConsumer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
+import java.util.ArrayList;
+import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.ForkJoinWorkerThread;
+import java.util.concurrent.RejectedExecutionException;
import java.util.stream.Collectors;
/**
* Implements Api for creating ratis pipelines.
*/
public class RatisPipelineProvider implements PipelineProvider {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RatisPipelineProvider.class);
+
private final NodeManager nodeManager;
private final PipelineStateManager stateManager;
private final Configuration conf;
+ // Set parallelism at 3, as now in Ratis we create 1 and 3 node pipelines.
+ private final int parallelisimForPool = 3;
+
+ private final ForkJoinPool.ForkJoinWorkerThreadFactory factory =
+ (pool -> {
+final ForkJoinWorkerThread worker = ForkJoinPool.
+defaultForkJoinWorkerThreadFactory.newThread(pool);
+worker.setName("ratisCreatePipeline" + worker.getPoolIndex());
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243536)
Time Spent: 9h (was: 8h 50m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243537&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243537 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 16/May/19 18:21 Start Date: 16/May/19 18:21 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#discussion_r284837740 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManager.java ## @@ -75,4 +75,6 @@ void finalizeAndDestroyPipeline(Pipeline pipeline, boolean onTimeout) void startPipelineCreator(); void triggerPipelineCreation(); + + PipelineFactory getPipelineFactory(); Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243537) Time Spent: 9h 10m (was: 9h) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 9h 10m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243529&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243529
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 18:20
Start Date: 16/May/19 18:20
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284837312
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -133,7 +173,86 @@ public Pipeline create(ReplicationFactor factor,
.build();
}
+
+ @Override
+ public void shutdown() {
+forkJoinPool.shutdownNow();
Review comment:
This is done based on arpit's comment, as on an unclean shutdown this
terminate abruptly. So we can use shutdownNow(), instead of awaitTermination in
normal case too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243529)
Time Spent: 8h 50m (was: 8h 40m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 50m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243212&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243212
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 09:27
Start Date: 16/May/19 09:27
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284609966
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineFactory.java
##
@@ -61,4 +61,13 @@ public Pipeline create(ReplicationType type,
ReplicationFactor factor,
List nodes) {
return providers.get(type).create(factor, nodes);
}
+
+ @VisibleForTesting
+ public PipelineProvider getProvider(ReplicationType type) {
Review comment:
We don't need to expose this api for now. In the test where it is used, we
can call pipelineManager.createPipeline instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243212)
Time Spent: 7h 10m (was: 7h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243214&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243214
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 09:27
Start Date: 16/May/19 09:27
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284614538
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -133,7 +173,86 @@ public Pipeline create(ReplicationFactor factor,
.build();
}
+
+ @Override
+ public void shutdown() {
+forkJoinPool.shutdownNow();
+ }
+
protected void initializePipeline(Pipeline pipeline) throws IOException {
-RatisPipelineUtils.createPipeline(pipeline, conf);
+createPipeline(pipeline);
+ }
+
+ /**
+ * Sends ratis command to create pipeline on all the datanodes.
+ *
+ * @param pipeline - Pipeline to be created
+ * @throws IOException if creation fails
+ */
+ public void createPipeline(Pipeline pipeline)
Review comment:
We can remove this fn and replace it with initializePipeline.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243214)
Time Spent: 7.5h (was: 7h 20m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243218&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243218
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 09:27
Start Date: 16/May/19 09:27
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284615716
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
##
@@ -346,6 +347,11 @@ public void triggerPipelineCreation() {
backgroundPipelineCreator.triggerPipelineCreation();
}
+ @Override
+ public PipelineFactory getPipelineFactory() {
Review comment:
We dont need to expose this api. We can add a shutdown call for the
pipelineFactory in the SCMPipelineManager#close fn.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243218)
Time Spent: 8h 10m (was: 8h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243215&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243215
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 09:27
Start Date: 16/May/19 09:27
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284612918
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -133,7 +173,86 @@ public Pipeline create(ReplicationFactor factor,
.build();
}
+
+ @Override
+ public void shutdown() {
+forkJoinPool.shutdownNow();
Review comment:
We should also wait for the tasks to finish. We need to use awaitTermination
call. We can use timeout of 60 seconds? That is what is used in Scheduler class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243215)
Time Spent: 7h 40m (was: 7.5h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243220&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243220
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 09:27
Start Date: 16/May/19 09:27
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284617241
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##
@@ -1017,6 +1017,9 @@ public void stop() {
} catch (Exception ex) {
LOG.error("SCM Metadata store stop failed", ex);
}
+
+// shutdown pipeline provider.
+pipelineManager.getPipelineFactory().shutdown();
Review comment:
We can remove this call.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243220)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243219&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243219
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 09:27
Start Date: 16/May/19 09:27
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284614769
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -133,7 +173,86 @@ public Pipeline create(ReplicationFactor factor,
.build();
}
+
+ @Override
+ public void shutdown() {
+forkJoinPool.shutdownNow();
+ }
+
protected void initializePipeline(Pipeline pipeline) throws IOException {
-RatisPipelineUtils.createPipeline(pipeline, conf);
+createPipeline(pipeline);
+ }
+
+ /**
+ * Sends ratis command to create pipeline on all the datanodes.
+ *
+ * @param pipeline - Pipeline to be created
+ * @throws IOException if creation fails
+ */
+ public void createPipeline(Pipeline pipeline)
+ throws IOException {
+final RaftGroup group = RatisHelper.newRaftGroup(pipeline);
+LOG.debug("creating pipeline:{} with {}", pipeline.getId(), group);
+callRatisRpc(pipeline.getNodes(),
+(raftClient, peer) -> {
+ RaftClientReply reply = raftClient.groupAdd(group, peer.getId());
+ if (reply == null || !reply.isSuccess()) {
+String msg = "Pipeline initialization failed for pipeline:"
++ pipeline.getId() + " node:" + peer.getId();
+LOG.error(msg);
+throw new IOException(msg);
+ }
+});
+ }
+
+ private void callRatisRpc(List datanodes,
+ CheckedBiConsumer< RaftClient, RaftPeer, IOException> rpc)
+ throws IOException {
+if (datanodes.isEmpty()) {
+ return;
+}
+
+final String rpcType = conf
+.get(ScmConfigKeys.DFS_CONTAINER_RATIS_RPC_TYPE_KEY,
+ScmConfigKeys.DFS_CONTAINER_RATIS_RPC_TYPE_DEFAULT);
+final RetryPolicy retryPolicy = RatisHelper.createRetryPolicy(conf);
+final List< IOException > exceptions =
+Collections.synchronizedList(new ArrayList<>());
+final int maxOutstandingRequests =
+HddsClientUtils.getMaxOutstandingRequests(conf);
+final GrpcTlsConfig tlsConfig = RatisHelper.createTlsClientConfig(new
+SecurityConfig(conf));
+final TimeDuration requestTimeout =
+RatisHelper.getClientRequestTimeout(conf);
+try {
+ forkJoinPool.submit(() -> {
Review comment:
Can we please verify that none of the threads are waiting for the parallel
stream call to finish?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243219)
Time Spent: 8h 20m (was: 8h 10m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243221&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243221
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 09:27
Start Date: 16/May/19 09:27
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284617457
##
File path:
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/MockRatisPipelineProvider.java
##
@@ -37,4 +37,9 @@ public MockRatisPipelineProvider(NodeManager nodeManager,
protected void initializePipeline(Pipeline pipeline) throws IOException {
// do nothing as the datanodes do not exists
}
+
+ @Override
+ public void shutdown() {
+
Review comment:
NIT - Can we add a comment like // do nothing?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243221)
Time Spent: 8.5h (was: 8h 20m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243222&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243222
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 09:27
Start Date: 16/May/19 09:27
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284617612
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineCreateAndDestory.java
##
@@ -97,8 +97,11 @@ public void testPipelineCreationOnNodeRestart() throws
Exception {
}
// try creating another pipeline now
+RatisPipelineProvider ratisPipelineProvider = (RatisPipelineProvider)
+pipelineManager.getPipelineFactory().getProvider(
+HddsProtos.ReplicationType.RATIS);
try {
- RatisPipelineUtils.createPipeline(pipelines.get(0), conf);
+ ratisPipelineProvider.createPipeline(pipelines.get(0));
Review comment:
We can use pipelineManager#create call instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243222)
Time Spent: 8h 40m (was: 8.5h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243217&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243217
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 09:27
Start Date: 16/May/19 09:27
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284616930
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SimplePipelineProvider.java
##
@@ -72,4 +72,9 @@ public Pipeline create(ReplicationFactor factor,
.setNodes(nodes)
.build();
}
+
+ @Override
+ public void shutdown() {
+
Review comment:
NIT - Can we add a comment like //do nothing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243217)
Time Spent: 8h (was: 7h 50m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243216&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243216
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 16/May/19 09:27
Start Date: 16/May/19 09:27
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r284613883
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -24,35 +24,75 @@
import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeState;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.hdds.scm.client.HddsClientUtils;
import
org.apache.hadoop.hdds.scm.container.placement.algorithms.ContainerPlacementPolicy;
import
org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom;
import org.apache.hadoop.hdds.scm.node.NodeManager;
import org.apache.hadoop.hdds.scm.pipeline.Pipeline.PipelineState;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.io.MultipleIOException;
+import org.apache.ratis.RatisHelper;
+import org.apache.ratis.client.RaftClient;
+import org.apache.ratis.grpc.GrpcTlsConfig;
+import org.apache.ratis.protocol.RaftClientReply;
+import org.apache.ratis.protocol.RaftGroup;
+import org.apache.ratis.protocol.RaftPeer;
+import org.apache.ratis.retry.RetryPolicy;
+import org.apache.ratis.rpc.SupportedRpcType;
+import org.apache.ratis.util.TimeDuration;
+import org.apache.ratis.util.function.CheckedBiConsumer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
+import java.util.ArrayList;
+import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.ForkJoinWorkerThread;
+import java.util.concurrent.RejectedExecutionException;
import java.util.stream.Collectors;
/**
* Implements Api for creating ratis pipelines.
*/
public class RatisPipelineProvider implements PipelineProvider {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RatisPipelineProvider.class);
+
private final NodeManager nodeManager;
private final PipelineStateManager stateManager;
private final Configuration conf;
+ // Set parallelism at 3, as now in Ratis we create 1 and 3 node pipelines.
+ private final int parallelisimForPool = 3;
+
+ private final ForkJoinPool.ForkJoinWorkerThreadFactory factory =
+ (pool -> {
+final ForkJoinWorkerThread worker = ForkJoinPool.
+defaultForkJoinWorkerThreadFactory.newThread(pool);
+worker.setName("ratisCreatePipeline" + worker.getPoolIndex());
Review comment:
NIT - "ratisCreatePipeline" - Can we make it all capital?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243216)
Time Spent: 7h 50m (was: 7h 40m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=243213&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243213 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 16/May/19 09:27 Start Date: 16/May/19 09:27 Worklog Time Spent: 10m Work Description: lokeshj1703 commented on pull request #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#discussion_r284610403 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManager.java ## @@ -75,4 +75,6 @@ void finalizeAndDestroyPipeline(Pipeline pipeline, boolean onTimeout) void startPipelineCreator(); void triggerPipelineCreation(); + + PipelineFactory getPipelineFactory(); Review comment: We don't need to expose this api as well. The shutdown for pipelineFactory should be called from PipelineManager itself. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243213) Time Spent: 7h 20m (was: 7h 10m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 7h 20m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=241308&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241308 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 13/May/19 20:56 Start Date: 13/May/19 20:56 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-491983496 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 49 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 66 | Maven dependency ordering for branch | | +1 | mvninstall | 408 | trunk passed | | +1 | compile | 198 | trunk passed | | +1 | checkstyle | 53 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 898 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 125 | trunk passed | | 0 | spotbugs | 239 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 422 | trunk passed | | -0 | patch | 266 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for patch | | +1 | mvninstall | 395 | the patch passed | | +1 | compile | 199 | the patch passed | | +1 | javac | 199 | the patch passed | | +1 | checkstyle | 57 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 703 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 121 | the patch passed | | +1 | findbugs | 436 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 148 | hadoop-hdds in the patch failed. | | -1 | unit | 1249 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 5662 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-714/10/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/714 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c9ed3643581d 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 389e640 | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/10/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/10/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/10/testReport/ | | Max. process+thread count | 4620 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/10/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 241308) Time Spent: 7h (was: 6h 50m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-req
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=241241&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241241 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 13/May/19 19:14 Start Date: 13/May/19 19:14 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-491949750 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 29 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 62 | Maven dependency ordering for branch | | +1 | mvninstall | 395 | trunk passed | | +1 | compile | 199 | trunk passed | | +1 | checkstyle | 51 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 818 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 126 | trunk passed | | 0 | spotbugs | 240 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 416 | trunk passed | | -0 | patch | 274 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 392 | the patch passed | | +1 | compile | 206 | the patch passed | | +1 | javac | 206 | the patch passed | | -0 | checkstyle | 31 | hadoop-hdds: The patch generated 17 new + 0 unchanged - 0 fixed = 17 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 650 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 114 | the patch passed | | +1 | findbugs | 440 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 129 | hadoop-hdds in the patch failed. | | -1 | unit | 1493 | hadoop-ozone in the patch failed. | | +1 | asflicense | 31 | The patch does not generate ASF License warnings. | | | | 5762 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestContainerStateMachineFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.web.TestOzoneRestWithMiniCluster | | | hadoop.ozone.TestContainerStateMachineIdempotency | | | hadoop.ozone.client.rpc.TestHybridPipelineOnDatanode | | | hadoop.ozone.scm.TestGetCommittedBlockLengthAndPutKey | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.scm.TestContainerSmallFile | | | hadoop.ozone.client.rpc.TestContainerStateMachine | | | hadoop.ozone.client.rpc.TestWatchForCommit | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.om.TestOzoneManagerHA | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-714/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/714 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b68105a4635e 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 5847e00 | | Default Java | 1.8.0_191 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/8/artifact/out/diff-checkstyle-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/8/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/8/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/8/testReport/ | | Max. process+thread count | 5275 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/8/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=241196&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241196 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 13/May/19 17:37 Start Date: 13/May/19 17:37 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-491917096 Thank You @lokeshj1703 for the review. I have addressed review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 241196) Time Spent: 6h 40m (was: 6.5h) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 6h 40m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=241195&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241195
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 13/May/19 17:37
Start Date: 13/May/19 17:37
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r283456973
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineFactory.java
##
@@ -38,12 +38,12 @@
private Map providers;
PipelineFactory(NodeManager nodeManager, PipelineStateManager stateManager,
- Configuration conf) {
+ Configuration conf, RatisPipelineUtils ratisPipelineUtils) {
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 241195)
Time Spent: 6.5h (was: 6h 20m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=240335&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-240335
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 10/May/19 17:22
Start Date: 10/May/19 17:22
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r282971370
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineFactory.java
##
@@ -38,12 +38,12 @@
private Map providers;
PipelineFactory(NodeManager nodeManager, PipelineStateManager stateManager,
- Configuration conf) {
+ Configuration conf, RatisPipelineUtils ratisPipelineUtils) {
Review comment:
@bharatviswa504 Thanks for adding the changes! An instance of
RatisPipelineUtils does not seem right as it is a utility class. Could we move
the functions of RatisPipelineUtils to RatisPipelineProvider instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 240335)
Time Spent: 6h 20m (was: 6h 10m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=240005&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-240005 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 10/May/19 01:59 Start Date: 10/May/19 01:59 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-491126699 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 30 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 61 | Maven dependency ordering for branch | | +1 | mvninstall | 391 | trunk passed | | +1 | compile | 197 | trunk passed | | +1 | checkstyle | 46 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 786 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 122 | trunk passed | | 0 | spotbugs | 239 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 425 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for patch | | +1 | mvninstall | 439 | the patch passed | | +1 | compile | 192 | the patch passed | | +1 | javac | 192 | the patch passed | | -0 | checkstyle | 28 | hadoop-hdds: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 631 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 121 | the patch passed | | +1 | findbugs | 430 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 131 | hadoop-hdds in the patch failed. | | -1 | unit | 1308 | hadoop-ozone in the patch failed. | | +1 | asflicense | 38 | The patch does not generate ASF License warnings. | | | | 6389 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestWatchForCommit | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-714/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/714 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b4b5ff3f75e2 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b832e17 | | Default Java | 1.8.0_191 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/7/artifact/out/diff-checkstyle-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/7/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/7/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/7/testReport/ | | Max. process+thread count | 5060 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/7/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 240005) Time Spent: 6h 10m (was: 6h) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-re
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=239989&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239989
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 10/May/19 00:12
Start Date: 10/May/19 00:12
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r282709424
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -41,16 +41,35 @@
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.ForkJoinWorkerThread;
+import java.util.concurrent.RejectedExecutionException;
/**
* Utility class for Ratis pipelines. Contains methods to create and destroy
* ratis pipelines.
*/
-final class RatisPipelineUtils {
+public final class RatisPipelineUtils {
private static final Logger LOG =
LoggerFactory.getLogger(RatisPipelineUtils.class);
+ // Set parallelism at 3, as now in Ratis we create 1 and 3 node pipelines.
+ private static final int PARALLELISIM_FOR_POOL = 3;
+
+ private static final ForkJoinPool.ForkJoinWorkerThreadFactory FACTORY =
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239989)
Time Spent: 6h (was: 5h 50m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=239988&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239988 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 10/May/19 00:11 Start Date: 10/May/19 00:11 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-491109791 Thank You @lokeshj1703 for the review. I have addressed the review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 239988) Time Spent: 5h 50m (was: 5h 40m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 5h 50m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233714&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233714 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 26/Apr/19 19:47 Start Date: 26/Apr/19 19:47 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-487179072 Test failures are not related to this patch. I will commit this shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233714) Time Spent: 5h 40m (was: 5.5h) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 5h 40m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233434&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233434
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 26/Apr/19 10:46
Start Date: 26/Apr/19 10:46
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278896085
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -41,16 +41,35 @@
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.ForkJoinWorkerThread;
+import java.util.concurrent.RejectedExecutionException;
/**
* Utility class for Ratis pipelines. Contains methods to create and destroy
* ratis pipelines.
*/
-final class RatisPipelineUtils {
+public final class RatisPipelineUtils {
private static final Logger LOG =
LoggerFactory.getLogger(RatisPipelineUtils.class);
+ // Set parallelism at 3, as now in Ratis we create 1 and 3 node pipelines.
+ private static final int PARALLELISIM_FOR_POOL = 3;
+
+ private static final ForkJoinPool.ForkJoinWorkerThreadFactory FACTORY =
Review comment:
Can we avoid making it static? The problem with static occurs in the
MiniOzoneCluster tests. Once SCM is stopped by one of the tests, the fork join
pool will be shutdown and will not be available again for execution. I think
this might be a reason for unit test failures.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233434)
Time Spent: 5h 20m (was: 5h 10m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233435&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233435
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 26/Apr/19 10:46
Start Date: 26/Apr/19 10:46
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278895492
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +165,33 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
Review comment:
Can you please check if one of the threads is not used up in waiting for
parallel stream to finish execution? If it does then there are only two threads
available for making a rpc call.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233435)
Time Spent: 5.5h (was: 5h 20m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233199&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233199 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 26/Apr/19 00:22 Start Date: 26/Apr/19 00:22 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-486882561 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 26 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1049 | trunk passed | | +1 | compile | 34 | trunk passed | | +1 | checkstyle | 25 | trunk passed | | +1 | mvnsite | 34 | trunk passed | | +1 | shadedclient | 733 | branch has no errors when building and testing our client artifacts. | | +1 | findbugs | 58 | trunk passed | | +1 | javadoc | 26 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 38 | the patch passed | | +1 | compile | 24 | the patch passed | | +1 | javac | 24 | the patch passed | | +1 | checkstyle | 16 | the patch passed | | +1 | mvnsite | 26 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 737 | patch has no errors when building and testing our client artifacts. | | +1 | findbugs | 46 | the patch passed | | +1 | javadoc | 21 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 106 | server-scm in the patch failed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 3100 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-714/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/714 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 39bcea67402b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b5dcf64 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/6/artifact/out/patch-unit-hadoop-hdds_server-scm.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/6/testReport/ | | Max. process+thread count | 445 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/6/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233199) Time Spent: 5h 10m (was: 5h) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 5h 10m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233180&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233180 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 25/Apr/19 23:41 Start Date: 25/Apr/19 23:41 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-486875564 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233180) Time Spent: 5h (was: 4h 50m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233179&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233179 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 25/Apr/19 23:39 Start Date: 25/Apr/19 23:39 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-486875182 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1119 | trunk passed | | +1 | compile | 69 | trunk passed | | +1 | checkstyle | 24 | trunk passed | | +1 | mvnsite | 33 | trunk passed | | +1 | shadedclient | 765 | branch has no errors when building and testing our client artifacts. | | +1 | findbugs | 42 | trunk passed | | +1 | javadoc | 24 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 34 | the patch passed | | +1 | compile | 24 | the patch passed | | +1 | javac | 24 | the patch passed | | +1 | checkstyle | 15 | the patch passed | | +1 | mvnsite | 26 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 826 | patch has no errors when building and testing our client artifacts. | | +1 | findbugs | 46 | the patch passed | | +1 | javadoc | 19 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 101 | server-scm in the patch failed. | | +1 | asflicense | 26 | The patch does not generate ASF License warnings. | | | | 3298 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-714/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/714 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux cfa19566ca15 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b5dcf64 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/5/artifact/out/patch-unit-hadoop-hdds_server-scm.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/5/testReport/ | | Max. process+thread count | 342 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/5/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233179) Time Spent: 4h 50m (was: 4h 40m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 4h 50m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233176&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233176
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 23:30
Start Date: 25/Apr/19 23:30
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278767040
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +165,37 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233176)
Time Spent: 4h 40m (was: 4.5h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233158&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233158
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 23:03
Start Date: 25/Apr/19 23:03
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278762154
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +165,37 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
+ LOG.error("Execution exception occurred during createPipeline", ex);
+ throw new IOException("Execution exception occurred during " +
+ "createPipeline", ex);
+} catch (RejectedExecutionException ex) {
+ LOG.error("RejectedExecutionException, occurred during " +
Review comment:
We return IOException, so in case of IOException we have no logging right?
It has only break;
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233158)
Time Spent: 4h 20m (was: 4h 10m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233159&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233159
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 23:03
Start Date: 25/Apr/19 23:03
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278760822
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +165,37 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
+ LOG.error("Execution exception occurred during createPipeline", ex);
+ throw new IOException("Execution exception occurred during " +
+ "createPipeline", ex);
+} catch (RejectedExecutionException ex) {
+ LOG.error("RejectedExecutionException, occurred during " +
Review comment:
Logging here, because in the actual createPipelines method in
BackGroudPipelineCreator which calls this method when we throw IOException they
break from while loop. That is the reason for the logging. Let me know if we
still don't want to log here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233159)
Time Spent: 4.5h (was: 4h 20m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233157&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233157
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 23:03
Start Date: 25/Apr/19 23:03
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278762154
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +165,37 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
+ LOG.error("Execution exception occurred during createPipeline", ex);
+ throw new IOException("Execution exception occurred during " +
+ "createPipeline", ex);
+} catch (RejectedExecutionException ex) {
+ LOG.error("RejectedExecutionException, occurred during " +
Review comment:
We return IOException, so in case of IOException we have not logging right?
It has only break;
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233157)
Time Spent: 4h 10m (was: 4h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233156&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233156
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 23:01
Start Date: 25/Apr/19 23:01
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #714: HDDS-1406. Avoid
usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278761757
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +165,37 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
+ LOG.error("Execution exception occurred during createPipeline", ex);
+ throw new IOException("Execution exception occurred during " +
+ "createPipeline", ex);
+} catch (RejectedExecutionException ex) {
+ LOG.error("RejectedExecutionException, occurred during " +
Review comment:
That method is already logging right:
```
pipelineManager.createPipeline(type, factor);
} catch (IOException ioe) {
break;
} catch (Throwable t) {
LOG.error("Error while creating pipelines {}", t);
break;
}
```
So if we log here the exception will be logged twice. Perhaps I'm looking at
it wrong.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233156)
Time Spent: 4h (was: 3h 50m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233151&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233151
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 22:56
Start Date: 25/Apr/19 22:56
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278760822
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +165,37 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
+ LOG.error("Execution exception occurred during createPipeline", ex);
+ throw new IOException("Execution exception occurred during " +
+ "createPipeline", ex);
+} catch (RejectedExecutionException ex) {
+ LOG.error("RejectedExecutionException, occurred during " +
Review comment:
Logging here, because in the actual createPipelines method in
BackGroudPipelineCreator which calls this method when we return IOException
they break from while loop. That is the reason for the logging. Let me know if
we still don't want to log here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233151)
Time Spent: 3h 50m (was: 3h 40m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233147&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233147
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 22:52
Start Date: 25/Apr/19 22:52
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #714: HDDS-1406. Avoid
usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278759739
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +165,37 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
+ LOG.error("Execution exception occurred during createPipeline", ex);
+ throw new IOException("Execution exception occurred during " +
+ "createPipeline", ex);
+} catch (RejectedExecutionException ex) {
+ LOG.error("RejectedExecutionException, occurred during " +
Review comment:
Same here, don't log.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233147)
Time Spent: 3.5h (was: 3h 20m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233148&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233148
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 22:52
Start Date: 25/Apr/19 22:52
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #714: HDDS-1406. Avoid
usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278759705
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +165,37 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
+ LOG.error("Execution exception occurred during createPipeline", ex);
Review comment:
Don't log here, just throw IOException that wraps the ExecutionException.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233148)
Time Spent: 3.5h (was: 3h 20m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233149&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233149
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 22:52
Start Date: 25/Apr/19 22:52
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #714: HDDS-1406. Avoid
usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278759939
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +165,37 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
Review comment:
You can simplify the code a bit by catching multiple exceptions in one
clause. e.g.
catch(ExecutionException|RejectedExecutionException e)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233149)
Time Spent: 3h 40m (was: 3.5h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233143&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233143
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 22:44
Start Date: 25/Apr/19 22:44
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278758443
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +164,28 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
Review comment:
DOne
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233143)
Time Spent: 3h 10m (was: 3h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233141&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233141 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 25/Apr/19 22:44 Start Date: 25/Apr/19 22:44 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-486863874 Thank You @arp7 for the review. Addressed review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233141) Time Spent: 2h 50m (was: 2h 40m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233142&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233142
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 22:44
Start Date: 25/Apr/19 22:44
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278758367
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##
@@ -1010,6 +1011,9 @@ public void stop() {
} catch (Exception ex) {
LOG.error("SCM Metadata store stop failed", ex);
}
+
+// shutdown RatisPipelineUtils pool.
+RatisPipelineUtils.POOL.shutdown();
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233142)
Time Spent: 3h (was: 2h 50m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=233144&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233144
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 22:44
Start Date: 25/Apr/19 22:44
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #714:
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278758443
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +164,28 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233144)
Time Spent: 3h 20m (was: 3h 10m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=232476&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-232476
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 01:36
Start Date: 25/Apr/19 01:36
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #714: HDDS-1406. Avoid
usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278372274
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##
@@ -1010,6 +1011,9 @@ public void stop() {
} catch (Exception ex) {
LOG.error("SCM Metadata store stop failed", ex);
}
+
+// shutdown RatisPipelineUtils pool.
+RatisPipelineUtils.POOL.shutdown();
Review comment:
shutdown will wait for previously submitted tasks to complete. You can
probably call shutdownNow since we don't care about the completion on shutdown.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 232476)
Time Spent: 2h 40m (was: 2.5h)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=232477&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-232477
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 01:36
Start Date: 25/Apr/19 01:36
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #714: HDDS-1406. Avoid
usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278371736
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +164,28 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
Review comment:
What about RejectedExecutionException. we should probably catch that also.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 232477)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=232475&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-232475
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 25/Apr/19 01:36
Start Date: 25/Apr/19 01:36
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #714: HDDS-1406. Avoid
usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r278372087
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,19 +164,28 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
-});
+try {
+ POOL.submit(() -> {
+datanodes.parallelStream().forEach(d -> {
+ final RaftPeer p = RatisHelper.toRaftPeer(d);
+ try (RaftClient client = RatisHelper
+ .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
+ retryPolicy, maxOutstandingRequests, tlsConfig,
+ requestTimeout)) {
+rpc.accept(client, p);
+ } catch (IOException ioe) {
+String errMsg =
+"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
+LOG.error(errMsg, ioe);
+exceptions.add(new IOException(errMsg, ioe));
+ }
+});
+ }).get();
+} catch (ExecutionException ex) {
Review comment:
We should also not swallow the ExecutionException and InterruptedException.
Instead convert them to something like PipelineCreationException which extends
IOException. Then createPipeline can catch it and handle it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 232475)
Time Spent: 2.5h (was: 2h 20m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=226845&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226845 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 12/Apr/19 18:45 Start Date: 12/Apr/19 18:45 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-482682418 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 63 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1109 | trunk passed | | +1 | compile | 31 | trunk passed | | +1 | checkstyle | 21 | trunk passed | | +1 | mvnsite | 31 | trunk passed | | +1 | shadedclient | 777 | branch has no errors when building and testing our client artifacts. | | +1 | findbugs | 40 | trunk passed | | +1 | javadoc | 24 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 33 | the patch passed | | +1 | compile | 25 | the patch passed | | +1 | javac | 25 | the patch passed | | +1 | checkstyle | 14 | the patch passed | | +1 | mvnsite | 26 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 809 | patch has no errors when building and testing our client artifacts. | | +1 | findbugs | 45 | the patch passed | | +1 | javadoc | 19 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 182 | server-scm in the patch passed. | | +1 | asflicense | 26 | The patch does not generate ASF License warnings. | | | | 3380 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-714/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/714 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 4c374b190af4 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 0c1fec3 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/3/testReport/ | | Max. process+thread count | 394 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/3/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226845) Time Spent: 2h 20m (was: 2h 10m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=226844&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226844 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 12/Apr/19 18:44 Start Date: 12/Apr/19 18:44 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-482682282 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 25 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1066 | trunk passed | | +1 | compile | 49 | trunk passed | | +1 | checkstyle | 22 | trunk passed | | +1 | mvnsite | 32 | trunk passed | | +1 | shadedclient | 665 | branch has no errors when building and testing our client artifacts. | | +1 | findbugs | 46 | trunk passed | | +1 | javadoc | 28 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 36 | the patch passed | | +1 | compile | 27 | the patch passed | | +1 | javac | 27 | the patch passed | | +1 | checkstyle | 16 | the patch passed | | +1 | mvnsite | 27 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 748 | patch has no errors when building and testing our client artifacts. | | +1 | findbugs | 48 | the patch passed | | +1 | javadoc | 22 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 98 | server-scm in the patch passed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 3077 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-714/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/714 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 98a0fe5d23aa 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 0c1fec3 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/4/testReport/ | | Max. process+thread count | 411 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/4/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226844) Time Spent: 2h 10m (was: 2h) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=226821&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226821 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 12/Apr/19 17:49 Start Date: 12/Apr/19 17:49 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-482663923 Thank You @lokeshj1703 for the review. Fixed review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226821) Time Spent: 2h (was: 1h 50m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=226820&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226820 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 12/Apr/19 17:47 Start Date: 12/Apr/19 17:47 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#discussion_r275004441 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java ## @@ -51,6 +52,15 @@ private static final Logger LOG = LoggerFactory.getLogger(RatisPipelineUtils.class); + // Set parallelism at 3, as now in Ratis we create 1 and 3 node pipelines. + private static int parallelismForPool = + (Runtime.getRuntime().availableProcessors() > 3) ? 3 : + Runtime.getRuntime().availableProcessors(); + + private static ForkJoinPool pool = new ForkJoinPool(parallelismForPool); Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226820) Time Spent: 1h 50m (was: 1h 40m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=226246&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226246 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 11/Apr/19 18:08 Start Date: 11/Apr/19 18:08 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-482231334 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1130 | trunk passed | | +1 | compile | 30 | trunk passed | | +1 | checkstyle | 20 | trunk passed | | +1 | mvnsite | 31 | trunk passed | | +1 | shadedclient | 781 | branch has no errors when building and testing our client artifacts. | | +1 | findbugs | 48 | trunk passed | | +1 | javadoc | 23 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 37 | the patch passed | | +1 | compile | 26 | the patch passed | | +1 | javac | 26 | the patch passed | | +1 | checkstyle | 15 | the patch passed | | +1 | mvnsite | 30 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 788 | patch has no errors when building and testing our client artifacts. | | +1 | findbugs | 54 | the patch passed | | +1 | javadoc | 20 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 119 | server-scm in the patch failed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 3297 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-714/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/714 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 74a655c7064c 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / a0468c5 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/2/artifact/out/patch-unit-hadoop-hdds_server-scm.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/2/testReport/ | | Max. process+thread count | 354 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/2/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226246) Time Spent: 1h 40m (was: 1.5h) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=226241&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226241 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 11/Apr/19 18:01 Start Date: 11/Apr/19 18:01 Worklog Time Spent: 10m Work Description: lokeshj1703 commented on pull request #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#discussion_r274560102 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java ## @@ -51,6 +52,15 @@ private static final Logger LOG = LoggerFactory.getLogger(RatisPipelineUtils.class); + // Set parallelism at 3, as now in Ratis we create 1 and 3 node pipelines. + private static int parallelismForPool = + (Runtime.getRuntime().availableProcessors() > 3) ? 3 : + Runtime.getRuntime().availableProcessors(); + + private static ForkJoinPool pool = new ForkJoinPool(parallelismForPool); Review comment: We can use three threads as these threads would ultimately be blocked on network IO and will not be CPU intensive. Also we need to shutdown the fork join pool. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226241) Time Spent: 1.5h (was: 1h 20m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=226228&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226228 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 11/Apr/19 17:44 Start Date: 11/Apr/19 17:44 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-48402 Thank You @arp7 for offline review comments. I have fixed them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226228) Time Spent: 1h 20m (was: 1h 10m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=226185&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226185 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 11/Apr/19 17:18 Start Date: 11/Apr/19 17:18 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-482210193 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226185) Time Spent: 1h 10m (was: 1h) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=226183&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226183 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 11/Apr/19 17:18 Start Date: 11/Apr/19 17:18 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#discussion_r274539471 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java ## @@ -51,6 +52,15 @@ private static final Logger LOG = LoggerFactory.getLogger(RatisPipelineUtils.class); + // Set parallelism at 3, as now in Ratis we create 1 and 3 node pipelines. + private static int parallelismForPool = + (Runtime.getRuntime().availableProcessors() > 3) ? 3 : + Runtime.getRuntime().availableProcessors(); + + private static ForkJoinPool pool = new ForkJoinPool(parallelismForPool); Review comment: Used ForkJoinPool can be used for normal tasks also. Refer NonFrok Join Clients. https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ForkJoinPool.html Used this ForJoinPool here to take advantage of its WorkSteal Algorithm in allocating/claiming threads. I think this might be useful. More can be read from the documentation. And if no of available processors is less than 3, even setting to 3 will not have any advantage. This is the reason for setting this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226183) Time Spent: 1h (was: 50m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=226097&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226097 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 11/Apr/19 15:51 Start Date: 11/Apr/19 15:51 Worklog Time Spent: 10m Work Description: lokeshj1703 commented on pull request #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#discussion_r274493607 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java ## @@ -51,6 +52,15 @@ private static final Logger LOG = LoggerFactory.getLogger(RatisPipelineUtils.class); + // Set parallelism at 3, as now in Ratis we create 1 and 3 node pipelines. + private static int parallelismForPool = + (Runtime.getRuntime().availableProcessors() > 3) ? 3 : + Runtime.getRuntime().availableProcessors(); + + private static ForkJoinPool pool = new ForkJoinPool(parallelismForPool); Review comment: We can use a simple executor as fork join pool is an executor for fork join tasks. Also we can always keep the number of threads to three as there is a network call involved in the task. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226097) Time Spent: 50m (was: 40m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=226095&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226095 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 11/Apr/19 15:49 Start Date: 11/Apr/19 15:49 Worklog Time Spent: 10m Work Description: lokeshj1703 commented on pull request #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#discussion_r274493607 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java ## @@ -51,6 +52,15 @@ private static final Logger LOG = LoggerFactory.getLogger(RatisPipelineUtils.class); + // Set parallelism at 3, as now in Ratis we create 1 and 3 node pipelines. + private static int parallelismForPool = + (Runtime.getRuntime().availableProcessors() > 3) ? 3 : + Runtime.getRuntime().availableProcessors(); + + private static ForkJoinPool pool = new ForkJoinPool(parallelismForPool); Review comment: We can use a simple executor as fork join pool is an executor for fork join tasks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226095) Time Spent: 40m (was: 0.5h) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[
https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=226094&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226094
]
ASF GitHub Bot logged work on HDDS-1406:
Author: ASF GitHub Bot
Created on: 11/Apr/19 15:47
Start Date: 11/Apr/19 15:47
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #714: HDDS-1406.
Avoid usage of commonPool in RatisPipelineUtils.
URL: https://github.com/apache/hadoop/pull/714#discussion_r274492733
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -146,18 +156,21 @@ private static void callRatisRpc(List
datanodes,
SecurityConfig(ozoneConf));
final TimeDuration requestTimeout =
RatisHelper.getClientRequestTimeout(ozoneConf);
-datanodes.parallelStream().forEach(d -> {
- final RaftPeer p = RatisHelper.toRaftPeer(d);
- try (RaftClient client = RatisHelper
- .newRaftClient(SupportedRpcType.valueOfIgnoreCase(rpcType), p,
- retryPolicy, maxOutstandingRequests, tlsConfig, requestTimeout))
{
-rpc.accept(client, p);
- } catch (IOException ioe) {
-String errMsg =
-"Failed invoke Ratis rpc " + rpc + " for " + d.getUuid();
-LOG.error(errMsg, ioe);
-exceptions.add(new IOException(errMsg, ioe));
- }
+pool.submit(() -> {
Review comment:
I think we need to make submit call for each datanode.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 226094)
Time Spent: 0.5h (was: 20m)
> Avoid usage of commonPool in RatisPipelineUtils
> ---
>
> Key: HDDS-1406
> URL: https://issues.apache.org/jira/browse/HDDS-1406
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We use parallelStream in during createPipline, this internally uses
> commonPool. Use Our own ForkJoinPool with parallelisim set with number of
> processors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1406) Avoid usage of commonPool in RatisPipelineUtils
[ https://issues.apache.org/jira/browse/HDDS-1406?focusedWorklogId=225278&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-225278 ] ASF GitHub Bot logged work on HDDS-1406: Author: ASF GitHub Bot Created on: 09/Apr/19 21:28 Start Date: 09/Apr/19 21:28 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #714: HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. URL: https://github.com/apache/hadoop/pull/714#issuecomment-481447227 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 455 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 1316 | root in trunk failed. | | +1 | compile | 64 | trunk passed | | +1 | checkstyle | 22 | trunk passed | | +1 | mvnsite | 35 | trunk passed | | +1 | shadedclient | 749 | branch has no errors when building and testing our client artifacts. | | +1 | findbugs | 45 | trunk passed | | +1 | javadoc | 25 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 35 | the patch passed | | +1 | compile | 27 | the patch passed | | +1 | javac | 27 | the patch passed | | +1 | checkstyle | 15 | the patch passed | | +1 | mvnsite | 26 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 804 | patch has no errors when building and testing our client artifacts. | | +1 | findbugs | 46 | the patch passed | | +1 | javadoc | 19 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 97 | server-scm in the patch passed. | | +1 | asflicense | 26 | The patch does not generate ASF License warnings. | | | | 3884 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-714/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/714 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 131df6f78a2d 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 312d4d4 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/1/artifact/out/branch-mvninstall-root.txt | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/1/testReport/ | | Max. process+thread count | 394 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-714/1/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 225278) Time Spent: 20m (was: 10m) > Avoid usage of commonPool in RatisPipelineUtils > --- > > Key: HDDS-1406 > URL: https://issues.apache.org/jira/browse/HDDS-1406 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > We use parallelStream in during createPipline, this internally uses > commonPool. Use Our own ForkJoinPool with parallelisim set with number of > processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
