[jira] [Commented] (HDFS-3246) pRead equivalent for direct read path
[
https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771671#comment-16771671
]
Hadoop QA commented on HDFS-3246:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project/hadoop-hdfs-native-client {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 4m
16s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 4m 16s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 16s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 44s{color} | {color:orange} root: The patch generated 35 new + 49 unchanged
- 0 fixed = 84 total (was 49) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project/hadoop-hdfs-native-client {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
2s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
44s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 43s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}225m 20s{color} |
{color:black} {color} |
\\
\\
||
[jira] [Commented] (HDFS-14235) Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon
[
https://issues.apache.org/jira/browse/HDFS-14235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771638#comment-16771638
]
Ranith Sardar commented on HDFS-14235:
--
Thanks [~surendrasingh]. I will update very soon.
> Handle ArrayIndexOutOfBoundsException in
> DataNodeDiskMetrics#slowDiskDetectionDaemon
> -
>
> Key: HDFS-14235
> URL: https://issues.apache.org/jira/browse/HDFS-14235
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-14235.000.patch, HDFS-14235.001.patch, NPE.png,
> exception.png
>
>
> below code throwing exception because {{volumeIterator.next()}} called two
> time without checking hashNext().
> {code:java}
> while (volumeIterator.hasNext()) {
> FsVolumeSpi volume = volumeIterator.next();
> DataNodeVolumeMetrics metrics = volumeIterator.next().getMetrics();
> String volumeName = volume.getBaseURI().getPath();
> metadataOpStats.put(volumeName,
> metrics.getMetadataOperationMean());
> readIoStats.put(volumeName, metrics.getReadIoMean());
> writeIoStats.put(volumeName, metrics.getWriteIoMean());
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14235) Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon
[
https://issues.apache.org/jira/browse/HDFS-14235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771576#comment-16771576
]
Surendra Singh Lilhore edited comment on HDFS-14235 at 2/19/19 6:55 AM:
[~RANith], one minor comment.
Pls correct the new variable description, it is not clear.
was (Author: surendrasingh):
+1
> Handle ArrayIndexOutOfBoundsException in
> DataNodeDiskMetrics#slowDiskDetectionDaemon
> -
>
> Key: HDFS-14235
> URL: https://issues.apache.org/jira/browse/HDFS-14235
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-14235.000.patch, HDFS-14235.001.patch, NPE.png,
> exception.png
>
>
> below code throwing exception because {{volumeIterator.next()}} called two
> time without checking hashNext().
> {code:java}
> while (volumeIterator.hasNext()) {
> FsVolumeSpi volume = volumeIterator.next();
> DataNodeVolumeMetrics metrics = volumeIterator.next().getMetrics();
> String volumeName = volume.getBaseURI().getPath();
> metadataOpStats.put(volumeName,
> metrics.getMetadataOperationMean());
> readIoStats.put(volumeName, metrics.getReadIoMean());
> writeIoStats.put(volumeName, metrics.getWriteIoMean());
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1085) Create an OM API to serve snapshots to Recon server
[ https://issues.apache.org/jira/browse/HDDS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771624#comment-16771624 ] Hudson commented on HDDS-1085: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15993 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15993/]) HDDS-1085. Create an OM API to serve snapshots to Recon server. (aengineer: rev 588b4c4d78d3cc7d684b07176dad9bd7ec603ff1) * (add) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBCheckpointSnapshot.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java * (edit) hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/TestOmUtils.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java * (add) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBCheckpointManager.java * (edit) hadoop-ozone/common/pom.xml * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManagerHttpServer.java * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDbSnapshotServlet.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java * (edit) hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestRDBStore.java > Create an OM API to serve snapshots to Recon server > --- > > Key: HDDS-1085 > URL: https://issues.apache.org/jira/browse/HDDS-1085 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Siddharth Wagle >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1085-000.patch, HDDS-1085-001.patch, > HDDS-1085-002.patch, HDDS-1085-003.patch > > Time Spent: 10m > Remaining Estimate: 0h > > We need to add an API to OM so that we can serve snapshots from the OM server. > - The snapshot should be streamed to fsck server with the ability to > throttle network utilization (like TransferFsImage) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-134) SCM CA: OM sends CSR and uses certificate issued by SCM
[ https://issues.apache.org/jira/browse/HDDS-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771622#comment-16771622 ] Anu Engineer commented on HDDS-134: --- [~ajayydv] Can you please rebase this patch, not applying to the head any more. Thanks > SCM CA: OM sends CSR and uses certificate issued by SCM > --- > > Key: HDDS-134 > URL: https://issues.apache.org/jira/browse/HDDS-134 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-134.00.patch, HDDS-134.01.patch, HDDS-134.02.patch, > HDDS-134.03.patch > > > Initialize OM keypair and get SCM signed certificate. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1085) Create an OM API to serve snapshots to Recon server
[ https://issues.apache.org/jira/browse/HDDS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-1085: --- Resolution: Fixed Fix Version/s: 0.4.0 Status: Resolved (was: Patch Available) [~elek] Thanks for the reviews. [~avijayan] Thanks for the contribution. [~swagle] Thanks for reporting this issue. I have committed this to the trunk branch with the understanding that we will add counters and possible move these as first class operations on OM, perhaps invoked via RPC to create snapshot and http to be read an existing snapshot instead of doing all that operation in a single blocking call. This is a good start, so let make forward progress. [~avijayan] Would it be possible for your file some Jiras to track these work items? Thanks > Create an OM API to serve snapshots to Recon server > --- > > Key: HDDS-1085 > URL: https://issues.apache.org/jira/browse/HDDS-1085 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Siddharth Wagle >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1085-000.patch, HDDS-1085-001.patch, > HDDS-1085-002.patch, HDDS-1085-003.patch > > Time Spent: 10m > Remaining Estimate: 0h > > We need to add an API to OM so that we can serve snapshots from the OM server. > - The snapshot should be streamed to fsck server with the ability to > throttle network utilization (like TransferFsImage) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[
https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771614#comment-16771614
]
Hadoop QA commented on HDDS-1130:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
46s{color} | {color:red} hadoop-ozone/tools generated 3 new + 0 unchanged - 0
fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 16s{color}
| {color:red} tools in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-ozone/tools |
| | Write to static field
org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.blockManager from
instance method
org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At
BenchMarkBlockManager.java:from instance method
org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At
BenchMarkBlockManager.java:[line 110] |
| | Write to static field
org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.pipelineManager from
instance method
org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At
BenchMarkBlockManager.java:from instance method
org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At
BenchMarkBlockManager.java:[line 105] |
| | Write to static field
org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.scm from instance method
org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At
BenchMarkBlockManager.java:from instance method
org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At
BenchMarkBlockManager.java:[line 104] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[
https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771596#comment-16771596
]
Yiqun Lin commented on HDDS-1130:
-
The patch almost looks good to me, two minor comments:
* Can we define a static variable {{NUM_PIPELINES}} to replace hard-coded
value {{10}}?
* We can use {{GenericTestUtils#waitFor}} to replace thread sleep:
{noformat}
GenericTestUtils.waitFor(() -> {
return !blockManager.isScmInChillMode();
}, 100, 6);
{noformat}
> Make BenchMarkBlockManager multi-threaded
> -
>
> Key: HDDS-1130
> URL: https://issues.apache.org/jira/browse/HDDS-1130
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1130.001.patch
>
>
> Currently BenchMarkBlockManager is run by a single thread. We can make it
> multi-threaded in order to have a better understanding of allocateBlock call
> performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1086) Remove RaftClient from OM
[ https://issues.apache.org/jira/browse/HDDS-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771589#comment-16771589 ] Shashikant Banerjee commented on HDDS-1086: --- Thanks [~hanishakoneru] for reporting this. I have one question here: 1) RaftClient has many useful features like, async API, asynchronous retries, ordering the request in the client side by maintaining a sliding window so that it never serves requests out of order etc. With all these features built in, removing Raft Client here may lead to handle some of these cases in Ozone. The alternative approach would be to implement Short Circuit read/write in Ratis itself to address the concern here. What do you think? > Remove RaftClient from OM > - > > Key: HDDS-1086 > URL: https://issues.apache.org/jira/browse/HDDS-1086 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: HA, OM >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-1086.001.patch > > > Currently we run RaftClient in OM which takes the incoming client requests > and submits it to the OM's Ratis server. This hop can be avoided if OM > submits the incoming client request directly to its Ratis server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771587#comment-16771587
]
Hadoop QA commented on HDFS-14295:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
1s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 51s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}162m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14295 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959178/HDFS-14295.3.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c58061efffd6 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7587f97 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26256/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26256/testReport/ |
| Max. process+thread count | 3466 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Updated] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDDS-1130: -- Status: Patch Available (was: Open) > Make BenchMarkBlockManager multi-threaded > - > > Key: HDDS-1130 > URL: https://issues.apache.org/jira/browse/HDDS-1130 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1130.001.patch > > > Currently BenchMarkBlockManager is run by a single thread. We can make it > multi-threaded in order to have a better understanding of allocateBlock call > performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDDS-1130: -- Attachment: HDDS-1130.001.patch > Make BenchMarkBlockManager multi-threaded > - > > Key: HDDS-1130 > URL: https://issues.apache.org/jira/browse/HDDS-1130 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1130.001.patch > > > Currently BenchMarkBlockManager is run by a single thread. We can make it > multi-threaded in order to have a better understanding of allocateBlock call > performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14235) Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon
[
https://issues.apache.org/jira/browse/HDFS-14235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771576#comment-16771576
]
Surendra Singh Lilhore commented on HDFS-14235:
---
+1
> Handle ArrayIndexOutOfBoundsException in
> DataNodeDiskMetrics#slowDiskDetectionDaemon
> -
>
> Key: HDFS-14235
> URL: https://issues.apache.org/jira/browse/HDFS-14235
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-14235.000.patch, HDFS-14235.001.patch, NPE.png,
> exception.png
>
>
> below code throwing exception because {{volumeIterator.next()}} called two
> time without checking hashNext().
> {code:java}
> while (volumeIterator.hasNext()) {
> FsVolumeSpi volume = volumeIterator.next();
> DataNodeVolumeMetrics metrics = volumeIterator.next().getMetrics();
> String volumeName = volume.getBaseURI().getPath();
> metadataOpStats.put(volumeName,
> metrics.getMetadataOperationMean());
> readIoStats.put(volumeName, metrics.getReadIoMean());
> writeIoStats.put(volumeName, metrics.getWriteIoMean());
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
Lokesh Jain created HDDS-1130: - Summary: Make BenchMarkBlockManager multi-threaded Key: HDDS-1130 URL: https://issues.apache.org/jira/browse/HDDS-1130 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Reporter: Lokesh Jain Assignee: Lokesh Jain Fix For: 0.4.0 Currently BenchMarkBlockManager is run by a single thread. We can make it multi-threaded in order to have a better understanding of allocateBlock call performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14219) ConcurrentModificationException occurs in datanode occasionally
[
https://issues.apache.org/jira/browse/HDFS-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771557#comment-16771557
]
Hadoop QA commented on HDFS-14219:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
6s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
51s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}173m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 2m
24s{color} | {color:red} The patch generated 414 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}206m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-hdfs:44 |
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestQuota |
| | hadoop.hdfs.TestFileAppend4 |
| Timed out junit tests | org.apache.hadoop.hdfs.TestEncryptionZones |
| | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
| | org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA |
| | org.apache.hadoop.hdfs.TestParallelShortCircuitRead |
| | org.apache.hadoop.hdfs.TestHdfsAdmin |
| | org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| | org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | org.apache.hadoop.hdfs.TestDatanodeRegistration |
| | org.apache.hadoop.hdfs.TestSetrepDecreasing |
| | org.apache.hadoop.hdfs.TestReservedRawPaths |
| | org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | org.apache.hadoop.hdfs.TestDataTransferKeepalive |
| | org.apache.hadoop.hdfs.TestDatanodeDeath |
| | org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart |
| | org.apache.hadoop.hdfs.TestPread |
| | org.apache.hadoop.hdfs.TestSafeMode |
| | org.apache.hadoop.hdfs.TestDFSFinalize |
| | org.apache.hadoop.hdfs.web.TestWebHdfsTokens |
| | org.apache.hadoop.hdfs.TestFileAppendRestart |
| | org.apache.hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter |
| | org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS |
| | org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
| | org.apache.hadoop.hdfs.TestBlockStoragePolicy |
| | org.apache.hadoop.hdfs.TestRollingUpgrade |
| | org.apache.hadoop.hdfs.TestLease |
| | org.apache.hadoop.hdfs.TestDFSInputStream |
| |
[jira] [Assigned] (HDDS-1094) Performance testing infrastructure : Special handling for zero-filled chunks on the Datanode
[ https://issues.apache.org/jira/browse/HDDS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Supratim Deka reassigned HDDS-1094: --- Assignee: Supratim Deka > Performance testing infrastructure : Special handling for zero-filled chunks > on the Datanode > > > Key: HDDS-1094 > URL: https://issues.apache.org/jira/browse/HDDS-1094 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > > Goal: > Make Ozone chunk Read/Write operations CPU/network bound for specially > constructed performance micro benchmarks. > Remove disk bandwidth and latency constraints - running ozone data path > against extreme low-latency & high throughput storage will expose performance > bottlenecks in the flow. But low-latency storage(NVME flash drives, Storage > class memory etc) is expensive and availability is limited. Is there a > workaround which achieves similar running conditions for the software without > actually having the low latency storage? At least for specially constructed > datasets - for example zero-filled blocks (*not* zero-length blocks). > Required characteristics of the solution: > No changes in Ozone client, OM and SCM. Changes limited to Datanode, Minimal > footprint in datanode code. > Possible High level Approach: > The ChunkManager and ChunkUtils can enable writeChunk for zero-filled chunks > to be dropped without actually writing to the local filesystem. Similarly, if > readChunk can construct a zero-filled buffer without reading from the local > filesystem whenever it detects a zero-filled chunk. Specifics of how to > detect and record a zero-filled chunk can be discussed on this jira. Also > discuss how to control this behaviour and make it available only for internal > testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-3246) pRead equivalent for direct read path
[ https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HDFS-3246: --- Target Version/s: (was: ) Status: Patch Available (was: Open) > pRead equivalent for direct read path > - > > Key: HDFS-3246 > URL: https://issues.apache.org/jira/browse/HDFS-3246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, performance >Affects Versions: 3.0.0-alpha1 >Reporter: Henry Robinson >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-3246.001.patch > > > There is no pread equivalent in ByteBufferReadable. We should consider adding > one. It would be relatively easy to implement for the distributed case > (certainly compared to HDFS-2834), since DFSInputStream does most of the > heavy lifting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-3246) pRead equivalent for direct read path
[ https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HDFS-3246: --- Attachment: HDFS-3246.001.patch > pRead equivalent for direct read path > - > > Key: HDFS-3246 > URL: https://issues.apache.org/jira/browse/HDFS-3246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, performance >Affects Versions: 3.0.0-alpha1 >Reporter: Henry Robinson >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-3246.001.patch > > > There is no pread equivalent in ByteBufferReadable. We should consider adding > one. It would be relatively easy to implement for the distributed case > (certainly compared to HDFS-2834), since DFSInputStream does most of the > heavy lifting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-699) Detect Ozone Network topology
[ https://issues.apache.org/jira/browse/HDDS-699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771543#comment-16771543 ] Sammi Chen commented on HDDS-699: - Thanks [~szetszwo] and [~xyao] for the great suggestion. I uploaded 02 patch. Here is the change list, # removes the lock on NetworkToplogy # NodeImpl and InnerNodeImpl become thread safe # add concurrent access network topology unit test # add NodeSchemaLoader and NodeSchemaManager unit test The total test coverage now is class(100%), method(97%) and line(90%). > Detect Ozone Network topology > - > > Key: HDDS-699 > URL: https://issues.apache.org/jira/browse/HDDS-699 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-699.00.patch, HDDS-699.01.patch, HDDS-699.02.patch > > > Traditionally this has been implemented in Hadoop via script or customizable > java class. One thing we want to add here is the flexible multi-level support > instead of fixed levels like DC/Rack/NG/Node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-699) Detect Ozone Network topology
[ https://issues.apache.org/jira/browse/HDDS-699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-699: Attachment: HDDS-699.02.patch > Detect Ozone Network topology > - > > Key: HDDS-699 > URL: https://issues.apache.org/jira/browse/HDDS-699 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-699.00.patch, HDDS-699.01.patch, HDDS-699.02.patch > > > Traditionally this has been implemented in Hadoop via script or customizable > java class. One thing we want to add here is the flexible multi-level support > instead of fixed levels like DC/Rack/NG/Node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1122) Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure
[
https://issues.apache.org/jira/browse/HDDS-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771536#comment-16771536
]
Hudson commented on HDDS-1122:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15992 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15992/])
HDDS-1122. Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit (yqlin:
rev 67af509097d8df4df161d53e3e511b312e8ac3dd)
* (edit)
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OMRatisHelper.java
> Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure
>
>
> Key: HDDS-1122
> URL: https://issues.apache.org/jira/browse/HDDS-1122
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Aravindan Vijayan
>Assignee: Yiqun Lin
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1122.001.patch
>
>
> {code}
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerRatisServer.testSubmitRatisRequest
> Failing for the past 7 builds (Since Failed#2281 )
> Took 1.8 sec.
> Error Message
> Message missing required fields: status
> Stacktrace
> com.google.protobuf.UninitializedMessageException: Message missing required
> fields: status
> at
> com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:770)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMResponse$Builder.build(OzoneManagerProtocolProtos.java:11664)
> at
> org.apache.hadoop.ozone.om.ratis.OMRatisHelper.getErrorResponse(OMRatisHelper.java:113)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommand(OzoneManagerRatisClient.java:128)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerRatisServer.testSubmitRatisRequest(TestOzoneManagerRatisServer.java:125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1122) Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure
[
https://issues.apache.org/jira/browse/HDDS-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDDS-1122:
Resolution: Fixed
Fix Version/s: 0.4.0
Status: Resolved (was: Patch Available)
Committed this.
Thanks [~anu] for the review and [~avijayan] for the report.
> Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure
>
>
> Key: HDDS-1122
> URL: https://issues.apache.org/jira/browse/HDDS-1122
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Aravindan Vijayan
>Assignee: Yiqun Lin
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1122.001.patch
>
>
> {code}
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerRatisServer.testSubmitRatisRequest
> Failing for the past 7 builds (Since Failed#2281 )
> Took 1.8 sec.
> Error Message
> Message missing required fields: status
> Stacktrace
> com.google.protobuf.UninitializedMessageException: Message missing required
> fields: status
> at
> com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:770)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMResponse$Builder.build(OzoneManagerProtocolProtos.java:11664)
> at
> org.apache.hadoop.ozone.om.ratis.OMRatisHelper.getErrorResponse(OMRatisHelper.java:113)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommand(OzoneManagerRatisClient.java:128)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerRatisServer.testSubmitRatisRequest(TestOzoneManagerRatisServer.java:125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14295.3.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Patch Available (was: Open)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
[
https://issues.apache.org/jira/browse/HDFS-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771503#comment-16771503
]
Hudson commented on HDFS-14296:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15991 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15991/])
HDFS-14296. Prefer ArrayList over LinkedList in VolumeScanner. (inigoiri: rev
7587f97127d0cde21c39e98b7a33b60195112dc3)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
> Prefer ArrayList over LinkedList in VolumeScanner
> -
>
> Key: HDFS-14296
> URL: https://issues.apache.org/jira/browse/HDFS-14296
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14296.1.patch
>
>
> {quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
> {{listIterator}} operations run in constant time. - ArrayList
> {quote}
> However, for a {{LinkedList}}, the entire list must be traversed to get to
> the desired index.
> Like
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
> Most of the time, the List is being iterated, which is quicker over a
> primitive array than walking a LinkedList.
> There is one place where an item is removed, potentially from the middle of
> the list
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
> but the speed of removing from the middle of the list isn't bad; it's a
> system native array shift, and it only happens on the off chance that a block
> pool is removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
[
https://issues.apache.org/jira/browse/HDFS-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14296:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
> Prefer ArrayList over LinkedList in VolumeScanner
> -
>
> Key: HDFS-14296
> URL: https://issues.apache.org/jira/browse/HDFS-14296
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14296.1.patch
>
>
> {quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
> {{listIterator}} operations run in constant time. - ArrayList
> {quote}
> However, for a {{LinkedList}}, the entire list must be traversed to get to
> the desired index.
> Like
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
> Most of the time, the List is being iterated, which is quicker over a
> primitive array than walking a LinkedList.
> There is one place where an item is removed, potentially from the middle of
> the list
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
> but the speed of removing from the middle of the list isn't bad; it's a
> system native array shift, and it only happens on the off chance that a block
> pool is removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
[
https://issues.apache.org/jira/browse/HDFS-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771492#comment-16771492
]
Íñigo Goiri commented on HDFS-14296:
Thanks [~belugabehr] for the patch.
Committed to trunk.
> Prefer ArrayList over LinkedList in VolumeScanner
> -
>
> Key: HDFS-14296
> URL: https://issues.apache.org/jira/browse/HDFS-14296
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14296.1.patch
>
>
> {quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
> {{listIterator}} operations run in constant time. - ArrayList
> {quote}
> However, for a {{LinkedList}}, the entire list must be traversed to get to
> the desired index.
> Like
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
> Most of the time, the List is being iterated, which is quicker over a
> primitive array than walking a LinkedList.
> There is one place where an item is removed, potentially from the middle of
> the list
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
> but the speed of removing from the middle of the list isn't bad; it's a
> system native array shift, and it only happens on the off chance that a block
> pool is removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
[
https://issues.apache.org/jira/browse/HDFS-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771471#comment-16771471
]
BELUGA BEHR commented on HDFS-14296:
This is a good reference.
https://stackoverflow.com/questions/322715/when-to-use-linkedlist-over-arraylist-in-java
> Prefer ArrayList over LinkedList in VolumeScanner
> -
>
> Key: HDFS-14296
> URL: https://issues.apache.org/jira/browse/HDFS-14296
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14296.1.patch
>
>
> {quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
> {{listIterator}} operations run in constant time. - ArrayList
> {quote}
> However, for a {{LinkedList}}, the entire list must be traversed to get to
> the desired index.
> Like
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
> Most of the time, the List is being iterated, which is quicker over a
> primitive array than walking a LinkedList.
> There is one place where an item is removed, potentially from the middle of
> the list
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
> but the speed of removing from the middle of the list isn't bad; it's a
> system native array shift, and it only happens on the off chance that a block
> pool is removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14130) Make ZKFC ObserverNode aware
[
https://issues.apache.org/jira/browse/HDFS-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771465#comment-16771465
]
Chao Sun commented on HDFS-14130:
-
Seems the test failure on \{{testManualFailoverWithDFSHAAdmin}} is related.
> Make ZKFC ObserverNode aware
>
>
> Key: HDFS-14130
> URL: https://issues.apache.org/jira/browse/HDFS-14130
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
>Affects Versions: HDFS-12943
>Reporter: Konstantin Shvachko
>Assignee: xiangheng
>Priority: Major
> Attachments: HDFS-14130-HDFS-12943.001.patch,
> HDFS-14130-HDFS-12943.003.patch, HDFS-14130-HDFS-12943.004.patch,
> HDFS-14130-HDFS-12943.005.patch, HDFS-14130-HDFS-12943.006.patch,
> HDFS-14130-HDFS-12943.007.patch, HDFS-14130.008.patch
>
>
> Need to fix automatic failover with ZKFC. Currently it does not know about
> ObserverNodes trying to convert them to SBNs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14219) ConcurrentModificationException occurs in datanode occasionally
[ https://issues.apache.org/jira/browse/HDFS-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Jie updated HDFS-14219: --- Attachment: HDFS-14129-branch-2.8.001.patch > ConcurrentModificationException occurs in datanode occasionally > --- > > Key: HDFS-14219 > URL: https://issues.apache.org/jira/browse/HDFS-14219 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.2 >Reporter: Tao Jie >Assignee: Tao Jie >Priority: Minor > Attachments: HDFS-14129-branch-2.8.001.patch, HDFS-14219.001.patch > > > ERROR occasionally occurs in datanode log: > ERROR BPServiceActor.java:792 - Exception in BPOfferService for Block pool > BP-1687106048-10.14.9.17-1535259994856 (Datanode Uuid > c17e635a-912f-4488-b4e8-93a58a27b5db) service to osscmh-9-21/10.14.9.21:8070 > java.util.ConcurrentModificationException: modification=62852685 != > iterModification = 62852684 > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.ensureNext(LightWeightGSet.java:305) > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.hasNext(LightWeightGSet.java:322) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1872) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:349) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:648) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:790) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
[
https://issues.apache.org/jira/browse/HDFS-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771460#comment-16771460
]
Hadoop QA commented on HDFS-14296:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}118m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}176m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14296 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959168/HDFS-14296.1.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6d5fa61697b6 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 235e3da |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26253/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26253/testReport/ |
| Max. process+thread count | 2906 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-14284) RBF: Log Router identifier when reporting exceptions
[
https://issues.apache.org/jira/browse/HDFS-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771453#comment-16771453
]
Chao Sun commented on HDFS-14284:
-
Agree this could be useful. Perhaps we can make this optional by adding another
constructor and field to {{RemoteException}}? cc [~crh] and [~fengnanli].
> RBF: Log Router identifier when reporting exceptions
>
>
> Key: HDFS-14284
> URL: https://issues.apache.org/jira/browse/HDFS-14284
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Priority: Major
>
> The typical setup is to use multiple Routers through
> ConfiguredFailoverProxyProvider.
> In a regular HA Namenode setup, it is easy to know which NN was used.
> However, in RBF, any Router can be the one reporting the exception and it is
> hard to know which was the one.
> We should have a way to identify which Router/Namenode was the one triggering
> the exception.
> This would also apply with Observer Namenodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14235) Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon
[
https://issues.apache.org/jira/browse/HDFS-14235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771446#comment-16771446
]
Hadoop QA commented on HDFS-14235:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 14s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14235 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959169/HDFS-14235.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 43a3b8cffc96 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 235e3da |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26254/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-9596) Remove Shuffle Method From DFSUtil
[ https://issues.apache.org/jira/browse/HDFS-9596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771435#comment-16771435 ] Hudson commented on HDFS-9596: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15990 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15990/]) HDFS-9596. Remove Shuffle Method From DFSUtil. Contributed by BELUGA (inigoiri: rev 1de25d134f64d815f9b43606fa426ece5ddbc430) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBlockPlacementPolicyRackFaultTolerant.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java > Remove Shuffle Method From DFSUtil > -- > > Key: HDFS-9596 > URL: https://issues.apache.org/jira/browse/HDFS-9596 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 3.3.0 > > Attachments: HDFS-9596.2.patch, HDFS-9596.3.patch, HDFS-9596.4.patch, > HDFS-9596.5.patch, HDFS-9596.6.patch, HDFS-9596.7.patch, > Shuffle.HDFS-9596.patch > > > DFSUtil contains a "shuffle" routine that shuffles arrays. With a little > wrapping, Java Collections class can shuffle. This code is superfluous and > undocumented. The method returns an array, though it does not specify if it > returns the same array or a new, copy, array of shuffled items. > It is only used in one place, so it is better to simply remove and substitute > this implementation with the java Collections implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
[
https://issues.apache.org/jira/browse/HDFS-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771434#comment-16771434
]
Íñigo Goiri commented on HDFS-14296:
Thanks [~belugabehr], I remember you had a link with a benchmark comparing
LinkedList and ArrayList.
Can you post it for completeness here?
As this is mostly iteration, ArrayList should be better.
+1 on [^HDFS-14296.1.patch].
> Prefer ArrayList over LinkedList in VolumeScanner
> -
>
> Key: HDFS-14296
> URL: https://issues.apache.org/jira/browse/HDFS-14296
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14296.1.patch
>
>
> {quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
> {{listIterator}} operations run in constant time. - ArrayList
> {quote}
> However, for a {{LinkedList}}, the entire list must be traversed to get to
> the desired index.
> Like
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
> Most of the time, the List is being iterated, which is quicker over a
> primitive array than walking a LinkedList.
> There is one place where an item is removed, potentially from the middle of
> the list
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
> but the speed of removing from the middle of the list isn't bad; it's a
> system native array shift, and it only happens on the off chance that a block
> pool is removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9596) Remove Shuffle Method From DFSUtil
[ https://issues.apache.org/jira/browse/HDFS-9596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771433#comment-16771433 ] Íñigo Goiri commented on HDFS-9596: --- Thanks [~belugabehr] for the patch. Committed to trunk. > Remove Shuffle Method From DFSUtil > -- > > Key: HDFS-9596 > URL: https://issues.apache.org/jira/browse/HDFS-9596 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 3.3.0 > > Attachments: HDFS-9596.2.patch, HDFS-9596.3.patch, HDFS-9596.4.patch, > HDFS-9596.5.patch, HDFS-9596.6.patch, HDFS-9596.7.patch, > Shuffle.HDFS-9596.patch > > > DFSUtil contains a "shuffle" routine that shuffles arrays. With a little > wrapping, Java Collections class can shuffle. This code is superfluous and > undocumented. The method returns an array, though it does not specify if it > returns the same array or a new, copy, array of shuffled items. > It is only used in one place, so it is better to simply remove and substitute > this implementation with the java Collections implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9596) Remove Shuffle Method From DFSUtil
[ https://issues.apache.org/jira/browse/HDFS-9596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-9596: -- Summary: Remove Shuffle Method From DFSUtil (was: Remove "Shuffle" Method From DFSUtil) > Remove Shuffle Method From DFSUtil > -- > > Key: HDFS-9596 > URL: https://issues.apache.org/jira/browse/HDFS-9596 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HDFS-9596.2.patch, HDFS-9596.3.patch, HDFS-9596.4.patch, > HDFS-9596.5.patch, HDFS-9596.6.patch, HDFS-9596.7.patch, > Shuffle.HDFS-9596.patch > > > DFSUtil contains a "shuffle" routine that shuffles arrays. With a little > wrapping, Java Collections class can shuffle. This code is superfluous and > undocumented. The method returns an array, though it does not specify if it > returns the same array or a new, copy, array of shuffled items. > It is only used in one place, so it is better to simply remove and substitute > this implementation with the java Collections implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9596) Remove Shuffle Method From DFSUtil
[ https://issues.apache.org/jira/browse/HDFS-9596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-9596: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) > Remove Shuffle Method From DFSUtil > -- > > Key: HDFS-9596 > URL: https://issues.apache.org/jira/browse/HDFS-9596 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 3.3.0 > > Attachments: HDFS-9596.2.patch, HDFS-9596.3.patch, HDFS-9596.4.patch, > HDFS-9596.5.patch, HDFS-9596.6.patch, HDFS-9596.7.patch, > Shuffle.HDFS-9596.patch > > > DFSUtil contains a "shuffle" routine that shuffles arrays. With a little > wrapping, Java Collections class can shuffle. This code is superfluous and > undocumented. The method returns an array, though it does not specify if it > returns the same array or a new, copy, array of shuffled items. > It is only used in one place, so it is better to simply remove and substitute > this implementation with the java Collections implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9596) Remove "Shuffle" Method From DFSUtil
[ https://issues.apache.org/jira/browse/HDFS-9596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771432#comment-16771432 ] Íñigo Goiri commented on HDFS-9596: --- +1 on [^HDFS-9596.7.patch]. Committing. > Remove "Shuffle" Method From DFSUtil > > > Key: HDFS-9596 > URL: https://issues.apache.org/jira/browse/HDFS-9596 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HDFS-9596.2.patch, HDFS-9596.3.patch, HDFS-9596.4.patch, > HDFS-9596.5.patch, HDFS-9596.6.patch, HDFS-9596.7.patch, > Shuffle.HDFS-9596.patch > > > DFSUtil contains a "shuffle" routine that shuffles arrays. With a little > wrapping, Java Collections class can shuffle. This code is superfluous and > undocumented. The method returns an array, though it does not specify if it > returns the same array or a new, copy, array of shuffled items. > It is only used in one place, so it is better to simply remove and substitute > this implementation with the java Collections implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771426#comment-16771426
]
Hadoop QA commented on HDFS-14295:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14295 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959166/HDFS-14295.3.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux cdeb9089ad58 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 235e3da |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26252/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26252/testReport/ |
| Max. process+thread count | 4817 (vs. ulimit of 1) |
| modules |
[jira] [Updated] (HDFS-9596) Remove "Shuffle" Method From DFSUtil
[ https://issues.apache.org/jira/browse/HDFS-9596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-9596: -- Attachment: (was: HDFS-9596.7.patch) > Remove "Shuffle" Method From DFSUtil > > > Key: HDFS-9596 > URL: https://issues.apache.org/jira/browse/HDFS-9596 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HDFS-9596.2.patch, HDFS-9596.3.patch, HDFS-9596.4.patch, > HDFS-9596.5.patch, HDFS-9596.6.patch, HDFS-9596.7.patch, > Shuffle.HDFS-9596.patch > > > DFSUtil contains a "shuffle" routine that shuffles arrays. With a little > wrapping, Java Collections class can shuffle. This code is superfluous and > undocumented. The method returns an array, though it does not specify if it > returns the same array or a new, copy, array of shuffled items. > It is only used in one place, so it is better to simply remove and substitute > this implementation with the java Collections implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9596) Remove "Shuffle" Method From DFSUtil
[ https://issues.apache.org/jira/browse/HDFS-9596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-9596: -- Attachment: (was: HDFS-9596.7.patch) > Remove "Shuffle" Method From DFSUtil > > > Key: HDFS-9596 > URL: https://issues.apache.org/jira/browse/HDFS-9596 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HDFS-9596.2.patch, HDFS-9596.3.patch, HDFS-9596.4.patch, > HDFS-9596.5.patch, HDFS-9596.6.patch, HDFS-9596.7.patch, > Shuffle.HDFS-9596.patch > > > DFSUtil contains a "shuffle" routine that shuffles arrays. With a little > wrapping, Java Collections class can shuffle. This code is superfluous and > undocumented. The method returns an array, though it does not specify if it > returns the same array or a new, copy, array of shuffled items. > It is only used in one place, so it is better to simply remove and substitute > this implementation with the java Collections implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771431#comment-16771431
]
Hadoop QA commented on HDFS-14292:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
49s{color} | {color:green} hadoop-hdfs-project generated 0 new + 539 unchanged
- 1 fixed = 539 total (was 540) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 58s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
528 unchanged - 2 fixed = 530 total (was 530) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
42s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadWhileWriting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestDataTransferKeepalive |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.TestLeaseRecovery |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.TestDataTransferProtocol |
| | hadoop.hdfs.server.namenode.TestDeleteRace |
| | hadoop.hdfs.TestReplaceDatanodeFailureReplication |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| |
[jira] [Commented] (HDFS-9596) Remove "Shuffle" Method From DFSUtil
[ https://issues.apache.org/jira/browse/HDFS-9596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771417#comment-16771417 ] BELUGA BEHR commented on HDFS-9596: --- [~giovanni.fumarola] [~elgoiri] Can either of you fine folks finish this one up for me? :) > Remove "Shuffle" Method From DFSUtil > > > Key: HDFS-9596 > URL: https://issues.apache.org/jira/browse/HDFS-9596 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HDFS-9596.2.patch, HDFS-9596.3.patch, HDFS-9596.4.patch, > HDFS-9596.5.patch, HDFS-9596.6.patch, HDFS-9596.7.patch, HDFS-9596.7.patch, > HDFS-9596.7.patch, Shuffle.HDFS-9596.patch > > > DFSUtil contains a "shuffle" routine that shuffles arrays. With a little > wrapping, Java Collections class can shuffle. This code is superfluous and > undocumented. The method returns an array, though it does not specify if it > returns the same array or a new, copy, array of shuffled items. > It is only used in one place, so it is better to simply remove and substitute > this implementation with the java Collections implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14249) RBF: Tooling to identify the subcluster location of a file
[
https://issues.apache.org/jira/browse/HDFS-14249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771416#comment-16771416
]
Hadoop QA commented on HDFS-14249:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
21s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
46s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 23m
54s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}209m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14249 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959156/HDFS-14249-HDFS-13891.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle cc |
| uname | Linux e3350b8a3ffe 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (HDFS-14235) Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon
[
https://issues.apache.org/jira/browse/HDFS-14235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771409#comment-16771409
]
Ranith Sardar commented on HDFS-14235:
--
Thanks [~surendrasingh], for reviewing the patch. I have updated accordingly
and added the new patch.
> Handle ArrayIndexOutOfBoundsException in
> DataNodeDiskMetrics#slowDiskDetectionDaemon
> -
>
> Key: HDFS-14235
> URL: https://issues.apache.org/jira/browse/HDFS-14235
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-14235.000.patch, HDFS-14235.001.patch, NPE.png,
> exception.png
>
>
> below code throwing exception because {{volumeIterator.next()}} called two
> time without checking hashNext().
> {code:java}
> while (volumeIterator.hasNext()) {
> FsVolumeSpi volume = volumeIterator.next();
> DataNodeVolumeMetrics metrics = volumeIterator.next().getMetrics();
> String volumeName = volume.getBaseURI().getPath();
> metadataOpStats.put(volumeName,
> metrics.getMetadataOperationMean());
> readIoStats.put(volumeName, metrics.getReadIoMean());
> writeIoStats.put(volumeName, metrics.getWriteIoMean());
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14235) Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon
[
https://issues.apache.org/jira/browse/HDFS-14235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ranith Sardar updated HDFS-14235:
-
Attachment: HDFS-14235.001.patch
> Handle ArrayIndexOutOfBoundsException in
> DataNodeDiskMetrics#slowDiskDetectionDaemon
> -
>
> Key: HDFS-14235
> URL: https://issues.apache.org/jira/browse/HDFS-14235
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-14235.000.patch, HDFS-14235.001.patch, NPE.png,
> exception.png
>
>
> below code throwing exception because {{volumeIterator.next()}} called two
> time without checking hashNext().
> {code:java}
> while (volumeIterator.hasNext()) {
> FsVolumeSpi volume = volumeIterator.next();
> DataNodeVolumeMetrics metrics = volumeIterator.next().getMetrics();
> String volumeName = volume.getBaseURI().getPath();
> metadataOpStats.put(volumeName,
> metrics.getMetadataOperationMean());
> readIoStats.put(volumeName, metrics.getReadIoMean());
> writeIoStats.put(volumeName, metrics.getWriteIoMean());
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
BELUGA BEHR created HDFS-14296:
--
Summary: Prefer ArrayList over LinkedList in VolumeScanner
Key: HDFS-14296
URL: https://issues.apache.org/jira/browse/HDFS-14296
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Assignee: BELUGA BEHR
Attachments: HDFS-14296.1.patch
{quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
{{listIterator}} operations run in constant time. - ArrayList
{quote}
However, for a {{LinkedList}}, the entire list must be traversed to get to the
desired index.
Like
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
Most of the time, the List is being iterated, which is quicker over a primitive
array than walking a LinkedList.
There is one place where an item is removed, potentially from the middle of the
list
[here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
but the speed of removing from the middle of the list isn't bad; it's a system
native array shift, and it only happens on the off chance that a block pool is
removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
[
https://issues.apache.org/jira/browse/HDFS-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14296:
---
Status: Patch Available (was: Open)
> Prefer ArrayList over LinkedList in VolumeScanner
> -
>
> Key: HDFS-14296
> URL: https://issues.apache.org/jira/browse/HDFS-14296
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14296.1.patch
>
>
> {quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
> {{listIterator}} operations run in constant time. - ArrayList
> {quote}
> However, for a {{LinkedList}}, the entire list must be traversed to get to
> the desired index.
> Like
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
> Most of the time, the List is being iterated, which is quicker over a
> primitive array than walking a LinkedList.
> There is one place where an item is removed, potentially from the middle of
> the list
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
> but the speed of removing from the middle of the list isn't bad; it's a
> system native array shift, and it only happens on the off chance that a block
> pool is removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
[
https://issues.apache.org/jira/browse/HDFS-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14296:
---
Attachment: HDFS-14296.1.patch
> Prefer ArrayList over LinkedList in VolumeScanner
> -
>
> Key: HDFS-14296
> URL: https://issues.apache.org/jira/browse/HDFS-14296
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14296.1.patch
>
>
> {quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
> {{listIterator}} operations run in constant time. - ArrayList
> {quote}
> However, for a {{LinkedList}}, the entire list must be traversed to get to
> the desired index.
> Like
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
> Most of the time, the List is being iterated, which is quicker over a
> primitive array than walking a LinkedList.
> There is one place where an item is removed, potentially from the middle of
> the list
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
> but the speed of removing from the middle of the list isn't bad; it's a
> system native array shift, and it only happens on the off chance that a block
> pool is removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771385#comment-16771385
]
BELUGA BEHR edited comment on HDFS-14295 at 2/18/19 9:25 PM:
-
Also note that this means that some of these threads are being counted in the
MXBean, so you could see cases where the MXBean reported a number higher than
{{dfs.datanode.max.transfer.thread}} which is a bit confusing.
[Check it out
here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2197-L2201]
was (Author: belugabehr):
Also note that this means that some of these threads are being counted in the
MXBean, so you could see cases where the MXBean reported a number higher than
{{dfs.datanode.max.transfer.thread}} which is a bit confusing.
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2197-L2201]
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771385#comment-16771385
]
BELUGA BEHR commented on HDFS-14295:
Also note that this means that some of these threads are being counted in the
MXBean, so you could see cases where the MXBean reported a number higher than
{{dfs.datanode.max.transfer.thread}} which is a bit confusing.
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2197-L2201]
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771376#comment-16771376
]
BELUGA BEHR commented on HDFS-14295:
One thing I'll point out that's a bit off, which I address in this patch, ...
There are two places in the code where a {{DataTransfer}} thread is started. In
[one
place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
it's started in a default thread group. In [another
place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
it's started in the
[dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
thread group.
I do not think it's correct to include any of these threads in the
{{dataXceiverServer}} thread group. Anything submitted to the
{{dataXceiverServer}} should probably be tied to the
{{dfs.datanode.max.transfer.threads}} configurations, and neither of these
methods are. Instead, they should be submitted into the same thread pool with
its own thread group (probably the default thread group, unless someone
suggests otherwise) and is what I have included in this patch.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14294) Deprecate DFSUtilClient#getSmallBufferSize
[
https://issues.apache.org/jira/browse/HDFS-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771378#comment-16771378
]
BELUGA BEHR commented on HDFS-14294:
{{javac}} is complaining about the use of deprecated methods... which makes
sense since this patch proposes to deprecate a method that previously was not.
> Deprecate DFSUtilClient#getSmallBufferSize
> --
>
> Key: HDFS-14294
> URL: https://issues.apache.org/jira/browse/HDFS-14294
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14294.1.patch
>
>
> {code:java|title=DFSUtilClient.java}
> public static int getIoFileBufferSize(Configuration conf) {
> return conf.getInt(
> CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_KEY,
> CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_DEFAULT);
> }
> public static int getSmallBufferSize(Configuration conf) {
> return Math.min(getIoFileBufferSize(conf) / 2, 512);
> }
> {code}
> This concept of a "small buffer size" seems a bit overkill. First of all,
> it's not documented that such a thing exists and that by adjusting
> {{dfs.stream-buffer-size}} an administrator is also scaling these other
> buffer sizes. Seconds, I think any "small" buffer size should just use the
> default JDK buffer sizes. Anything that benefits from being larger than the
> default JDK size should be the controlled by {{IO_FILE_BUFFER_SIZE_KEY}} /
> {{dfs.stream-buffer-size}}. For reference, the default JDK size is 8K :
> [HDFS-14293]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14295.3.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Patch Available (was: Open)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14219) ConcurrentModificationException occurs in datanode occasionally
[ https://issues.apache.org/jira/browse/HDFS-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771367#comment-16771367 ] Arpit Agarwal commented on HDFS-14219: -- +1 thanks for finding and fixing this [~Tao Jie]. Could you rename the patch to HDFS-14129-branch-2.8.001.patch so Yetus can apply it to the right branch? > ConcurrentModificationException occurs in datanode occasionally > --- > > Key: HDFS-14219 > URL: https://issues.apache.org/jira/browse/HDFS-14219 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.2 >Reporter: Tao Jie >Assignee: Tao Jie >Priority: Minor > Attachments: HDFS-14219.001.patch > > > ERROR occasionally occurs in datanode log: > ERROR BPServiceActor.java:792 - Exception in BPOfferService for Block pool > BP-1687106048-10.14.9.17-1535259994856 (Datanode Uuid > c17e635a-912f-4488-b4e8-93a58a27b5db) service to osscmh-9-21/10.14.9.21:8070 > java.util.ConcurrentModificationException: modification=62852685 != > iterModification = 62852684 > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.ensureNext(LightWeightGSet.java:305) > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.hasNext(LightWeightGSet.java:322) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1872) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:349) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:648) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:790) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14292.3.patch
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Patch Available (was: Open)
New patch; rebased.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Open (was: Patch Available)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Description:
I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
{{hdfs-site.xml}}. It is described as "Specifies the maximum number of threads
to use for transferring data in and out of the DN." The default value is
4096. I found it interesting because 4096 threads sounds like a lot to me.
I'm not sure how a system with 8-16 cores would react to this large a thread
count. Intuitively, I would say that the overhead of context switching would
be immense.
During mt investigation, I discovered the
[following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
setup in the {{DataXceiverServer}} class:
# A peer connects to a DataNode
# A new thread is spun up to service this connection
# The thread runs to completion
# The tread dies
It would perhaps be better if we used a thread pool to better manage the
lifecycle of the service threads and to allow the DataNode to re-use existing
threads, saving on the need to create and spin-up threads on demand.
In this JIRA, I have added a couple of things:
# Added a thread pool to {{DataXceiverServer}} class that, on demand, will
create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
completed its prior duties will stay idle for up to 60 seconds (configurable),
it will be retired if no new work has arrived.
# Added new methods to the {{Peer}} Interface to allow for better logging and
less code within each Thread ({{DataXceiver}}).
# Updated the Thread code ({{DataXceiver}}) regarding its interactions with
{{blockReceiver}} instance variable
was:
I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
{{hdfs-site.xml}}. It is described as "Specifies the maximum number of threads
to use for transferring data in and out of the DN." The default value is
4096. I found it interesting because 4096 threads sounds like a lot to me.
I'm not sure how a system with 8-16 cores would react to this large a thread
count. Intuitively, I would say that the overhead of context switching would
be immense.
During mt investigation, I discovered the
[following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
setup in the {{DataXceiverServer}} class:
# A peer connects to a DataNode
# A new thread is spun up to service this connection
# The thread runs to completion
# The tread dies
It would perhaps be better if we used a thread pool to better manage the
lifecycle of the service threads and to allow the DataNode to re-use existing
threads, saving on the need to create and spin-up threads on demand.
In this JIRA, I have added a couple of things:
# Added a thread pool that will always maintain a single thread running, always
awaiting a new connection should one arrive. On-demand, it will create up to
{{dfs.datanode.max.transfer.threads}}. A thread that has completed its prior
duties will stay idle for up to 30 seconds, it will be retired if no new work
has arrived.
# Added new methods to the {{Peer}} Interface to allow for better logging and
less code within each Thread ({{DataXceiver}}).
# Updated the Thread code ({{DataXceiver}}) regarding its interactions with
{{blockReceiver}} instance variable
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It
[jira] [Commented] (HDFS-14188) Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter
[
https://issues.apache.org/jira/browse/HDFS-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771361#comment-16771361
]
Wei-Chiu Chuang commented on HDFS-14188:
+1
> Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a
> parameter
> ---
>
> Key: HDFS-14188
> URL: https://issues.apache.org/jira/browse/HDFS-14188
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.1.1
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Attachments: HDFS-14188.001.patch, HDFS-14188.002.patch,
> HDFS-14188.003.patch, HDFS-14188.004.patch, HDFS-14188.005.patch
>
>
> hdfs ec -verifyClusterSetup command verifies if there are enough data nodes
> and racks for the enabled erasure coding policies
> I think it would be beneficial if it could accept an erasure coding policy as
> a parameter optionally. For example the following command would run the
> verify for only the RS-6-3-1024k policy.
> {code:java}
> hdfs ec -verifyClusterSetup -policy RS-6-3-1024k
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771355#comment-16771355
]
Hadoop QA commented on HDFS-14295:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 49s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 154 unchanged - 0 fixed = 156 total (was 154) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
17s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 24s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestHDFSServerPorts |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14295 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959152/HDFS-14295.2.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c3f4664ae5ca 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 920a896 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HDFS-14118) Use DNS to resolve Namenodes and Routers
[ https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771349#comment-16771349 ] Íñigo Goiri commented on HDFS-14118: Some people in the watchers list has been working on the clients for observer NNs and similar. Anybody up for review? Otherwise, next week I'll give a final look and go with it. > Use DNS to resolve Namenodes and Routers > > > Key: HDFS-14118 > URL: https://issues.apache.org/jira/browse/HDFS-14118 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: DNS testing log, HDFS-14118.001.patch, > HDFS-14118.002.patch, HDFS-14118.003.patch, HDFS-14118.004.patch, > HDFS-14118.005.patch, HDFS-14118.006.patch, HDFS-14118.007.patch, > HDFS-14118.008.patch, HDFS-14118.009.patch, HDFS-14118.010.patch, > HDFS-14118.011.patch, HDFS-14118.012.patch, HDFS-14118.013.patch, > HDFS-14118.014.patch, HDFS-14118.015.patch, HDFS-14118.016.patch, > HDFS-14118.017.patch, HDFS-14118.018.patch, HDFS-14118.019.patch, > HDFS-14118.patch > > > Clients will need to know about routers to talk to the HDFS cluster > (obviously), and having routers updating (adding/removing) will have to make > every client change, which is a painful process. > DNS can be used here to resolve the single domain name clients knows to a > list of routers in the current config. However, DNS won't be able to consider > only resolving to the working router based on certain health thresholds. > There are some ways about how this can be solved. One way is to have a > separate script to regularly check the status of the router and update the > DNS records if a router fails the health thresholds. In this way, security > might be carefully considered for this way. Another way is to have the client > do the normal connecting/failover after they get the list of routers, which > requires the change of current failover proxy provider. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9596) Remove "Shuffle" Method From DFSUtil
[
https://issues.apache.org/jira/browse/HDFS-9596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771333#comment-16771333
]
Hadoop QA commented on HDFS-9596:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 100 unchanged - 1 fixed = 100 total (was 101) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 4s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-9596 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959150/HDFS-9596.7.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ab0126f717da 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 920a896 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26247/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26247/testReport/ |
| Max. process+thread count | 4249 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-14287) DataXceiverServer May Double-Close PeerServer
[
https://issues.apache.org/jira/browse/HDFS-14287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771331#comment-16771331
]
Hadoop QA commented on HDFS-14287:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14287 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959151/HDFS-14287.1.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c2d1847fe161 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 920a896 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26246/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26246/testReport/ |
| Max. process+thread count | 4057 (vs. ulimit of 1)
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Open (was: Patch Available)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool that will always maintain a single thread running,
> always awaiting a new connection should one arrive. On-demand, it will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 30 seconds, it will be
> retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14287) DataXceiverServer May Double-Close PeerServer
[
https://issues.apache.org/jira/browse/HDFS-14287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14287:
---
Attachment: (was: HDFS-14287.1.patch)
> DataXceiverServer May Double-Close PeerServer
> -
>
> Key: HDFS-14287
> URL: https://issues.apache.org/jira/browse/HDFS-14287
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14287.1.patch
>
>
> Based on my work with [HDFS-14258], I noticed that when a user calls the
> {{kill()}} method in {{DataXceiverServer}}, it closes the {{PeerServer}}.
> This causes the thread responsible for receiving connections from the
> {{PeerServer}} to interrupt and then exit. On the way out, this thread also
> [closes|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L246-L253]
> the {{PeerServer}} for good measure. I'm not sure what the affects of
> closing the {{PeerServer}} are twice, but it should be avoided.
> To be clear, this is not a regressions caused by [HDFS-14258], if anything,
> [HDFS-14258] makes it easier to fix. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771323#comment-16771323
]
Hadoop QA commented on HDFS-14292:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-14292 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14292 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959159/HDFS-14292.2.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26250/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool that will always maintain a single thread running,
> always awaiting a new connection should one arrive. On-demand, it will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 30 seconds, it will be
> retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1098) Introduce Retry Policy in Ozone Client
[ https://issues.apache.org/jira/browse/HDDS-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771318#comment-16771318 ] Shashikant Banerjee commented on HDDS-1098: --- Patch v0 is dependent on HDDS-726. > Introduce Retry Policy in Ozone Client > -- > > Key: HDDS-1098 > URL: https://issues.apache.org/jira/browse/HDDS-1098 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1098.000.patch > > > Currently , ozone client indefinitely retries to failover to a new block > write in case it encounters exceptions during write. The aim here is to bring > in retry policy in the client so that indefinite retries should go away. > Ozone client code will also be refactored as a part of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Patch Available (was: Open)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool that will always maintain a single thread running,
> always awaiting a new connection should one arrive. On-demand, it will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 30 seconds, it will be
> retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14292.2.patch
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool that will always maintain a single thread running,
> always awaiting a new connection should one arrive. On-demand, it will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 30 seconds, it will be
> retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: (was: HDFS-14292.2.patc)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool that will always maintain a single thread running,
> always awaiting a new connection should one arrive. On-demand, it will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 30 seconds, it will be
> retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14292.2.patc
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool that will always maintain a single thread running,
> always awaiting a new connection should one arrive. On-demand, it will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 30 seconds, it will be
> retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1098) Introduce Retry Policy in Ozone Client
[ https://issues.apache.org/jira/browse/HDDS-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-1098: -- Attachment: HDDS-1098.000.patch > Introduce Retry Policy in Ozone Client > -- > > Key: HDDS-1098 > URL: https://issues.apache.org/jira/browse/HDDS-1098 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1098.000.patch > > > Currently , ozone client indefinitely retries to failover to a new block > write in case it encounters exceptions during write. The aim here is to bring > in retry policy in the client so that indefinite retries should go away. > Ozone client code will also be refactored as a part of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14287) DataXceiverServer May Double-Close PeerServer
[
https://issues.apache.org/jira/browse/HDFS-14287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771309#comment-16771309
]
Hudson commented on HDFS-14287:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15989 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15989/])
HDFS-14287. DataXceiverServer May Double-Close PeerServer. Contributed
(inigoiri: rev 235e3da90a4212d0c204afaef09db3408abfab82)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
> DataXceiverServer May Double-Close PeerServer
> -
>
> Key: HDFS-14287
> URL: https://issues.apache.org/jira/browse/HDFS-14287
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14287.1.patch
>
>
> Based on my work with [HDFS-14258], I noticed that when a user calls the
> {{kill()}} method in {{DataXceiverServer}}, it closes the {{PeerServer}}.
> This causes the thread responsible for receiving connections from the
> {{PeerServer}} to interrupt and then exit. On the way out, this thread also
> [closes|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L246-L253]
> the {{PeerServer}} for good measure. I'm not sure what the affects of
> closing the {{PeerServer}} are twice, but it should be avoided.
> To be clear, this is not a regressions caused by [HDFS-14258], if anything,
> [HDFS-14258] makes it easier to fix. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14287) DataXceiverServer May Double-Close PeerServer
[
https://issues.apache.org/jira/browse/HDFS-14287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14287:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
> DataXceiverServer May Double-Close PeerServer
> -
>
> Key: HDFS-14287
> URL: https://issues.apache.org/jira/browse/HDFS-14287
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14287.1.patch
>
>
> Based on my work with [HDFS-14258], I noticed that when a user calls the
> {{kill()}} method in {{DataXceiverServer}}, it closes the {{PeerServer}}.
> This causes the thread responsible for receiving connections from the
> {{PeerServer}} to interrupt and then exit. On the way out, this thread also
> [closes|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L246-L253]
> the {{PeerServer}} for good measure. I'm not sure what the affects of
> closing the {{PeerServer}} are twice, but it should be avoided.
> To be clear, this is not a regressions caused by [HDFS-14258], if anything,
> [HDFS-14258] makes it easier to fix. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14287) DataXceiverServer May Double-Close PeerServer
[
https://issues.apache.org/jira/browse/HDFS-14287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771307#comment-16771307
]
Íñigo Goiri commented on HDFS-14287:
Thanks [~belugabehr] for the patch.
Committed to trunk.
> DataXceiverServer May Double-Close PeerServer
> -
>
> Key: HDFS-14287
> URL: https://issues.apache.org/jira/browse/HDFS-14287
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14287.1.patch
>
>
> Based on my work with [HDFS-14258], I noticed that when a user calls the
> {{kill()}} method in {{DataXceiverServer}}, it closes the {{PeerServer}}.
> This causes the thread responsible for receiving connections from the
> {{PeerServer}} to interrupt and then exit. On the way out, this thread also
> [closes|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L246-L253]
> the {{PeerServer}} for good measure. I'm not sure what the affects of
> closing the {{PeerServer}} are twice, but it should be avoided.
> To be clear, this is not a regressions caused by [HDFS-14258], if anything,
> [HDFS-14258] makes it easier to fix. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14291) EC: GetServerDefaults To Support Default Erasure Coding Policy
[ https://issues.apache.org/jira/browse/HDFS-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771308#comment-16771308 ] Wei-Chiu Chuang commented on HDFS-14291: Hi [~ayushtkn] thanks for the patch. The patch itself looks good to me. Out of curiosity, what would be the use case for the proposed change? > EC: GetServerDefaults To Support Default Erasure Coding Policy > -- > > Key: HDFS-14291 > URL: https://issues.apache.org/jira/browse/HDFS-14291 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14291-01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14188) Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter
[
https://issues.apache.org/jira/browse/HDFS-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771306#comment-16771306
]
Hadoop QA commented on HDFS-14188:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14188 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959142/HDFS-14188.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ba49d0e1f81e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f2fb653 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26245/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26245/testReport/ |
| Max. process+thread count | 5107 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Updated] (HDDS-1129) Fix findbug/checkstyle errors hdds projects
[ https://issues.apache.org/jira/browse/HDDS-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-1129: --- Status: Patch Available (was: Open) > Fix findbug/checkstyle errors hdds projects > --- > > Key: HDDS-1129 > URL: https://issues.apache.org/jira/browse/HDDS-1129 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HDDS-1114 fixed all the findbug/checkstyle problems but in the mean time new > patches are committed with newer error. > Here I would like to cleanup the projects again. > (Except the static field in RatisPipelineProvider which will be ignored in > this patch and tracked in HDDS-1128) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14287) DataXceiverServer May Double-Close PeerServer
[
https://issues.apache.org/jira/browse/HDFS-14287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771303#comment-16771303
]
Íñigo Goiri commented on HDFS-14287:
Interesting that we actually had a "closed" variable but wasn't properly used.
For clarity, I'm removing the multiple attachments of the same patch.
TestJournalNodeSync is failing but it seems unrelated.
+1
> DataXceiverServer May Double-Close PeerServer
> -
>
> Key: HDFS-14287
> URL: https://issues.apache.org/jira/browse/HDFS-14287
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14287.1.patch
>
>
> Based on my work with [HDFS-14258], I noticed that when a user calls the
> {{kill()}} method in {{DataXceiverServer}}, it closes the {{PeerServer}}.
> This causes the thread responsible for receiving connections from the
> {{PeerServer}} to interrupt and then exit. On the way out, this thread also
> [closes|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L246-L253]
> the {{PeerServer}} for good measure. I'm not sure what the affects of
> closing the {{PeerServer}} are twice, but it should be avoided.
> To be clear, this is not a regressions caused by [HDFS-14258], if anything,
> [HDFS-14258] makes it easier to fix. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14287) DataXceiverServer May Double-Close PeerServer
[
https://issues.apache.org/jira/browse/HDFS-14287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14287:
---
Attachment: (was: HDFS-14287.1.patch)
> DataXceiverServer May Double-Close PeerServer
> -
>
> Key: HDFS-14287
> URL: https://issues.apache.org/jira/browse/HDFS-14287
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14287.1.patch
>
>
> Based on my work with [HDFS-14258], I noticed that when a user calls the
> {{kill()}} method in {{DataXceiverServer}}, it closes the {{PeerServer}}.
> This causes the thread responsible for receiving connections from the
> {{PeerServer}} to interrupt and then exit. On the way out, this thread also
> [closes|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L246-L253]
> the {{PeerServer}} for good measure. I'm not sure what the affects of
> closing the {{PeerServer}} are twice, but it should be avoided.
> To be clear, this is not a regressions caused by [HDFS-14258], if anything,
> [HDFS-14258] makes it easier to fix. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14249) RBF: Tooling to identify the subcluster location of a file
[ https://issues.apache.org/jira/browse/HDFS-14249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771300#comment-16771300 ] Íñigo Goiri commented on HDFS-14249: Thanks [~ayushtkn] for the review. I think I tackled all your comments. The highlight is that I made it support directories and added tests using the RPC server. > RBF: Tooling to identify the subcluster location of a file > -- > > Key: HDFS-14249 > URL: https://issues.apache.org/jira/browse/HDFS-14249 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14249-HDFS-13891.000.patch, > HDFS-14249-HDFS-13891.001.patch, HDFS-14249-HDFS-13891.002.patch > > > Mount points can spread files across multiple subclusters depennding on a > policy (e.g., HASH, HASH_ALL). Administrators would need a way to identify > the location. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14249) RBF: Tooling to identify the subcluster location of a file
[ https://issues.apache.org/jira/browse/HDFS-14249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14249: --- Attachment: HDFS-14249-HDFS-13891.002.patch > RBF: Tooling to identify the subcluster location of a file > -- > > Key: HDFS-14249 > URL: https://issues.apache.org/jira/browse/HDFS-14249 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14249-HDFS-13891.000.patch, > HDFS-14249-HDFS-13891.001.patch, HDFS-14249-HDFS-13891.002.patch > > > Mount points can spread files across multiple subclusters depennding on a > policy (e.g., HASH, HASH_ALL). Administrators would need a way to identify > the location. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-726) Ozone Client should update SCM to move the container out of allocation path in case a write transaction fails
[ https://issues.apache.org/jira/browse/HDDS-726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-726: - Attachment: HDDS-726.005.patch > Ozone Client should update SCM to move the container out of allocation path > in case a write transaction fails > - > > Key: HDDS-726 > URL: https://issues.apache.org/jira/browse/HDDS-726 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-726.000.patch, HDDS-726.001.patch, > HDDS-726.002.patch, HDDS-726.003.patch, HDDS-726.004.patch, HDDS-726.005.patch > > > Once an container write transaction fails, it will be marked corrupted. Once > Ozone client gets an exception in such case it should tell SCM to move the > container out of allocation path. SCM will eventually close the container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-726) Ozone Client should update SCM to move the container out of allocation path in case a write transaction fails
[ https://issues.apache.org/jira/browse/HDDS-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771290#comment-16771290 ] Shashikant Banerjee commented on HDDS-726: -- Thanks [~msingh] for the review comments. Patch v5 addresses your review comments. > Ozone Client should update SCM to move the container out of allocation path > in case a write transaction fails > - > > Key: HDDS-726 > URL: https://issues.apache.org/jira/browse/HDDS-726 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-726.000.patch, HDDS-726.001.patch, > HDDS-726.002.patch, HDDS-726.003.patch, HDDS-726.004.patch, HDDS-726.005.patch > > > Once an container write transaction fails, it will be marked corrupted. Once > Ozone client gets an exception in such case it should tell SCM to move the > container out of allocation path. SCM will eventually close the container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1127) Fix failing and intermittent Ozone unit tests
[ https://issues.apache.org/jira/browse/HDDS-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771284#comment-16771284 ] Anu Engineer commented on HDDS-1127: +1, Please do. Let us also spend some time on these tests, some times they are flaky by design. So it might be a good idea to rewrite them or modify code to be more testable. > Fix failing and intermittent Ozone unit tests > - > > Key: HDDS-1127 > URL: https://issues.apache.org/jira/browse/HDDS-1127 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Blocker > > Full Ozone build with acceptance + unit tests takes ~1.5 hour. > In the last 30 hours I executed a new full build at every 2 hours and > collected all the results. > We have ~1200 test method and ~15 are failed more than 4 times out of the 17 > run. > I propose the following method to fix them: > # Turn them off immediately (@Skip) to get real data for the pre-commits > # Create a Jira for every failing tests *with assignee* (I would choose an > assignee based on the history of the unit test). We can adjust the assignee > later but I would prefer use a default person instead of creating unassigned > jira-s. > # Fix them and enable the tests again. > Failing tests are the following: > |Package|Class|Test|84|83|82|81|80|79|78|77|76|75|74|73|72|71|70|69|68|67|66|65|FAILED| > |org.apache.hadoop.hdds.scm.chillmode|TestSCMChillModeManager|testDisableChillMode|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|N/A|17| > |org.apache.hadoop.hdds.scm.node|TestDeadNodeHandler|testOnMessage|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|N/A|17| > |org.apache.hadoop.hdds.scm.node|TestDeadNodeHandler|testOnMessageReplicaFailure|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|N/A|17| > |org.apache.hadoop.ozone|TestSecureOzoneCluster|testDelegationToken|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|N/A|17| > |org.apache.hadoop.ozone|TestSecureOzoneCluster|testDelegationTokenRenewal|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|N/A|17| > |org.apache.hadoop.ozone.container.ozoneimpl|TestOzoneContainerWithTLS|testCreateOzoneContainer[0]|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|N/A|17| > |org.apache.hadoop.ozone.container.ozoneimpl|TestOzoneContainerWithTLS|testCreateOzoneContainer[1]|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|N/A|17| > |[org.apache.hadoop.ozone.om|http://org.apache.hadoop.ozone.om/]|TestOzoneManager|testAccessVolume|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|N/A|17| > |[org.apache.hadoop.ozone.om|http://org.apache.hadoop.ozone.om/]|TestOzoneManager|testOmInitializationFailure|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|N/A|17| > |[org.apache.hadoop.ozone.om|http://org.apache.hadoop.ozone.om/]|TestOzoneManager|testRenameKey|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|N/A|17| > |[org.apache.hadoop.ozone.om|http://org.apache.hadoop.ozone.om/]|TestOzoneManagerHA|testTwoOMNodesDown|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|N/A|17| > |org.apache.hadoop.ozone.om.ratis|TestOzoneManagerRatisServer|testSubmitRatisRequest|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|FAILED|FAILED|FAILED|FAILED|FAILED|FAILED|N/A|N/A|17| > |org.apache.hadoop.ozone.freon|TestFreonWithDatanodeFastRestart|testRestart|FAILED|FAILED|FAILED|PASSED|FAILED|PASSED|FAILED|PASSED|FAILED|FAILED|FAILED|N/A|FAILED|PASSED|PASSED|FAILED|FAILED|PASSED|N/A|N/A|11| > |org.apache.hadoop.hdds.scm.container|TestContainerStateManagerIntegration|testGetMatchingContainerMultipleThreads|PASSED|PASSED|PASSED|PASSED|FAILED|PASSED|PASSED|PASSED|PASSED|PASSED|PASSED|N/A|PASSED|FAILED|PASSED|PASSED|FAILED|FAILED|N/A|N/A|4| >
[jira] [Commented] (HDDS-1122) Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure
[
https://issues.apache.org/jira/browse/HDDS-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771274#comment-16771274
]
Anu Engineer commented on HDDS-1122:
Thank you for the fix. +1.
> Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure
>
>
> Key: HDDS-1122
> URL: https://issues.apache.org/jira/browse/HDDS-1122
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Aravindan Vijayan
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDDS-1122.001.patch
>
>
> {code}
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerRatisServer.testSubmitRatisRequest
> Failing for the past 7 builds (Since Failed#2281 )
> Took 1.8 sec.
> Error Message
> Message missing required fields: status
> Stacktrace
> com.google.protobuf.UninitializedMessageException: Message missing required
> fields: status
> at
> com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:770)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMResponse$Builder.build(OzoneManagerProtocolProtos.java:11664)
> at
> org.apache.hadoop.ozone.om.ratis.OMRatisHelper.getErrorResponse(OMRatisHelper.java:113)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommand(OzoneManagerRatisClient.java:128)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerRatisServer.testSubmitRatisRequest(TestOzoneManagerRatisServer.java:125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1122) Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure
[
https://issues.apache.org/jira/browse/HDDS-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDDS-1122:
---
Target Version/s: 0.4.0
> Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure
>
>
> Key: HDDS-1122
> URL: https://issues.apache.org/jira/browse/HDDS-1122
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Aravindan Vijayan
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDDS-1122.001.patch
>
>
> {code}
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerRatisServer.testSubmitRatisRequest
> Failing for the past 7 builds (Since Failed#2281 )
> Took 1.8 sec.
> Error Message
> Message missing required fields: status
> Stacktrace
> com.google.protobuf.UninitializedMessageException: Message missing required
> fields: status
> at
> com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:770)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMResponse$Builder.build(OzoneManagerProtocolProtos.java:11664)
> at
> org.apache.hadoop.ozone.om.ratis.OMRatisHelper.getErrorResponse(OMRatisHelper.java:113)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommand(OzoneManagerRatisClient.java:128)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerRatisServer.testSubmitRatisRequest(TestOzoneManagerRatisServer.java:125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1088) Add blockade Tests to test Replica Manager
[ https://issues.apache.org/jira/browse/HDDS-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-1088: --- Target Version/s: 0.4.0 > Add blockade Tests to test Replica Manager > -- > > Key: HDDS-1088 > URL: https://issues.apache.org/jira/browse/HDDS-1088 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Nilotpal Nandi >Assignee: Nilotpal Nandi >Priority: Major > Attachments: HDDS-1088.001.patch > > > We need to add tests for testing Replica Manager for scenarios like loss of > node, adding new nodes, under-replicated containers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771267#comment-16771267
]
Hadoop QA commented on HDFS-14295:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 46s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 154 unchanged - 0 fixed = 156 total (was 154) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14295 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959136/HDFS-14295.1.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b2c21499f7b5 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f2fb653 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26244/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771268#comment-16771268
]
BELUGA BEHR commented on HDFS-14292:
Yup, some of the unit tests are checking to make sure that the threads are not
being re-used. Ugh.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool that will always maintain a single thread running,
> always awaiting a new connection should one arrive. On-demand, it will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 30 seconds, it will be
> retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1085) Create an OM API to serve snapshots to Recon server
[
https://issues.apache.org/jira/browse/HDDS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771263#comment-16771263
]
Elek, Marton commented on HDDS-1085:
{quote}bq.[~elek] For this HTTP Servlet, authentication will be provided by
Spnego automatically. For authorization, we plan to add a validation step in
the servlet to make sure sure only 'admin' users can download the checkpoint.
These will probably be the other OM instances and the FCSK server. I can create
a followup JIRA for that. Does that sound OK?
{quote}
Sure, for me everything is fine (if it works), I was interested only about the
reasoning behind REST. I don't know how Java HTTP client works with Spnego but
there should be a solution. But this conversation can be moved to HDDS-1084, I
am fine with commit this patch and improve it gradually.
> Create an OM API to serve snapshots to Recon server
> ---
>
> Key: HDDS-1085
> URL: https://issues.apache.org/jira/browse/HDDS-1085
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Siddharth Wagle
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-1085-000.patch, HDDS-1085-001.patch,
> HDDS-1085-002.patch, HDDS-1085-003.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We need to add an API to OM so that we can serve snapshots from the OM server.
> - The snapshot should be streamed to fsck server with the ability to
> throttle network utilization (like TransferFsImage)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Patch Available (was: Open)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14295.2.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1129) Fix findbug/checkstyle errors in hadoop-hdds projects
[ https://issues.apache.org/jira/browse/HDDS-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-1129: --- Summary: Fix findbug/checkstyle errors in hadoop-hdds projects (was: Fix findbug/checkstyle errors hdds projects) > Fix findbug/checkstyle errors in hadoop-hdds projects > - > > Key: HDDS-1129 > URL: https://issues.apache.org/jira/browse/HDDS-1129 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HDDS-1114 fixed all the findbug/checkstyle problems but in the mean time new > patches are committed with newer error. > Here I would like to cleanup the projects again. > (Except the static field in RatisPipelineProvider which will be ignored in > this patch and tracked in HDDS-1128) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
