[jira] [Updated] (HDFS-9139) Enable parallel JUnit tests for HDFS Pre-commit
[ https://issues.apache.org/jira/browse/HDFS-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-9139: Attachment: HDFS-9139.03.patch Corrected the test-patch.sh to enable parallel tests > Enable parallel JUnit tests for HDFS Pre-commit > > > Key: HDFS-9139 > URL: https://issues.apache.org/jira/browse/HDFS-9139 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-9139.01.patch, HDFS-9139.02.patch, > HDFS-9139.03.patch > > > Forked from HADOOP-11984, > With the initial and significant work from [~cnauroth], this Jira is to track > and support parallel tests' run for HDFS Precommit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8740) Move DistributedFileSystem to hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8740:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
I've committed the patch to trunk and branch-2. Thanks [~liuml07] for the
contribution.
> Move DistributedFileSystem to hadoop-hdfs-client
>
>
> Key: HDFS-8740
> URL: https://issues.apache.org/jira/browse/HDFS-8740
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Yi Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8740.000.patch
>
>
> This jira tracks efforts of moving
> {{org.apache.hadoop.hdfs.DistributedFileSystem}} class from {{hadoop-hdfs}}
> to {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8740) Move DistributedFileSystem to hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909822#comment-14909822
]
Hudson commented on HDFS-8740:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8528 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8528/])
HDFS-8740. Move DistributedFileSystem to hadoop-hdfs-client. Contributed by
Mingliang Liu. (wheat9: rev 1c030c6e58dc83152f933323bb7743ad47f5af27)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
> Move DistributedFileSystem to hadoop-hdfs-client
>
>
> Key: HDFS-8740
> URL: https://issues.apache.org/jira/browse/HDFS-8740
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Yi Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8740.000.patch
>
>
> This jira tracks efforts of moving
> {{org.apache.hadoop.hdfs.DistributedFileSystem}} class from {{hadoop-hdfs}}
> to {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9139) Enable parallel JUnit tests for HDFS Pre-commit
[
https://issues.apache.org/jira/browse/HDFS-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909721#comment-14909721
]
Hadoop QA commented on HDFS-9139:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | reexec | 0m 0s | dev-support patch detected. |
| {color:red}-1{color} | pre-patch | 17m 50s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:blue}0{color} | @author | 0m 0s | Skipping @author checks as
test-patch has been patched. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 14 new or modified test files. |
| {color:green}+1{color} | javac | 7m 53s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 4s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 7s | There were no new checkstyle
issues. |
| {color:red}-1{color} | shellcheck | 0m 8s | The applied patch generated 1
new shellcheck (v0.3.3) issues (total was 32, now 33). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 2s | Tests passed in hadoop-hdfs.
|
| | | 207m 18s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12762587/HDFS-9139.02.patch |
| Optional Tests | shellcheck javadoc javac unit findbugs checkstyle |
| git revision | trunk / f0f984e |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12701/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| shellcheck |
https://builds.apache.org/job/PreCommit-HDFS-Build/12701/artifact/patchprocess/diffpatchshellcheck.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12701/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12701/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12701/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12701/console |
This message was automatically generated.
> Enable parallel JUnit tests for HDFS Pre-commit
>
>
> Key: HDFS-9139
> URL: https://issues.apache.org/jira/browse/HDFS-9139
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-9139.01.patch, HDFS-9139.02.patch
>
>
> Forked from HADOOP-11984,
> With the initial and significant work from [~cnauroth], this Jira is to track
> and support parallel tests' run for HDFS Precommit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909735#comment-14909735 ] 郭同 commented on HDFS-2576: -- http://stackoverflow.com/questions/32779439/how-to-let-the-hdfss-replica-blocks-position-be-set-by-myself/32808390#32808390 > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8740) Move DistributedFileSystem to hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909839#comment-14909839
]
Hudson commented on HDFS-8740:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #449 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/449/])
HDFS-8740. Move DistributedFileSystem to hadoop-hdfs-client. Contributed by
Mingliang Liu. (wheat9: rev 1c030c6e58dc83152f933323bb7743ad47f5af27)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Move DistributedFileSystem to hadoop-hdfs-client
>
>
> Key: HDFS-8740
> URL: https://issues.apache.org/jira/browse/HDFS-8740
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Yi Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8740.000.patch
>
>
> This jira tracks efforts of moving
> {{org.apache.hadoop.hdfs.DistributedFileSystem}} class from {{hadoop-hdfs}}
> to {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8740) Move DistributedFileSystem to hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909818#comment-14909818
]
Haohui Mai commented on HDFS-8740:
--
+1
> Move DistributedFileSystem to hadoop-hdfs-client
>
>
> Key: HDFS-8740
> URL: https://issues.apache.org/jira/browse/HDFS-8740
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Yi Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8740.000.patch
>
>
> This jira tracks efforts of moving
> {{org.apache.hadoop.hdfs.DistributedFileSystem}} class from {{hadoop-hdfs}}
> to {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9139) Enable parallel JUnit tests for HDFS Pre-commit
[ https://issues.apache.org/jira/browse/HDFS-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909878#comment-14909878 ] Hadoop QA commented on HDFS-9139: - (!) A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HDFS-Build/12703/console in case of problems. > Enable parallel JUnit tests for HDFS Pre-commit > > > Key: HDFS-9139 > URL: https://issues.apache.org/jira/browse/HDFS-9139 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-9139.01.patch, HDFS-9139.02.patch, > HDFS-9139.03.patch > > > Forked from HADOOP-11984, > With the initial and significant work from [~cnauroth], this Jira is to track > and support parallel tests' run for HDFS Precommit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8740) Move DistributedFileSystem to hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909898#comment-14909898
]
Hudson commented on HDFS-8740:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2366 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2366/])
HDFS-8740. Move DistributedFileSystem to hadoop-hdfs-client. Contributed by
Mingliang Liu. (wheat9: rev 1c030c6e58dc83152f933323bb7743ad47f5af27)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
> Move DistributedFileSystem to hadoop-hdfs-client
>
>
> Key: HDFS-8740
> URL: https://issues.apache.org/jira/browse/HDFS-8740
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Yi Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8740.000.patch
>
>
> This jira tracks efforts of moving
> {{org.apache.hadoop.hdfs.DistributedFileSystem}} class from {{hadoop-hdfs}}
> to {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8740) Move DistributedFileSystem to hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909896#comment-14909896
]
Hudson commented on HDFS-8740:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2393 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2393/])
HDFS-8740. Move DistributedFileSystem to hadoop-hdfs-client. Contributed by
Mingliang Liu. (wheat9: rev 1c030c6e58dc83152f933323bb7743ad47f5af27)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
> Move DistributedFileSystem to hadoop-hdfs-client
>
>
> Key: HDFS-8740
> URL: https://issues.apache.org/jira/browse/HDFS-8740
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Yi Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8740.000.patch
>
>
> This jira tracks efforts of moving
> {{org.apache.hadoop.hdfs.DistributedFileSystem}} class from {{hadoop-hdfs}}
> to {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9139) Enable parallel JUnit tests for HDFS Pre-commit
[
https://issues.apache.org/jira/browse/HDFS-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909903#comment-14909903
]
Hadoop QA commented on HDFS-9139:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | reexec | 0m 0s | dev-support patch detected. |
| {color:red}-1{color} | pre-patch | 16m 25s | Findbugs (version ) appears to
be broken on trunk. |
| {color:blue}0{color} | @author | 0m 0s | Skipping @author checks as
test-patch has been patched. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 14 new or modified test files. |
| {color:green}+1{color} | javac | 9m 17s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 11m 13s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 35s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 35s | There were no new checkstyle
issues. |
| {color:green}+1{color} | shellcheck | 0m 10s | There were no new shellcheck
(v0.3.3) issues. |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 2m 4s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 49s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 44s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 50m 19s | Tests failed in hadoop-hdfs. |
| | | 97m 53s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.TestSafeMode |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12762600/HDFS-9139.03.patch |
| Optional Tests | shellcheck javadoc javac unit findbugs checkstyle |
| git revision | trunk / 1c030c6 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12703/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12703/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12703/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12703/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12703/console |
This message was automatically generated.
> Enable parallel JUnit tests for HDFS Pre-commit
>
>
> Key: HDFS-9139
> URL: https://issues.apache.org/jira/browse/HDFS-9139
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-9139.01.patch, HDFS-9139.02.patch,
> HDFS-9139.03.patch
>
>
> Forked from HADOOP-11984,
> With the initial and significant work from [~cnauroth], this Jira is to track
> and support parallel tests' run for HDFS Precommit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8740) Move DistributedFileSystem to hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909887#comment-14909887
]
Hudson commented on HDFS-8740:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #1188 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1188/])
HDFS-8740. Move DistributedFileSystem to hadoop-hdfs-client. Contributed by
Mingliang Liu. (wheat9: rev 1c030c6e58dc83152f933323bb7743ad47f5af27)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
> Move DistributedFileSystem to hadoop-hdfs-client
>
>
> Key: HDFS-8740
> URL: https://issues.apache.org/jira/browse/HDFS-8740
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Yi Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8740.000.patch
>
>
> This jira tracks efforts of moving
> {{org.apache.hadoop.hdfs.DistributedFileSystem}} class from {{hadoop-hdfs}}
> to {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8740) Move DistributedFileSystem to hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909900#comment-14909900
]
Hudson commented on HDFS-8740:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #426 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/426/])
HDFS-8740. Move DistributedFileSystem to hadoop-hdfs-client. Contributed by
Mingliang Liu. (wheat9: rev 1c030c6e58dc83152f933323bb7743ad47f5af27)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
> Move DistributedFileSystem to hadoop-hdfs-client
>
>
> Key: HDFS-8740
> URL: https://issues.apache.org/jira/browse/HDFS-8740
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Yi Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8740.000.patch
>
>
> This jira tracks efforts of moving
> {{org.apache.hadoop.hdfs.DistributedFileSystem}} class from {{hadoop-hdfs}}
> to {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8740) Move DistributedFileSystem to hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909881#comment-14909881
]
Hudson commented on HDFS-8740:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #455 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/455/])
HDFS-8740. Move DistributedFileSystem to hadoop-hdfs-client. Contributed by
Mingliang Liu. (wheat9: rev 1c030c6e58dc83152f933323bb7743ad47f5af27)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/CorruptFileBlockIterator.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
> Move DistributedFileSystem to hadoop-hdfs-client
>
>
> Key: HDFS-8740
> URL: https://issues.apache.org/jira/browse/HDFS-8740
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Yi Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8740.000.patch
>
>
> This jira tracks efforts of moving
> {{org.apache.hadoop.hdfs.DistributedFileSystem}} class from {{hadoop-hdfs}}
> to {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909739#comment-14909739 ] 郭同 commented on HDFS-2576: -- http://stackoverflow.com/questions/32779439/how-to-let-the-hdfss-replica-blocks-position-be-set-by-myself/32808390#32808390 > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9149) Consider multi datacenter when sortByDistance
[
https://issues.apache.org/jira/browse/HDFS-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909740#comment-14909740
]
He Tianyi commented on HDFS-9149:
-
The default {{NetworkTopology}} is not dc aware. Perhaps implement a custom one
with datacenter location awareness and override {{sortByDistance}}?
> Consider multi datacenter when sortByDistance
> -
>
> Key: HDFS-9149
> URL: https://issues.apache.org/jira/browse/HDFS-9149
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: He Xiaoqiao
>
> {{sortByDistance}} doesn't consider multi-datacenter when read data, so there
> my be reading data via other datacenter when hadoop deployment with multi-IDC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9139) Enable parallel JUnit tests for HDFS Pre-commit
[ https://issues.apache.org/jira/browse/HDFS-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909749#comment-14909749 ] Hadoop QA commented on HDFS-9139: - (!) A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HDFS-Build/12702/console in case of problems. > Enable parallel JUnit tests for HDFS Pre-commit > > > Key: HDFS-9139 > URL: https://issues.apache.org/jira/browse/HDFS-9139 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-9139.01.patch, HDFS-9139.02.patch, > HDFS-9139.03.patch > > > Forked from HADOOP-11984, > With the initial and significant work from [~cnauroth], this Jira is to track > and support parallel tests' run for HDFS Precommit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9150) Consider implementing lazy seek in S3AInputStream
Rajesh Balamohan created HDFS-9150: -- Summary: Consider implementing lazy seek in S3AInputStream Key: HDFS-9150 URL: https://issues.apache.org/jira/browse/HDFS-9150 Project: Hadoop HDFS Issue Type: Bug Components: HDFS Reporter: Rajesh Balamohan - Currently, "read(long position, byte[] buffer, int offset, int length)" is not implemented in S3AInputStream (unlike DFSInputStream). So, "readFully(long position, byte[] buffer, int offset, int length)" in S3AInputStream goes through the default implementation of seek(), read(), seek() in FSInputStream. - However, seek() in S3AInputStream involves re-opening of connection to S3 everytime (https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L115). - It would be good to consider having a lazy seek implementation to reduce connection overheads to S3. (e.g Presto implements lazy seek. https://github.com/facebook/presto/blob/master/presto-hive/src/main/java/com/facebook/presto/hive/PrestoS3FileSystem.java#L623) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9147) Fix the setting of visibleLength in ExternalBlockReader
[ https://issues.apache.org/jira/browse/HDFS-9147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909952#comment-14909952 ] Yi Liu commented on HDFS-9147: -- +1, the jenkins seems having issues, let me re-trigger. > Fix the setting of visibleLength in ExternalBlockReader > --- > > Key: HDFS-9147 > URL: https://issues.apache.org/jira/browse/HDFS-9147 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.8.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-9147.001.patch > > > BlockReaderFactory needs to take the start offset into consideration when > setting the visibleLength to use in ExternalBlockReader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9079) Erasure coding: preallocate multiple generation stamps and serialize updates from data streamers
[
https://issues.apache.org/jira/browse/HDFS-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909979#comment-14909979
]
Walter Su commented on HDFS-9079:
-
I see what you're doing. Some comments.
1. setupPipelineForAppendOrRecovery() will trim bad nodes. When
nodes.length==0, the failed streamer won't call updateBlockForPipeline().
That's one reason you need HDFS-9040.
{code}
LocatedBlock updateBlockForPipeline() throws IOException {
-if (LOG.isDebugEnabled()) {
- LOG.debug("updateBlockForPipeline(), " + this);
+if (!getErrorState().hasExternalErrorOnly()) {
+ // If the streamer itself has encountered an error, bump GS
+ coordinator.addEvent(new BlockMetadataCoordinator.DNFailureEvent(index));
+ Preconditions.checkState(coordinator.getProposedGenStamp()
+ >= block.getGenerationStamp());
}
{code}
2. The old way calls {{updatePipeline}} to refresh storedBlock once one
streamer failed. The internal block of the failed streamer has old GS so it
won't accepted by blockReport.
The new way delays {{updatePipeline}}. One failure doesn't call it, only
endBlock() will. Assume one failed block has gs=1001, one failed block has
gs=1002, the others have gs=1003. The storedBlock has gs=1001, right? Assume
client gets killed before endBlock(). Now every blocks can be accepted by
blockReport. It affects lease recovery's judgement. since lease recovery need 6
healthy blocks so it can start.
But I think it can be fixed easily. If a block with gs=1003 reported, the
blocks with gs=1001,1002 are bad obviously. So no problem.
3. updatePipeline() is called when overlapping failures finally get handled, or
just before endBlock()? I'm confused.
> Erasure coding: preallocate multiple generation stamps and serialize updates
> from data streamers
>
>
> Key: HDFS-9079
> URL: https://issues.apache.org/jira/browse/HDFS-9079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-9079-HDFS-7285.00.patch
>
>
> A non-striped DataStreamer goes through the following steps in error handling:
> {code}
> 1) Finds error => 2) Asks NN for new GS => 3) Gets new GS from NN => 4)
> Applies new GS to DN (createBlockOutputStream) => 5) Ack from DN => 6)
> Updates block on NN
> {code}
> To simplify the above we can preallocate GS when NN creates a new striped
> block group ({{FSN#createNewBlock}}). For each new striped block group we can
> reserve {{NUM_PARITY_BLOCKS}} GS's. Then steps 1~3 in the above sequence can
> be saved. If more than {{NUM_PARITY_BLOCKS}} errors have happened we
> shouldn't try to further recover anyway.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9147) Fix the setting of visibleLength in ExternalBlockReader
[
https://issues.apache.org/jira/browse/HDFS-9147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14910030#comment-14910030
]
Hadoop QA commented on HDFS-9147:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 20m 5s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 0s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 9s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 55s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 27s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 23s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 8s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 165m 12s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 30s | Tests passed in
hadoop-hdfs-client. |
| | | 216m 48s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.web.TestWebHDFSOAuth2 |
| Timed out tests | org.apache.hadoop.hdfs.TestDecommission |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12762575/HDFS-9147.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 1c030c6 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12704/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12704/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12704/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12704/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12704/console |
This message was automatically generated.
> Fix the setting of visibleLength in ExternalBlockReader
> ---
>
> Key: HDFS-9147
> URL: https://issues.apache.org/jira/browse/HDFS-9147
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.8.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-9147.001.patch
>
>
> BlockReaderFactory needs to take the start offset into consideration when
> setting the visibleLength to use in ExternalBlockReader.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9152) Get input/output error while copying 800 small files to NFS Gateway mount point
Weiwei Yang created HDFS-9152: - Summary: Get input/output error while copying 800 small files to NFS Gateway mount point Key: HDFS-9152 URL: https://issues.apache.org/jira/browse/HDFS-9152 Project: Hadoop HDFS Issue Type: Bug Components: nfs Affects Versions: 2.7.1 Reporter: Weiwei Yang We have around 800 3-5K files on local file system, we have nfs gateway mounted on /hdfs/, when we tried to copy these files to HDFS by cp ~/userdata/* /hdfs/user/cqdemo/demo3.data/ most of files are failed because of cp: writing `/hdfs/user/cqdemo/demo3.data/TRAFF_201408011220.csv': Input/output error cp: writing `/hdfs/user/cqdemo/demo3.data/TRAFF_201408011221.csv': Input/output error cp: writing `/hdfs/user/cqdemo/demo3.data/TRAFF_201408011222.csv': Input/output error for same set of files, I tried to use hadoop dfs -put cimm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9151) Mover should print the exit status/reason on console like balancer tool.
Archana T created HDFS-9151:
---
Summary: Mover should print the exit status/reason on console like
balancer tool.
Key: HDFS-9151
URL: https://issues.apache.org/jira/browse/HDFS-9151
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer & mover
Reporter: Archana T
Assignee: Surendra Singh Lilhore
Priority: Minor
Mover should print exit reason on console --
In cases where there is No blocks to move or unavailable Storages or any other,
Mover tool gives No information on exit reason on the console--
{code}
# ./hdfs mover
...
Sep 28, 2015 12:31:25 PM Mover took 10sec
# echo $?
0
# ./hdfs mover
...
Sep 28, 2015 12:33:10 PM Mover took 1sec
# echo $?
254
{code}
Unlike Balancer prints exit reason
example--
#./hdfs balancer
...
{color:red}The cluster is balanced. Exiting...{color}
Sep 28, 2015 12:18:02 PM Balancing took 1.744 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9053) Support large directories efficiently using B-Tree
[ https://issues.apache.org/jira/browse/HDFS-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14910048#comment-14910048 ] Yi Liu commented on HDFS-9053: -- Hi [~jingzhao], could you help to do further reviews? Thanks a lot. > Support large directories efficiently using B-Tree > -- > > Key: HDFS-9053 > URL: https://issues.apache.org/jira/browse/HDFS-9053 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Yi Liu >Assignee: Yi Liu >Priority: Critical > Attachments: HDFS-9053 (BTree with simple benchmark).patch, HDFS-9053 > (BTree).patch, HDFS-9053.001.patch, HDFS-9053.002.patch > > > This is a long standing issue, we were trying to improve this in the past. > Currently we use an ArrayList for the children under a directory, and the > children are ordered in the list, for insert/delete/search, the time > complexity is O(log n), but insertion/deleting causes re-allocations and > copies of big arrays, so the operations are costly. For example, if the > children grow to 1M size, the ArrayList will resize to > 1M capacity, so need > > 1M * 4bytes = 4M continuous heap memory, it easily causes full GC in HDFS > cluster where namenode heap memory is already highly used. I recap the 3 > main issues: > # Insertion/deletion operations in large directories are expensive because > re-allocations and copies of big arrays. > # Dynamically allocate several MB continuous heap memory which will be > long-lived can easily cause full GC problem. > # Even most children are removed later, but the directory INode still > occupies same size heap memory, since the ArrayList will never shrink. > This JIRA is similar to HDFS-7174 created by [~kihwal], but use B-Tree to > solve the problem suggested by [~shv]. > So the target of this JIRA is to implement a low memory footprint B-Tree and > use it to replace ArrayList. > If the elements size is not large (less than the maximum degree of B-Tree > node), the B-Tree only has one root node which contains an array for the > elements. And if the size grows large enough, it will split automatically, > and if elements are removed, then B-Tree nodes can merge automatically (see > more: https://en.wikipedia.org/wiki/B-tree). It will solve the above 3 > issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7663) Erasure Coding: lease recovery / append on striped file
[
https://issues.apache.org/jira/browse/HDFS-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14910002#comment-14910002
]
Walter Su commented on HDFS-7663:
-
bq. Lease/Block recovery is actually a different topic and is a tricky problem.
As you suggested in HDFS-9040, we can create a separate jira for it..
They have strong relationship when appending is to append a paritial block.
bq. Looks like till now we do not have a correct solution for EC file lease
recovery (at least the protocol for DataNodes to do the recovery work)?
Sorry, we are not going to use {{BlockRecoveryCommand}}?
bq. 3. Appending to a partial block group is actually very similar to "keep
writing after calling hsync/hflush", since we need to overwrite the parity data
of the last stripe. We have an initial proposal in the latest design doc in
HDFS-7285 about this part.
I didn't know client can overwrite/truncate replica on DN. Can you tell me how
to do it? If so, I think appending to a partial block group is a good option.
> Erasure Coding: lease recovery / append on striped file
> ---
>
> Key: HDFS-7663
> URL: https://issues.apache.org/jira/browse/HDFS-7663
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Walter Su
> Attachments: HDFS-7663.00.txt
>
>
> Append should be easy if we have variable length block support from
> HDFS-3689, i.e., the new data will be appended to a new block. We need to
> revisit whether and how to support appending data to the original last block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-488) Implement moveToLocal HDFS command
[
https://issues.apache.org/jira/browse/HDFS-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pooja Gupta reassigned HDFS-488:
Assignee: Pooja Gupta (was: Steven Capo)
> Implement moveToLocal HDFS command
> ---
>
> Key: HDFS-488
> URL: https://issues.apache.org/jira/browse/HDFS-488
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Ravi Phulari
>Assignee: Pooja Gupta
>Priority: Minor
> Labels: newbie
> Attachments: HDFS-488.patch, Screen Shot 2014-07-23 at 12.28.23 PM
> 1.png
>
>
> Surprisingly executing HDFS FsShell command -moveToLocal outputs - "Option
> '-moveToLocal' is not implemented yet."
>
> {code}
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -moveToLocal bt t
> Option '-moveToLocal' is not implemented yet.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9149) Consider multi datacenter when sortByDistance
[
https://issues.apache.org/jira/browse/HDFS-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909767#comment-14909767
]
He Xiaoqiao commented on HDFS-9149:
---
[~He Tianyi] i think it is not necessary to extend {{NetworkTopology}} to
satisfy dc aware, maybe we can improve {{getWeight}} with higher layer in the
topology for better locality.
> Consider multi datacenter when sortByDistance
> -
>
> Key: HDFS-9149
> URL: https://issues.apache.org/jira/browse/HDFS-9149
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: He Xiaoqiao
>
> {{sortByDistance}} doesn't consider multi-datacenter when read data, so there
> my be reading data via other datacenter when hadoop deployment with multi-IDC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9149) Consider multi datacenter when sortByDistance
[
https://issues.apache.org/jira/browse/HDFS-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909768#comment-14909768
]
He Xiaoqiao commented on HDFS-9149:
---
[~He Tianyi] i think it is not necessary to extend {{NetworkTopology}} to
satisfy dc aware, maybe we can improve {{getWeight}} with higher layer in the
topology for better locality.
> Consider multi datacenter when sortByDistance
> -
>
> Key: HDFS-9149
> URL: https://issues.apache.org/jira/browse/HDFS-9149
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: He Xiaoqiao
>
> {{sortByDistance}} doesn't consider multi-datacenter when read data, so there
> my be reading data via other datacenter when hadoop deployment with multi-IDC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9139) Enable parallel JUnit tests for HDFS Pre-commit
[
https://issues.apache.org/jira/browse/HDFS-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909787#comment-14909787
]
Hadoop QA commented on HDFS-9139:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | reexec | 0m 0s | dev-support patch detected. |
| {color:red}-1{color} | pre-patch | 15m 42s | Findbugs (version ) appears to
be broken on trunk. |
| {color:blue}0{color} | @author | 0m 0s | Skipping @author checks as
test-patch has been patched. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 14 new or modified test files. |
| {color:green}+1{color} | javac | 7m 50s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 2s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 32s | There were no new checkstyle
issues. |
| {color:green}+1{color} | shellcheck | 0m 9s | There were no new shellcheck
(v0.3.3) issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 30s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 12s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 51m 39s | Tests failed in hadoop-hdfs. |
| | | 94m 15s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12762600/HDFS-9139.03.patch |
| Optional Tests | shellcheck javadoc javac unit findbugs checkstyle |
| git revision | trunk / f0f984e |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12702/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12702/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12702/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12702/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12702/console |
This message was automatically generated.
> Enable parallel JUnit tests for HDFS Pre-commit
>
>
> Key: HDFS-9139
> URL: https://issues.apache.org/jira/browse/HDFS-9139
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-9139.01.patch, HDFS-9139.02.patch,
> HDFS-9139.03.patch
>
>
> Forked from HADOOP-11984,
> With the initial and significant work from [~cnauroth], this Jira is to track
> and support parallel tests' run for HDFS Precommit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9139) Enable parallel JUnit tests for HDFS Pre-commit
[ https://issues.apache.org/jira/browse/HDFS-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-9139: Attachment: HDFS-9139.02.patch After HADOOP-11984 is in, updated the patch. > Enable parallel JUnit tests for HDFS Pre-commit > > > Key: HDFS-9139 > URL: https://issues.apache.org/jira/browse/HDFS-9139 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-9139.01.patch, HDFS-9139.02.patch > > > Forked from HADOOP-11984, > With the initial and significant work from [~cnauroth], this Jira is to track > and support parallel tests' run for HDFS Precommit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9139) Enable parallel JUnit tests for HDFS Pre-commit
[ https://issues.apache.org/jira/browse/HDFS-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909659#comment-14909659 ] Hadoop QA commented on HDFS-9139: - (!) A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HDFS-Build/12701/console in case of problems. > Enable parallel JUnit tests for HDFS Pre-commit > > > Key: HDFS-9139 > URL: https://issues.apache.org/jira/browse/HDFS-9139 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-9139.01.patch, HDFS-9139.02.patch > > > Forked from HADOOP-11984, > With the initial and significant work from [~cnauroth], this Jira is to track > and support parallel tests' run for HDFS Precommit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9080) update htrace version to 4.0.1
[
https://issues.apache.org/jira/browse/HDFS-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909664#comment-14909664
]
Hadoop QA commented on HDFS-9080:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 25m 35s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 11 new or modified test files. |
| {color:green}+1{color} | javac | 8m 2s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 17s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 3m 0s | Site still builds. |
| {color:red}-1{color} | checkstyle | 2m 34s | The applied patch generated 3
new checkstyle issues (total was 719, now 710). |
| {color:red}-1{color} | whitespace | 1m 18s | The patch has 4 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 6m 26s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 22m 12s | Tests failed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 165m 20s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 28s | Tests passed in
hadoop-hdfs-client. |
| | | 248m 24s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common |
| Failed unit tests | hadoop.fs.TestSymlinkLocalFSFileSystem |
| | hadoop.net.TestDNS |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.TestRemoteBlockReader2 |
| | hadoop.hdfs.TestBlockReaderFactory |
| | hadoop.hdfs.TestRemoteBlockReader |
| | hadoop.hdfs.TestDFSOutputStream |
| | hadoop.hdfs.TestClientBlockVerification |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.web.TestWebHDFSOAuth2 |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12762576/HDFS-9080.012.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / 0b31c23 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12699/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12699/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12699/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12699/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12699/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12699/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12699/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12699/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12699/console |
This message was automatically generated.
> update htrace version to 4.0.1
> --
>
> Key: HDFS-9080
> URL: https://issues.apache.org/jira/browse/HDFS-9080
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-9080.001.patch, HDFS-9080.002.patch,
> HDFS-9080.003.patch, HDFS-9080.004.patch, HDFS-9080.005.patch,
> HDFS-9080.006.patch, HDFS-9080.007.patch, HDFS-9080.009.patch,
> HDFS-9080.010.patch, HDFS-9080.011.patch, HDFS-9080.012.patch,
> tracing-fsshell-put.png
>
>
> Update the HTrace library version Hadoop uses to htrace 4.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9147) Fix the setting of visibleLength in ExternalBlockReader
[
https://issues.apache.org/jira/browse/HDFS-9147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909629#comment-14909629
]
Hadoop QA commented on HDFS-9147:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 59s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 0s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 18s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 6s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 27s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 12s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 29m 7s | Tests failed in hadoop-hdfs. |
| {color:red}-1{color} | hdfs tests | 0m 20s | Tests failed in
hadoop-hdfs-client. |
| | | 80m 0s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestFileStatus |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.server.namenode.TestINodeFile |
| | hadoop.fs.contract.hdfs.TestHDFSContractOpen |
| | hadoop.hdfs.server.datanode.TestFsDatasetCache |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart |
| | hadoop.hdfs.TestFileCreationDelete |
| | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.hdfs.TestEncryptionZonesWithHA |
| | hadoop.fs.contract.hdfs.TestHDFSContractMkdir |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.namenode.TestNameNodeXAttr |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.TestDatanodeDeath |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages |
| | hadoop.hdfs.server.datanode.TestCachingStrategy |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.cli.TestXAttrCLI |
| | hadoop.hdfs.server.namenode.TestDeleteRace |
| | hadoop.hdfs.server.namenode.TestParallelImageWrite |
| | hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional |
| | hadoop.hdfs.server.namenode.TestSaveNamespace |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.TestDFSRename |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA |
| |
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement |
| | hadoop.hdfs.TestFileCreationEmpty |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.hdfs.server.datanode.TestDeleteBlockPool |
| | hadoop.hdfs.TestRemoteBlockReader2 |
| | hadoop.hdfs.server.namenode.TestStorageRestore |
| | hadoop.hdfs.server.namenode.TestFileLimit |
| | hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.fs.contract.hdfs.TestHDFSContractSetTimes |
| | hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots |
| | hadoop.hdfs.qjournal.TestNNWithQJM |
| | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| | hadoop.hdfs.TestDFSFinalize |
| | hadoop.hdfs.server.namenode.TestSecureNameNode |
| | hadoop.hdfs.server.namenode.TestFileContextAcl |
| | hadoop.hdfs.TestDatanodeRegistration |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestDataTransferProtocol |
| | hadoop.fs.viewfs.TestViewFsWithXAttrs |
| | hadoop.hdfs.server.datanode.TestDiskError |
| | hadoop.hdfs.TestFsShellPermission |
| | hadoop.hdfs.server.namenode.TestMalformedURLs |
| | hadoop.hdfs.TestBalancerBandwidth |
| | hadoop.hdfs.TestReadWhileWriting |
| | hadoop.hdfs.qjournal.server.TestJournalNode |
| | hadoop.fs.TestSWebHdfsFileContextMainOperations |
| | hadoop.hdfs.TestDisableConnCache |
| |
[jira] [Commented] (HDFS-9087) Add some jitter to DataNode.checkDiskErrorThread
[ https://issues.apache.org/jira/browse/HDFS-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909621#comment-14909621 ] Hudson commented on HDFS-9087: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #453 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/453/]) HDFS-9087. Add some jitter to DataNode.checkDiskErrorThread (Elliott Clark via Colin P. McCabe) (cmccabe: rev 0b31c237f2622e256726fc5d7698f0f195dbdbc1) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Add some jitter to DataNode.checkDiskErrorThread > > > Key: HDFS-9087 > URL: https://issues.apache.org/jira/browse/HDFS-9087 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-9087-v0.patch, HDFS-9087-v1.patch, > HDFS-9087-v2.patch, HDFS-9087-v3.patch > > > If all datanodes are started across a cluster at the same time (or errors in > the network cause ioexceptions) there can be storms where lots of datanodes > check their disks at the exact same time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9087) Add some jitter to DataNode.checkDiskErrorThread
[ https://issues.apache.org/jira/browse/HDFS-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909634#comment-14909634 ] Hudson commented on HDFS-9087: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2364 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2364/]) HDFS-9087. Add some jitter to DataNode.checkDiskErrorThread (Elliott Clark via Colin P. McCabe) (cmccabe: rev 0b31c237f2622e256726fc5d7698f0f195dbdbc1) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java > Add some jitter to DataNode.checkDiskErrorThread > > > Key: HDFS-9087 > URL: https://issues.apache.org/jira/browse/HDFS-9087 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-9087-v0.patch, HDFS-9087-v1.patch, > HDFS-9087-v2.patch, HDFS-9087-v3.patch > > > If all datanodes are started across a cluster at the same time (or errors in > the network cause ioexceptions) there can be storms where lots of datanodes > check their disks at the exact same time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9087) Add some jitter to DataNode.checkDiskErrorThread
[ https://issues.apache.org/jira/browse/HDFS-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909635#comment-14909635 ] Hudson commented on HDFS-9087: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #424 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/424/]) HDFS-9087. Add some jitter to DataNode.checkDiskErrorThread (Elliott Clark via Colin P. McCabe) (cmccabe: rev 0b31c237f2622e256726fc5d7698f0f195dbdbc1) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Add some jitter to DataNode.checkDiskErrorThread > > > Key: HDFS-9087 > URL: https://issues.apache.org/jira/browse/HDFS-9087 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-9087-v0.patch, HDFS-9087-v1.patch, > HDFS-9087-v2.patch, HDFS-9087-v3.patch > > > If all datanodes are started across a cluster at the same time (or errors in > the network cause ioexceptions) there can be storms where lots of datanodes > check their disks at the exact same time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9087) Add some jitter to DataNode.checkDiskErrorThread
[ https://issues.apache.org/jira/browse/HDFS-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909639#comment-14909639 ] Hudson commented on HDFS-9087: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2391 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2391/]) HDFS-9087. Add some jitter to DataNode.checkDiskErrorThread (Elliott Clark via Colin P. McCabe) (cmccabe: rev 0b31c237f2622e256726fc5d7698f0f195dbdbc1) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Add some jitter to DataNode.checkDiskErrorThread > > > Key: HDFS-9087 > URL: https://issues.apache.org/jira/browse/HDFS-9087 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-9087-v0.patch, HDFS-9087-v1.patch, > HDFS-9087-v2.patch, HDFS-9087-v3.patch > > > If all datanodes are started across a cluster at the same time (or errors in > the network cause ioexceptions) there can be storms where lots of datanodes > check their disks at the exact same time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
