[jira] [Commented] (HDFS-8828) Utilize Snapshot diff report to build copy list in distcp
[
https://issues.apache.org/jira/browse/HDFS-8828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662836#comment-14662836
]
Hadoop QA commented on HDFS-8828:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 40s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 27s | The applied patch generated 1
new checkstyle issues (total was 120, now 121). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 21s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 47s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 6m 23s | Tests passed in
hadoop-distcp. |
| | | 42m 54s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749397/HDFS-8828.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f73bdd |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11941/artifact/patchprocess/diffcheckstylehadoop-distcp.txt
|
| hadoop-distcp test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11941/artifact/patchprocess/testrun_hadoop-distcp.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11941/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11941/console |
This message was automatically generated.
> Utilize Snapshot diff report to build copy list in distcp
> -
>
> Key: HDFS-8828
> URL: https://issues.apache.org/jira/browse/HDFS-8828
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: distcp, snapshots
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: HDFS-8828.001.patch, HDFS-8828.002.patch,
> HDFS-8828.003.patch, HDFS-8828.004.patch
>
>
> Some users reported huge time cost to build file copy list in distcp. (30
> hours for 1.6M files). We can leverage snapshot diff report to build file
> copy list including files/dirs which are changes only between two snapshots
> (or a snapshot and a normal dir). It speed up the process in two folds: 1.
> less copy list building time. 2. less file copy MR jobs.
> HDFS snapshot diff report provide information about file/directory creation,
> deletion, rename and modification between two snapshots or a snapshot and a
> normal directory. HDFS-7535 synchronize deletion and rename, then fallback to
> the default distcp. So it still relies on default distcp to building complete
> list of files under the source dir. This patch only puts creation and
> modification files into the copy list based on snapshot diff report. We can
> minimize the number of files to copy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8877) Allow bypassing some minor exceptions while loading editlog
[ https://issues.apache.org/jira/browse/HDFS-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-8877. - Resolution: Not A Problem > Allow bypassing some minor exceptions while loading editlog > --- > > Key: HDFS-8877 > URL: https://issues.apache.org/jira/browse/HDFS-8877 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > > Usually when users hit editlog corruption like HDFS-7587, before upgrading to > a new version with the bug fix, a customized jar has to be provided by > developers first to bypass the exception while loading edits. The whole > process is usually painful. > If we can confirm the corruption/exception is a known issue and can be > ignored after upgrading to the newer version, it may be helpful to have the > capability for users/developers to specify certain types/numbers of > exceptions that can be bypassed while loading edits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8877) Allow bypassing some minor exceptions while loading editlog
[ https://issues.apache.org/jira/browse/HDFS-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662822#comment-14662822 ] Jing Zhao commented on HDFS-8877: - [~cnauroth] just pointed me to the existing "recovery mode" functionality. Actually the recovery mode should be able to solve a lot of cases we hit here. I will close the jira by now. > Allow bypassing some minor exceptions while loading editlog > --- > > Key: HDFS-8877 > URL: https://issues.apache.org/jira/browse/HDFS-8877 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > > Usually when users hit editlog corruption like HDFS-7587, before upgrading to > a new version with the bug fix, a customized jar has to be provided by > developers first to bypass the exception while loading edits. The whole > process is usually painful. > If we can confirm the corruption/exception is a known issue and can be > ignored after upgrading to the newer version, it may be helpful to have the > capability for users/developers to specify certain types/numbers of > exceptions that can be bypassed while loading edits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8859) Improve DataNode (ReplicaMap) memory footprint to save about 45%
[
https://issues.apache.org/jira/browse/HDFS-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-8859:
-
Attachment: HDFS-8859.002.patch
Fix the test failures and enhance the test.
> Improve DataNode (ReplicaMap) memory footprint to save about 45%
>
>
> Key: HDFS-8859
> URL: https://issues.apache.org/jira/browse/HDFS-8859
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Yi Liu
>Assignee: Yi Liu
>Priority: Critical
> Attachments: HDFS-8859.001.patch, HDFS-8859.002.patch
>
>
> By using following approach we can save about *45%* memory footprint for each
> block replica in DataNode memory (This JIRA only talks about *ReplicaMap* in
> DataNode), the details are:
> In ReplicaMap,
> {code}
> private final Map> map =
> new HashMap>();
> {code}
> Currently we use a HashMap {{Map}} to store the replicas
> in memory. The key is block id of the block replica which is already
> included in {{ReplicaInfo}}, so this memory can be saved. Also HashMap Entry
> has a object overhead. We can implement a lightweight Set which is similar
> to {{LightWeightGSet}}, but not a fixed size ({{LightWeightGSet}} uses fix
> size for the entries array, usually it's a big value, an example is
> {{BlocksMap}}, this can avoid full gc since no need to resize), also we
> should be able to get Element through key.
> Following is comparison of memory footprint If we implement a lightweight set
> as described:
> We can save:
> {noformat}
> SIZE (bytes) ITEM
> 20The Key: Long (12 bytes object overhead + 8
> bytes long)
> 12HashMap Entry object overhead
> 4 reference to the key in Entry
> 4 reference to the value in Entry
> 4 hash in Entry
> {noformat}
> Total: -44 bytes
> We need to add:
> {noformat}
> SIZE (bytes) ITEM
> 4 a reference to next element in ReplicaInfo
> {noformat}
> Total: +4 bytes
> So totally we can save 40bytes for each block replica
> And currently one finalized replica needs around 46 bytes (notice: we ignore
> memory alignment here).
> We can save 1 - (4 + 46) / (44 + 46) = *45%* memory for each block replica
> in DataNode.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[
https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662770#comment-14662770
]
Hadoop QA commented on HDFS-8078:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 21m 22s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 7m 56s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 41s | The applied patch generated 1
new checkstyle issues (total was 39, now 38). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 6m 27s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 23m 0s | Tests failed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 176m 36s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 30s | Tests passed in
hadoop-hdfs-client. |
| | | 251m 33s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.net.TestNetUtils |
| | hadoop.hdfs.server.blockmanagement.TestDatanodeManager |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.namenode.TestFsck |
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749341/HDFS-8078.11.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f73bdd |
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/11939/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11939/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11939/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11939/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11939/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11939/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11939/console |
This message was automatically generated.
> HDFS client gets errors trying to to connect to IPv6 DataNode
> -
>
> Key: HDFS-8078
> URL: https://issues.apache.org/jira/browse/HDFS-8078
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
>Reporter: Nate Edel
>Assignee: Nate Edel
> Labels: BB2015-05-TBR, ipv6
> Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, HDFS-8078.9.patch
>
>
> 1st exception, on put:
> 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception
> java.lang.IllegalArgumentException: Does not contain a valid host:port
> authority: 2401:db00:1010:70ba:face:0:8:0:50010
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
> Appears to actually stem from code in DataNodeID which assumes it's safe to
> append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for
> IPv6. Net
[jira] [Updated] (HDFS-8828) Utilize Snapshot diff report to build copy list in distcp
[ https://issues.apache.org/jira/browse/HDFS-8828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated HDFS-8828: --- Attachment: HDFS-8828.004.patch > Utilize Snapshot diff report to build copy list in distcp > - > > Key: HDFS-8828 > URL: https://issues.apache.org/jira/browse/HDFS-8828 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp, snapshots >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: HDFS-8828.001.patch, HDFS-8828.002.patch, > HDFS-8828.003.patch, HDFS-8828.004.patch > > > Some users reported huge time cost to build file copy list in distcp. (30 > hours for 1.6M files). We can leverage snapshot diff report to build file > copy list including files/dirs which are changes only between two snapshots > (or a snapshot and a normal dir). It speed up the process in two folds: 1. > less copy list building time. 2. less file copy MR jobs. > HDFS snapshot diff report provide information about file/directory creation, > deletion, rename and modification between two snapshots or a snapshot and a > normal directory. HDFS-7535 synchronize deletion and rename, then fallback to > the default distcp. So it still relies on default distcp to building complete > list of files under the source dir. This patch only puts creation and > modification files into the copy list based on snapshot diff report. We can > minimize the number of files to copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8805) Archival Storage: getStoragePolicy should not need superuser privilege
[
https://issues.apache.org/jira/browse/HDFS-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662768#comment-14662768
]
Brahma Reddy Battula commented on HDFS-8805:
[~jingzhao] If I remove from {{getFileInfo(FSDirectory, String, boolean,
boolean, boolean)}}, I need to remove from {{getFileInfo(FSDirectory, String,
INodesInPath, boolean, boolean)}} right, same variable
({{includeStoragePolicy}}) is passed from first one second one right..? do I
miss something..?
> Archival Storage: getStoragePolicy should not need superuser privilege
> --
>
> Key: HDFS-8805
> URL: https://issues.apache.org/jira/browse/HDFS-8805
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover, namenode
>Reporter: Hui Zheng
>Assignee: Brahma Reddy Battula
> Fix For: 2.6.0
>
> Attachments: HDFS-8805-002.patch, HDFS-8805-003.patch, HDFS-8805.patch
>
>
> The result of getStoragePolicy command is always 'unspecified' even we has
> set a StoragePolicy on a directory.But the real placement of blocks is
> correct.
> The result of fsck is not correct either.
> {code}
> $ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD
> Set storage policy COLD on /tmp/cold
> $ hdfs storagepolicies -getStoragePolicy -path /tmp/cold
> The storage policy of /tmp/cold is unspecified
> $ hdfs fsck -storagepolicies /tmp/cold
> Blocks NOT satisfying the specified storage policy:
> Storage Policy Specified Storage Policy # of blocks
> % of blocks
> ARCHIVE:4(COLD) HOT 5
>55.5556%
> ARCHIVE:3(COLD) HOT 4
>44.%
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662762#comment-14662762
]
Hadoop QA commented on HDFS-8865:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 26m 11s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 8m 59s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 20s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 55s | The applied patch generated
10 new checkstyle issues (total was 491, now 498). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 44s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 9s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 175m 32s | Tests failed in hadoop-hdfs. |
| | | 231m 23s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749347/HDFS-8865.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f73bdd |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11938/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11938/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11938/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11938/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11938/console |
This message was automatically generated.
> Improve quota initialization performance
>
>
> Key: HDFS-8865
> URL: https://issues.apache.org/jira/browse/HDFS-8865
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-8865.patch
>
>
> After replaying edits, the whole file system tree is recursively scanned in
> order to initialize the quota. For big name space, this can take a very long
> time. Since this is done during namenode failover, it also affects failover
> latency.
> By using the Fork-Join framework, I was able to greatly reduce the
> initialization time. The following is the test result using the fsimage from
> one of the big name nodes we have.
> || threads || seconds||
> | 1 (existing) | 55|
> | 1 (fork-join) | 68 |
> | 4 | 16 |
> | 8 | 8 |
> | 12 | 6 |
> | 16 | 5 |
> | 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8622) Implement GETCONTENTSUMMARY operation for WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662749#comment-14662749
]
Akira AJISAKA commented on HDFS-8622:
-
Thank you for the update. Some comments from me. (Mainly naming of the
variables)
{code}
Path parentDir = new Path("/dir1");
{code}
1. I'm thinking the name of the variables should be the same as the name of the
path for readability.
{code}
Path targetDirForLinks = new Path("/targetDirForLinks");
{code}
2. Since the directory is not a target of a symlink, dirForLinks is better for
me.
{code}
Path linkPath = new Path("/link1");
Path linkPathDir = new Path("/linkdir");
Path linkForDir = new Path("/targetDirForLinks/linkfordir1");
{code}
3. I'm thinking linkPathDir and the related test is not necessary because the
attribute of the target of a symlink is not related to the content summary of
the symlink. For the naming, link1, link2, ... is sufficient for me.
{code}
try (FSDataOutputStream o = hdfs.create(new Path(parentDir, "file4"))) {
o.write("123".getBytes());
}
Path filePath = new Path(parentDir,"/file4");
...
hdfs.createSymlink(filePath, linkPath, true);
{code}
4. We can define the {{filePath}} first and re-use it, or we can create a
symlink to file1 instead of defining {{filePath}}. The latter option is clear
for me.
5. Would you fix the checkstyle issue?
> Implement GETCONTENTSUMMARY operation for WebImageViewer
>
>
> Key: HDFS-8622
> URL: https://issues.apache.org/jira/browse/HDFS-8622
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jagadesh Kiran N
>Assignee: Jagadesh Kiran N
> Attachments: HDFS-8622-00.patch, HDFS-8622-01.patch,
> HDFS-8622-02.patch, HDFS-8622-03.patch, HDFS-8622-04.patch,
> HDFS-8622-05.patch, HDFS-8622-06.patch, HDFS-8622-07.patch, HDFS-8622-08.patch
>
>
> it would be better for administrators if {code} GETCONTENTSUMMARY {code} are
> supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662742#comment-14662742 ] Andrew Wang commented on HDFS-7285: --- [~szetszwo] I never said that, what I did say was the following: * A rebase workflow gets very difficult without the ability to squash patches, since you would otherwise spend a lot of time fixing conflicts in intermediate commits that don't even show up in HEAD. We have 180 some commits in the EC branch now, and is definitely at the "very difficult" stage of rebasing. * Moving refactors from branches to trunk is standard practice we've done many times before and is something I recommended we do here too. > Erasure Coding Support inside HDFS > -- > > Key: HDFS-7285 > URL: https://issues.apache.org/jira/browse/HDFS-7285 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Weihua Jiang >Assignee: Zhe Zhang > Attachments: Consolidated-20150707.patch, > Consolidated-20150806.patch, ECAnalyzer.py, ECParser.py, > HDFS-7285-initial-PoC.patch, HDFS-7285-merge-consolidated-01.patch, > HDFS-7285-merge-consolidated-trunk-01.patch, > HDFS-7285-merge-consolidated.trunk.03.patch, > HDFS-7285-merge-consolidated.trunk.04.patch, > HDFS-EC-Merge-PoC-20150624.patch, HDFS-EC-merge-consolidated-01.patch, > HDFS-bistriped.patch, HDFSErasureCodingDesign-20141028.pdf, > HDFSErasureCodingDesign-20141217.pdf, HDFSErasureCodingDesign-20150204.pdf, > HDFSErasureCodingDesign-20150206.pdf, HDFSErasureCodingPhaseITestPlan.pdf, > fsimage-analysis-20150105.pdf > > > Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice > of data reliability, comparing to the existing HDFS 3-replica approach. For > example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, > with storage overhead only being 40%. This makes EC a quite attractive > alternative for big data storage, particularly for cold data. > Facebook had a related open source project called HDFS-RAID. It used to be > one of the contribute packages in HDFS but had been removed since Hadoop 2.0 > for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends > on MapReduce to do encoding and decoding tasks; 2) it can only be used for > cold files that are intended not to be appended anymore; 3) the pure Java EC > coding implementation is extremely slow in practical use. Due to these, it > might not be a good idea to just bring HDFS-RAID back. > We (Intel and Cloudera) are working on a design to build EC into HDFS that > gets rid of any external dependencies, makes it self-contained and > independently maintained. This design lays the EC feature on the storage type > support and considers compatible with existing HDFS features like caching, > snapshot, encryption, high availability and etc. This design will also > support different EC coding schemes, implementations and policies for > different deployment scenarios. By utilizing advanced libraries (e.g. Intel > ISA-L library), an implementation can greatly improve the performance of EC > encoding/decoding and makes the EC solution even more attractive. We will > post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8880) NameNode should periodically log metrics to separate appender
[ https://issues.apache.org/jira/browse/HDFS-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8880: Attachment: HDFS-8880.01.patch > NameNode should periodically log metrics to separate appender > - > > Key: HDFS-8880 > URL: https://issues.apache.org/jira/browse/HDFS-8880 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8880.01.patch > > > The NameNode can periodically log metrics to help debugging when the cluster > is not setup with another metrics monitoring scheme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8880) NameNode metrics logging
[ https://issues.apache.org/jira/browse/HDFS-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8880: Attachment: namenode-metrics.log Sample log file attached. > NameNode metrics logging > > > Key: HDFS-8880 > URL: https://issues.apache.org/jira/browse/HDFS-8880 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8880.01.patch, namenode-metrics.log > > > The NameNode can periodically log metrics to help debugging when the cluster > is not setup with another metrics monitoring scheme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8880) NameNode should periodically log metrics
[ https://issues.apache.org/jira/browse/HDFS-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8880: Status: Patch Available (was: Open) > NameNode should periodically log metrics > > > Key: HDFS-8880 > URL: https://issues.apache.org/jira/browse/HDFS-8880 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8880.01.patch > > > The NameNode can periodically log metrics to help debugging when the cluster > is not setup with another metrics monitoring scheme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HDFS-8880) NameNode should periodically log metrics to separate appender
[
https://issues.apache.org/jira/browse/HDFS-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662651#comment-14662651
]
Arpit Agarwal edited comment on HDFS-8880 at 8/7/15 11:37 PM:
--
Patch to log NameNode metrics once every 10 minutes by default to a separate
_name-metrics.log_ appender.
Metrics logging can be turned off by setting
{{dfs.namenode.metrics.logger.period.seconds}} to a non-positive value.
The logging filters out {{TabularData}} and {{CompositeData}} metrics and also
truncates string values at 128 bytes to avoid logging excessively.
Updated patch with tests to follow soon.
was (Author: arpitagarwal):
Patch to log NameNode metrics once every 10 minutes by default to a separate
_name-metrics.log_ appender.
The logging filters out {{TabularData}} and {{CompositeData}} metrics and also
truncates string values at 128 bytes to avoid logging excessively.
Updated patch with tests to follow soon.
> NameNode should periodically log metrics to separate appender
> -
>
> Key: HDFS-8880
> URL: https://issues.apache.org/jira/browse/HDFS-8880
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8880.01.patch
>
>
> The NameNode can periodically log metrics to help debugging when the cluster
> is not setup with another metrics monitoring scheme.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8880) NameNode should periodically log metrics to separate appender
[ https://issues.apache.org/jira/browse/HDFS-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8880: Attachment: (was: HDFS-8880.01.patch) > NameNode should periodically log metrics to separate appender > - > > Key: HDFS-8880 > URL: https://issues.apache.org/jira/browse/HDFS-8880 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8880.01.patch > > > The NameNode can periodically log metrics to help debugging when the cluster > is not setup with another metrics monitoring scheme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8880) NameNode should periodically log metrics
[
https://issues.apache.org/jira/browse/HDFS-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-8880:
Attachment: HDFS-8880.01.patch
Patch to log NameNode metrics once every 10 minutes by default to a separate
_name-metrics.log_ appender.
The logging filters out {{TabularData}} and {{CompositeData}} metrics and also
truncates string values at 128 bytes to avoid logging excessively.
Updated patch with tests to follow soon.
> NameNode should periodically log metrics
>
>
> Key: HDFS-8880
> URL: https://issues.apache.org/jira/browse/HDFS-8880
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8880.01.patch
>
>
> The NameNode can periodically log metrics to help debugging when the cluster
> is not setup with another metrics monitoring scheme.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8880) NameNode should periodically log metrics to separate appender
[ https://issues.apache.org/jira/browse/HDFS-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8880: Summary: NameNode should periodically log metrics to separate appender (was: NameNode should periodically log metrics) > NameNode should periodically log metrics to separate appender > - > > Key: HDFS-8880 > URL: https://issues.apache.org/jira/browse/HDFS-8880 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8880.01.patch > > > The NameNode can periodically log metrics to help debugging when the cluster > is not setup with another metrics monitoring scheme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8880) NameNode metrics logging
[ https://issues.apache.org/jira/browse/HDFS-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8880: Summary: NameNode metrics logging (was: NameNode should periodically log metrics to separate appender) > NameNode metrics logging > > > Key: HDFS-8880 > URL: https://issues.apache.org/jira/browse/HDFS-8880 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8880.01.patch > > > The NameNode can periodically log metrics to help debugging when the cluster > is not setup with another metrics monitoring scheme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8880) NameNode should periodically log metrics
Arpit Agarwal created HDFS-8880: --- Summary: NameNode should periodically log metrics Key: HDFS-8880 URL: https://issues.apache.org/jira/browse/HDFS-8880 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Arpit Agarwal Assignee: Arpit Agarwal The NameNode can periodically log metrics to help debugging when the cluster is not setup with another metrics monitoring scheme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8879) Quota by storage type usage incorrectly initialized upon namenode restart
[
https://issues.apache.org/jira/browse/HDFS-8879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-8879:
Reporter: Kihwal Lee (was: Xiaoyu Yao)
> Quota by storage type usage incorrectly initialized upon namenode restart
> -
>
> Key: HDFS-8879
> URL: https://issues.apache.org/jira/browse/HDFS-8879
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Kihwal Lee
>Assignee: Xiaoyu Yao
>
> This was found by [~kihwal] as part of HDFS-8865 work in this
> [comment|https://issues.apache.org/jira/browse/HDFS-8865?focusedCommentId=14660904&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14660904].
> The unit test
> testQuotaByStorageTypePersistenceInFsImage/testQuotaByStorageTypePersistenceInFsEdit
> failed to detect this because they were using an obsolete
> FsDirectory instance. Once added the highlighted line below, the issue can be
> reproed.
> {code}
> >fsdir = cluster.getNamesystem().getFSDirectory();
> INode testDirNodeAfterNNRestart = fsdir.getINode4Write(testDir.toString());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-8879) Quota by storage type usage incorrectly initialized upon namenode restart
[
https://issues.apache.org/jira/browse/HDFS-8879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao reassigned HDFS-8879:
Assignee: Xiaoyu Yao
> Quota by storage type usage incorrectly initialized upon namenode restart
> -
>
> Key: HDFS-8879
> URL: https://issues.apache.org/jira/browse/HDFS-8879
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>
> This was found by [~kihwal] as part of HDFS-8865 work in this
> [comment|https://issues.apache.org/jira/browse/HDFS-8865?focusedCommentId=14660904&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14660904].
> The unit test
> testQuotaByStorageTypePersistenceInFsImage/testQuotaByStorageTypePersistenceInFsEdit
> failed to detect this because they were using an obsolete
> FsDirectory instance. Once added the highlighted line below, the issue can be
> reproed.
> {code}
> >fsdir = cluster.getNamesystem().getFSDirectory();
> INode testDirNodeAfterNNRestart = fsdir.getINode4Write(testDir.toString());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8879) Quota by storage type usage incorrectly initialized upon namenode restart
Xiaoyu Yao created HDFS-8879:
Summary: Quota by storage type usage incorrectly initialized upon
namenode restart
Key: HDFS-8879
URL: https://issues.apache.org/jira/browse/HDFS-8879
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.0
Reporter: Xiaoyu Yao
This was found by [~kihwal] as part of HDFS-8865 work in this
[comment|https://issues.apache.org/jira/browse/HDFS-8865?focusedCommentId=14660904&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14660904].
The unit test
testQuotaByStorageTypePersistenceInFsImage/testQuotaByStorageTypePersistenceInFsEdit
failed to detect this because they were using an obsolete
FsDirectory instance. Once added the highlighted line below, the issue can be
reproed.
{code}
>fsdir = cluster.getNamesystem().getFSDirectory();
INode testDirNodeAfterNNRestart = fsdir.getINode4Write(testDir.toString());
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8876) Make hard coded parameters used by balancer and other tools configurable
[
https://issues.apache.org/jira/browse/HDFS-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662631#comment-14662631
]
Chris Trezzo commented on HDFS-8876:
Thanks [~szetszwo]. I will take a look at HDFS-8818 and HDFS-8824.
> Make hard coded parameters used by balancer and other tools configurable
>
>
> Key: HDFS-8876
> URL: https://issues.apache.org/jira/browse/HDFS-8876
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> During investigation of how to speed up balancer, at least to the level
> specified by {{dfs.datanode.balance.bandwidthPerSec}}, we found that
> parameters such as {{MAX_BLOCKS_SIZE_TO_FETCH}} and
> {{SOURCE_BLOCKS_MIN_SIZE}} are hard coded. These parameters are related to
> block size and other configurable parameters used by balancer. So least we
> should make it configurable. In the longer term, it might be interesting to
> understand if we simplify all these related configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662585#comment-14662585 ] Xiaoyu Yao commented on HDFS-8865: -- [~kihwal], thanks for working on this improvement work and fixing the issue on quota by storage type usage update. Can we fix the quota by storage type update issue in a separate JIRA? > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-8865.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8865: - Status: Patch Available (was: Open) > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-8865.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8865: - Attachment: HDFS-8865.patch > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-8865.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662366#comment-14662366 ] Tsz Wo Nicholas Sze commented on HDFS-7285: --- [~zhz], could you describe the git rebase workflow we are using? In a separated discussion, [~andrew.wang] claimed that our git rebase workflow requires committing some patches to trunk before merging the branch. I wonder if you feel the same way. > Erasure Coding Support inside HDFS > -- > > Key: HDFS-7285 > URL: https://issues.apache.org/jira/browse/HDFS-7285 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Weihua Jiang >Assignee: Zhe Zhang > Attachments: Consolidated-20150707.patch, > Consolidated-20150806.patch, ECAnalyzer.py, ECParser.py, > HDFS-7285-initial-PoC.patch, HDFS-7285-merge-consolidated-01.patch, > HDFS-7285-merge-consolidated-trunk-01.patch, > HDFS-7285-merge-consolidated.trunk.03.patch, > HDFS-7285-merge-consolidated.trunk.04.patch, > HDFS-EC-Merge-PoC-20150624.patch, HDFS-EC-merge-consolidated-01.patch, > HDFS-bistriped.patch, HDFSErasureCodingDesign-20141028.pdf, > HDFSErasureCodingDesign-20141217.pdf, HDFSErasureCodingDesign-20150204.pdf, > HDFSErasureCodingDesign-20150206.pdf, HDFSErasureCodingPhaseITestPlan.pdf, > fsimage-analysis-20150105.pdf > > > Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice > of data reliability, comparing to the existing HDFS 3-replica approach. For > example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, > with storage overhead only being 40%. This makes EC a quite attractive > alternative for big data storage, particularly for cold data. > Facebook had a related open source project called HDFS-RAID. It used to be > one of the contribute packages in HDFS but had been removed since Hadoop 2.0 > for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends > on MapReduce to do encoding and decoding tasks; 2) it can only be used for > cold files that are intended not to be appended anymore; 3) the pure Java EC > coding implementation is extremely slow in practical use. Due to these, it > might not be a good idea to just bring HDFS-RAID back. > We (Intel and Cloudera) are working on a design to build EC into HDFS that > gets rid of any external dependencies, makes it self-contained and > independently maintained. This design lays the EC feature on the storage type > support and considers compatible with existing HDFS features like caching, > snapshot, encryption, high availability and etc. This design will also > support different EC coding schemes, implementations and policies for > different deployment scenarios. By utilizing advanced libraries (e.g. Intel > ISA-L library), an implementation can greatly improve the performance of EC > encoding/decoding and makes the EC solution even more attractive. We will > post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8866) Typo in docs: Rumtime -> Runtime
[
https://issues.apache.org/jira/browse/HDFS-8866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662357#comment-14662357
]
Hudson commented on HDFS-8866:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8279 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8279/])
HDFS-8866. Typo in docs: Rumtime -> Runtime. Contributed by Gabor Liptak.
(jghoman: rev 8f73bdd06b16d5048ffb6071bbcecf849c6225db)
* hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Typo in docs: Rumtime -> Runtime
>
>
> Key: HDFS-8866
> URL: https://issues.apache.org/jira/browse/HDFS-8866
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, webhdfs
>Reporter: Jakob Homan
>Assignee: Gabor Liptak
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-8866.1.patch
>
>
> From WebHDFS site doc:
> {noformat}### HTTP Response Codes
> | Exceptions | HTTP Response Codes |
> |: |: |
> | `IllegalArgumentException ` | `400 Bad Request ` |
> | `UnsupportedOperationException` | `400 Bad Request ` |
> | `SecurityException ` | `401 Unauthorized ` |
> | `IOException ` | `403 Forbidden ` |
> | `FileNotFoundException ` | `404 Not Found ` |
> | `RumtimeException ` | `500 Internal Server Error` |{noformat}
> Everyone knows there's no exception to rum time. Rum time is mandatory, but
> irrelevant to WebHDFS. Let's make it Runtime...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8750) FIleSystem does not honor Configuration.getClassLoader() while loading FileSystem implementations
[
https://issues.apache.org/jira/browse/HDFS-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662349#comment-14662349
]
Hadoop QA commented on HDFS-8750:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 58s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 50s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 12s | The applied patch generated 1
new checkstyle issues (total was 140, now 140). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 22s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 56s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 24m 12s | Tests failed in
hadoop-common. |
| | | 64m 15s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.net.TestNetUtils |
| | hadoop.security.ssl.TestReloadingX509TrustManager |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749311/HDFS-8750.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f73bdd |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11937/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11937/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11937/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11937/console |
This message was automatically generated.
> FIleSystem does not honor Configuration.getClassLoader() while loading
> FileSystem implementations
> -
>
> Key: HDFS-8750
> URL: https://issues.apache.org/jira/browse/HDFS-8750
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, HDFS
>Reporter: Himanshu
>Assignee: Himanshu
> Attachments: HDFS-8750.001.patch, HDFS-8750.002.patch,
> HDFS-8750.003.patch, HDFS-8750.004.patch
>
>
> In FileSystem.loadFileSystems(), at
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L2652
> a "scheme" -> "FileSystem implementation" map is created from the jars
> available on classpath. It uses Thread.currentThread().getClassLoader() via
> ServiceLoader.load(FileSystem.class)
> Instead, loadFileSystems() should take Configuration as an argument and
> should first check if a classloader is configured in
> configuration.getClassLoader(), if yes then
> ServiceLoader.load(FileSystem.class, configuration.getClassLoader()) should
> be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662327#comment-14662327 ] Nate Edel commented on HDFS-8078: - Separate from the question of a feature branch, here's the patch with common patterns refactored into NetUtils (mostly out of HADOOP-12122) per [~ste...@apache.org]'s comment. While this is a very safe change, I'd ask you hold off on detailed review until [~newanja] and I can re-complete some testing, and I one of us can add some more unit tests. I'm submitting it here to get QABot's opinion at this point. > HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HDFS-8078 > URL: https://issues.apache.org/jira/browse/HDFS-8078 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, HDFS-8078.9.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HDFS-8078: Attachment: HDFS-8078.11.patch > HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HDFS-8078 > URL: https://issues.apache.org/jira/browse/HDFS-8078 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, HDFS-8078.9.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HDFS-8078: Status: Patch Available (was: Open) > HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HDFS-8078 > URL: https://issues.apache.org/jira/browse/HDFS-8078 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, HDFS-8078.9.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HDFS-8078: Status: Open (was: Patch Available) > HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HDFS-8078 > URL: https://issues.apache.org/jira/browse/HDFS-8078 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.9.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8876) Make hard coded parameters used by balancer and other tools configurable
[
https://issues.apache.org/jira/browse/HDFS-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662306#comment-14662306
]
Tsz Wo Nicholas Sze commented on HDFS-8876:
---
Thanks for filing the JIRAs. I already have HDFS-8818 to speed up Balancer.
In my tests, it can speed up from 4GB per iteration to 800GB per iteration
while the duration of each iteration is only increased from ~1 minutes to 1-2
minutes.
I also have patches to remove some useless hard coded parameters and make the
other hard coded parameters configurable; see HDFS-8818 and HDFS-8824.
> Make hard coded parameters used by balancer and other tools configurable
>
>
> Key: HDFS-8876
> URL: https://issues.apache.org/jira/browse/HDFS-8876
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> During investigation of how to speed up balancer, at least to the level
> specified by {{dfs.datanode.balance.bandwidthPerSec}}, we found that
> parameters such as {{MAX_BLOCKS_SIZE_TO_FETCH}} and
> {{SOURCE_BLOCKS_MIN_SIZE}} are hard coded. These parameters are related to
> block size and other configurable parameters used by balancer. So least we
> should make it configurable. In the longer term, it might be interesting to
> understand if we simplify all these related configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6860) BlockStateChange logs are too noisy
[ https://issues.apache.org/jira/browse/HDFS-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-6860: - Attachment: HDFS-6860.branch-2.6.patch Attach a 2.6 patch that includes HDFS-6860 and slf4j changes HDFS-7706, HDFS-7712 and HADOOP-11430. > BlockStateChange logs are too noisy > --- > > Key: HDFS-6860 > URL: https://issues.apache.org/jira/browse/HDFS-6860 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Chang Li > Labels: BB2015-05-TBR, newbie > Fix For: 2.8.0 > > Attachments: HDFS-6860.00.patch, HDFS-6860.01.patch, > HDFS-6860.branch-2.6.patch, HDFS6860.patch, HDFS6860.patch > > > Block State Change logs are too noisy at the default INFO level and affect NN > performance on busy clusters. > Most of these state changes can be logged at debug level instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8877) Allow bypassing some minor exceptions while loading editlog
[
https://issues.apache.org/jira/browse/HDFS-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662259#comment-14662259
]
Jing Zhao commented on HDFS-8877:
-
Maybe a simple way as the first step can be: 1) to provide a new configuration
property for users to specify the specific editlog ops that certain exceptions
can be ignored (i.e., logging the exception instead of failing the edits
loading), and 2), we change the editlog loading code and allow some specific
exceptions to be ignored. E.g., when loading {{OP_CLOS}} or
{{OP_REASSIGN_LEASE}}, instead of directly throwing IOException if the file is
not under construction, we change the code so that this check can only lead to
a warn level log if the configuration is set.
> Allow bypassing some minor exceptions while loading editlog
> ---
>
> Key: HDFS-8877
> URL: https://issues.apache.org/jira/browse/HDFS-8877
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
>
> Usually when users hit editlog corruption like HDFS-7587, before upgrading to
> a new version with the bug fix, a customized jar has to be provided by
> developers first to bypass the exception while loading edits. The whole
> process is usually painful.
> If we can confirm the corruption/exception is a known issue and can be
> ignored after upgrading to the newer version, it may be helpful to have the
> capability for users/developers to specify certain types/numbers of
> exceptions that can be bypassed while loading edits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8875) Optimize the wait time in Balancer for federation scenario
[
https://issues.apache.org/jira/browse/HDFS-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662250#comment-14662250
]
Chris Trezzo commented on HDFS-8875:
bq. I thought the balancer is going to wait until one namespace finish moving
before going to the next namespace/iteration, no?
Good point. I was thinking in terms of a setup where the balancer is killed
periodically and the first namespace never finishes... If the balancer is never
restarted then you are right, I don't see a point to the shuffle.
> Optimize the wait time in Balancer for federation scenario
> --
>
> Key: HDFS-8875
> URL: https://issues.apache.org/jira/browse/HDFS-8875
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> Balancer has wait time between two consecutive iterations. That is to give
> some time for block movement to be fully committed ( return from replaceBlock
> doesn't mean the NN's blockmap has been updated and the block has been
> invalidated on the source node.).
> This wait time could be 23 seconds if {{dfs.heartbeat.interval}} is set to 10
> and {{dfs.namenode.replication.interval}} is to 3. In the case of federation,
> given we iterate through all namespaces in each iteration, this wait time
> becomes unnecessary as while balancer is processing the next namespace, it
> gives the previous namespace it just finished time to commit.
> In addition, Balancer calls {{Collections.shuffle(connectors);}} It doesn't
> seem necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8878) An HDFS built-in DistCp
Linxiao Jin created HDFS-8878: - Summary: An HDFS built-in DistCp Key: HDFS-8878 URL: https://issues.apache.org/jira/browse/HDFS-8878 Project: Hadoop HDFS Issue Type: New Feature Reporter: Linxiao Jin Assignee: Linxiao Jin For now, we use DistCp to do directory copy, which works quite good. However, it would be better if there is an HDFS built-in, efficient, directory copy tool. It could be faster by cut off the redundant communication between HDFS, YARN and MapReduce. It could also release the resource DistCp consumed in job tracker and YARN and easier for debugging. We need more discussion on the new protocol between NN and DN from different clusters to achieve HDFS-level command sending and data transfer. One available hacky solution could be, the srcNN get the block distribution of the target file, ask each datanode to start a DFSClient and copy their local shortcircuited block as a file in dst cluster. After all the block-file in dst cluster is completed, use a DFSClient to concat them together to form the target destination file. There might be some optimized solution by implement a newly designed protocol to communicate over cluster rather than DFSClient and use methods from lower bottom layer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8875) Optimize the wait time in Balancer for federation scenario
[
https://issues.apache.org/jira/browse/HDFS-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662244#comment-14662244
]
Ming Ma commented on HDFS-8875:
---
bq. For the Collections.shuffle(connectors); call: I can see this being
advantageous in the scenario where the balancer is constantly behind. With the
shuffle, you won't always start with the same namespace.
I thought the balancer is going to wait until one namespace finish moving
before going to the next namespace/iteration, no?
bq. Even with federation, we still might run into the case where we would want
to sleep between iterations
Agree. I didn't mean to get rid of the wait time. The optimization could be
like what you suggested.
Another thing is if we should add parallelism for different namespaces.
> Optimize the wait time in Balancer for federation scenario
> --
>
> Key: HDFS-8875
> URL: https://issues.apache.org/jira/browse/HDFS-8875
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> Balancer has wait time between two consecutive iterations. That is to give
> some time for block movement to be fully committed ( return from replaceBlock
> doesn't mean the NN's blockmap has been updated and the block has been
> invalidated on the source node.).
> This wait time could be 23 seconds if {{dfs.heartbeat.interval}} is set to 10
> and {{dfs.namenode.replication.interval}} is to 3. In the case of federation,
> given we iterate through all namespaces in each iteration, this wait time
> becomes unnecessary as while balancer is processing the next namespace, it
> gives the previous namespace it just finished time to commit.
> In addition, Balancer calls {{Collections.shuffle(connectors);}} It doesn't
> seem necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8877) Allow bypassing some minor exceptions while loading editlog
Jing Zhao created HDFS-8877: --- Summary: Allow bypassing some minor exceptions while loading editlog Key: HDFS-8877 URL: https://issues.apache.org/jira/browse/HDFS-8877 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Jing Zhao Assignee: Jing Zhao Usually when users hit editlog corruption like HDFS-7587, before upgrading to a new version with the bug fix, a customized jar has to be provided by developers first to bypass the exception while loading edits. The whole process is usually painful. If we can confirm the corruption/exception is a known issue and can be ignored after upgrading to the newer version, it may be helpful to have the capability for users/developers to specify certain types/numbers of exceptions that can be bypassed while loading edits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8866) Typo in docs: Rumtime -> Runtime
[
https://issues.apache.org/jira/browse/HDFS-8866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HDFS-8866:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
+rum. I've committed this. Thanks, Gabor!
> Typo in docs: Rumtime -> Runtime
>
>
> Key: HDFS-8866
> URL: https://issues.apache.org/jira/browse/HDFS-8866
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, webhdfs
>Reporter: Jakob Homan
>Assignee: Gabor Liptak
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-8866.1.patch
>
>
> From WebHDFS site doc:
> {noformat}### HTTP Response Codes
> | Exceptions | HTTP Response Codes |
> |: |: |
> | `IllegalArgumentException ` | `400 Bad Request ` |
> | `UnsupportedOperationException` | `400 Bad Request ` |
> | `SecurityException ` | `401 Unauthorized ` |
> | `IOException ` | `403 Forbidden ` |
> | `FileNotFoundException ` | `404 Not Found ` |
> | `RumtimeException ` | `500 Internal Server Error` |{noformat}
> Everyone knows there's no exception to rum time. Rum time is mandatory, but
> irrelevant to WebHDFS. Let's make it Runtime...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8875) Optimize the wait time in Balancer for federation scenario
[
https://issues.apache.org/jira/browse/HDFS-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662231#comment-14662231
]
Chris Trezzo commented on HDFS-8875:
[~mingma] Couple of thoughts about this issue:
# For the {{Collections.shuffle(connectors);}} call: I can see this being
advantageous in the scenario where the balancer is constantly behind. With the
shuffle, you won't always start with the same namespace.
# For the sleep time between iterations: Even with federation, we still might
run into the case where we would want to sleep between iterations. For example,
lets say that only one namespace still needs balancing. The other namespaces
quickly finish their iteration and we are back to the same namespace. Maybe we
can check for the amount of time each iteration took. If that time is greater
than sleep time, then we skip the sleep.
> Optimize the wait time in Balancer for federation scenario
> --
>
> Key: HDFS-8875
> URL: https://issues.apache.org/jira/browse/HDFS-8875
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> Balancer has wait time between two consecutive iterations. That is to give
> some time for block movement to be fully committed ( return from replaceBlock
> doesn't mean the NN's blockmap has been updated and the block has been
> invalidated on the source node.).
> This wait time could be 23 seconds if {{dfs.heartbeat.interval}} is set to 10
> and {{dfs.namenode.replication.interval}} is to 3. In the case of federation,
> given we iterate through all namespaces in each iteration, this wait time
> becomes unnecessary as while balancer is processing the next namespace, it
> gives the previous namespace it just finished time to commit.
> In addition, Balancer calls {{Collections.shuffle(connectors);}} It doesn't
> seem necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8827) Erasure Coding: Fix NPE when NameNode processes over-replicated striped blocks
[ https://issues.apache.org/jira/browse/HDFS-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8827: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7285 Status: Resolved (was: Patch Available) I've committed this. Thanks for the contribution, [~walter.k.su] and [~tfukudom]! > Erasure Coding: Fix NPE when NameNode processes over-replicated striped blocks > -- > > Key: HDFS-8827 > URL: https://issues.apache.org/jira/browse/HDFS-8827 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Takuya Fukudome >Assignee: Walter Su > Fix For: HDFS-7285 > > Attachments: HDFS-8827-HDFS-7285.04.patch, > HDFS-8827-HDFS-7285.05.patch, HDFS-8827.1.patch, HDFS-8827.2.patch, > HDFS-8827.3.patch, processing-over-replica-npe.log > > > In our test cluster, when namenode processed over replicated striped blocks, > null pointer exception(NPE) occurred. This happened under below situation: 1) > some datanodes shutdown. 2) namenode recovers block group which lost internal > blocks. 3) restart the stopped datanodes. 4) namenode processes over > replicated striped blocks. 5) NPE occurs > I think BlockPlacementPolicyDefault#chooseReplicaToDelete will return null in > this situation which causes this NPE problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662206#comment-14662206 ] Xiaoyu Yao commented on HDFS-8747: -- bq. Regarding nested encryption zones, one request we've had is setting "/" as an encryption zone, then subdirs as different EZs. This guarantees that all data in HDFS is encrypted, and gives the flexibility of using different EZ keys for subdirs if desired. That is not difficult to support as long as we don't allow create zone with existing files. Do you expect the root zone to be created implicitly by NN if this certain key is enabled? > Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption > Zones > -- > > Key: HDFS-8747 > URL: https://issues.apache.org/jira/browse/HDFS-8747 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-8747-07092015.pdf, HDFS-8747-07152015.pdf, > HDFS-8747-07292015.pdf > > > HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to > allow create encryption zone on top of a single HDFS directory. Files under > the root directory of the encryption zone will be encrypted/decrypted > transparently upon HDFS client write or read operations. > Generally, it does not support rename(without data copying) across encryption > zones or between encryption zone and non-encryption zone because different > security settings of encryption zones. However, there are certain use cases > where efficient rename support is desired. This JIRA is to propose better > support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft > Delete” (a.k.a. trash) with HDFS encryption zones. > “Scratch Space” is widely used in Hadoop jobs, which requires efficient > rename support. Temporary files from MR jobs are usually stored in staging > area outside encryption zone such as “/tmp” directory and then rename to > targeted directories as specified once the data is ready to be further > processed. > Below is a summary of supported/unsupported cases from latest Hadoop: > * Rename within the encryption zone is supported > * Rename the entire encryption zone by moving the root directory of the zone > is allowed. > * Rename sub-directory/file from encryption zone to non-encryption zone is > not allowed. > * Rename sub-directory/file from encryption zone A to encryption zone B is > not allowed. > * Rename from non-encryption zone to encryption zone is not allowed. > “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that > helps prevent accidental deletion of files and directories. If trash is > enabled and a file or directory is deleted using the Hadoop shell, the file > is moved to the .Trash directory of the user's home directory instead of > being deleted. Deleted files are initially moved (renamed) to the Current > sub-directory of the .Trash directory with original path being preserved. > Files and directories in the trash can be restored simply by moving them to a > location outside the .Trash directory. > Due to the limited rename support, delete sub-directory/file within > encryption zone with trash feature is not allowed. Client has to use > -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved > the error message but without a complete solution to the problem. > We propose to solve the problem by generalizing the mapping between > encryption zone and its underlying HDFS directories from 1:1 today to 1:N. > The encryption zone should allow non-overlapped directories such as scratch > space or soft delete "trash" locations to be added/removed dynamically after > creation. This way, rename for "scratch space" and "soft delete" can be > better supported without breaking the assumption that rename is only > supported "within the zone". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8867) Enable optimized block reports
[ https://issues.apache.org/jira/browse/HDFS-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662198#comment-14662198 ] Rushabh S Shah commented on HDFS-8867: -- [~jingzhao]: Daryn has a patch and he will upload on Monday. I have to delete the previous comment since I copied the wrong user. > Enable optimized block reports > -- > > Key: HDFS-8867 > URL: https://issues.apache.org/jira/browse/HDFS-8867 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Rushabh S Shah >Assignee: Daryn Sharp > Fix For: 2.7.2 > > > Opening this ticket on behalf of [~daryn] > HDFS-7435 introduced a more efficiently encoded block report format designed > to improve performance and reduce GC load on the NN and DNs. The NN is not > advertising this capability to the DNs so old-style reports are still being > used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8876) Make hard coded parameters used by balancer and other tools configurable
[
https://issues.apache.org/jira/browse/HDFS-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated HDFS-8876:
---
Issue Type: Sub-task (was: Improvement)
Parent: HDFS-8825
> Make hard coded parameters used by balancer and other tools configurable
>
>
> Key: HDFS-8876
> URL: https://issues.apache.org/jira/browse/HDFS-8876
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> During investigation of how to speed up balancer, at least to the level
> specified by {{dfs.datanode.balance.bandwidthPerSec}}, we found that
> parameters such as {{MAX_BLOCKS_SIZE_TO_FETCH}} and
> {{SOURCE_BLOCKS_MIN_SIZE}} are hard coded. These parameters are related to
> block size and other configurable parameters used by balancer. So least we
> should make it configurable. In the longer term, it might be interesting to
> understand if we simplify all these related configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8876) Make hard coded parameters used by balancer and other tools configurable
[
https://issues.apache.org/jira/browse/HDFS-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662190#comment-14662190
]
Chris Trezzo commented on HDFS-8876:
[~jingzhao] will do!
> Make hard coded parameters used by balancer and other tools configurable
>
>
> Key: HDFS-8876
> URL: https://issues.apache.org/jira/browse/HDFS-8876
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> During investigation of how to speed up balancer, at least to the level
> specified by {{dfs.datanode.balance.bandwidthPerSec}}, we found that
> parameters such as {{MAX_BLOCKS_SIZE_TO_FETCH}} and
> {{SOURCE_BLOCKS_MIN_SIZE}} are hard coded. These parameters are related to
> block size and other configurable parameters used by balancer. So least we
> should make it configurable. In the longer term, it might be interesting to
> understand if we simplify all these related configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HDFS-8867) Enable optimized block reports
[ https://issues.apache.org/jira/browse/HDFS-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8867: Comment: was deleted (was: Cool, thanks!) > Enable optimized block reports > -- > > Key: HDFS-8867 > URL: https://issues.apache.org/jira/browse/HDFS-8867 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Rushabh S Shah >Assignee: Daryn Sharp > Fix For: 2.7.2 > > > Opening this ticket on behalf of [~daryn] > HDFS-7435 introduced a more efficiently encoded block report format designed > to improve performance and reduce GC load on the NN and DNs. The NN is not > advertising this capability to the DNs so old-style reports are still being > used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8875) Optimize the wait time in Balancer for federation scenario
[
https://issues.apache.org/jira/browse/HDFS-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated HDFS-8875:
---
Issue Type: Sub-task (was: Improvement)
Parent: HDFS-8825
> Optimize the wait time in Balancer for federation scenario
> --
>
> Key: HDFS-8875
> URL: https://issues.apache.org/jira/browse/HDFS-8875
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> Balancer has wait time between two consecutive iterations. That is to give
> some time for block movement to be fully committed ( return from replaceBlock
> doesn't mean the NN's blockmap has been updated and the block has been
> invalidated on the source node.).
> This wait time could be 23 seconds if {{dfs.heartbeat.interval}} is set to 10
> and {{dfs.namenode.replication.interval}} is to 3. In the case of federation,
> given we iterate through all namespaces in each iteration, this wait time
> becomes unnecessary as while balancer is processing the next namespace, it
> gives the previous namespace it just finished time to commit.
> In addition, Balancer calls {{Collections.shuffle(connectors);}} It doesn't
> seem necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8874) Add DN metrics for balancer and other block movement scenarios
[
https://issues.apache.org/jira/browse/HDFS-8874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated HDFS-8874:
---
Issue Type: Sub-task (was: Improvement)
Parent: HDFS-8825
> Add DN metrics for balancer and other block movement scenarios
> --
>
> Key: HDFS-8874
> URL: https://issues.apache.org/jira/browse/HDFS-8874
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> For balancer, mover and migrator (HDFS-8789), we want to know how close it is
> to the DN's throttling thresholds. Although DN has existing metrics such as
> {{BytesWritten}}, {{BytesRead}}, {{CopyBlockOpNumOps}} and
> {{ReplaceBlockOpNumOps}}, there is no metrics to indicate the number of bytes
> moved.
> We can add {{ReplaceBlockBytesWritten}} and {{CopyBlockBytesRead}} to account
> for the bytes moved in ReplaceBlock and CopyBlock operations. In addition, we
> can also add throttling metrics for {{DataTransferThrottler}} and
> {{BlockBalanceThrottler}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HDFS-8876) Make hard coded parameters used by balancer and other tools configurable
[
https://issues.apache.org/jira/browse/HDFS-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-8876 started by Chris Trezzo.
--
> Make hard coded parameters used by balancer and other tools configurable
>
>
> Key: HDFS-8876
> URL: https://issues.apache.org/jira/browse/HDFS-8876
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> During investigation of how to speed up balancer, at least to the level
> specified by {{dfs.datanode.balance.bandwidthPerSec}}, we found that
> parameters such as {{MAX_BLOCKS_SIZE_TO_FETCH}} and
> {{SOURCE_BLOCKS_MIN_SIZE}} are hard coded. These parameters are related to
> block size and other configurable parameters used by balancer. So least we
> should make it configurable. In the longer term, it might be interesting to
> understand if we simplify all these related configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HDFS-8867) Enable optimized block reports
[ https://issues.apache.org/jira/browse/HDFS-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8867: - Comment: was deleted (was: [~jinzhao]: daryn has a patch and he will upload on monday.) > Enable optimized block reports > -- > > Key: HDFS-8867 > URL: https://issues.apache.org/jira/browse/HDFS-8867 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Rushabh S Shah >Assignee: Daryn Sharp > Fix For: 2.7.2 > > > Opening this ticket on behalf of [~daryn] > HDFS-7435 introduced a more efficiently encoded block report format designed > to improve performance and reduce GC load on the NN and DNs. The NN is not > advertising this capability to the DNs so old-style reports are still being > used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8867) Enable optimized block reports
[ https://issues.apache.org/jira/browse/HDFS-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662180#comment-14662180 ] Jing Zhao commented on HDFS-8867: - Cool, thanks! > Enable optimized block reports > -- > > Key: HDFS-8867 > URL: https://issues.apache.org/jira/browse/HDFS-8867 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Rushabh S Shah >Assignee: Daryn Sharp > Fix For: 2.7.2 > > > Opening this ticket on behalf of [~daryn] > HDFS-7435 introduced a more efficiently encoded block report format designed > to improve performance and reduce GC load on the NN and DNs. The NN is not > advertising this capability to the DNs so old-style reports are still being > used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8772) Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662178#comment-14662178
]
Hudson commented on HDFS-8772:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8278 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8278/])
HDFS-8772. Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails.
Contributed by Walter Su. (wang: rev 98a27d110129c7b32455035831480f1c6197260b)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyIsHot.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails
>
>
> Key: HDFS-8772
> URL: https://issues.apache.org/jira/browse/HDFS-8772
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: 2.8.0
>
> Attachments: HDFS-8772-branch-2.04.patch, HDFS-8772.01.patch,
> HDFS-8772.02.patch, HDFS-8772.03.patch, HDFS-8772.04.patch
>
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/11596/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11598/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11599/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11600/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11606/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11608/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11612/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11618/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11650/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11655/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11659/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11663/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11664/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11667/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11669/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11676/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11677/testReport/
> {noformat}
> java.lang.AssertionError: expected:<0> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot.testDatanodeRestarts(TestStandbyIsHot.java:188)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8876) Make hard coded parameters used by balancer and other tools configurable
[
https://issues.apache.org/jira/browse/HDFS-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662177#comment-14662177
]
Jing Zhao commented on HDFS-8876:
-
Thanks for reporting the issue, Ming! Maybe we can move all balancer related
improvement under HDFS-8825?
> Make hard coded parameters used by balancer and other tools configurable
>
>
> Key: HDFS-8876
> URL: https://issues.apache.org/jira/browse/HDFS-8876
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> During investigation of how to speed up balancer, at least to the level
> specified by {{dfs.datanode.balance.bandwidthPerSec}}, we found that
> parameters such as {{MAX_BLOCKS_SIZE_TO_FETCH}} and
> {{SOURCE_BLOCKS_MIN_SIZE}} are hard coded. These parameters are related to
> block size and other configurable parameters used by balancer. So least we
> should make it configurable. In the longer term, it might be interesting to
> understand if we simplify all these related configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8867) Enable optimized block reports
[ https://issues.apache.org/jira/browse/HDFS-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662176#comment-14662176 ] Rushabh S Shah commented on HDFS-8867: -- [~jinzhao]: daryn has a patch and he will upload on monday. > Enable optimized block reports > -- > > Key: HDFS-8867 > URL: https://issues.apache.org/jira/browse/HDFS-8867 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Rushabh S Shah >Assignee: Daryn Sharp > Fix For: 2.7.2 > > > Opening this ticket on behalf of [~daryn] > HDFS-7435 introduced a more efficiently encoded block report format designed > to improve performance and reduce GC load on the NN and DNs. The NN is not > advertising this capability to the DNs so old-style reports are still being > used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8867) Enable optimized block reports
[
https://issues.apache.org/jira/browse/HDFS-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662169#comment-14662169
]
Jing Zhao commented on HDFS-8867:
-
Looks like the issue here is that {{CAPABILITIES_SUPPORTED}} has not been
correctly initialized because the enum {{Capability}} is not loaded?
> Enable optimized block reports
> --
>
> Key: HDFS-8867
> URL: https://issues.apache.org/jira/browse/HDFS-8867
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Daryn Sharp
> Fix For: 2.7.2
>
>
> Opening this ticket on behalf of [~daryn]
> HDFS-7435 introduced a more efficiently encoded block report format designed
> to improve performance and reduce GC load on the NN and DNs. The NN is not
> advertising this capability to the DNs so old-style reports are still being
> used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8876) Make hard coded parameters used by balancer and other tools configurable
Ming Ma created HDFS-8876:
-
Summary: Make hard coded parameters used by balancer and other
tools configurable
Key: HDFS-8876
URL: https://issues.apache.org/jira/browse/HDFS-8876
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ming Ma
Assignee: Chris Trezzo
During investigation of how to speed up balancer, at least to the level
specified by {{dfs.datanode.balance.bandwidthPerSec}}, we found that parameters
such as {{MAX_BLOCKS_SIZE_TO_FETCH}} and {{SOURCE_BLOCKS_MIN_SIZE}} are hard
coded. These parameters are related to block size and other configurable
parameters used by balancer. So least we should make it configurable. In the
longer term, it might be interesting to understand if we simplify all these
related configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HDFS-8875) Optimize the wait time in Balancer for federation scenario
[
https://issues.apache.org/jira/browse/HDFS-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-8875 started by Chris Trezzo.
--
> Optimize the wait time in Balancer for federation scenario
> --
>
> Key: HDFS-8875
> URL: https://issues.apache.org/jira/browse/HDFS-8875
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> Balancer has wait time between two consecutive iterations. That is to give
> some time for block movement to be fully committed ( return from replaceBlock
> doesn't mean the NN's blockmap has been updated and the block has been
> invalidated on the source node.).
> This wait time could be 23 seconds if {{dfs.heartbeat.interval}} is set to 10
> and {{dfs.namenode.replication.interval}} is to 3. In the case of federation,
> given we iterate through all namespaces in each iteration, this wait time
> becomes unnecessary as while balancer is processing the next namespace, it
> gives the previous namespace it just finished time to commit.
> In addition, Balancer calls {{Collections.shuffle(connectors);}} It doesn't
> seem necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8875) Optimize the wait time in Balancer for federation scenario
Ming Ma created HDFS-8875:
-
Summary: Optimize the wait time in Balancer for federation scenario
Key: HDFS-8875
URL: https://issues.apache.org/jira/browse/HDFS-8875
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ming Ma
Assignee: Chris Trezzo
Balancer has wait time between two consecutive iterations. That is to give some
time for block movement to be fully committed ( return from replaceBlock
doesn't mean the NN's blockmap has been updated and the block has been
invalidated on the source node.).
This wait time could be 23 seconds if {{dfs.heartbeat.interval}} is set to 10
and {{dfs.namenode.replication.interval}} is to 3. In the case of federation,
given we iterate through all namespaces in each iteration, this wait time
becomes unnecessary as while balancer is processing the next namespace, it
gives the previous namespace it just finished time to commit.
In addition, Balancer calls {{Collections.shuffle(connectors);}} It doesn't
seem necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HDFS-8874) Add DN metrics for balancer and other block movement scenarios
[
https://issues.apache.org/jira/browse/HDFS-8874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-8874 started by Chris Trezzo.
--
> Add DN metrics for balancer and other block movement scenarios
> --
>
> Key: HDFS-8874
> URL: https://issues.apache.org/jira/browse/HDFS-8874
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Chris Trezzo
>
> For balancer, mover and migrator (HDFS-8789), we want to know how close it is
> to the DN's throttling thresholds. Although DN has existing metrics such as
> {{BytesWritten}}, {{BytesRead}}, {{CopyBlockOpNumOps}} and
> {{ReplaceBlockOpNumOps}}, there is no metrics to indicate the number of bytes
> moved.
> We can add {{ReplaceBlockBytesWritten}} and {{CopyBlockBytesRead}} to account
> for the bytes moved in ReplaceBlock and CopyBlock operations. In addition, we
> can also add throttling metrics for {{DataTransferThrottler}} and
> {{BlockBalanceThrottler}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8827) Erasure Coding: Fix NPE when NameNode processes over-replicated striped blocks
[ https://issues.apache.org/jira/browse/HDFS-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662143#comment-14662143 ] Jing Zhao commented on HDFS-8827: - +1. I will commit it shortly. > Erasure Coding: Fix NPE when NameNode processes over-replicated striped blocks > -- > > Key: HDFS-8827 > URL: https://issues.apache.org/jira/browse/HDFS-8827 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Takuya Fukudome >Assignee: Walter Su > Attachments: HDFS-8827-HDFS-7285.04.patch, > HDFS-8827-HDFS-7285.05.patch, HDFS-8827.1.patch, HDFS-8827.2.patch, > HDFS-8827.3.patch, processing-over-replica-npe.log > > > In our test cluster, when namenode processed over replicated striped blocks, > null pointer exception(NPE) occurred. This happened under below situation: 1) > some datanodes shutdown. 2) namenode recovers block group which lost internal > blocks. 3) restart the stopped datanodes. 4) namenode processes over > replicated striped blocks. 5) NPE occurs > I think BlockPlacementPolicyDefault#chooseReplicaToDelete will return null in > this situation which causes this NPE problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8827) Erasure Coding: Fix NPE when NameNode processes over-replicated striped blocks
[ https://issues.apache.org/jira/browse/HDFS-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8827: Summary: Erasure Coding: Fix NPE when NameNode processes over-replicated striped blocks (was: Erasure Coding: When namenode processes over replicated striped block, NPE will be occur in ReplicationMonitor) > Erasure Coding: Fix NPE when NameNode processes over-replicated striped blocks > -- > > Key: HDFS-8827 > URL: https://issues.apache.org/jira/browse/HDFS-8827 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Takuya Fukudome >Assignee: Walter Su > Attachments: HDFS-8827-HDFS-7285.04.patch, > HDFS-8827-HDFS-7285.05.patch, HDFS-8827.1.patch, HDFS-8827.2.patch, > HDFS-8827.3.patch, processing-over-replica-npe.log > > > In our test cluster, when namenode processed over replicated striped blocks, > null pointer exception(NPE) occurred. This happened under below situation: 1) > some datanodes shutdown. 2) namenode recovers block group which lost internal > blocks. 3) restart the stopped datanodes. 4) namenode processes over > replicated striped blocks. 5) NPE occurs > I think BlockPlacementPolicyDefault#chooseReplicaToDelete will return null in > this situation which causes this NPE problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8775) SASL support for data transfer protocol in libhdfspp
[ https://issues.apache.org/jira/browse/HDFS-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662141#comment-14662141 ] Bob Hansen commented on HDFS-8775: -- In the tests, we need some negative tests: * How does sasl_digest handle: + blank requests/replies + malformed requests + Network errors > SASL support for data transfer protocol in libhdfspp > > > Key: HDFS-8775 > URL: https://issues.apache.org/jira/browse/HDFS-8775 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-8775.000.patch > > > This jira proposes to implement basic SASL support for the data transfer > protocol which allows libhdfspp to talk to secure clusters. > Support for encryption is deferred to subsequent jiras. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8815) DFS getStoragePolicy implementation using single RPC call
[
https://issues.apache.org/jira/browse/HDFS-8815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662136#comment-14662136
]
Surendra Singh Lilhore commented on HDFS-8815:
--
Thanks [~vinayrpet] for reviews and commit
Thanks [~arpitagarwal] for review..
> DFS getStoragePolicy implementation using single RPC call
> -
>
> Key: HDFS-8815
> URL: https://issues.apache.org/jira/browse/HDFS-8815
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Fix For: 2.8.0
>
> Attachments: HDFS-8815-001.patch, HDFS-8815-002.patch,
> HDFS-8815-003.patch, HDFS-8815-004.patch
>
>
> HADOOP-12161 introduced a new {{FileSystem#getStoragePolicy}} call. The DFS
> implementation of the call requires two RPC calls, the first to fetch the
> storage policy ID and the second to fetch the policy suite to map the policy
> ID to a {{BlockStoragePolicySpi}}.
> Fix the implementation to require a single RPC call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8805) Archival Storage: getStoragePolicy should not need superuser privilege
[
https://issues.apache.org/jira/browse/HDFS-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662129#comment-14662129
]
Jing Zhao commented on HDFS-8805:
-
Thanks for updating the patch, [~brahmareddy]! The new patch removes the
{{includeStoragePolicy}} parameter from both {{getFileInfo}} methods. Actually
we still need the parameter for the first one: {{getFileInfo(FSDirectory,
String, INodesInPath, boolean, boolean)}}, considering it is called by
{{getAuditFileInfo}} which does not require the storage policy information. We
only need to remove the parameter from the other getFileInfo:
{{getFileInfo(FSDirectory, String, boolean, boolean, boolean)}}.
> Archival Storage: getStoragePolicy should not need superuser privilege
> --
>
> Key: HDFS-8805
> URL: https://issues.apache.org/jira/browse/HDFS-8805
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover, namenode
>Reporter: Hui Zheng
>Assignee: Brahma Reddy Battula
> Fix For: 2.6.0
>
> Attachments: HDFS-8805-002.patch, HDFS-8805-003.patch, HDFS-8805.patch
>
>
> The result of getStoragePolicy command is always 'unspecified' even we has
> set a StoragePolicy on a directory.But the real placement of blocks is
> correct.
> The result of fsck is not correct either.
> {code}
> $ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD
> Set storage policy COLD on /tmp/cold
> $ hdfs storagepolicies -getStoragePolicy -path /tmp/cold
> The storage policy of /tmp/cold is unspecified
> $ hdfs fsck -storagepolicies /tmp/cold
> Blocks NOT satisfying the specified storage policy:
> Storage Policy Specified Storage Policy # of blocks
> % of blocks
> ARCHIVE:4(COLD) HOT 5
>55.5556%
> ARCHIVE:3(COLD) HOT 4
>44.%
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8874) Add DN metrics for balancer and other block movement scenarios
Ming Ma created HDFS-8874:
-
Summary: Add DN metrics for balancer and other block movement
scenarios
Key: HDFS-8874
URL: https://issues.apache.org/jira/browse/HDFS-8874
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ming Ma
Assignee: Chris Trezzo
For balancer, mover and migrator (HDFS-8789), we want to know how close it is
to the DN's throttling thresholds. Although DN has existing metrics such as
{{BytesWritten}}, {{BytesRead}}, {{CopyBlockOpNumOps}} and
{{ReplaceBlockOpNumOps}}, there is no metrics to indicate the number of bytes
moved.
We can add {{ReplaceBlockBytesWritten}} and {{CopyBlockBytesRead}} to account
for the bytes moved in ReplaceBlock and CopyBlock operations. In addition, we
can also add throttling metrics for {{DataTransferThrottler}} and
{{BlockBalanceThrottler}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8775) SASL support for data transfer protocol in libhdfspp
[ https://issues.apache.org/jira/browse/HDFS-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662123#comment-14662123 ] Bob Hansen commented on HDFS-8775: -- The lack of comments makes it very hard to verify that the code is doing what it should. All I can say is "the code kinda does what it looks like it is trying to do." Especially in the digest handshake protocol, a definition of the correct behavior would go a long way to being able to confirm that the behavior is correct. Things you might want to look into: sasl_authenticator.h: * What is the TEST_mock_cnonce for? Do we have a mechanism to strip it out of release code? Can we #ifdef it out? sasl_digest.h: * Why is kMaxBufferSize 64k? Does that relate to some other constant (in which case, can we import the symbol?) or is it just "64k should be enough for anybody"? * GenerateCNonce(): is RAND_pseudo good enough for security in this case, or should we be using (transitively) /dev/random? * ParseFirstChallenge(): This will silently accept many malformed requests like + foo + foo,bar,baz + ~~~=~~~ * ParseFirstChallenge(): requires a "nonce" field in the message, but doesn't use it * GetMD5Digest(): we should check the return values of the OpenSSL calls > SASL support for data transfer protocol in libhdfspp > > > Key: HDFS-8775 > URL: https://issues.apache.org/jira/browse/HDFS-8775 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-8775.000.patch > > > This jira proposes to implement basic SASL support for the data transfer > protocol which allows libhdfspp to talk to secure clusters. > Support for encryption is deferred to subsequent jiras. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8750) FIleSystem does not honor Configuration.getClassLoader() while loading FileSystem implementations
[ https://issues.apache.org/jira/browse/HDFS-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu updated HDFS-8750: --- Attachment: HDFS-8750.004.patch > FIleSystem does not honor Configuration.getClassLoader() while loading > FileSystem implementations > - > > Key: HDFS-8750 > URL: https://issues.apache.org/jira/browse/HDFS-8750 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs, HDFS >Reporter: Himanshu >Assignee: Himanshu > Attachments: HDFS-8750.001.patch, HDFS-8750.002.patch, > HDFS-8750.003.patch, HDFS-8750.004.patch > > > In FileSystem.loadFileSystems(), at > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L2652 > a "scheme" -> "FileSystem implementation" map is created from the jars > available on classpath. It uses Thread.currentThread().getClassLoader() via > ServiceLoader.load(FileSystem.class) > Instead, loadFileSystems() should take Configuration as an argument and > should first check if a classloader is configured in > configuration.getClassLoader(), if yes then > ServiceLoader.load(FileSystem.class, configuration.getClassLoader()) should > be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8772) Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-8772:
--
Resolution: Fixed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Thanks Walter again for working on this, committed to trunk and branch-2.
> Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails
>
>
> Key: HDFS-8772
> URL: https://issues.apache.org/jira/browse/HDFS-8772
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: 2.8.0
>
> Attachments: HDFS-8772-branch-2.04.patch, HDFS-8772.01.patch,
> HDFS-8772.02.patch, HDFS-8772.03.patch, HDFS-8772.04.patch
>
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/11596/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11598/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11599/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11600/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11606/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11608/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11612/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11618/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11650/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11655/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11659/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11663/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11664/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11667/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11669/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11676/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11677/testReport/
> {noformat}
> java.lang.AssertionError: expected:<0> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot.testDatanodeRestarts(TestStandbyIsHot.java:188)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8873) throttle directoryScanner
Nathan Roberts created HDFS-8873: Summary: throttle directoryScanner Key: HDFS-8873 URL: https://issues.apache.org/jira/browse/HDFS-8873 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.7.1 Reporter: Nathan Roberts The new 2-level directory layout can make directory scans expensive in terms of disk seeks (see HDFS-8791) for details. It would be good if the directoryScanner() had a configurable duty cycle that would reduce its impact on disk performance (much like the approach in HDFS-8617). Without such a throttle, disks can go 100% busy for many minutes at a time (assuming the common case of all inodes in cache but no directory blocks cached, 64K seeks are required for full directory listing which translates to 655 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8622) Implement GETCONTENTSUMMARY operation for WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662021#comment-14662021
]
Hadoop QA commented on HDFS-8622:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 59s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 59s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 53s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 3m 6s | Site still builds. |
| {color:red}-1{color} | checkstyle | 1m 23s | The applied patch generated 1
new checkstyle issues (total was 123, now 124). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 22s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 37s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 143m 31s | Tests failed in hadoop-hdfs. |
| | | 195m 0s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| Timed out tests |
org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions |
| | org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749248/HDFS-8622-08.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / b6265d3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11936/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11936/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11936/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11936/console |
This message was automatically generated.
> Implement GETCONTENTSUMMARY operation for WebImageViewer
>
>
> Key: HDFS-8622
> URL: https://issues.apache.org/jira/browse/HDFS-8622
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jagadesh Kiran N
>Assignee: Jagadesh Kiran N
> Attachments: HDFS-8622-00.patch, HDFS-8622-01.patch,
> HDFS-8622-02.patch, HDFS-8622-03.patch, HDFS-8622-04.patch,
> HDFS-8622-05.patch, HDFS-8622-06.patch, HDFS-8622-07.patch, HDFS-8622-08.patch
>
>
> it would be better for administrators if {code} GETCONTENTSUMMARY {code} are
> supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8828) Utilize Snapshot diff report to build copy list in distcp
[
https://issues.apache.org/jira/browse/HDFS-8828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661976#comment-14661976
]
Yongjun Zhang commented on HDFS-8828:
-
Thanks [~yufeigu] for the new rev and [~jingzhao] for looking into.
I did more review of rev 3, below are my comments:
# {{static DiffInfo[] getDiffsForListBuilding(SnapshotDiffReport report)}
## Please add one line in javadoc to state that DELETE is handled in
{{DistCpSync#sync}}
## Add a comment for {{if(entry.getSourcePath().length <= 0)}}, what it means
when the source path is <= 0.
# {{isParentOf(Path parent, Path child) }}
## it need to check whether the matching point is a path delimiter; in
addition,
## {{if (childPath.equals(parentPath))}} can be optimized with checking length
instead of content, once the {{startWith}} matches.
# In {{getRenameItem:
## The comment in the body: suggest to change
// The same path string may appear in:
// 1. both renamed and modified snapshot diff entries,
// 2. both renamed and created snapshot diff entries
// Case 1 is the about same file/directory, whereas case 2 is about two
different files/directories.
// We are finding case 1 here, thus we check against DiffType.MODIFY.
## Add a space in {{}else if}}
## Add a comment in the {{else if}} branch saying that this item can be either
modified or created.
# In {{static Path getTargetPath(Path sourcePath, DiffInfo renameItem)}}
## change "by the rename item" to "based on the rename item"
## {{StringBuffer sb = new StringBuffer(sourcePath.toString());}} to after the
first return statement.
# In {{getDiffList(DistCpOptions options)}},
## The two calls to {{finalListWithTarget.add(diff);}} in
{{getDiffList(DistCpOptions options)}} can be consolidated.
## The check of {{if (diff.target != null)}} meant to check whether the entry
is not rename, can we change it to check the type instead of target to be more
clear?
# Need some improvement in javadoc for {{public void
doBuildListingWithSnapshotDiff}}. Suggest to replace "We need to handle rename
item here since some created/modified items could be renamed as well." with "An
item can be created/modified and renamed, in which case, the target path is put
into the list".
I did not review the test code yet, hope you can revisit and can catch
something yourself. I will review after this round.
Thanks.
> Utilize Snapshot diff report to build copy list in distcp
> -
>
> Key: HDFS-8828
> URL: https://issues.apache.org/jira/browse/HDFS-8828
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: distcp, snapshots
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: HDFS-8828.001.patch, HDFS-8828.002.patch,
> HDFS-8828.003.patch
>
>
> Some users reported huge time cost to build file copy list in distcp. (30
> hours for 1.6M files). We can leverage snapshot diff report to build file
> copy list including files/dirs which are changes only between two snapshots
> (or a snapshot and a normal dir). It speed up the process in two folds: 1.
> less copy list building time. 2. less file copy MR jobs.
> HDFS snapshot diff report provide information about file/directory creation,
> deletion, rename and modification between two snapshots or a snapshot and a
> normal directory. HDFS-7535 synchronize deletion and rename, then fallback to
> the default distcp. So it still relies on default distcp to building complete
> list of files under the source dir. This patch only puts creation and
> modification files into the copy list based on snapshot diff report. We can
> minimize the number of files to copy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8623) Refactor NameNode handling of invalid, corrupt, and under-recovery blocks
[ https://issues.apache.org/jira/browse/HDFS-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661914#comment-14661914 ] Hudson commented on HDFS-8623: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2207 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2207/]) Revert "HDFS-8623. Refactor NameNode handling of invalid, corrupt, and under-recovery blocks. Contributed by Zhe Zhang." (jing9: rev 663eba0ab1c73b45f98e46ffc87ad8fd91584046) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstructionContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNodeCount.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java > Refactor NameNode handling of invalid, corrupt, and under-recovery blocks > - > > Key: HDFS-8623 > URL: https://issues.apache.org/jira/browse/HDFS-8623 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.7.0 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8623.00.patch, HDFS-8623.01.patch, > HDFS-8623.02.patch, HDFS-8623.03.patch, HDFS-8623.04.patch, > HDFS-8623.05.patch, HDFS-8623.06.patch > > > In order to support striped blocks in invalid, corrupt, and under-recovery > blocks handling, HDFS-7907 introduces some refactors. This JIRA aims to merge > these changes to trunk first to minimize and cleanup HDFS-7285 merge patch so > that it only contains striping/EC logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661912#comment-14661912
]
Hudson commented on HDFS-8856:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2207 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2207/])
HDFS-8856. Make LeaseManager#countPath O(1). (Contributed by Arpit Agarwal)
(arp: rev 6d4eee718a3fe1450a627128eb94728011bd9b68)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0
>
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch, HDFS-8856.04.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Refactor BlockInfo class hierarchy with static helper class
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661913#comment-14661913
]
Hudson commented on HDFS-8499:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2207 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2207/])
Revert "HDFS-8499. Refactor BlockInfo class hierarchy with static helper class.
Contributed by Zhe Zhang." (jing9: rev f4c523b69ba55b1fd35e8995c3011a9f546ac835)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBlockUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileUnderConstructionFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstructionContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoUnderConstruction.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirTruncateOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ContiguousBlockStorageOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotTestHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
> Refactor BlockInfo class hierarchy with static helper class
> ---
>
> Key: HDFS-8499
> URL: https://issues.apache.org/jira/browse/HDFS-8499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
> HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
> HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
> HDFS-8499.UCFeature.patch, HDFS-bistriped.patch,
> revert-HDFS-8499-HDFS-8623.patch
>
>
> In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
> common abstraction for striped and contiguous UC
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661888#comment-14661888
]
Hudson commented on HDFS-8856:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2226 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2226/])
HDFS-8856. Make LeaseManager#countPath O(1). (Contributed by Arpit Agarwal)
(arp: rev 6d4eee718a3fe1450a627128eb94728011bd9b68)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0
>
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch, HDFS-8856.04.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8623) Refactor NameNode handling of invalid, corrupt, and under-recovery blocks
[ https://issues.apache.org/jira/browse/HDFS-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661890#comment-14661890 ] Hudson commented on HDFS-8623: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2226 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2226/]) Revert "HDFS-8623. Refactor NameNode handling of invalid, corrupt, and under-recovery blocks. Contributed by Zhe Zhang." (jing9: rev 663eba0ab1c73b45f98e46ffc87ad8fd91584046) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNodeCount.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstructionContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Refactor NameNode handling of invalid, corrupt, and under-recovery blocks > - > > Key: HDFS-8623 > URL: https://issues.apache.org/jira/browse/HDFS-8623 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.7.0 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8623.00.patch, HDFS-8623.01.patch, > HDFS-8623.02.patch, HDFS-8623.03.patch, HDFS-8623.04.patch, > HDFS-8623.05.patch, HDFS-8623.06.patch > > > In order to support striped blocks in invalid, corrupt, and under-recovery > blocks handling, HDFS-7907 introduces some refactors. This JIRA aims to merge > these changes to trunk first to minimize and cleanup HDFS-7285 merge patch so > that it only contains striping/EC logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Refactor BlockInfo class hierarchy with static helper class
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661889#comment-14661889
]
Hudson commented on HDFS-8499:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2226 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2226/])
Revert "HDFS-8499. Refactor BlockInfo class hierarchy with static helper class.
Contributed by Zhe Zhang." (jing9: rev f4c523b69ba55b1fd35e8995c3011a9f546ac835)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstructionContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileUnderConstructionFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBlockUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirTruncateOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ContiguousBlockStorageOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotTestHelper.java
> Refactor BlockInfo class hierarchy with static helper class
> ---
>
> Key: HDFS-8499
> URL: https://issues.apache.org/jira/browse/HDFS-8499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
> HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
> HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
> HDFS-8499.UCFeature.patch, HDFS-bistriped.patch,
> revert-HDFS-8499-HDFS-8623.patch
>
>
> In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
> common abstraction for striped and cont
[jira] [Updated] (HDFS-8872) Reporting of missing blocks is different in fsck and namenode ui/metasave
[ https://issues.apache.org/jira/browse/HDFS-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-8872: - Description: Namenode ui and metasave will not report a block as missing if the only replica is on decommissioning/decomissioned node while fsck will show it as MISSING. Since decommissioned node can be formatted/removed anytime, we can actually lose the block. Its better to alert on namenode ui if the only copy is on decomissioned/decommissioning node. was: Namenode ui and metasave will not report a block as missing if the only replica is on decommissioning/decomissioned node where fsck will show it as MISSING. Since decommissioned node can be formatted/removed anytime, we can actually lose the block. Its better to alert on namenode ui if the only copy is on decomissioned/decommissioning node. > Reporting of missing blocks is different in fsck and namenode ui/metasave > - > > Key: HDFS-8872 > URL: https://issues.apache.org/jira/browse/HDFS-8872 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Fix For: 2.7.2 > > > Namenode ui and metasave will not report a block as missing if the only > replica is on decommissioning/decomissioned node while fsck will show it as > MISSING. > Since decommissioned node can be formatted/removed anytime, we can actually > lose the block. > Its better to alert on namenode ui if the only copy is on > decomissioned/decommissioning node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661861#comment-14661861
]
Hudson commented on HDFS-8856:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #277 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/277/])
HDFS-8856. Make LeaseManager#countPath O(1). (Contributed by Arpit Agarwal)
(arp: rev 6d4eee718a3fe1450a627128eb94728011bd9b68)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0
>
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch, HDFS-8856.04.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Refactor BlockInfo class hierarchy with static helper class
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661862#comment-14661862
]
Hudson commented on HDFS-8499:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #277 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/277/])
Revert "HDFS-8499. Refactor BlockInfo class hierarchy with static helper class.
Contributed by Zhe Zhang." (jing9: rev f4c523b69ba55b1fd35e8995c3011a9f546ac835)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstructionContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotTestHelper.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ContiguousBlockStorageOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileUnderConstructionFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirTruncateOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBlockUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
> Refactor BlockInfo class hierarchy with static helper class
> ---
>
> Key: HDFS-8499
> URL: https://issues.apache.org/jira/browse/HDFS-8499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
> HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
> HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
> HDFS-8499.UCFeature.patch, HDFS-bistriped.patch,
> revert-HDFS-8499-HDFS-8623.patch
>
>
> In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
> common abstraction for stripe
[jira] [Commented] (HDFS-8623) Refactor NameNode handling of invalid, corrupt, and under-recovery blocks
[ https://issues.apache.org/jira/browse/HDFS-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661863#comment-14661863 ] Hudson commented on HDFS-8623: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #277 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/277/]) Revert "HDFS-8623. Refactor NameNode handling of invalid, corrupt, and under-recovery blocks. Contributed by Zhe Zhang." (jing9: rev 663eba0ab1c73b45f98e46ffc87ad8fd91584046) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNodeCount.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstructionContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java > Refactor NameNode handling of invalid, corrupt, and under-recovery blocks > - > > Key: HDFS-8623 > URL: https://issues.apache.org/jira/browse/HDFS-8623 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.7.0 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8623.00.patch, HDFS-8623.01.patch, > HDFS-8623.02.patch, HDFS-8623.03.patch, HDFS-8623.04.patch, > HDFS-8623.05.patch, HDFS-8623.06.patch > > > In order to support striped blocks in invalid, corrupt, and under-recovery > blocks handling, HDFS-7907 introduces some refactors. This JIRA aims to merge > these changes to trunk first to minimize and cleanup HDFS-7285 merge patch so > that it only contains striping/EC logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Refactor BlockInfo class hierarchy with static helper class
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661839#comment-14661839
]
Hudson commented on HDFS-8499:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #269 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/269/])
Revert "HDFS-8499. Refactor BlockInfo class hierarchy with static helper class.
Contributed by Zhe Zhang." (jing9: rev f4c523b69ba55b1fd35e8995c3011a9f546ac835)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileUnderConstructionFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ContiguousBlockStorageOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotTestHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirTruncateOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBlockUnderConstruction.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstructionContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoUnderConstruction.java
> Refactor BlockInfo class hierarchy with static helper class
> ---
>
> Key: HDFS-8499
> URL: https://issues.apache.org/jira/browse/HDFS-8499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
> HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
> HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
> HDFS-8499.UCFeature.patch, HDFS-bistriped.patch,
> revert-HDFS-8499-HDFS-8623.patch
>
>
> In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
> common abstraction for striped and cont
[jira] [Commented] (HDFS-8623) Refactor NameNode handling of invalid, corrupt, and under-recovery blocks
[ https://issues.apache.org/jira/browse/HDFS-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661840#comment-14661840 ] Hudson commented on HDFS-8623: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #269 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/269/]) Revert "HDFS-8623. Refactor NameNode handling of invalid, corrupt, and under-recovery blocks. Contributed by Zhe Zhang." (jing9: rev 663eba0ab1c73b45f98e46ffc87ad8fd91584046) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstructionContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNodeCount.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java > Refactor NameNode handling of invalid, corrupt, and under-recovery blocks > - > > Key: HDFS-8623 > URL: https://issues.apache.org/jira/browse/HDFS-8623 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.7.0 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8623.00.patch, HDFS-8623.01.patch, > HDFS-8623.02.patch, HDFS-8623.03.patch, HDFS-8623.04.patch, > HDFS-8623.05.patch, HDFS-8623.06.patch > > > In order to support striped blocks in invalid, corrupt, and under-recovery > blocks handling, HDFS-7907 introduces some refactors. This JIRA aims to merge > these changes to trunk first to minimize and cleanup HDFS-7285 merge patch so > that it only contains striping/EC logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661838#comment-14661838
]
Hudson commented on HDFS-8856:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #269 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/269/])
HDFS-8856. Make LeaseManager#countPath O(1). (Contributed by Arpit Agarwal)
(arp: rev 6d4eee718a3fe1450a627128eb94728011bd9b68)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0
>
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch, HDFS-8856.04.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-8836) Skip newline on empty files with getMerge -nl
[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru reassigned HDFS-8836: - Assignee: kanaka kumar avvaru (was: Jagadesh Kiran N) > Skip newline on empty files with getMerge -nl > - > > Key: HDFS-8836 > URL: https://issues.apache.org/jira/browse/HDFS-8836 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.6.0, 2.7.1 >Reporter: Jan Filipiak >Assignee: kanaka kumar avvaru >Priority: Trivial > > Hello everyone, > I recently was in the need of using the new line option -nl with getMerge > because the files I needed to merge simply didn't had one. I was merging all > the files from one directory and unfortunately this directory also included > empty files, which effectively led to multiple newlines append after some > files. I needed to remove them manually afterwards. > In this situation it is maybe good to have another argument that allows > skipping empty files. > Thing one could try to implement this feature: > The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't > return the number of bytes copied which would be convenient as one could > skip append the new line when 0 bytes where copied or one would check the > file size before. > I posted this Idea on the mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E > but I didn't really get many responses, so I thought I my try this way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8859) Improve DataNode (ReplicaMap) memory footprint to save about 45%
[
https://issues.apache.org/jira/browse/HDFS-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661761#comment-14661761
]
Hadoop QA commented on HDFS-8859:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 46s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 41s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 14s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 24s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 22m 31s | Tests failed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 188m 11s | Tests failed in hadoop-hdfs. |
| | | 251m 55s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.net.TestNetUtils |
| | hadoop.ha.TestZKFailoverController |
| | hadoop.hdfs.server.namenode.TestParallelImageWrite |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.server.namenode.ha.TestStandbyIsHot |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.TestDatanodeLayoutUpgrade |
| | hadoop.hdfs.server.namenode.ha.TestDNFencing |
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749223/HDFS-8859.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b6265d3 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11934/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11934/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11934/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11934/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11934/console |
This message was automatically generated.
> Improve DataNode (ReplicaMap) memory footprint to save about 45%
>
>
> Key: HDFS-8859
> URL: https://issues.apache.org/jira/browse/HDFS-8859
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Yi Liu
>Assignee: Yi Liu
>Priority: Critical
> Attachments: HDFS-8859.001.patch
>
>
> By using following approach we can save about *45%* memory footprint for each
> block replica in DataNode memory (This JIRA only talks about *ReplicaMap* in
> DataNode), the details are:
> In ReplicaMap,
> {code}
> private final Map> map =
> new HashMap>();
> {code}
> Currently we use a HashMap {{Map}} to store the replicas
> in memory. The key is block id of the block replica which is already
> included in {{ReplicaInfo}}, so this memory can be saved. Also HashMap Entry
> has a object overhead. We can implement a lightweight Set which is similar
> to {{LightWeightGSet}}, but not a fixed size ({{LightWeightGSet}} uses fix
> size for the entries array, usually it's a big value, an example is
> {{BlocksMap}}, this can avoid full gc since no need to resize), also we
> should be able to get Element through key.
> Following is comparison of memory footprint If we implement a lightweight set
> as described:
> We can save:
> {noformat}
> SIZE (bytes) ITEM
> 20The Key: Long (12 bytes object overhead + 8
> bytes long)
> 12HashMap Entry object overhead
> 4 reference to the key in Entry
> 4 r
[jira] [Updated] (HDFS-8622) Implement GETCONTENTSUMMARY operation for WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadesh Kiran N updated HDFS-8622:
---
Attachment: HDFS-8622-08.patch
Hi [~ajisakaa] ,updated your comments.please review the patch
> Implement GETCONTENTSUMMARY operation for WebImageViewer
>
>
> Key: HDFS-8622
> URL: https://issues.apache.org/jira/browse/HDFS-8622
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jagadesh Kiran N
>Assignee: Jagadesh Kiran N
> Attachments: HDFS-8622-00.patch, HDFS-8622-01.patch,
> HDFS-8622-02.patch, HDFS-8622-03.patch, HDFS-8622-04.patch,
> HDFS-8622-05.patch, HDFS-8622-06.patch, HDFS-8622-07.patch, HDFS-8622-08.patch
>
>
> it would be better for administrators if {code} GETCONTENTSUMMARY {code} are
> supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8852) HDFS architecture documentation of version 2.x is outdated about append write support
[ https://issues.apache.org/jira/browse/HDFS-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661679#comment-14661679 ] Ajith S commented on HDFS-8852: --- +1 will update accordingly. Thanks [~ajisakaa] > HDFS architecture documentation of version 2.x is outdated about append write > support > - > > Key: HDFS-8852 > URL: https://issues.apache.org/jira/browse/HDFS-8852 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Reporter: Hong Dai Thanh >Assignee: Ajith S > Labels: newbie > > In the [latest version of the > documentation|http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Simple_Coherency_Model], > and also documentation for all releases with version 2, it’s mentioned that > “A file once created, written, and closed need not be changed. “ and “There > is a plan to support appending-writes to files in the future.” > > However, as far as I know, HDFS has supported append write since 0.21, based > on [HDFS-265|https://issues.apache.org/jira/browse/HDFS-265] and [the old > version of the documentation in > 2012|https://web.archive.org/web/20121221171824/http://hadoop.apache.org/docs/hdfs/current/hdfs_design.html#Appending-Writes+and+File+Syncs] > Various posts on the Internet also suggests that append write has been > available in HDFS, and will always be available in Hadoop version 2 branch. > > Can we update the documentation to reflect the current status? > (Please also review whether the documentation should also be updated for > version 0.21 and above, and the version 1.x branch) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661639#comment-14661639
]
Hudson commented on HDFS-8856:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1010 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1010/])
HDFS-8856. Make LeaseManager#countPath O(1). (Contributed by Arpit Agarwal)
(arp: rev 6d4eee718a3fe1450a627128eb94728011bd9b68)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0
>
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch, HDFS-8856.04.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8623) Refactor NameNode handling of invalid, corrupt, and under-recovery blocks
[ https://issues.apache.org/jira/browse/HDFS-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661641#comment-14661641 ] Hudson commented on HDFS-8623: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1010 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1010/]) Revert "HDFS-8623. Refactor NameNode handling of invalid, corrupt, and under-recovery blocks. Contributed by Zhe Zhang." (jing9: rev 663eba0ab1c73b45f98e46ffc87ad8fd91584046) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstructionContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNodeCount.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Refactor NameNode handling of invalid, corrupt, and under-recovery blocks > - > > Key: HDFS-8623 > URL: https://issues.apache.org/jira/browse/HDFS-8623 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.7.0 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8623.00.patch, HDFS-8623.01.patch, > HDFS-8623.02.patch, HDFS-8623.03.patch, HDFS-8623.04.patch, > HDFS-8623.05.patch, HDFS-8623.06.patch > > > In order to support striped blocks in invalid, corrupt, and under-recovery > blocks handling, HDFS-7907 introduces some refactors. This JIRA aims to merge > these changes to trunk first to minimize and cleanup HDFS-7285 merge patch so > that it only contains striping/EC logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Refactor BlockInfo class hierarchy with static helper class
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661640#comment-14661640
]
Hudson commented on HDFS-8499:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1010 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1010/])
Revert "HDFS-8499. Refactor BlockInfo class hierarchy with static helper class.
Contributed by Zhe Zhang." (jing9: rev f4c523b69ba55b1fd35e8995c3011a9f546ac835)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotTestHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBlockUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ContiguousBlockStorageOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileUnderConstructionFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstructionContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirTruncateOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoUnderConstruction.java
> Refactor BlockInfo class hierarchy with static helper class
> ---
>
> Key: HDFS-8499
> URL: https://issues.apache.org/jira/browse/HDFS-8499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
> HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
> HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
> HDFS-8499.UCFeature.patch, HDFS-bistriped.patch,
> revert-HDFS-8499-HDFS-8623.patch
>
>
> In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
> common abstraction for striped and contiguous UC
[jira] [Commented] (HDFS-8623) Refactor NameNode handling of invalid, corrupt, and under-recovery blocks
[ https://issues.apache.org/jira/browse/HDFS-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661631#comment-14661631 ] Hudson commented on HDFS-8623: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #280 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/280/]) Revert "HDFS-8623. Refactor NameNode handling of invalid, corrupt, and under-recovery blocks. Contributed by Zhe Zhang." (jing9: rev 663eba0ab1c73b45f98e46ffc87ad8fd91584046) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNodeCount.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstructionContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java > Refactor NameNode handling of invalid, corrupt, and under-recovery blocks > - > > Key: HDFS-8623 > URL: https://issues.apache.org/jira/browse/HDFS-8623 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.7.0 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8623.00.patch, HDFS-8623.01.patch, > HDFS-8623.02.patch, HDFS-8623.03.patch, HDFS-8623.04.patch, > HDFS-8623.05.patch, HDFS-8623.06.patch > > > In order to support striped blocks in invalid, corrupt, and under-recovery > blocks handling, HDFS-7907 introduces some refactors. This JIRA aims to merge > these changes to trunk first to minimize and cleanup HDFS-7285 merge patch so > that it only contains striping/EC logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661629#comment-14661629
]
Hudson commented on HDFS-8856:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #280 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/280/])
HDFS-8856. Make LeaseManager#countPath O(1). (Contributed by Arpit Agarwal)
(arp: rev 6d4eee718a3fe1450a627128eb94728011bd9b68)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0
>
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch, HDFS-8856.04.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Refactor BlockInfo class hierarchy with static helper class
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661630#comment-14661630
]
Hudson commented on HDFS-8499:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #280 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/280/])
Revert "HDFS-8499. Refactor BlockInfo class hierarchy with static helper class.
Contributed by Zhe Zhang." (jing9: rev f4c523b69ba55b1fd35e8995c3011a9f546ac835)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ContiguousBlockStorageOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBlockUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotTestHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileUnderConstructionFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirTruncateOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstructionContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
> Refactor BlockInfo class hierarchy with static helper class
> ---
>
> Key: HDFS-8499
> URL: https://issues.apache.org/jira/browse/HDFS-8499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
> HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
> HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
> HDFS-8499.UCFeature.patch, HDFS-bistriped.patch,
> revert-HDFS-8499-HDFS-8623.patch
>
>
> In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
> common abstraction for striped and cont
[jira] [Commented] (HDFS-8784) BlockInfo#numNodes should be numStorages
[ https://issues.apache.org/jira/browse/HDFS-8784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661619#comment-14661619 ] Jagadesh Kiran N commented on HDFS-8784: hi [~arpitagarwal] , please review the same > BlockInfo#numNodes should be numStorages > > > Key: HDFS-8784 > URL: https://issues.apache.org/jira/browse/HDFS-8784 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Jagadesh Kiran N > Attachments: HDFS-8784-00.patch, HDFS-8784-01.patch > > > The method actually returns the number of storages holding a block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661600#comment-14661600
]
Rakesh R commented on HDFS-8854:
Nice Work [~walter.k.su]!
I've referred {{HDFS-8854-HDFS-7285.02.patch}} and have few comments, please
see:
# Since ErasureCodingPolicy has {{cellSize}}, can we avoid separate variable
{{cellSize}} in DFSStripedInputStream.java, DFSStripedOutputStream.java classes.
# typo: {{ecPoilcy}}, {{ecPoilcies}}. Please correct this to {{ecPolicy}},
{{ecPolicies}}
# one general suggestion - while commenting or logging or #toString() instead
of {{EC polices}}, good to use {{erasure coding policies}} explicitly. I
remember sometime back there were few discussions to use this way
> Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
> --
>
> Key: HDFS-8854
> URL: https://issues.apache.org/jira/browse/HDFS-8854
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8854-Consolidated-20150806.02.txt,
> HDFS-8854-HDFS-7285.00.patch, HDFS-8854-HDFS-7285.01.patch,
> HDFS-8854-HDFS-7285.02.patch, HDFS-8854.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661552#comment-14661552
]
Hadoop QA commented on HDFS-8854:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | patch | 0m 1s | The patch file was not named
according to hadoop's naming conventions. Please see
https://wiki.apache.org/hadoop/HowToContribute for instructions. |
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749226/HDFS-8854-Consolidated-20150806.02.txt
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b6265d3 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11935/console |
This message was automatically generated.
> Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
> --
>
> Key: HDFS-8854
> URL: https://issues.apache.org/jira/browse/HDFS-8854
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8854-Consolidated-20150806.02.txt,
> HDFS-8854-HDFS-7285.00.patch, HDFS-8854-HDFS-7285.01.patch,
> HDFS-8854-HDFS-7285.02.patch, HDFS-8854.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8854: Attachment: HDFS-8854-Consolidated-20150806.02.txt > Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs > -- > > Key: HDFS-8854 > URL: https://issues.apache.org/jira/browse/HDFS-8854 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Walter Su >Assignee: Walter Su > Attachments: HDFS-8854-Consolidated-20150806.02.txt, > HDFS-8854-HDFS-7285.00.patch, HDFS-8854-HDFS-7285.01.patch, > HDFS-8854-HDFS-7285.02.patch, HDFS-8854.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
