[jira] [Commented] (HDFS-7716) Erasure Coding: extend BlockInfo to handle EC info
[
https://issues.apache.org/jira/browse/HDFS-7716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306789#comment-14306789
]
Jing Zhao commented on HDFS-7716:
-
Thanks for the comments, Zhe! I'm currently still working on a patch. For your
first two questions, I still plan to use triplets since currently the block
report processing logic depends on this data structure. The difference between
{{BlockReplicationInfo}} and {{BlockGroupInfo}} mainly lies in that
{{BlockGroupInfo}} requires the triplets to be sorted based on the block
sequence in the group. Thus the {{addStorage}}, {{removeStorage}} and
{{numNodes}} will be different in these two classes. In the meanwhile, to
handle over-replicated blocks in a block group (due to temporary datanode
failure etc.) we may need extra data structure in {{BlockGroupInfo}} to
explicitly record the block index of each triplet.
> Erasure Coding: extend BlockInfo to handle EC info
> --
>
> Key: HDFS-7716
> URL: https://issues.apache.org/jira/browse/HDFS-7716
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
>
> The current BlockInfo's implementation only supports the replication
> mechanism. To use the same blocksMap handling block group and its data/parity
> blocks, we need to define a new BlockGroupInfo class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7716) Erasure Coding: extend BlockInfo to handle EC info
[
https://issues.apache.org/jira/browse/HDFS-7716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306764#comment-14306764
]
Zhe Zhang commented on HDFS-7716:
-
Thanks Jing! The high-level plan looks good. Below are some detailed items we
can further discuss:
# Shall we use {{triplets}} to record block locations in an block group?
# If we use {{triplets}} for both replicated blocks and EC block groups, I
don't clearly see which fields and methods to offload to
{{BlockReplicationInfo}}. Thoughts?
# We discussed recording the EC schema in {{BlockGroupInfo}} to avoid
frequently checking INode. Before HDFS-7337 we can simply record the number of
data and parity blocks (2 {{short}} variables).
# An INode feature for {{BlockGroupInfo}} sounds good. Before our meetup last
week, HDFS-7339's [patch |
https://issues.apache.org/jira/secure/attachment/12693731/HDFS-7339-006.patch]
used that approach. We might be able to reuse some code.
# [~andrew.wang] and I have discussed how to support snapshots when a file is
converted between replicated and EC forms. If we keep both the regular
{{BlockInfo}} array and the {{BlockGroupInfo}} feature, we should be able to
calculate snapshots.
> Erasure Coding: extend BlockInfo to handle EC info
> --
>
> Key: HDFS-7716
> URL: https://issues.apache.org/jira/browse/HDFS-7716
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
>
> The current BlockInfo's implementation only supports the replication
> mechanism. To use the same blocksMap handling block group and its data/parity
> blocks, we need to define a new BlockGroupInfo class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7738) Add more negative tests for truncate
[
https://issues.apache.org/jira/browse/HDFS-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306717#comment-14306717
]
Hadoop QA commented on HDFS-7738:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696653/h7738_20150204.patch
against trunk revision 0b567f4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestFileCreation
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9436//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9436//console
This message is automatically generated.
> Add more negative tests for truncate
>
>
> Key: HDFS-7738
> URL: https://issues.apache.org/jira/browse/HDFS-7738
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: h7738_20150204.patch
>
>
> The following are negative test cases for truncate.
> - new length > old length
> - truncating a directory
> - truncating a non-existing file
> - truncating a file without write permission
> - truncating a file opened for append
> - truncating a file in safemode
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6054) MiniQJMHACluster should not use static port to avoid binding failure in unit test
[
https://issues.apache.org/jira/browse/HDFS-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306690#comment-14306690
]
Hadoop QA commented on HDFS-6054:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696646/HDFS-6054.002.patch
against trunk revision 9112f09.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-hdfs-project/hadoop-hdfs
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9435//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9435//console
This message is automatically generated.
> MiniQJMHACluster should not use static port to avoid binding failure in unit
> test
> -
>
> Key: HDFS-6054
> URL: https://issues.apache.org/jira/browse/HDFS-6054
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Brandon Li
>Assignee: Yongjun Zhang
> Attachments: HDFS-6054.001.patch, HDFS-6054.002.patch
>
>
> One example of the test failues: TestFailureToReadEdits
> {noformat}
> Error Message
> Port in use: localhost:10003
> Stacktrace
> java.net.BindException: Port in use: localhost:10003
> at sun.nio.ch.Net.bind(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:845)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:786)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:132)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:593)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:492)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:650)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:635)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1283)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:966)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:851)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:697)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:374)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.setUpCluster(TestFailureToReadEdits.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7738) Add more negative tests for truncate
[
https://issues.apache.org/jira/browse/HDFS-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306573#comment-14306573
]
Yi Liu commented on HDFS-7738:
--
{quote}
I rather keep this patch simple. Sounds good?
{quote}
I'm OK for this. +1.
{quote}
Since AlreadyBeingCreatedException is in org.apache.hadoop.hdfs.protocol,
changing is to something else is a bigger change which need more thought
{quote}
Yes, but actually {{AlreadyBeingCreatedException}} is only explicitly thrown by
{{ClientProtocol#create}} . My thought is we can define a separate recover
lease exception which extends IOException and let
{{AlreadyBeingCreatedException}} extends it, otherwise people may see already
being created exception when he does an append/truncate operation, that's odd.
Of course, it's a minor improvement, we can also do it in a follow-on JIRA if
you think it's necessary.
> Add more negative tests for truncate
>
>
> Key: HDFS-7738
> URL: https://issues.apache.org/jira/browse/HDFS-7738
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: h7738_20150204.patch
>
>
> The following are negative test cases for truncate.
> - new length > old length
> - truncating a directory
> - truncating a non-existing file
> - truncating a file without write permission
> - truncating a file opened for append
> - truncating a file in safemode
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7738) Add more negative tests for truncate
[ https://issues.apache.org/jira/browse/HDFS-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306563#comment-14306563 ] Tsz Wo Nicholas Sze commented on HDFS-7738: --- Thanks for the quick review. Since AlreadyBeingCreatedException is in org.apache.hadoop.hdfs.protocol, changing is to something else is a bigger change which need more thought. I rather keep this patch simple. Sounds good? > Add more negative tests for truncate > > > Key: HDFS-7738 > URL: https://issues.apache.org/jira/browse/HDFS-7738 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Fix For: 2.7.0 > > Attachments: h7738_20150204.patch > > > The following are negative test cases for truncate. > - new length > old length > - truncating a directory > - truncating a non-existing file > - truncating a file without write permission > - truncating a file opened for append > - truncating a file in safemode -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7738) Add more negative tests for truncate
[
https://issues.apache.org/jira/browse/HDFS-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306558#comment-14306558
]
Yi Liu edited comment on HDFS-7738 at 2/5/15 3:12 AM:
--
I notice the patch distinguishes recover lease exception string for
create/truncate/append file and recover lease, it's great, I also had the same
thought, but the exception is still {{AlreadyBeingCreatedException}}, should we
redefine it?
Besides this, I'm +1 for the patch and pending Jenkins.
was (Author: hitliuyi):
I notice the patch distinguishes recover lease exception string for
create/truncate/append/recover file, it's great, I also had the same thought,
but the exception is still {{AlreadyBeingCreatedException}}, should we redefine
it?
Besides this, I'm +1 for the patch and pending Jenkins.
> Add more negative tests for truncate
>
>
> Key: HDFS-7738
> URL: https://issues.apache.org/jira/browse/HDFS-7738
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: h7738_20150204.patch
>
>
> The following are negative test cases for truncate.
> - new length > old length
> - truncating a directory
> - truncating a non-existing file
> - truncating a file without write permission
> - truncating a file opened for append
> - truncating a file in safemode
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7738) Add more negative tests for truncate
[
https://issues.apache.org/jira/browse/HDFS-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306558#comment-14306558
]
Yi Liu commented on HDFS-7738:
--
I notice the patch distinguishes recover lease exception string for
create/truncate/append/recover file, it's great, I also had the same thought,
but the exception is still {{AlreadyBeingCreatedException}}, should we redefine
it?
Besides this, I'm +1 for the patch and pending Jenkins.
> Add more negative tests for truncate
>
>
> Key: HDFS-7738
> URL: https://issues.apache.org/jira/browse/HDFS-7738
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: h7738_20150204.patch
>
>
> The following are negative test cases for truncate.
> - new length > old length
> - truncating a directory
> - truncating a non-existing file
> - truncating a file without write permission
> - truncating a file opened for append
> - truncating a file in safemode
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7738) Add more negative tests for truncate
[ https://issues.apache.org/jira/browse/HDFS-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-7738: -- Status: Patch Available (was: Open) > Add more negative tests for truncate > > > Key: HDFS-7738 > URL: https://issues.apache.org/jira/browse/HDFS-7738 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Fix For: 2.7.0 > > Attachments: h7738_20150204.patch > > > The following are negative test cases for truncate. > - new length > old length > - truncating a directory > - truncating a non-existing file > - truncating a file without write permission > - truncating a file opened for append > - truncating a file in safemode -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7738) Add more negative tests for truncate
[ https://issues.apache.org/jira/browse/HDFS-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-7738: -- Attachment: h7738_20150204.patch h7738_20150204.patch: adds the new tests and revises the exception message in FSNamesystem.recoverLeaseInternal(..). > Add more negative tests for truncate > > > Key: HDFS-7738 > URL: https://issues.apache.org/jira/browse/HDFS-7738 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Fix For: 2.7.0 > > Attachments: h7738_20150204.patch > > > The following are negative test cases for truncate. > - new length > old length > - truncating a directory > - truncating a non-existing file > - truncating a file without write permission > - truncating a file opened for append > - truncating a file in safemode -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306543#comment-14306543 ] Junping Du commented on HDFS-6261: -- Sounds like a reasonable plan, [~decster]! I may need more time for HADOOP-11495 because some other JIRA and efforts, please feel free to take that one if you can do it now and I will help to review it. Thanks [~aw] for comments also! > Add document for enabling node group layer in HDFS > -- > > Key: HDFS-6261 > URL: https://issues.apache.org/jira/browse/HDFS-6261 > Project: Hadoop HDFS > Issue Type: Task > Components: documentation >Reporter: Wenwu Peng >Assignee: Binglin Chang > Labels: documentation > Attachments: 2-layer-topology.png, 3-layer-topology.png, > 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, > HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch > > > Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there > is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) > and how to do configuration. so we need to doc it. > 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current > 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7349) Support DFS command for the EC encoding
[ https://issues.apache.org/jira/browse/HDFS-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306525#comment-14306525 ] Kai Zheng commented on HDFS-7349: - OK no problem. I'm working on the way from low level coders to upper level codecs in relevant JIRAs, which will make schemas configurable, pluggable and workable, having the two predefined as system defaults. Meanwhile, we may have the schemas with their parameters hard-coded as you previously suggested in places they're needed and later I will resolve them. > Support DFS command for the EC encoding > --- > > Key: HDFS-7349 > URL: https://issues.apache.org/jira/browse/HDFS-7349 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-7349-001.patch, HDFS-7349-002.patch > > > Support implementation of the following commands > *hdfs dfs -convertToEC * >: Converts all blocks under this path to EC form (if not already in > EC form, and if can be coded). > *hdfs dfs -convertToRep * >: Converts all blocks under this path to be replicated form. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7738) Add more negative tests for truncate
Tsz Wo Nicholas Sze created HDFS-7738: - Summary: Add more negative tests for truncate Key: HDFS-7738 URL: https://issues.apache.org/jira/browse/HDFS-7738 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Priority: Minor The following are negative test cases for truncate. - new length > old length - truncating a directory - truncating a non-existing file - truncating a file without write permission - truncating a file opened for append - truncating a file in safemode -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7371) Loading predefined EC schemas from configuration
[ https://issues.apache.org/jira/browse/HDFS-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306507#comment-14306507 ] Kai Zheng commented on HDFS-7371: - Thanks Zhe. Will look at the design and think about it. > Loading predefined EC schemas from configuration > > > Key: HDFS-7371 > URL: https://issues.apache.org/jira/browse/HDFS-7371 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > > System administrator can configure multiple EC codecs in hdfs-site.xml file, > and codec instances or schemas in a new configuration file named > ec-schema.xml in the conf folder. A codec can be referenced by its instance > or schema using the codec name, and a schema can be utilized and specified by > the schema name for a folder or EC ZONE to enforce EC. Once a schema is used > to define an EC ZONE, then its associated parameter values will be stored as > xattributes and respected thereafter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306506#comment-14306506 ] Binglin Chang commented on HDFS-6261: - bq. I'd prefer to see this get merged into the RackAwareness documentation rather than building a completely new doc OK. Will update the patch once HADOOP-11495 is resolved. > Add document for enabling node group layer in HDFS > -- > > Key: HDFS-6261 > URL: https://issues.apache.org/jira/browse/HDFS-6261 > Project: Hadoop HDFS > Issue Type: Task > Components: documentation >Reporter: Wenwu Peng >Assignee: Binglin Chang > Labels: documentation > Attachments: 2-layer-topology.png, 3-layer-topology.png, > 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, > HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch > > > Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there > is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) > and how to do configuration. so we need to doc it. > 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current > 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6054) MiniQJMHACluster should not use static port to avoid binding failure in unit test
[
https://issues.apache.org/jira/browse/HDFS-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306491#comment-14306491
]
Yongjun Zhang commented on HDFS-6054:
-
Hi [~kihwal],
I just uploaded rev 002 to make the port random. I also make both
MiniQJMHACluster and SHARED_DIR_HA report the basePort. In addition, I think
it's better to cleanup the cluster before retrying in MiniQJMHACluster, so
added the code to do so.
I did similar test like you did this time, started with using 1 as the
base, while making the port 10001 unavailable with {{nc -l 10001}}; then later
changed to use random even in the first try as in the patch.
Thanks.
> MiniQJMHACluster should not use static port to avoid binding failure in unit
> test
> -
>
> Key: HDFS-6054
> URL: https://issues.apache.org/jira/browse/HDFS-6054
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Brandon Li
>Assignee: Yongjun Zhang
> Attachments: HDFS-6054.001.patch, HDFS-6054.002.patch
>
>
> One example of the test failues: TestFailureToReadEdits
> {noformat}
> Error Message
> Port in use: localhost:10003
> Stacktrace
> java.net.BindException: Port in use: localhost:10003
> at sun.nio.ch.Net.bind(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:845)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:786)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:132)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:593)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:492)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:650)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:635)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1283)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:966)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:851)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:697)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:374)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.setUpCluster(TestFailureToReadEdits.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7349) Support DFS command for the EC encoding
[ https://issues.apache.org/jira/browse/HDFS-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306489#comment-14306489 ] Zhe Zhang commented on HDFS-7349: - I agree with selecting from predefined schemas. How about we start with RS63 and XOR21? > Support DFS command for the EC encoding > --- > > Key: HDFS-7349 > URL: https://issues.apache.org/jira/browse/HDFS-7349 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-7349-001.patch, HDFS-7349-002.patch > > > Support implementation of the following commands > *hdfs dfs -convertToEC * >: Converts all blocks under this path to EC form (if not already in > EC form, and if can be coded). > *hdfs dfs -convertToRep * >: Converts all blocks under this path to be replicated form. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6054) MiniQJMHACluster should not use static port to avoid binding failure in unit test
[
https://issues.apache.org/jira/browse/HDFS-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-6054:
Attachment: HDFS-6054.002.patch
> MiniQJMHACluster should not use static port to avoid binding failure in unit
> test
> -
>
> Key: HDFS-6054
> URL: https://issues.apache.org/jira/browse/HDFS-6054
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Brandon Li
>Assignee: Yongjun Zhang
> Attachments: HDFS-6054.001.patch, HDFS-6054.002.patch
>
>
> One example of the test failues: TestFailureToReadEdits
> {noformat}
> Error Message
> Port in use: localhost:10003
> Stacktrace
> java.net.BindException: Port in use: localhost:10003
> at sun.nio.ch.Net.bind(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:845)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:786)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:132)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:593)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:492)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:650)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:635)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1283)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:966)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:851)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:697)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:374)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.setUpCluster(TestFailureToReadEdits.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6832) Fix the usage of 'hdfs namenode' command
[
https://issues.apache.org/jira/browse/HDFS-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-6832:
---
Status: Open (was: Patch Available)
Cancelling patch since it no longer applies.
> Fix the usage of 'hdfs namenode' command
>
>
> Key: HDFS-6832
> URL: https://issues.apache.org/jira/browse/HDFS-6832
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.1
>Reporter: Akira AJISAKA
>Assignee: skrho
>Priority: Minor
> Labels: newbie
> Attachments: hdfs-6832.txt, hdfs-6832_001.txt, hdfs-6832_002.txt
>
>
> {code}
> [root@trunk ~]# hdfs namenode -help
> Usage: java NameNode [-backup] |
> [-checkpoint] |
> [-format [-clusterid cid ] [-force] [-nonInteractive] ] |
> [-upgrade [-clusterid cid] [-renameReserved] ] |
> [-upgradeOnly [-clusterid cid] [-renameReserved] ] |
> [-rollback] |
> [-rollingUpgrade ] |
> [-finalize] |
> [-importCheckpoint] |
> [-initializeSharedEdits] |
> [-bootstrapStandby] |
> [-recover [ -force] ] |
> [-metadataVersion ] ]
> {code}
> There're some issues in the usage to be fixed.
> # "Usage: java NameNode" should be "Usage: hdfs namenode"
> # -rollingUpgrade started option should be added
> # The last ']' should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6832) Fix the usage of 'hdfs namenode' command
[
https://issues.apache.org/jira/browse/HDFS-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306439#comment-14306439
]
Hadoop QA commented on HDFS-6832:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665701/hdfs-6832_002.txt
against trunk revision cc6bbfc.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9434//console
This message is automatically generated.
> Fix the usage of 'hdfs namenode' command
>
>
> Key: HDFS-6832
> URL: https://issues.apache.org/jira/browse/HDFS-6832
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.1
>Reporter: Akira AJISAKA
>Assignee: skrho
>Priority: Minor
> Labels: newbie
> Attachments: hdfs-6832.txt, hdfs-6832_001.txt, hdfs-6832_002.txt
>
>
> {code}
> [root@trunk ~]# hdfs namenode -help
> Usage: java NameNode [-backup] |
> [-checkpoint] |
> [-format [-clusterid cid ] [-force] [-nonInteractive] ] |
> [-upgrade [-clusterid cid] [-renameReserved] ] |
> [-upgradeOnly [-clusterid cid] [-renameReserved] ] |
> [-rollback] |
> [-rollingUpgrade ] |
> [-finalize] |
> [-importCheckpoint] |
> [-initializeSharedEdits] |
> [-bootstrapStandby] |
> [-recover [ -force] ] |
> [-metadataVersion ] ]
> {code}
> There're some issues in the usage to be fixed.
> # "Usage: java NameNode" should be "Usage: hdfs namenode"
> # -rollingUpgrade started option should be added
> # The last ']' should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-402) Display the server version in dfsadmin -report
[ https://issues.apache.org/jira/browse/HDFS-402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-402: -- Status: Open (was: Patch Available) Cancelling the patch as it no longer applies. > Display the server version in dfsadmin -report > -- > > Key: HDFS-402 > URL: https://issues.apache.org/jira/browse/HDFS-402 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: Uma Maheswara Rao G >Priority: Minor > Labels: newbie > Attachments: HDFS-402.patch, HDFS-402.patch, HDFS-402.patch, > hdfs-402.txt > > > As part of HADOOP-5094, it was requested to include the server version in the > dfsadmin -report, to avoid the need to screen scrape to get this information: > bq. Please do provide the server version, so there is a quick and non-taxing > way of determine what is the current running version on the namenode. > Currently there is nothing in the dfs client protocol to query this > information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-316: -- Status: Open (was: Patch Available) Cancelling since patch no longer applies. > Balancer should run for a configurable # of iterations > -- > > Key: HDFS-316 > URL: https://issues.apache.org/jira/browse/HDFS-316 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.4.1 >Reporter: Brian Bockelman >Assignee: Xiaoyu Yao >Priority: Minor > Labels: newbie > Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch > > > The balancer currently exits if nothing has changed after 5 iterations. > Our site would like to constantly balance a stream of incoming data; we would > like to be able to set the number of iterations it "does nothing" for before > exiting; even better would be if we set it to a negative number and could > continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-1918) DataXceiver double logs every IOE out of readBlock
[ https://issues.apache.org/jira/browse/HDFS-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-1918: --- Resolution: Unresolved Status: Resolved (was: Patch Available) Closing as Unresolved. Code base has moved on since this was filed. If this is still an issue, please open a new JIRA. Thanks. > DataXceiver double logs every IOE out of readBlock > -- > > Key: HDFS-1918 > URL: https://issues.apache.org/jira/browse/HDFS-1918 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.20.2 >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Trivial > Attachments: HDFS-1918.patch > > > DataXceiver will log an IOE twice because opReadBlock() will catch it, log a > WARN, then throw it again only to be caught in run() as a Throwable and > logged as an ERROR. As far as I can tell all the information is the same in > both messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-2304) log libhdfs messages to syslog
[ https://issues.apache.org/jira/browse/HDFS-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-2304: --- Status: Open (was: Patch Available) old, outdated patch. cancelling. > log libhdfs messages to syslog > -- > > Key: HDFS-2304 > URL: https://issues.apache.org/jira/browse/HDFS-2304 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs >Affects Versions: 2.0.0-alpha, 1.0.0 >Reporter: Eli Collins >Assignee: Stephen Fritz >Priority: Minor > Labels: newbie > Attachments: HDFS-2304.patch > > > libhdfs currently logs to stdout. Not all applications that link to libhdfs > capture stdout so it would be useful to have the logs goto syslog as well (eg > see fuse_dfs.h macros that log to both). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3728) Update Httpfs documentation
[ https://issues.apache.org/jira/browse/HDFS-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-3728: --- Status: Open (was: Patch Available) Cancelling patch since it no longer applies. > Update Httpfs documentation > --- > > Key: HDFS-3728 > URL: https://issues.apache.org/jira/browse/HDFS-3728 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.2-alpha, 1.0.3, 3.0.0 >Reporter: Santhosh Srinivasan >Priority: Minor > Labels: newbie > Attachments: HDFS-3728.patch > > > Link: > http://hadoop.apache.org/common/docs/current/hadoop-hdfs-httpfs/index.html > Section: How HttpFS and Hadoop HDFS Proxy differ? > # Change seening to seen > # "HttpFS uses a clean HTTP REST API making its use with HTTP tools more > intuitive." is very subjective. Can it be rephrased or removed? > Link: > http://hadoop.apache.org/common/docs/current/hadoop-hdfs-httpfs/ServerSetup.html > Section: Configure HttpFS > # Change "...add to the httpfs-site.xml file the httpfs.hadoop.config.dir > property set to..." to "add to the httpfs-site.xml file the > httpfs.hadoop.config.dir property and set the value to ..." > Section: Configure Hadoop > # Change defined to define > Section: Restart Hadoop > # Typo - to (not ot) > Section: Start/Stop HttpFS > # lists (plural) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7371) Loading predefined EC schemas from configuration
[ https://issues.apache.org/jira/browse/HDFS-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306421#comment-14306421 ] Zhe Zhang commented on HDFS-7371: - Thanks Kai. It's an interesting and non-trivial question what should be configured on command line and what should go into config XML files. We can refer to [encryption design | http://www.cloudera.com/content/cloudera/en/documentation/core/v5-2-x/topics/cdh_sg_hdfs_encryption.html]. > Loading predefined EC schemas from configuration > > > Key: HDFS-7371 > URL: https://issues.apache.org/jira/browse/HDFS-7371 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > > System administrator can configure multiple EC codecs in hdfs-site.xml file, > and codec instances or schemas in a new configuration file named > ec-schema.xml in the conf folder. A codec can be referenced by its instance > or schema using the codec name, and a schema can be utilized and specified by > the schema name for a folder or EC ZONE to enforce EC. Once a schema is used > to define an EC ZONE, then its associated parameter values will be stored as > xattributes and respected thereafter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-4922) Improve the short-circuit document
[ https://issues.apache.org/jira/browse/HDFS-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-4922: --- Target Version/s: 2.1.0-beta, 3.0.0 (was: 3.0.0, 2.1.0-beta) Status: Open (was: Patch Available) Cancelling patch since it no longer applies. > Improve the short-circuit document > -- > > Key: HDFS-4922 > URL: https://issues.apache.org/jira/browse/HDFS-4922 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, hdfs-client >Affects Versions: 2.1.0-beta, 3.0.0 >Reporter: Fengdong Yu >Assignee: Fengdong Yu >Priority: Minor > Attachments: HDFS-4922-002.patch, HDFS-4922-003.patch, > HDFS-4922-004.patch, HDFS-4922-005.patch, HDFS-4922-006.patch, HDFS-4922.patch > > > explain the default value and add one configure key, which doesn't show in > the document, but exists in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-4919) Improve documentation of dfs.permissions.enabled flag.
[ https://issues.apache.org/jira/browse/HDFS-4919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-4919: --- Target Version/s: 2.1.0-beta, 3.0.0 (was: 3.0.0, 2.1.0-beta) Status: Open (was: Patch Available) Cancelling patch since it no longer applies. > Improve documentation of dfs.permissions.enabled flag. > -- > > Key: HDFS-4919 > URL: https://issues.apache.org/jira/browse/HDFS-4919 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 2.1.0-beta, 3.0.0 >Reporter: Chris Nauroth > Attachments: HDFS-4919.patch > > > The description of dfs.permissions.enabled in hdfs-default.xml does not state > that permissions are always checked on certain calls regardless of this > configuration. The HDFS permissions guide still mentions the deprecated > dfs.permissions property instead of the currently supported > dfs.permissions.enabled. > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Configuration_Parameters -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-6261: --- Status: Open (was: Patch Available) Cancelling the current patch due to non-applicability and due to comments. > Add document for enabling node group layer in HDFS > -- > > Key: HDFS-6261 > URL: https://issues.apache.org/jira/browse/HDFS-6261 > Project: Hadoop HDFS > Issue Type: Task > Components: documentation >Reporter: Wenwu Peng >Assignee: Binglin Chang > Labels: documentation > Attachments: 2-layer-topology.png, 3-layer-topology.png, > 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, > HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch > > > Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there > is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) > and how to do configuration. so we need to doc it. > 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current > 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6832) Fix the usage of 'hdfs namenode' command
[
https://issues.apache.org/jira/browse/HDFS-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-6832:
---
Status: Patch Available (was: Open)
> Fix the usage of 'hdfs namenode' command
>
>
> Key: HDFS-6832
> URL: https://issues.apache.org/jira/browse/HDFS-6832
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.1
>Reporter: Akira AJISAKA
>Assignee: skrho
>Priority: Minor
> Labels: newbie
> Attachments: hdfs-6832.txt, hdfs-6832_001.txt, hdfs-6832_002.txt
>
>
> {code}
> [root@trunk ~]# hdfs namenode -help
> Usage: java NameNode [-backup] |
> [-checkpoint] |
> [-format [-clusterid cid ] [-force] [-nonInteractive] ] |
> [-upgrade [-clusterid cid] [-renameReserved] ] |
> [-upgradeOnly [-clusterid cid] [-renameReserved] ] |
> [-rollback] |
> [-rollingUpgrade ] |
> [-finalize] |
> [-importCheckpoint] |
> [-initializeSharedEdits] |
> [-bootstrapStandby] |
> [-recover [ -force] ] |
> [-metadataVersion ] ]
> {code}
> There're some issues in the usage to be fixed.
> # "Usage: java NameNode" should be "Usage: hdfs namenode"
> # -rollingUpgrade started option should be added
> # The last ']' should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7546) Document, and set an accepting default for dfs.namenode.kerberos.principal.pattern
[ https://issues.apache.org/jira/browse/HDFS-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306414#comment-14306414 ] Allen Wittenauer commented on HDFS-7546: +1 lgtm > Document, and set an accepting default for > dfs.namenode.kerberos.principal.pattern > -- > > Key: HDFS-7546 > URL: https://issues.apache.org/jira/browse/HDFS-7546 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 2.1.1-beta >Reporter: Harsh J >Assignee: Harsh J >Priority: Minor > Labels: supportability > Attachments: HDFS-7546.patch > > > This config is used in the SaslRpcClient, and the no-default breaks > cross-realm trust principals being used at clients. > Current location: > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslRpcClient.java#L309 > The config should be documented and the default should be set to * to > preserve the prior-to-introduction behaviour. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6832) Fix the usage of 'hdfs namenode' command
[
https://issues.apache.org/jira/browse/HDFS-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-6832:
---
Status: Open (was: Patch Available)
> Fix the usage of 'hdfs namenode' command
>
>
> Key: HDFS-6832
> URL: https://issues.apache.org/jira/browse/HDFS-6832
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.1
>Reporter: Akira AJISAKA
>Assignee: skrho
>Priority: Minor
> Labels: newbie
> Attachments: hdfs-6832.txt, hdfs-6832_001.txt, hdfs-6832_002.txt
>
>
> {code}
> [root@trunk ~]# hdfs namenode -help
> Usage: java NameNode [-backup] |
> [-checkpoint] |
> [-format [-clusterid cid ] [-force] [-nonInteractive] ] |
> [-upgrade [-clusterid cid] [-renameReserved] ] |
> [-upgradeOnly [-clusterid cid] [-renameReserved] ] |
> [-rollback] |
> [-rollingUpgrade ] |
> [-finalize] |
> [-importCheckpoint] |
> [-initializeSharedEdits] |
> [-bootstrapStandby] |
> [-recover [ -force] ] |
> [-metadataVersion ] ]
> {code}
> There're some issues in the usage to be fixed.
> # "Usage: java NameNode" should be "Usage: hdfs namenode"
> # -rollingUpgrade started option should be added
> # The last ']' should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7716) Erasure Coding: extend BlockInfo to handle EC info
[ https://issues.apache.org/jira/browse/HDFS-7716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306407#comment-14306407 ] Jing Zhao commented on HDFS-7716: - Here is the plan: # the class BlockInfo holds the common information and methods # create a new class BlockReplicationInfo extending BlockInfo for replication explicit logic # create a new class BlockGroupInfo extending BlockInfo for handling EC # we will still have BlockReplicationInfoUC and BlockGroupInfoUC to handle under construction stage # INodeFile still contains a BlockReplicationInfo array for replicated blocks. In case that the file is erasure coded, the file contains an extra inode feature to store the BlockGroupInfo array > Erasure Coding: extend BlockInfo to handle EC info > -- > > Key: HDFS-7716 > URL: https://issues.apache.org/jira/browse/HDFS-7716 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao > > The current BlockInfo's implementation only supports the replication > mechanism. To use the same blocksMap handling block group and its data/parity > blocks, we need to define a new BlockGroupInfo class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7270) Add congestion signaling capability to DataNode write protocol
[
https://issues.apache.org/jira/browse/HDFS-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306390#comment-14306390
]
Hadoop QA commented on HDFS-7270:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696598/HDFS-7270.004.patch
against trunk revision 5f4ef2d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHDFSAcl
org.apache.hadoop.hdfs.TestLeaseRecovery2
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9433//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9433//console
This message is automatically generated.
> Add congestion signaling capability to DataNode write protocol
> --
>
> Key: HDFS-7270
> URL: https://issues.apache.org/jira/browse/HDFS-7270
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-7270.000.patch, HDFS-7270.001.patch,
> HDFS-7270.002.patch, HDFS-7270.003.patch, HDFS-7270.004.patch
>
>
> When a client writes to HDFS faster than the disk bandwidth of the DNs, it
> saturates the disk bandwidth and put the DNs unresponsive. The client only
> backs off by aborting / recovering the pipeline, which leads to failed writes
> and unnecessary pipeline recovery.
> This jira proposes to add explicit congestion control mechanisms in the
> writing pipeline.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7733) NFS: readdir/readdirplus return null directory attribute on failure
[
https://issues.apache.org/jira/browse/HDFS-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306320#comment-14306320
]
Hudson commented on HDFS-7733:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #7010 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7010/])
HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure.
(Contributed by Arpit Agarwal) (arp: rev
c6f20007ebda509b39a7e4098b99e9b43d73d5b2)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java
> NFS: readdir/readdirplus return null directory attribute on failure
> ---
>
> Key: HDFS-7733
> URL: https://issues.apache.org/jira/browse/HDFS-7733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.6.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.6.1
>
> Attachments: HDFS-7733.01.patch
>
>
> NFS readdir and readdirplus operations return a null directory attribute on
> some failure paths. This causes clients to get a 'Stale file handle' error
> which can only be fixed by unmounting and remounting the share.
> The issue can be reproduced by running 'ls' against a large directory which
> is being actively modified, triggering the 'cookie mismatch' failure path.
> {code}
> } else {
> LOG.error("cookieverf mismatch. request cookieverf: " + cookieVerf
> + " dir cookieverf: " + dirStatus.getModificationTime());
> return new READDIRPLUS3Response(Nfs3Status.NFS3ERR_BAD_COOKIE);
> }
> {code}
> Thanks to [~brandonli] for catching the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7733) NFS: readdir/readdirplus return null directory attribute on failure
[
https://issues.apache.org/jira/browse/HDFS-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-7733:
Resolution: Fixed
Fix Version/s: 2.6.1
Target Version/s: (was: 2.7.0)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the review and helping debug this Brandon!
I committed it to trunk through branch-2.6.
> NFS: readdir/readdirplus return null directory attribute on failure
> ---
>
> Key: HDFS-7733
> URL: https://issues.apache.org/jira/browse/HDFS-7733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.6.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.6.1
>
> Attachments: HDFS-7733.01.patch
>
>
> NFS readdir and readdirplus operations return a null directory attribute on
> some failure paths. This causes clients to get a 'Stale file handle' error
> which can only be fixed by unmounting and remounting the share.
> The issue can be reproduced by running 'ls' against a large directory which
> is being actively modified, triggering the 'cookie mismatch' failure path.
> {code}
> } else {
> LOG.error("cookieverf mismatch. request cookieverf: " + cookieVerf
> + " dir cookieverf: " + dirStatus.getModificationTime());
> return new READDIRPLUS3Response(Nfs3Status.NFS3ERR_BAD_COOKIE);
> }
> {code}
> Thanks to [~brandonli] for catching the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7655) Expose truncate API for Web HDFS
[ https://issues.apache.org/jira/browse/HDFS-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306302#comment-14306302 ] Yi Liu commented on HDFS-7655: -- Thanks [~shv] for review too. Will update the patch to address your comments. > Expose truncate API for Web HDFS > > > Key: HDFS-7655 > URL: https://issues.apache.org/jira/browse/HDFS-7655 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Yi Liu >Assignee: Yi Liu > Attachments: HDFS-7655.001.patch, HDFS-7655.002.patch > > > This JIRA is to expose truncate API for Web HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-3107) HDFS truncate
[ https://issues.apache.org/jira/browse/HDFS-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306296#comment-14306296 ] Tsz Wo Nicholas Sze commented on HDFS-3107: --- [~shv], how is the remaining work? Hope that we still have progress after merging to branch-2. BTW, it seems that we do not have truncate tests with HA nor datanode restart. Is it true? > HDFS truncate > - > > Key: HDFS-3107 > URL: https://issues.apache.org/jira/browse/HDFS-3107 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Reporter: Lei Chang >Assignee: Plamen Jeliazkov > Fix For: 2.7.0 > > Attachments: HDFS-3107-13.patch, HDFS-3107-14.patch, > HDFS-3107-15.patch, HDFS-3107-HDFS-7056-combined.patch, HDFS-3107.008.patch, > HDFS-3107.15_branch2.patch, HDFS-3107.patch, HDFS-3107.patch, > HDFS-3107.patch, HDFS-3107.patch, HDFS-3107.patch, HDFS-3107.patch, > HDFS-3107.patch, HDFS-3107.patch, HDFS-3107.patch, HDFS-3107.patch, > HDFS-3107.patch, HDFS_truncate.pdf, HDFS_truncate.pdf, HDFS_truncate.pdf, > HDFS_truncate_semantics_Mar15.pdf, HDFS_truncate_semantics_Mar21.pdf, > editsStored, editsStored.xml > > Original Estimate: 1,344h > Remaining Estimate: 1,344h > > Systems with transaction support often need to undo changes made to the > underlying storage when a transaction is aborted. Currently HDFS does not > support truncate (a standard Posix operation) which is a reverse operation of > append, which makes upper layer applications use ugly workarounds (such as > keeping track of the discarded byte range per file in a separate metadata > store, and periodically running a vacuum process to rewrite compacted files) > to overcome this limitation of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7734) Class cast exception in NameNode#main
[
https://issues.apache.org/jira/browse/HDFS-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306295#comment-14306295
]
Yi Liu commented on HDFS-7734:
--
Thanks for Andrew, Kihwal and Arpit 's reviews and comments.
> Class cast exception in NameNode#main
> -
>
> Key: HDFS-7734
> URL: https://issues.apache.org/jira/browse/HDFS-7734
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>Assignee: Yi Liu
>Priority: Blocker
> Fix For: 2.7.0
>
> Attachments: HDFS-7734.001.patch, HDFS-7734.002.patch
>
>
> NameNode hits the following exception immediately on startup.
> {code}
> 15/02/03 15:50:25 ERROR namenode.NameNode: Failed to start namenode.
> java.lang.ClassCastException: org.apache.log4j.Logger cannot be cast to
> org.apache.commons.logging.Log
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1557)
> 15/02/03 15:50:25 INFO util.ExitUtil: Exiting with status 1
> {code}
> Location of the exception in NameNode.java:
> {code}
> public static void main(String argv[]) throws Exception {
> if (DFSUtil.parseHelpArgument(argv, NameNode.USAGE, System.out, true)) {
> System.exit(0);
> }
> try {
> StringUtils.startupShutdownMessage(NameNode.class, argv,
> (org.apache.commons.logging.Log)
> LogManager.getLogger(LOG.getName())); < Failed here.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-4014) Fix warnings found by findbugs2
[ https://issues.apache.org/jira/browse/HDFS-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA resolved HDFS-4014. - Resolution: Fixed Fix Version/s: 2.0.3-alpha Closing this issue since all of the sub-tasks were completed. > Fix warnings found by findbugs2 > > > Key: HDFS-4014 > URL: https://issues.apache.org/jira/browse/HDFS-4014 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0-alpha >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 2.0.3-alpha > > Attachments: findbugs.out.24.html, findbugs.out.25.html, > findbugs.out.26.html > > > The HDFS side of HADOOP-8594. Ubrella jira for fixing the warnings found by > findbugs 2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7411) Refactor and improve decommissioning logic into DecommissionManager
[ https://issues.apache.org/jira/browse/HDFS-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306261#comment-14306261 ] Tsz Wo Nicholas Sze commented on HDFS-7411: --- I do not intention to block the decommission improvement. However, it is really a bad idea to arbitrarily change the behavior of a public conf when keeping the existing behavior is easy, and mixing code refactoring with improvement in a big patch. [~andrew.wang], I am glad that you split the slf4j change from here to HDFS-7706 and filed another minor JIRA HDFS-7712 for the further work. If the work is included here, the blocker JIRA HDFS-7734 will be broken by the unnecessarily complicated patch here. In order to move faster, how about I volunteer doing the refactoring? > Refactor and improve decommissioning logic into DecommissionManager > --- > > Key: HDFS-7411 > URL: https://issues.apache.org/jira/browse/HDFS-7411 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.5.1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-7411.001.patch, hdfs-7411.002.patch, > hdfs-7411.003.patch, hdfs-7411.004.patch, hdfs-7411.005.patch, > hdfs-7411.006.patch, hdfs-7411.007.patch, hdfs-7411.008.patch, > hdfs-7411.009.patch, hdfs-7411.010.patch > > > Would be nice to split out decommission logic from DatanodeManager to > DecommissionManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7702) Move metadata across namenode - Effort to a real distributed namenode
[ https://issues.apache.org/jira/browse/HDFS-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306216#comment-14306216 ] Zhe Zhang commented on HDFS-7702: - Thanks [~raviprak] for clarifying. It does sound like a very good extension to federation. [~xiyunyue] Maybe the JIRA title and description should be updated to refer to federation? HDFS-7737 is currently named "Implement distributed namenode". > Move metadata across namenode - Effort to a real distributed namenode > - > > Key: HDFS-7702 > URL: https://issues.apache.org/jira/browse/HDFS-7702 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ray >Assignee: Ray > > Implement a tool can show in memory namespace tree structure with > weight(size) and a API can move metadata across different namenode. The > purpose is moving data efficiently and faster, without moving blocks on > datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7411) Refactor and improve decommissioning logic into DecommissionManager
[ https://issues.apache.org/jira/browse/HDFS-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306193#comment-14306193 ] Andrew Wang commented on HDFS-7411: --- I would definitely welcome some other opinions on the perceived incompatibility here. I think Colin and myself have already made our thoughts on the matter pretty clear above. Thanks in advance Kihwal. > Refactor and improve decommissioning logic into DecommissionManager > --- > > Key: HDFS-7411 > URL: https://issues.apache.org/jira/browse/HDFS-7411 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.5.1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-7411.001.patch, hdfs-7411.002.patch, > hdfs-7411.003.patch, hdfs-7411.004.patch, hdfs-7411.005.patch, > hdfs-7411.006.patch, hdfs-7411.007.patch, hdfs-7411.008.patch, > hdfs-7411.009.patch, hdfs-7411.010.patch > > > Would be nice to split out decommission logic from DatanodeManager to > DecommissionManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7270) Add congestion signaling capability to DataNode write protocol
[ https://issues.apache.org/jira/browse/HDFS-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-7270: - Attachment: HDFS-7270.004.patch > Add congestion signaling capability to DataNode write protocol > -- > > Key: HDFS-7270 > URL: https://issues.apache.org/jira/browse/HDFS-7270 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-7270.000.patch, HDFS-7270.001.patch, > HDFS-7270.002.patch, HDFS-7270.003.patch, HDFS-7270.004.patch > > > When a client writes to HDFS faster than the disk bandwidth of the DNs, it > saturates the disk bandwidth and put the DNs unresponsive. The client only > backs off by aborting / recovering the pipeline, which leads to failed writes > and unnecessary pipeline recovery. > This jira proposes to add explicit congestion control mechanisms in the > writing pipeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7702) Move metadata across namenode - Effort to a real distributed namenode
[ https://issues.apache.org/jira/browse/HDFS-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306187#comment-14306187 ] Ravi Prakash commented on HDFS-7702: HDFS Federation was static partitioning from what I remember. Even though the ambition to move subtrees across namenodes existed during the design phase, it never came to fruition. I'd be quite excited about this work. I ofcourse have my own opinions but I'll hold on to them for now. This would be a pretty big chunk of work. I would like to know who is signing up to do this. And I don't mean just you Ray! > Move metadata across namenode - Effort to a real distributed namenode > - > > Key: HDFS-7702 > URL: https://issues.apache.org/jira/browse/HDFS-7702 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ray >Assignee: Ray > > Implement a tool can show in memory namespace tree structure with > weight(size) and a API can move metadata across different namenode. The > purpose is moving data efficiently and faster, without moving blocks on > datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6054) MiniQJMHACluster should not use static port to avoid binding failure in unit test
[
https://issues.apache.org/jira/browse/HDFS-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306182#comment-14306182
]
Yongjun Zhang commented on HDFS-6054:
-
Hi [~kihwal],
Thanks a lot for looking into! I thought each trial we are using a random
number, but as you pointed out, it's not! My bad to not have examined it well
enough.
I'm uploading a revised patch shortly.
> MiniQJMHACluster should not use static port to avoid binding failure in unit
> test
> -
>
> Key: HDFS-6054
> URL: https://issues.apache.org/jira/browse/HDFS-6054
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Brandon Li
>Assignee: Yongjun Zhang
> Attachments: HDFS-6054.001.patch
>
>
> One example of the test failues: TestFailureToReadEdits
> {noformat}
> Error Message
> Port in use: localhost:10003
> Stacktrace
> java.net.BindException: Port in use: localhost:10003
> at sun.nio.ch.Net.bind(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:845)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:786)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:132)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:593)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:492)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:650)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:635)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1283)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:966)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:851)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:697)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:374)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.setUpCluster(TestFailureToReadEdits.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7411) Refactor and improve decommissioning logic into DecommissionManager
[ https://issues.apache.org/jira/browse/HDFS-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306167#comment-14306167 ] Kihwal Lee commented on HDFS-7411: -- HDFS-7735 is also being worked on. If we can convince our selves that the performance benefit of this change is great, it will be worth while to spend time to find a way to introduce this without incompatibility. The performance here can be viewed in two angles: 1) impact on name node and 2) decommissioning speed. I will also try to review the patch. I think everyone agrees that we need to improve decommissioning. > Refactor and improve decommissioning logic into DecommissionManager > --- > > Key: HDFS-7411 > URL: https://issues.apache.org/jira/browse/HDFS-7411 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.5.1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-7411.001.patch, hdfs-7411.002.patch, > hdfs-7411.003.patch, hdfs-7411.004.patch, hdfs-7411.005.patch, > hdfs-7411.006.patch, hdfs-7411.007.patch, hdfs-7411.008.patch, > hdfs-7411.009.patch, hdfs-7411.010.patch > > > Would be nice to split out decommission logic from DatanodeManager to > DecommissionManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6054) MiniQJMHACluster should not use static port to avoid binding failure in unit test
[
https://issues.apache.org/jira/browse/HDFS-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306152#comment-14306152
]
Kihwal Lee commented on HDFS-6054:
--
Before looking at the patch too hard, I tried running the test case before &
after with {{nc -l 10001}} running another terminal window to make the port
unavailable. The test didn't pass after the patch. Then I realized that the
patch simply retries and waits for the port to be freed.
{noformat}
2015-02-04 15:42:56,100 INFO ha.TestFailureToReadEdits
(TestFailureToReadEdits.java:setUpCluster(124))
- SHARED_DIR_HA: MiniQJMHACluster port conflicts, retried 716 times
{noformat}
Is there any harm in incrementing the port number in each retry? That seems to
make the test pass.
> MiniQJMHACluster should not use static port to avoid binding failure in unit
> test
> -
>
> Key: HDFS-6054
> URL: https://issues.apache.org/jira/browse/HDFS-6054
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Brandon Li
>Assignee: Yongjun Zhang
> Attachments: HDFS-6054.001.patch
>
>
> One example of the test failues: TestFailureToReadEdits
> {noformat}
> Error Message
> Port in use: localhost:10003
> Stacktrace
> java.net.BindException: Port in use: localhost:10003
> at sun.nio.ch.Net.bind(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:845)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:786)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:132)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:593)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:492)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:650)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:635)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1283)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:966)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:851)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:697)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:374)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.setUpCluster(TestFailureToReadEdits.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7270) Add congestion signaling capability to DataNode write protocol
[
https://issues.apache.org/jira/browse/HDFS-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306141#comment-14306141
]
Hadoop QA commented on HDFS-7270:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696311/HDFS-7270.003.patch
against trunk revision 064e077.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestSetrepIncreasing
org.apache.hadoop.hdfs.TestModTime
org.apache.hadoop.hdfs.TestDFSClientRetries
org.apache.hadoop.hdfs.TestSetrepDecreasing
org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
org.apache.hadoop.hdfs.server.namenode.TestDeleteRace
org.apache.hadoop.hdfs.TestPread
org.apache.hadoop.hdfs.server.namenode.TestFSDirectory
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate
org.apache.hadoop.hdfs.TestReadWhileWriting
org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
org.apache.hadoop.hdfs.server.mover.TestStorageMover
org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
org.apache.hadoop.hdfs.TestFileAppend
org.apache.hadoop.hdfs.TestEncryptedTransfer
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
org.apache.hadoop.hdfs.server.datanode.TestHSync
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
org.apache.hadoop.hdfs.TestFileAppendRestart
org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement
org.apache.hadoop.tracing.TestTracing
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
org.apache.hadoop.hdfs.web.TestWebHDFS
org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
org.apache.hadoop.hdfs.TestReplication
org.apache.hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN
org.apache.hadoop.hdfs.TestFileAppend4
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart
org.apache.hadoop.hdfs.TestWriteRead
org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus
org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics
org.apache.hadoop.hdfs.TestDecommission
org.apache.hadoop.hdfs.TestCrcCorruption
org.apache.hadoop.hdfs.server.namenode.TestINodeFile
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
org.
[jira] [Commented] (HDFS-7702) Move metadata across namenode - Effort to a real distributed namenode
[ https://issues.apache.org/jira/browse/HDFS-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306131#comment-14306131 ] Zhe Zhang commented on HDFS-7702: - How is this related to HDFS federation (HDFS-1052)? > Move metadata across namenode - Effort to a real distributed namenode > - > > Key: HDFS-7702 > URL: https://issues.apache.org/jira/browse/HDFS-7702 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ray >Assignee: Ray > > Implement a tool can show in memory namespace tree structure with > weight(size) and a API can move metadata across different namenode. The > purpose is moving data efficiently and faster, without moving blocks on > datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306034#comment-14306034
]
Hadoop QA commented on HDFS-7692:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696523/HDFS-7692.01.patch
against trunk revision 064e077.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9430//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9430//console
This message is automatically generated.
> DataStorage#addStorageLocations(...) should support MultiThread to speedup
> the upgrade of block pool at multi storage directories.
> --
>
> Key: HDFS-7692
> URL: https://issues.apache.org/jira/browse/HDFS-7692
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.5.2
>Reporter: Leitao Guo
> Attachments: HDFS-7692.01.patch
>
>
> {code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
> for (StorageLocation dataDir : dataDirs) {
> File root = dataDir.getFile();
> ... ...
> bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
> startOpt);
> addBlockPoolStorage(bpid, bpStorage);
> ... ...
> successVolumes.add(dataDir);
> }
> {code}
> In the above code the storage directories will be analyzed one by one, which
> is really time consuming when upgrading HDFS with datanodes have dozens of
> large volumes. MultiThread dataDirs analyzing should be supported here to
> speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305999#comment-14305999 ] Tsz Wo Nicholas Sze commented on HDFS-7285: --- Forgot to mention that the new design doc file is [HDFSErasureCodingDesign-20150204.pdf|https://issues.apache.org/jira/secure/attachment/12696562/HDFSErasureCodingDesign-20150204.pdf]. > Erasure Coding Support inside HDFS > -- > > Key: HDFS-7285 > URL: https://issues.apache.org/jira/browse/HDFS-7285 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Weihua Jiang >Assignee: Zhe Zhang > Attachments: ECAnalyzer.py, ECParser.py, > HDFSErasureCodingDesign-20141028.pdf, HDFSErasureCodingDesign-20141217.pdf, > HDFSErasureCodingDesign-20150204.pdf, fsimage-analysis-20150105.pdf > > > Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice > of data reliability, comparing to the existing HDFS 3-replica approach. For > example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, > with storage overhead only being 40%. This makes EC a quite attractive > alternative for big data storage, particularly for cold data. > Facebook had a related open source project called HDFS-RAID. It used to be > one of the contribute packages in HDFS but had been removed since Hadoop 2.0 > for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends > on MapReduce to do encoding and decoding tasks; 2) it can only be used for > cold files that are intended not to be appended anymore; 3) the pure Java EC > coding implementation is extremely slow in practical use. Due to these, it > might not be a good idea to just bring HDFS-RAID back. > We (Intel and Cloudera) are working on a design to build EC into HDFS that > gets rid of any external dependencies, makes it self-contained and > independently maintained. This design lays the EC feature on the storage type > support and considers compatible with existing HDFS features like caching, > snapshot, encryption, high availability and etc. This design will also > support different EC coding schemes, implementations and policies for > different deployment scenarios. By utilizing advanced libraries (e.g. Intel > ISA-L library), an implementation can greatly improve the performance of EC > encoding/decoding and makes the EC solution even more attractive. We will > post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6054) MiniQJMHACluster should not use static port to avoid binding failure in unit test
[
https://issues.apache.org/jira/browse/HDFS-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-6054:
-
Component/s: test
Target Version/s: 2.7.0
> MiniQJMHACluster should not use static port to avoid binding failure in unit
> test
> -
>
> Key: HDFS-6054
> URL: https://issues.apache.org/jira/browse/HDFS-6054
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Brandon Li
>Assignee: Yongjun Zhang
> Attachments: HDFS-6054.001.patch
>
>
> One example of the test failues: TestFailureToReadEdits
> {noformat}
> Error Message
> Port in use: localhost:10003
> Stacktrace
> java.net.BindException: Port in use: localhost:10003
> at sun.nio.ch.Net.bind(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:845)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:786)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:132)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:593)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:492)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:650)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:635)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1283)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:966)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:851)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:697)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:374)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.setUpCluster(TestFailureToReadEdits.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7704) DN heartbeat to Active NN may be blocked and expire if connection to Standby NN continues to time out.
[
https://issues.apache.org/jira/browse/HDFS-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305997#comment-14305997
]
Kihwal Lee commented on HDFS-7704:
--
{{processQueueMessages()}} is called outside of the try-catch block of
{{offerService()}}. It will be better to move it inside, at the end of the try
block and let the existing exception handling take care of the rest. Then we
can get rid of {{nnAddress]} that are passed only for logging. Make
{{reportTo()}} throw and {{offerService()}} always log exceptions with
{{nnAddress}}.
When a reporting fails, is it okay to simply drop it? Probably not. Take a look
at {{reportReceivedDeletedBlocks()}}. We should add a similar logic of putting
failed reports back to the queue to {{processQueueMessages()}}.
> DN heartbeat to Active NN may be blocked and expire if connection to Standby
> NN continues to time out.
> ---
>
> Key: HDFS-7704
> URL: https://issues.apache.org/jira/browse/HDFS-7704
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.5.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HDFS-7704-v2.patch, HDFS-7704-v3.patch,
> HDFS-7704-v4.patch, HDFS-7704.patch
>
>
> There are couple of synchronous calls in BPOfferservice (i.e reportBadBlocks
> and trySendErrorReport) which will wait for both of the actor threads to
> process this calls.
> This calls are made with writeLock acquired.
> When reportBadBlocks() is blocked at the RPC layer due to unreachable NN,
> subsequent heartbeat response processing has to wait for the write lock. It
> eventually gets through, but takes too long and it blocks the next heartbeat.
> In our HA cluster setup, the standby namenode was taking a long time to
> process the request.
> Requesting improvement in datanode to make the above calls asynchronous since
> these reports don't have any specific
> deadlines, so extra few seconds of delay should be acceptable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-7285: -- Attachment: HDFSErasureCodingDesign-20150204.pdf Revised the design doc as follows: - Revised the sections for -* Saving I/O bandwidth -* BlockGroup, -* ErasureCodec, -* ECClient, and -* NameNode Memory Usage Reduction. - Added new sections for -* EC Writer, -* Handling Datanode Failure during Write, -* Reading a Closed File, -* Reading a Being Written File, -* Hflush, -* Hsync, -* Append, -* Truncate, -* BlockGroup States, -* Generation Stamp, -* BlockGroup Recovery, -* EC Block Reconstruction, -* Collision with Random Block ID. > Erasure Coding Support inside HDFS > -- > > Key: HDFS-7285 > URL: https://issues.apache.org/jira/browse/HDFS-7285 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Weihua Jiang >Assignee: Zhe Zhang > Attachments: ECAnalyzer.py, ECParser.py, > HDFSErasureCodingDesign-20141028.pdf, HDFSErasureCodingDesign-20141217.pdf, > HDFSErasureCodingDesign-20150204.pdf, fsimage-analysis-20150105.pdf > > > Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice > of data reliability, comparing to the existing HDFS 3-replica approach. For > example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, > with storage overhead only being 40%. This makes EC a quite attractive > alternative for big data storage, particularly for cold data. > Facebook had a related open source project called HDFS-RAID. It used to be > one of the contribute packages in HDFS but had been removed since Hadoop 2.0 > for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends > on MapReduce to do encoding and decoding tasks; 2) it can only be used for > cold files that are intended not to be appended anymore; 3) the pure Java EC > coding implementation is extremely slow in practical use. Due to these, it > might not be a good idea to just bring HDFS-RAID back. > We (Intel and Cloudera) are working on a design to build EC into HDFS that > gets rid of any external dependencies, makes it self-contained and > independently maintained. This design lays the EC feature on the storage type > support and considers compatible with existing HDFS features like caching, > snapshot, encryption, high availability and etc. This design will also > support different EC coding schemes, implementations and policies for > different deployment scenarios. By utilizing advanced libraries (e.g. Intel > ISA-L library), an implementation can greatly improve the performance of EC > encoding/decoding and makes the EC solution even more attractive. We will > post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7702) Move metadata across namenode - Effort to a real distributed namenode
[ https://issues.apache.org/jira/browse/HDFS-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray updated HDFS-7702: -- Issue Type: Sub-task (was: New Feature) Parent: HDFS-7737 > Move metadata across namenode - Effort to a real distributed namenode > - > > Key: HDFS-7702 > URL: https://issues.apache.org/jira/browse/HDFS-7702 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ray >Assignee: Ray > > Implement a tool can show in memory namespace tree structure with > weight(size) and a API can move metadata across different namenode. The > purpose is moving data efficiently and faster, without moving blocks on > datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7737) Implement distributed namenode
Ray created HDFS-7737: - Summary: Implement distributed namenode Key: HDFS-7737 URL: https://issues.apache.org/jira/browse/HDFS-7737 Project: Hadoop HDFS Issue Type: New Feature Reporter: Ray Assignee: Ray Priority: Minor Try to add the following functions on HDFS: 1) Manually move metadata across different name nodes, without losing the original data locality 2) Automatically balance metadata across different name nodes with a novel subtree partition algorithm 3) Able to set a replication factor number for each name node, and each replicated name node can balance live read/write traffic 4) All the functionalities above are highly configurable, and the administrator can decide if he wants to use distributed name node setup or just HDFS federation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305930#comment-14305930
]
Lei (Eddy) Xu commented on HDFS-7722:
-
[~usrikanth] Would you mind to let me take on this one? I already have a patch
available for it.
Thanks a lot!
> DataNode#checkDiskError should also remove Storage when error is found.
> ---
>
> Key: HDFS-7722
> URL: https://issues.apache.org/jira/browse/HDFS-7722
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Srikanth Upputuri
>
> When {{DataNode#checkDiskError}} found disk errors, it removes all block
> metadatas from {{FsDatasetImpl}}. However, it does not removed the
> corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
> The result is that, we could not directly run {{reconfig}} to hot swap the
> failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7719) BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state associated with volumes
[
https://issues.apache.org/jira/browse/HDFS-7719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305920#comment-14305920
]
Lei (Eddy) Xu commented on HDFS-7719:
-
Thank you so much for reviewing and committing this, [~cmccabe]!
> BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state
> associated with volumes
>
>
> Key: HDFS-7719
> URL: https://issues.apache.org/jira/browse/HDFS-7719
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Fix For: 2.7.0
>
> Attachments: HDFS-7719.000.patch, HDFS-7719.001.patch,
> HDFS-7719.002.patch, HDFS-7719.003.patch
>
>
> The parameter of {{BlockPoolSliceStorage#removeVolumes()}} is a set of volume
> level directories, thus {{BlockPoolSliceStorage}} could not directly compare
> its own {{StorageDirs}} with this volume-level directory. The result of that
> is {{BlockPoolSliceStorage}} did not actually remove the targeted
> {{StorageDirectory}}.
> It will cause failure when remove a volume and then immediately add a volume
> back with the same mount point..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7719) BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state associated with volumes
[
https://issues.apache.org/jira/browse/HDFS-7719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-7719:
---
Resolution: Fixed
Fix Version/s: 2.7.0
Status: Resolved (was: Patch Available)
> BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state
> associated with volumes
>
>
> Key: HDFS-7719
> URL: https://issues.apache.org/jira/browse/HDFS-7719
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Fix For: 2.7.0
>
> Attachments: HDFS-7719.000.patch, HDFS-7719.001.patch,
> HDFS-7719.002.patch, HDFS-7719.003.patch
>
>
> The parameter of {{BlockPoolSliceStorage#removeVolumes()}} is a set of volume
> level directories, thus {{BlockPoolSliceStorage}} could not directly compare
> its own {{StorageDirs}} with this volume-level directory. The result of that
> is {{BlockPoolSliceStorage}} did not actually remove the targeted
> {{StorageDirectory}}.
> It will cause failure when remove a volume and then immediately add a volume
> back with the same mount point..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[
https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305912#comment-14305912
]
Hadoop QA commented on HDFS-316:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12672071/HDFS-316.2.patch
against trunk revision 40a4157.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9432//console
This message is automatically generated.
> Balancer should run for a configurable # of iterations
> --
>
> Key: HDFS-316
> URL: https://issues.apache.org/jira/browse/HDFS-316
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.4.1
>Reporter: Brian Bockelman
>Assignee: Xiaoyu Yao
>Priority: Minor
> Labels: newbie
> Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch
>
>
> The balancer currently exits if nothing has changed after 5 iterations.
> Our site would like to constantly balance a stream of incoming data; we would
> like to be able to set the number of iterations it "does nothing" for before
> exiting; even better would be if we set it to a negative number and could
> continuously run this as a daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-7018) Implement hdfs.h interface in libhdfs3
[ https://issues.apache.org/jira/browse/HDFS-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HDFS-7018. Resolution: Fixed Fix Version/s: HDFS-6994 Target Version/s: HDFS-6994 > Implement hdfs.h interface in libhdfs3 > -- > > Key: HDFS-7018 > URL: https://issues.apache.org/jira/browse/HDFS-7018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Fix For: HDFS-6994 > > Attachments: HDFS-7018-pnative.002.patch, > HDFS-7018-pnative.003.patch, HDFS-7018-pnative.004.patch, > HDFS-7018-pnative.005.patch, HDFS-7018.patch > > > Implement C interface for libhdfs3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7018) Implement hdfs.h interface in libhdfs3
[ https://issues.apache.org/jira/browse/HDFS-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7018: --- Summary: Implement hdfs.h interface in libhdfs3 (was: Implement C interface for libhdfs3) > Implement hdfs.h interface in libhdfs3 > -- > > Key: HDFS-7018 > URL: https://issues.apache.org/jira/browse/HDFS-7018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Attachments: HDFS-7018-pnative.002.patch, > HDFS-7018-pnative.003.patch, HDFS-7018-pnative.004.patch, > HDFS-7018-pnative.005.patch, HDFS-7018.patch > > > Implement C interface for libhdfs3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7018) Implement C interface for libhdfs3
[ https://issues.apache.org/jira/browse/HDFS-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305907#comment-14305907 ] Colin Patrick McCabe commented on HDFS-7018: +1. Thanks, Zhanwei. > Implement C interface for libhdfs3 > -- > > Key: HDFS-7018 > URL: https://issues.apache.org/jira/browse/HDFS-7018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Attachments: HDFS-7018-pnative.002.patch, > HDFS-7018-pnative.003.patch, HDFS-7018-pnative.004.patch, > HDFS-7018-pnative.005.patch, HDFS-7018.patch > > > Implement C interface for libhdfs3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-316: -- Status: Patch Available (was: Open) > Balancer should run for a configurable # of iterations > -- > > Key: HDFS-316 > URL: https://issues.apache.org/jira/browse/HDFS-316 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.4.1 >Reporter: Brian Bockelman >Assignee: Xiaoyu Yao >Priority: Minor > Labels: newbie > Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch > > > The balancer currently exits if nothing has changed after 5 iterations. > Our site would like to constantly balance a stream of incoming data; we would > like to be able to set the number of iterations it "does nothing" for before > exiting; even better would be if we set it to a negative number and could > continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305900#comment-14305900 ] Allen Wittenauer commented on HDFS-316: --- Re-fire the jenkins test > Balancer should run for a configurable # of iterations > -- > > Key: HDFS-316 > URL: https://issues.apache.org/jira/browse/HDFS-316 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.4.1 >Reporter: Brian Bockelman >Assignee: Xiaoyu Yao >Priority: Minor > Labels: newbie > Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch > > > The balancer currently exits if nothing has changed after 5 iterations. > Our site would like to constantly balance a stream of incoming data; we would > like to be able to set the number of iterations it "does nothing" for before > exiting; even better would be if we set it to a negative number and could > continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-316: -- Status: Open (was: Patch Available) > Balancer should run for a configurable # of iterations > -- > > Key: HDFS-316 > URL: https://issues.apache.org/jira/browse/HDFS-316 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.4.1 >Reporter: Brian Bockelman >Assignee: Xiaoyu Yao >Priority: Minor > Labels: newbie > Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch > > > The balancer currently exits if nothing has changed after 5 iterations. > Our site would like to constantly balance a stream of incoming data; we would > like to be able to set the number of iterations it "does nothing" for before > exiting; even better would be if we set it to a negative number and could > continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7719) BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state associated with volumes
[
https://issues.apache.org/jira/browse/HDFS-7719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305889#comment-14305889
]
Hudson commented on HDFS-7719:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7005 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7005/])
HDFS-7719. BlockPoolSliceStorage#removeVolumes fails to remove some in-memory
state associated with volumes. (Lei (Eddy) Xu via Colin P. McCabe) (cmccabe:
rev 40a415799b1ff3602fbb461765f8b36f1133bda2)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
> BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state
> associated with volumes
>
>
> Key: HDFS-7719
> URL: https://issues.apache.org/jira/browse/HDFS-7719
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7719.000.patch, HDFS-7719.001.patch,
> HDFS-7719.002.patch, HDFS-7719.003.patch
>
>
> The parameter of {{BlockPoolSliceStorage#removeVolumes()}} is a set of volume
> level directories, thus {{BlockPoolSliceStorage}} could not directly compare
> its own {{StorageDirs}} with this volume-level directory. The result of that
> is {{BlockPoolSliceStorage}} did not actually remove the targeted
> {{StorageDirectory}}.
> It will cause failure when remove a volume and then immediately add a volume
> back with the same mount point..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7734) Class cast exception in NameNode#main
[
https://issues.apache.org/jira/browse/HDFS-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305883#comment-14305883
]
Arpit Agarwal commented on HDFS-7734:
-
Thank you for picking this up [~hitliuyi] and the prompt fix. It was a blocker
for any new development against trunk. :-)
> Class cast exception in NameNode#main
> -
>
> Key: HDFS-7734
> URL: https://issues.apache.org/jira/browse/HDFS-7734
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>Assignee: Yi Liu
>Priority: Blocker
> Fix For: 2.7.0
>
> Attachments: HDFS-7734.001.patch, HDFS-7734.002.patch
>
>
> NameNode hits the following exception immediately on startup.
> {code}
> 15/02/03 15:50:25 ERROR namenode.NameNode: Failed to start namenode.
> java.lang.ClassCastException: org.apache.log4j.Logger cannot be cast to
> org.apache.commons.logging.Log
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1557)
> 15/02/03 15:50:25 INFO util.ExitUtil: Exiting with status 1
> {code}
> Location of the exception in NameNode.java:
> {code}
> public static void main(String argv[]) throws Exception {
> if (DFSUtil.parseHelpArgument(argv, NameNode.USAGE, System.out, true)) {
> System.exit(0);
> }
> try {
> StringUtils.startupShutdownMessage(NameNode.class, argv,
> (org.apache.commons.logging.Log)
> LogManager.getLogger(LOG.getName())); < Failed here.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6994) libhdfs3 - A native C/C++ HDFS client
[ https://issues.apache.org/jira/browse/HDFS-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305880#comment-14305880 ] Colin Patrick McCabe commented on HDFS-6994: Hi Haohui, That is interesting. Why do you feel libhdfs3 and the Java client cannot support "thousands of files concurrently"? What are you doing differently that you believe will be better for this application? > libhdfs3 - A native C/C++ HDFS client > - > > Key: HDFS-6994 > URL: https://issues.apache.org/jira/browse/HDFS-6994 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Attachments: HDFS-6994-rpc-8.patch, HDFS-6994.patch > > > Hi All > I just got the permission to open source libhdfs3, which is a native C/C++ > HDFS client based on Hadoop RPC protocol and HDFS Data Transfer Protocol. > libhdfs3 provide the libhdfs style C interface and a C++ interface. Support > both HADOOP RPC version 8 and 9. Support Namenode HA and Kerberos > authentication. > libhdfs3 is currently used by HAWQ of Pivotal > I'd like to integrate libhdfs3 into HDFS source code to benefit others. > You can find libhdfs3 code from github > https://github.com/PivotalRD/libhdfs3 > http://pivotalrd.github.io/libhdfs3/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7719) BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state associated with volumes
[
https://issues.apache.org/jira/browse/HDFS-7719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305860#comment-14305860
]
Colin Patrick McCabe commented on HDFS-7719:
I ran TestDFSMkdirs locally and can confirm that it succeeds. This is
definitely Jenkins flakiness. Committed to 2.7
> BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state
> associated with volumes
>
>
> Key: HDFS-7719
> URL: https://issues.apache.org/jira/browse/HDFS-7719
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7719.000.patch, HDFS-7719.001.patch,
> HDFS-7719.002.patch, HDFS-7719.003.patch
>
>
> The parameter of {{BlockPoolSliceStorage#removeVolumes()}} is a set of volume
> level directories, thus {{BlockPoolSliceStorage}} could not directly compare
> its own {{StorageDirs}} with this volume-level directory. The result of that
> is {{BlockPoolSliceStorage}} did not actually remove the targeted
> {{StorageDirectory}}.
> It will cause failure when remove a volume and then immediately add a volume
> back with the same mount point..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7719) BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state associated with volumes
[
https://issues.apache.org/jira/browse/HDFS-7719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305849#comment-14305849
]
Colin Patrick McCabe edited comment on HDFS-7719 at 2/4/15 8:01 PM:
+1. Thanks, Eddy
{code}
java.lang.NoClassDefFoundError:
org/apache/hadoop/security/proto/SecurityProtos$GetDelegationTokenRequestProto$1
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
{code}
This looks unrelated. I'm not sure why we see these {{NoClassDefFoundError}}
messages intermittently on Jenkins. The last time we looked into this, it was
an Ansible issue with JDK classes getting moved around during the test run... I
thought we had fixed that though.
was (Author: cmccabe):
+1. Thanks, Eddy
{code}
java.lang.NoClassDefFoundError:
org/apache/hadoop/security/proto/SecurityProtos$GetDelegationTokenRequestProto$1
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
{code}
This looks unrelated. I'm not sure why we see these {{NoClassDefFoundError}}
messages intermittently on Jenkins. The last time we looked into this, it was
a Puppet issue with JDK classes getting moved around during the test run... I
thought we had fixed that though.
> BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state
> associated with volumes
>
>
> Key: HDFS-7719
> URL: https://issues.apache.org/jira/browse/HDFS-7719
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7719.000.patch, HDFS-7719.001.patch,
> HDFS-7719.002.patch, HDFS-7719.003.patch
>
>
> The parameter of {{BlockPoolSliceStorage#removeVolumes()}} is a set of volume
> level directories, thus {{BlockPoolSliceStorage}} could not directly compare
> its own {{StorageDirs}} with this volume-level directory. The result of that
> is {{BlockPoolSliceStorage}} did not actually remove the targeted
> {{StorageDirectory}}.
> It will cause failure when remove a volume and then immediately add a volume
> back with the same mount point..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7719) BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state associated with volumes
[
https://issues.apache.org/jira/browse/HDFS-7719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305849#comment-14305849
]
Colin Patrick McCabe commented on HDFS-7719:
+1. Thanks, Eddy
{code}
java.lang.NoClassDefFoundError:
org/apache/hadoop/security/proto/SecurityProtos$GetDelegationTokenRequestProto$1
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
{code}
This looks unrelated. I'm not sure why we see these {{NoClassDefFoundError}}
messages intermittently on Jenkins. The last time we looked into this, it was
a Puppet issue with JDK classes getting moved around during the test run... I
thought we had fixed that though.
> BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state
> associated with volumes
>
>
> Key: HDFS-7719
> URL: https://issues.apache.org/jira/browse/HDFS-7719
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7719.000.patch, HDFS-7719.001.patch,
> HDFS-7719.002.patch, HDFS-7719.003.patch
>
>
> The parameter of {{BlockPoolSliceStorage#removeVolumes()}} is a set of volume
> level directories, thus {{BlockPoolSliceStorage}} could not directly compare
> its own {{StorageDirs}} with this volume-level directory. The result of that
> is {{BlockPoolSliceStorage}} did not actually remove the targeted
> {{StorageDirectory}}.
> It will cause failure when remove a volume and then immediately add a volume
> back with the same mount point..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7719) BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state associated with volumes
[
https://issues.apache.org/jira/browse/HDFS-7719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-7719:
---
Summary: BlockPoolSliceStorage#removeVolumes fails to remove some in-memory
state associated with volumes (was: BlockPoolSliceStorage could not remove
storageDir.)
> BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state
> associated with volumes
>
>
> Key: HDFS-7719
> URL: https://issues.apache.org/jira/browse/HDFS-7719
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7719.000.patch, HDFS-7719.001.patch,
> HDFS-7719.002.patch, HDFS-7719.003.patch
>
>
> The parameter of {{BlockPoolSliceStorage#removeVolumes()}} is a set of volume
> level directories, thus {{BlockPoolSliceStorage}} could not directly compare
> its own {{StorageDirs}} with this volume-level directory. The result of that
> is {{BlockPoolSliceStorage}} did not actually remove the targeted
> {{StorageDirectory}}.
> It will cause failure when remove a volume and then immediately add a volume
> back with the same mount point..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7734) Class cast exception in NameNode#main
[
https://issues.apache.org/jira/browse/HDFS-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305824#comment-14305824
]
Hudson commented on HDFS-7734:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7004 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7004/])
HDFS-7734. Class cast exception in NameNode#main. Contributed by Yi Liu. (wang:
rev 9175105eeaecf0a1d60b57989b73ce45cee4689b)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LogAdapter.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SignalLogger.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
> Class cast exception in NameNode#main
> -
>
> Key: HDFS-7734
> URL: https://issues.apache.org/jira/browse/HDFS-7734
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>Assignee: Yi Liu
>Priority: Blocker
> Fix For: 2.7.0
>
> Attachments: HDFS-7734.001.patch, HDFS-7734.002.patch
>
>
> NameNode hits the following exception immediately on startup.
> {code}
> 15/02/03 15:50:25 ERROR namenode.NameNode: Failed to start namenode.
> java.lang.ClassCastException: org.apache.log4j.Logger cannot be cast to
> org.apache.commons.logging.Log
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1557)
> 15/02/03 15:50:25 INFO util.ExitUtil: Exiting with status 1
> {code}
> Location of the exception in NameNode.java:
> {code}
> public static void main(String argv[]) throws Exception {
> if (DFSUtil.parseHelpArgument(argv, NameNode.USAGE, System.out, true)) {
> System.exit(0);
> }
> try {
> StringUtils.startupShutdownMessage(NameNode.class, argv,
> (org.apache.commons.logging.Log)
> LogManager.getLogger(LOG.getName())); < Failed here.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7734) Class cast exception in NameNode#main
[
https://issues.apache.org/jira/browse/HDFS-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-7734:
--
Resolution: Fixed
Fix Version/s: 2.7.0
Status: Resolved (was: Patch Available)
Committed to trunk and branch-2, thanks for the patch Yi, and Arpit/Kihwal too
for discussion.
> Class cast exception in NameNode#main
> -
>
> Key: HDFS-7734
> URL: https://issues.apache.org/jira/browse/HDFS-7734
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>Assignee: Yi Liu
>Priority: Blocker
> Fix For: 2.7.0
>
> Attachments: HDFS-7734.001.patch, HDFS-7734.002.patch
>
>
> NameNode hits the following exception immediately on startup.
> {code}
> 15/02/03 15:50:25 ERROR namenode.NameNode: Failed to start namenode.
> java.lang.ClassCastException: org.apache.log4j.Logger cannot be cast to
> org.apache.commons.logging.Log
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1557)
> 15/02/03 15:50:25 INFO util.ExitUtil: Exiting with status 1
> {code}
> Location of the exception in NameNode.java:
> {code}
> public static void main(String argv[]) throws Exception {
> if (DFSUtil.parseHelpArgument(argv, NameNode.USAGE, System.out, true)) {
> System.exit(0);
> }
> try {
> StringUtils.startupShutdownMessage(NameNode.class, argv,
> (org.apache.commons.logging.Log)
> LogManager.getLogger(LOG.getName())); < Failed here.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7734) Class cast exception in NameNode#main
[
https://issues.apache.org/jira/browse/HDFS-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305790#comment-14305790
]
Kihwal Lee commented on HDFS-7734:
--
bq. Yi's patch looks good to me, should we just commit this for now, and figure
out additional testing later?
+1 on this approach.
> Class cast exception in NameNode#main
> -
>
> Key: HDFS-7734
> URL: https://issues.apache.org/jira/browse/HDFS-7734
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>Assignee: Yi Liu
>Priority: Blocker
> Attachments: HDFS-7734.001.patch, HDFS-7734.002.patch
>
>
> NameNode hits the following exception immediately on startup.
> {code}
> 15/02/03 15:50:25 ERROR namenode.NameNode: Failed to start namenode.
> java.lang.ClassCastException: org.apache.log4j.Logger cannot be cast to
> org.apache.commons.logging.Log
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1557)
> 15/02/03 15:50:25 INFO util.ExitUtil: Exiting with status 1
> {code}
> Location of the exception in NameNode.java:
> {code}
> public static void main(String argv[]) throws Exception {
> if (DFSUtil.parseHelpArgument(argv, NameNode.USAGE, System.out, true)) {
> System.exit(0);
> }
> try {
> StringUtils.startupShutdownMessage(NameNode.class, argv,
> (org.apache.commons.logging.Log)
> LogManager.getLogger(LOG.getName())); < Failed here.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7734) Class cast exception in NameNode#main
[
https://issues.apache.org/jira/browse/HDFS-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305771#comment-14305771
]
Andrew Wang commented on HDFS-7734:
---
Yi's patch looks good to me, should we just commit this for now, and figure out
additional testing later? We rely on miniclusters pretty heavily for testing,
and I'm sort of okay with the little gap between {{main}} and
{{createNameNode}}.
> Class cast exception in NameNode#main
> -
>
> Key: HDFS-7734
> URL: https://issues.apache.org/jira/browse/HDFS-7734
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>Assignee: Yi Liu
>Priority: Blocker
> Attachments: HDFS-7734.001.patch, HDFS-7734.002.patch
>
>
> NameNode hits the following exception immediately on startup.
> {code}
> 15/02/03 15:50:25 ERROR namenode.NameNode: Failed to start namenode.
> java.lang.ClassCastException: org.apache.log4j.Logger cannot be cast to
> org.apache.commons.logging.Log
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1557)
> 15/02/03 15:50:25 INFO util.ExitUtil: Exiting with status 1
> {code}
> Location of the exception in NameNode.java:
> {code}
> public static void main(String argv[]) throws Exception {
> if (DFSUtil.parseHelpArgument(argv, NameNode.USAGE, System.out, true)) {
> System.exit(0);
> }
> try {
> StringUtils.startupShutdownMessage(NameNode.class, argv,
> (org.apache.commons.logging.Log)
> LogManager.getLogger(LOG.getName())); < Failed here.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7704) DN heartbeat to Active NN may be blocked and expire if connection to Standby NN continues to time out.
[ https://issues.apache.org/jira/browse/HDFS-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305742#comment-14305742 ] Charles Lamb commented on HDFS-7704: Oh, sorry, one more comment. In the test, to be consistent with the code that is already there, you can add an import static for Assert.assertTrue rather than importing (non-static) Assert. Or, the opposite (eliminate the import statics). > DN heartbeat to Active NN may be blocked and expire if connection to Standby > NN continues to time out. > --- > > Key: HDFS-7704 > URL: https://issues.apache.org/jira/browse/HDFS-7704 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 2.5.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-7704-v2.patch, HDFS-7704-v3.patch, > HDFS-7704-v4.patch, HDFS-7704.patch > > > There are couple of synchronous calls in BPOfferservice (i.e reportBadBlocks > and trySendErrorReport) which will wait for both of the actor threads to > process this calls. > This calls are made with writeLock acquired. > When reportBadBlocks() is blocked at the RPC layer due to unreachable NN, > subsequent heartbeat response processing has to wait for the write lock. It > eventually gets through, but takes too long and it blocks the next heartbeat. > In our HA cluster setup, the standby namenode was taking a long time to > process the request. > Requesting improvement in datanode to make the above calls asynchronous since > these reports don't have any specific > deadlines, so extra few seconds of delay should be acceptable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leitao Guo updated HDFS-7692:
-
Release Note: Please help review the patch. Thanks!
Status: Patch Available (was: Open)
> DataStorage#addStorageLocations(...) should support MultiThread to speedup
> the upgrade of block pool at multi storage directories.
> --
>
> Key: HDFS-7692
> URL: https://issues.apache.org/jira/browse/HDFS-7692
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.5.2
>Reporter: Leitao Guo
> Attachments: HDFS-7692.01.patch
>
>
> {code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
> for (StorageLocation dataDir : dataDirs) {
> File root = dataDir.getFile();
> ... ...
> bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
> startOpt);
> addBlockPoolStorage(bpid, bpStorage);
> ... ...
> successVolumes.add(dataDir);
> }
> {code}
> In the above code the storage directories will be analyzed one by one, which
> is really time consuming when upgrading HDFS with datanodes have dozens of
> large volumes. MultiThread dataDirs analyzing should be supported here to
> speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leitao Guo updated HDFS-7692:
-
Attachment: HDFS-7692.01.patch
> DataStorage#addStorageLocations(...) should support MultiThread to speedup
> the upgrade of block pool at multi storage directories.
> --
>
> Key: HDFS-7692
> URL: https://issues.apache.org/jira/browse/HDFS-7692
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.5.2
>Reporter: Leitao Guo
> Attachments: HDFS-7692.01.patch
>
>
> {code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
> for (StorageLocation dataDir : dataDirs) {
> File root = dataDir.getFile();
> ... ...
> bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
> startOpt);
> addBlockPoolStorage(bpid, bpStorage);
> ... ...
> successVolumes.add(dataDir);
> }
> {code}
> In the above code the storage directories will be analyzed one by one, which
> is really time consuming when upgrading HDFS with datanodes have dozens of
> large volumes. MultiThread dataDirs analyzing should be supported here to
> speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7733) NFS: readdir/readdirplus return null directory attribute on failure
[
https://issues.apache.org/jira/browse/HDFS-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305682#comment-14305682
]
Brandon Li commented on HDFS-7733:
--
+1. The patch looks good to me. Thank you, Arpit, for the fix.
> NFS: readdir/readdirplus return null directory attribute on failure
> ---
>
> Key: HDFS-7733
> URL: https://issues.apache.org/jira/browse/HDFS-7733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.6.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-7733.01.patch
>
>
> NFS readdir and readdirplus operations return a null directory attribute on
> some failure paths. This causes clients to get a 'Stale file handle' error
> which can only be fixed by unmounting and remounting the share.
> The issue can be reproduced by running 'ls' against a large directory which
> is being actively modified, triggering the 'cookie mismatch' failure path.
> {code}
> } else {
> LOG.error("cookieverf mismatch. request cookieverf: " + cookieVerf
> + " dir cookieverf: " + dirStatus.getModificationTime());
> return new READDIRPLUS3Response(Nfs3Status.NFS3ERR_BAD_COOKIE);
> }
> {code}
> Thanks to [~brandonli] for catching the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305670#comment-14305670
]
Zhe Zhang commented on HDFS-7652:
-
This JIRA can only be tested with client striped writing.
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7734) Class cast exception in NameNode#main
[
https://issues.apache.org/jira/browse/HDFS-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305478#comment-14305478
]
Hadoop QA commented on HDFS-7734:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696452/HDFS-7734.002.patch
against trunk revision 42548f4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.ipc.TestCallQueueManager
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9429//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9429//console
This message is automatically generated.
> Class cast exception in NameNode#main
> -
>
> Key: HDFS-7734
> URL: https://issues.apache.org/jira/browse/HDFS-7734
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>Assignee: Yi Liu
>Priority: Blocker
> Attachments: HDFS-7734.001.patch, HDFS-7734.002.patch
>
>
> NameNode hits the following exception immediately on startup.
> {code}
> 15/02/03 15:50:25 ERROR namenode.NameNode: Failed to start namenode.
> java.lang.ClassCastException: org.apache.log4j.Logger cannot be cast to
> org.apache.commons.logging.Log
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1557)
> 15/02/03 15:50:25 INFO util.ExitUtil: Exiting with status 1
> {code}
> Location of the exception in NameNode.java:
> {code}
> public static void main(String argv[]) throws Exception {
> if (DFSUtil.parseHelpArgument(argv, NameNode.USAGE, System.out, true)) {
> System.exit(0);
> }
> try {
> StringUtils.startupShutdownMessage(NameNode.class, argv,
> (org.apache.commons.logging.Log)
> LogManager.getLogger(LOG.getName())); < Failed here.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7704) DN heartbeat to Active NN may be blocked and expire if connection to Standby NN continues to time out.
[ https://issues.apache.org/jira/browse/HDFS-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305452#comment-14305452 ] Charles Lamb commented on HDFS-7704: Hi [~shahrs87], I only have a few nits on the latest rev: BPServiceActor.java: Line continuations are 4 spaces. At line 257 you've introduced a line containing spaces. Also, you've removed the last newline of the file. BPServiceActorAction.java: Line continuations are 4 spaces (statements in a new block are indented 2 spaces). ErrorReportAction.java: s/A ErrorReportAction/An ErrorReportAction/ Can LOG be private? ReportBadBlockAction.java Can LOG be private? s/to namenode :/to namenode: / The block comment in that catch should be: /* * One common reason ... */ > DN heartbeat to Active NN may be blocked and expire if connection to Standby > NN continues to time out. > --- > > Key: HDFS-7704 > URL: https://issues.apache.org/jira/browse/HDFS-7704 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 2.5.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-7704-v2.patch, HDFS-7704-v3.patch, > HDFS-7704-v4.patch, HDFS-7704.patch > > > There are couple of synchronous calls in BPOfferservice (i.e reportBadBlocks > and trySendErrorReport) which will wait for both of the actor threads to > process this calls. > This calls are made with writeLock acquired. > When reportBadBlocks() is blocked at the RPC layer due to unreachable NN, > subsequent heartbeat response processing has to wait for the write lock. It > eventually gets through, but takes too long and it blocks the next heartbeat. > In our HA cluster setup, the standby namenode was taking a long time to > process the request. > Requesting improvement in datanode to make the above calls asynchronous since > these reports don't have any specific > deadlines, so extra few seconds of delay should be acceptable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7712) Switch blockStateChangeLog to use slf4j
[ https://issues.apache.org/jira/browse/HDFS-7712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305262#comment-14305262 ] Hudson commented on HDFS-7712: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2045 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2045/]) HDFS-7712. Switch blockStateChangeLog to use slf4j. (wang: rev 3ae38ec7dfa1aaf451cf889cec6cf862379af32a) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPipelines.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsWithMultipleNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend3.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestSymlinkHdfs.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreationDelete.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithMultipleNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsckWithMultipleNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java > Switch blockStateChangeLog to use slf4j > --- > > Key: HDFS-7712 > URL: https://issues.apache.org/jira/browse/HDFS-7712 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-7712.001.patch, hdfs-7712.002.patch, > hdfs-7712.003.patch, hdfs-7712.004.patch > > > As pointed out in HDFS-7706, updating blockStateChangeLog to use slf4j will > save a lot of string construction costs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7707) Edit log corruption due to delayed block removal again
[ https://issues.apache.org/jira/browse/HDFS-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305261#comment-14305261 ] Hudson commented on HDFS-7707: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2045 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2045/]) HDFS-7707. Edit log corruption due to delayed block removal again. Contributed by Yongjun Zhang (kihwal: rev 843806d03ab1a24f191782f42eb817505228eb9f) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeleteRace.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Edit log corruption due to delayed block removal again > -- > > Key: HDFS-7707 > URL: https://issues.apache.org/jira/browse/HDFS-7707 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Fix For: 2.7.0 > > Attachments: HDFS-7707.001.patch, HDFS-7707.002.patch, > HDFS-7707.003.patch, reproduceHDFS-7707.patch > > > Edit log corruption is seen again, even with the fix of HDFS-6825. > Prior to HDFS-6825 fix, if dirX is deleted recursively, an OP_CLOSE can get > into edit log for the fileY under dirX, thus corrupting the edit log > (restarting NN with the edit log would fail). > What HDFS-6825 does to fix this issue is, to detect whether fileY is already > deleted by checking the ancestor dirs on it's path, if any of them doesn't > exist, then fileY is already deleted, and don't put OP_CLOSE to edit log for > the file. > For this new edit log corruption, what I found was, the client first deleted > dirX recursively, then create another dir with exactly the same name as dirX > right away. Because HDFS-6825 count on the namespace checking (whether dirX > exists in its parent dir) to decide whether a file has been deleted, the > newly created dirX defeats this checking, thus OP_CLOSE for the already > deleted file gets into the edit log, due to delayed block removal. > What we need to do is to have a more robust way to detect whether a file has > been deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7721) The HDFS BlockScanner may run fast during the first hour
[
https://issues.apache.org/jira/browse/HDFS-7721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305267#comment-14305267
]
Hudson commented on HDFS-7721:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2045 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2045/])
HDFS-7721. The HDFS BlockScanner may run fast during the first hour (cmccabe)
(cmccabe: rev 115428176e1d919fe7d54d01b34dfda57d1b3950)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockScanner.java
> The HDFS BlockScanner may run fast during the first hour
>
>
> Key: HDFS-7721
> URL: https://issues.apache.org/jira/browse/HDFS-7721
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Colin Patrick McCabe
> Fix For: 3.0.0
>
> Attachments: HDFS-7721.001.patch
>
>
> -
> https://builds.apache.org/job/PreCommit-HDFS-Build/9375//testReport/org.apache.hadoop.hdfs.server.datanode/TestBlockScanner/testScanRateLimit/
> -
> https://builds.apache.org/job/PreCommit-HDFS-Build/9365//testReport/org.apache.hadoop.hdfs.server.datanode/TestBlockScanner/testScanRateLimit/
> {code}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hdfs.server.datanode.TestBlockScanner.testScanRateLimit(TestBlockScanner.java:439)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7707) Edit log corruption due to delayed block removal again
[ https://issues.apache.org/jira/browse/HDFS-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305208#comment-14305208 ] Hudson commented on HDFS-7707: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #95 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/95/]) HDFS-7707. Edit log corruption due to delayed block removal again. Contributed by Yongjun Zhang (kihwal: rev 843806d03ab1a24f191782f42eb817505228eb9f) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeleteRace.java > Edit log corruption due to delayed block removal again > -- > > Key: HDFS-7707 > URL: https://issues.apache.org/jira/browse/HDFS-7707 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Fix For: 2.7.0 > > Attachments: HDFS-7707.001.patch, HDFS-7707.002.patch, > HDFS-7707.003.patch, reproduceHDFS-7707.patch > > > Edit log corruption is seen again, even with the fix of HDFS-6825. > Prior to HDFS-6825 fix, if dirX is deleted recursively, an OP_CLOSE can get > into edit log for the fileY under dirX, thus corrupting the edit log > (restarting NN with the edit log would fail). > What HDFS-6825 does to fix this issue is, to detect whether fileY is already > deleted by checking the ancestor dirs on it's path, if any of them doesn't > exist, then fileY is already deleted, and don't put OP_CLOSE to edit log for > the file. > For this new edit log corruption, what I found was, the client first deleted > dirX recursively, then create another dir with exactly the same name as dirX > right away. Because HDFS-6825 count on the namespace checking (whether dirX > exists in its parent dir) to decide whether a file has been deleted, the > newly created dirX defeats this checking, thus OP_CLOSE for the already > deleted file gets into the edit log, due to delayed block removal. > What we need to do is to have a more robust way to detect whether a file has > been deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7712) Switch blockStateChangeLog to use slf4j
[ https://issues.apache.org/jira/browse/HDFS-7712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305209#comment-14305209 ] Hudson commented on HDFS-7712: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #95 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/95/]) HDFS-7712. Switch blockStateChangeLog to use slf4j. (wang: rev 3ae38ec7dfa1aaf451cf889cec6cf862379af32a) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsckWithMultipleNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreationDelete.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend3.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPipelines.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsWithMultipleNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithMultipleNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestSymlinkHdfs.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java > Switch blockStateChangeLog to use slf4j > --- > > Key: HDFS-7712 > URL: https://issues.apache.org/jira/browse/HDFS-7712 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-7712.001.patch, hdfs-7712.002.patch, > hdfs-7712.003.patch, hdfs-7712.004.patch > > > As pointed out in HDFS-7706, updating blockStateChangeLog to use slf4j will > save a lot of string construction costs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7721) The HDFS BlockScanner may run fast during the first hour
[
https://issues.apache.org/jira/browse/HDFS-7721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305214#comment-14305214
]
Hudson commented on HDFS-7721:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #95 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/95/])
HDFS-7721. The HDFS BlockScanner may run fast during the first hour (cmccabe)
(cmccabe: rev 115428176e1d919fe7d54d01b34dfda57d1b3950)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockScanner.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> The HDFS BlockScanner may run fast during the first hour
>
>
> Key: HDFS-7721
> URL: https://issues.apache.org/jira/browse/HDFS-7721
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Colin Patrick McCabe
> Fix For: 3.0.0
>
> Attachments: HDFS-7721.001.patch
>
>
> -
> https://builds.apache.org/job/PreCommit-HDFS-Build/9375//testReport/org.apache.hadoop.hdfs.server.datanode/TestBlockScanner/testScanRateLimit/
> -
> https://builds.apache.org/job/PreCommit-HDFS-Build/9365//testReport/org.apache.hadoop.hdfs.server.datanode/TestBlockScanner/testScanRateLimit/
> {code}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hdfs.server.datanode.TestBlockScanner.testScanRateLimit(TestBlockScanner.java:439)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7734) Class cast exception in NameNode#main
[
https://issues.apache.org/jira/browse/HDFS-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305190#comment-14305190
]
Kihwal Lee commented on HDFS-7734:
--
Since {{MiniDFSCluster}} directly calls {{createNamenode()}}, existing unit
tests didn't caught this bug. We can either add a new test case or add a smoke
test that brings up a pseudo distributed file system and runs simple tests
during precommit.
> Class cast exception in NameNode#main
> -
>
> Key: HDFS-7734
> URL: https://issues.apache.org/jira/browse/HDFS-7734
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>Assignee: Yi Liu
>Priority: Blocker
> Attachments: HDFS-7734.001.patch, HDFS-7734.002.patch
>
>
> NameNode hits the following exception immediately on startup.
> {code}
> 15/02/03 15:50:25 ERROR namenode.NameNode: Failed to start namenode.
> java.lang.ClassCastException: org.apache.log4j.Logger cannot be cast to
> org.apache.commons.logging.Log
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1557)
> 15/02/03 15:50:25 INFO util.ExitUtil: Exiting with status 1
> {code}
> Location of the exception in NameNode.java:
> {code}
> public static void main(String argv[]) throws Exception {
> if (DFSUtil.parseHelpArgument(argv, NameNode.USAGE, System.out, true)) {
> System.exit(0);
> }
> try {
> StringUtils.startupShutdownMessage(NameNode.class, argv,
> (org.apache.commons.logging.Log)
> LogManager.getLogger(LOG.getName())); < Failed here.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7734) Class cast exception in NameNode#main
[
https://issues.apache.org/jira/browse/HDFS-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305184#comment-14305184
]
Hadoop QA commented on HDFS-7734:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696437/HDFS-7734.002.patch
against trunk revision 26dee14.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9428//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9428//console
This message is automatically generated.
> Class cast exception in NameNode#main
> -
>
> Key: HDFS-7734
> URL: https://issues.apache.org/jira/browse/HDFS-7734
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>Assignee: Yi Liu
>Priority: Blocker
> Attachments: HDFS-7734.001.patch, HDFS-7734.002.patch
>
>
> NameNode hits the following exception immediately on startup.
> {code}
> 15/02/03 15:50:25 ERROR namenode.NameNode: Failed to start namenode.
> java.lang.ClassCastException: org.apache.log4j.Logger cannot be cast to
> org.apache.commons.logging.Log
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1557)
> 15/02/03 15:50:25 INFO util.ExitUtil: Exiting with status 1
> {code}
> Location of the exception in NameNode.java:
> {code}
> public static void main(String argv[]) throws Exception {
> if (DFSUtil.parseHelpArgument(argv, NameNode.USAGE, System.out, true)) {
> System.exit(0);
> }
> try {
> StringUtils.startupShutdownMessage(NameNode.class, argv,
> (org.apache.commons.logging.Log)
> LogManager.getLogger(LOG.getName())); < Failed here.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7712) Switch blockStateChangeLog to use slf4j
[ https://issues.apache.org/jira/browse/HDFS-7712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305147#comment-14305147 ] Kihwal Lee edited comment on HDFS-7712 at 2/4/15 2:27 PM: -- OMG, I missed that. was (Author: kihwal): OMG, I missed that. Precommit test runs will look deceivingly clean until we fix this. > Switch blockStateChangeLog to use slf4j > --- > > Key: HDFS-7712 > URL: https://issues.apache.org/jira/browse/HDFS-7712 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-7712.001.patch, hdfs-7712.002.patch, > hdfs-7712.003.patch, hdfs-7712.004.patch > > > As pointed out in HDFS-7706, updating blockStateChangeLog to use slf4j will > save a lot of string construction costs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7712) Switch blockStateChangeLog to use slf4j
[ https://issues.apache.org/jira/browse/HDFS-7712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305147#comment-14305147 ] Kihwal Lee commented on HDFS-7712: -- OMG, I missed that. Precommit test runs will look deceivingly clean until we fix this. > Switch blockStateChangeLog to use slf4j > --- > > Key: HDFS-7712 > URL: https://issues.apache.org/jira/browse/HDFS-7712 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-7712.001.patch, hdfs-7712.002.patch, > hdfs-7712.003.patch, hdfs-7712.004.patch > > > As pointed out in HDFS-7706, updating blockStateChangeLog to use slf4j will > save a lot of string construction costs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7707) Edit log corruption due to delayed block removal again
[ https://issues.apache.org/jira/browse/HDFS-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305126#comment-14305126 ] Hudson commented on HDFS-7707: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #91 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/91/]) HDFS-7707. Edit log corruption due to delayed block removal again. Contributed by Yongjun Zhang (kihwal: rev 843806d03ab1a24f191782f42eb817505228eb9f) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeleteRace.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Edit log corruption due to delayed block removal again > -- > > Key: HDFS-7707 > URL: https://issues.apache.org/jira/browse/HDFS-7707 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Fix For: 2.7.0 > > Attachments: HDFS-7707.001.patch, HDFS-7707.002.patch, > HDFS-7707.003.patch, reproduceHDFS-7707.patch > > > Edit log corruption is seen again, even with the fix of HDFS-6825. > Prior to HDFS-6825 fix, if dirX is deleted recursively, an OP_CLOSE can get > into edit log for the fileY under dirX, thus corrupting the edit log > (restarting NN with the edit log would fail). > What HDFS-6825 does to fix this issue is, to detect whether fileY is already > deleted by checking the ancestor dirs on it's path, if any of them doesn't > exist, then fileY is already deleted, and don't put OP_CLOSE to edit log for > the file. > For this new edit log corruption, what I found was, the client first deleted > dirX recursively, then create another dir with exactly the same name as dirX > right away. Because HDFS-6825 count on the namespace checking (whether dirX > exists in its parent dir) to decide whether a file has been deleted, the > newly created dirX defeats this checking, thus OP_CLOSE for the already > deleted file gets into the edit log, due to delayed block removal. > What we need to do is to have a more robust way to detect whether a file has > been deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7712) Switch blockStateChangeLog to use slf4j
[ https://issues.apache.org/jira/browse/HDFS-7712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305127#comment-14305127 ] Hudson commented on HDFS-7712: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #91 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/91/]) HDFS-7712. Switch blockStateChangeLog to use slf4j. (wang: rev 3ae38ec7dfa1aaf451cf889cec6cf862379af32a) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsckWithMultipleNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestSymlinkHdfs.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend3.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithMultipleNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreationDelete.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPipelines.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsWithMultipleNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java > Switch blockStateChangeLog to use slf4j > --- > > Key: HDFS-7712 > URL: https://issues.apache.org/jira/browse/HDFS-7712 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-7712.001.patch, hdfs-7712.002.patch, > hdfs-7712.003.patch, hdfs-7712.004.patch > > > As pointed out in HDFS-7706, updating blockStateChangeLog to use slf4j will > save a lot of string construction costs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7721) The HDFS BlockScanner may run fast during the first hour
[
https://issues.apache.org/jira/browse/HDFS-7721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305132#comment-14305132
]
Hudson commented on HDFS-7721:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #91 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/91/])
HDFS-7721. The HDFS BlockScanner may run fast during the first hour (cmccabe)
(cmccabe: rev 115428176e1d919fe7d54d01b34dfda57d1b3950)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockScanner.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
> The HDFS BlockScanner may run fast during the first hour
>
>
> Key: HDFS-7721
> URL: https://issues.apache.org/jira/browse/HDFS-7721
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Colin Patrick McCabe
> Fix For: 3.0.0
>
> Attachments: HDFS-7721.001.patch
>
>
> -
> https://builds.apache.org/job/PreCommit-HDFS-Build/9375//testReport/org.apache.hadoop.hdfs.server.datanode/TestBlockScanner/testScanRateLimit/
> -
> https://builds.apache.org/job/PreCommit-HDFS-Build/9365//testReport/org.apache.hadoop.hdfs.server.datanode/TestBlockScanner/testScanRateLimit/
> {code}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hdfs.server.datanode.TestBlockScanner.testScanRateLimit(TestBlockScanner.java:439)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7735) Optimize decommission Datanodes to reduce the impact on NameNode's performance
[ https://issues.apache.org/jira/browse/HDFS-7735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305122#comment-14305122 ] Kihwal Lee commented on HDFS-7735: -- HDFS-7411 is related. Please have a look. > Optimize decommission Datanodes to reduce the impact on NameNode's performance > -- > > Key: HDFS-7735 > URL: https://issues.apache.org/jira/browse/HDFS-7735 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: zhaoyunjiong >Assignee: zhaoyunjiong > Attachments: HDFS-7735.patch > > > When decommission DataNodes, by default, DecommissionManager will check > progress every 30 seconds, and it will hold writeLock of Namesystem. It > significantly impact NameNode's performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7712) Switch blockStateChangeLog to use slf4j
[ https://issues.apache.org/jira/browse/HDFS-7712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305100#comment-14305100 ] Hudson commented on HDFS-7712: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2026 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2026/]) HDFS-7712. Switch blockStateChangeLog to use slf4j. (wang: rev 3ae38ec7dfa1aaf451cf889cec6cf862379af32a) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend3.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsckWithMultipleNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsWithMultipleNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestSymlinkHdfs.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPipelines.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreationDelete.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithMultipleNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java > Switch blockStateChangeLog to use slf4j > --- > > Key: HDFS-7712 > URL: https://issues.apache.org/jira/browse/HDFS-7712 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-7712.001.patch, hdfs-7712.002.patch, > hdfs-7712.003.patch, hdfs-7712.004.patch > > > As pointed out in HDFS-7706, updating blockStateChangeLog to use slf4j will > save a lot of string construction costs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
