[jira] [Updated] (HDFS-5191) revisit zero-copy API in FSDataInputStream to make it more intuitive
[ https://issues.apache.org/jira/browse/HDFS-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5191: --- Attachment: HDFS-5191-caching.006.patch * add the ability for the ByteBufferPool to create non-direct buffers. * fix JNI support and add JNI support for non-direct buffers. * add some unit tests > revisit zero-copy API in FSDataInputStream to make it more intuitive > > > Key: HDFS-5191 > URL: https://issues.apache.org/jira/browse/HDFS-5191 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, libhdfs >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5191-caching.001.patch, > HDFS-5191-caching.003.patch, HDFS-5191-caching.006.patch > > > As per the discussion on HDFS-4953, we should revisit the zero-copy API to > make it more intuitive for new users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5167) Add metrics about the NameNode retry cache
[
https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772579#comment-13772579
]
Hadoop QA commented on HDFS-5167:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604162/HDFS-5167.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5007//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5007//console
This message is automatically generated.
> Add metrics about the NameNode retry cache
> --
>
> Key: HDFS-5167
> URL: https://issues.apache.org/jira/browse/HDFS-5167
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, namenode
>Affects Versions: 3.0.0
>Reporter: Jing Zhao
>Priority: Minor
> Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch,
> HDFS-5167.4.patch, HDFS-5167.5.patch
>
>
> It will be helpful to have metrics in NameNode about the retry cache, such as
> the retry count etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5023) TestSnapshotPathINodes.testAllowSnapshot is failing in branch-2
[
https://issues.apache.org/jira/browse/HDFS-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772575#comment-13772575
]
Tsuyoshi OZAWA commented on HDFS-5023:
--
testNonSnapshotPathNodes also failed in the same environment.
{code}
testNonSnapshotPathINodes(org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes)
Time elapsed: 0.002 sec <<< FAILURE!
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at
org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes.assertSnapshot(TestSnapshotPathINodes.java:124)
at
org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes.testNonSnapshotPathINodes(TestSnapshotPathINodes.java:149)
{code}
> TestSnapshotPathINodes.testAllowSnapshot is failing in branch-2
> ---
>
> Key: HDFS-5023
> URL: https://issues.apache.org/jira/browse/HDFS-5023
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots, test
>Affects Versions: 2.3.0
>Reporter: Ravi Prakash
> Labels: test
> Attachments:
> org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes-output.txt,
> TEST-org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes.xml
>
>
> The assertion on line 91 is failing. I am using Fedora 19 + JDK7.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5023) TestSnapshotPathINodes.testAllowSnapshot is failing in branch-2
[
https://issues.apache.org/jira/browse/HDFS-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772573#comment-13772573
]
Tsuyoshi OZAWA commented on HDFS-5023:
--
I also faced this problem with JDK7 + Ubuntu.
{code}
Running org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes
Tests run: 6, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 5.285 sec <<<
FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes
testAllowSnapshot(org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes)
Time elapsed: 0.006 sec <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertFalse(Assert.java:68)
at org.junit.Assert.assertFalse(Assert.java:79)
at
org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes.testAllowSnapshot(TestSnapshotPathINodes.java:91)
{code}
> TestSnapshotPathINodes.testAllowSnapshot is failing in branch-2
> ---
>
> Key: HDFS-5023
> URL: https://issues.apache.org/jira/browse/HDFS-5023
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots, test
>Affects Versions: 2.3.0
>Reporter: Ravi Prakash
> Labels: test
> Attachments:
> org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes-output.txt,
> TEST-org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes.xml
>
>
> The assertion on line 91 is failing. I am using Fedora 19 + JDK7.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3059) ssl-server.xml causes NullPointer
[
https://issues.apache.org/jira/browse/HDFS-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772571#comment-13772571
]
Jing Zhao commented on HDFS-3059:
-
I think this is a very useful feature. We just met the similar problem in both
NN and DN when starting http server with https enabled. By applying the patch
we can quickly identify the cause of the problem.
> ssl-server.xml causes NullPointer
> -
>
> Key: HDFS-3059
> URL: https://issues.apache.org/jira/browse/HDFS-3059
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, security
>Affects Versions: 0.20.205.0, 1.0.0
> Environment: in core-site.xml:
> {code:xml}
>
> hadoop.security.authentication
> kerberos
>
>
> hadoop.security.authorization
> true
>
> {code}
> in hdfs-site.xml:
> {code:xml}
>
> dfs.https.server.keystore.resource
> /etc/hadoop/conf/ssl-server.xml
>
>
> dfs.https.enable
> true
>
>
> ...other security props
>
> {code}
>Reporter: Evert Lammerts
>Priority: Minor
> Attachments: HDFS-3059.patch, HDFS-3059.patch.2
>
>
> If ssl is enabled (dfs.https.enable) but ssl-server.xml is not available, a
> DN will crash during startup while setting up an SSL socket with a
> NullPointerException:
> {noformat}12/03/07 17:08:36 DEBUG security.Krb5AndCertsSslSocketConnector:
> useKerb = false, useCerts = true
> jetty.ssl.password : jetty.ssl.keypassword : 12/03/07 17:08:36 INFO
> mortbay.log: jetty-6.1.26.cloudera.1
> 12/03/07 17:08:36 INFO mortbay.log: Started
> selectchannelconnec...@p-worker35.alley.sara.nl:1006
> 12/03/07 17:08:36 DEBUG security.Krb5AndCertsSslSocketConnector: Creating new
> KrbServerSocket for: 0.0.0.0
> 12/03/07 17:08:36 WARN mortbay.log: java.lang.NullPointerException
> 12/03/07 17:08:36 WARN mortbay.log: failed
> Krb5AndCertsSslSocketConnector@0.0.0.0:50475: java.io.IOException:
> !JsseListener: java.lang.NullPointerException
> 12/03/07 17:08:36 WARN mortbay.log: failed Server@604788d5:
> java.io.IOException: !JsseListener: java.lang.NullPointerException
> 12/03/07 17:08:36 INFO mortbay.log: Stopped
> Krb5AndCertsSslSocketConnector@0.0.0.0:50475
> 12/03/07 17:08:36 INFO mortbay.log: Stopped
> selectchannelconnec...@p-worker35.alley.sara.nl:1006
> 12/03/07 17:08:37 INFO datanode.DataNode: Waiting for threadgroup to exit,
> active threads is 0{noformat}
> The same happens if I set an absolute path to an existing
> dfs.https.server.keystore.resource - in this case the file cannot be found
> but not even a WARN is given.
> Since in dfs.https.server.keystore.resource we know we need to have 4
> properties specified (ssl.server.truststore.location,
> ssl.server.keystore.location, ssl.server.keystore.password, and
> ssl.server.keystore.keypassword) we should check if they are set and throw an
> IOException if they are not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.05.patch Some more patch cleanup to reduce the size of the change. > Datanode must support all the volumes as individual storages > > > Key: HDFS-4988 > URL: https://issues.apache.org/jira/browse/HDFS-4988 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Suresh Srinivas >Assignee: Arpit Agarwal > Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, > HDFS-4988.03.patch, HDFS-4988.04.patch, HDFS-4988.05.patch > > > Currently all the volumes on datanode is reported as a single storage. This > change proposes reporting them as individual storage. This requires: > # A unique storage ID for each storage > #* This needs to be generated during formatting > # There should be an option to allow existing disks to be reported as single > storage unit for backward compatibility. > # A functionality is also needed to split the existing all volumes as single > storage unit to to individual storage units. > # -Configuration must allow for each storage unit a storage type attribute. > (Now HDFS-5000)- > # Block reports must be sent on a per storage basis. In some cases (such > memory tier) block reports may need to be sent more frequently. That means > block reporting period must be on a per storage type basis. > My proposal is for new clusters to configure volumes by default as separate > storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5233) Use Datanode UUID to identify Datanodes
[ https://issues.apache.org/jira/browse/HDFS-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5233: Attachment: HDFS-5233.02.patch Updated patch. > Use Datanode UUID to identify Datanodes > --- > > Key: HDFS-5233 > URL: https://issues.apache.org/jira/browse/HDFS-5233 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-5233.01.patch, HDFS-5233.02.patch > > > StorageID currently identifies both a Datanode and a storage attached to the > Datanode. Once we start tracking multiple storages per datanode we would like > to deprecate StorageID in favor of a DatanodeUuid for the datanode and a > StorageUuid per storage. > This Jira track replacing StorageID with DatanodeUuid wherever it is used to > identify a Datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5167) Add metrics about the NameNode retry cache
[ https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated HDFS-5167: - Attachment: HDFS-5167.5.patch Fixed to pass broken tests about secondary namenode. > Add metrics about the NameNode retry cache > -- > > Key: HDFS-5167 > URL: https://issues.apache.org/jira/browse/HDFS-5167 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, namenode >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Priority: Minor > Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch, > HDFS-5167.4.patch, HDFS-5167.5.patch > > > It will be helpful to have metrics in NameNode about the retry cache, such as > the retry count etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5232) Protocol changes to transmit StorageUuid
[ https://issues.apache.org/jira/browse/HDFS-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5232: Attachment: HDFS-5232.03.patch > Protocol changes to transmit StorageUuid > > > Key: HDFS-5232 > URL: https://issues.apache.org/jira/browse/HDFS-5232 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, hdfs-client, namenode >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-5232.01.patch, HDFS-5232.02.patch, > HDFS-5232.03.patch > > > Currently the StorageID is used to identify both the Datanode and the > storage. This works because we treat all storages attached to the Datanode as > a single storage unit. > We plan to replace the StorageID with two independent IDs: > # Datanode UUID - this will be a string that uniquely identifies each > Datanode in the cluster. For upgraded clusters, the Datanode UUID will be the > StorageID prior to the upgrade. i.e. we will reuse the prior Storage ID. > # Storage UUID - Each storage attached to the datanode will be identified by > a UUID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5236) Change PathBasedCacheDirective APIs to be a single value rather than batch API
Andrew Wang created HDFS-5236:
-
Summary: Change PathBasedCacheDirective APIs to be a single value
rather than batch API
Key: HDFS-5236
URL: https://issues.apache.org/jira/browse/HDFS-5236
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-4949
Reporter: Andrew Wang
Assignee: Andrew Wang
It's kind of awkward that APIs like {{addPathBasedCacheDirective}} take a list
of Directives and return a list of Fallibles, since end users don't see a
checked exception and need to check each Fallible. It also complicates HDFS
code. Let's think about switching these APIs to instead take only a single
Directive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5234) Move RpcFrameDecoder out of the public API
[
https://issues.apache.org/jira/browse/HDFS-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772385#comment-13772385
]
Hadoop QA commented on HDFS-5234:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604130/HDFS-5234.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5006//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5006//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5006//console
This message is automatically generated.
> Move RpcFrameDecoder out of the public API
> --
>
> Key: HDFS-5234
> URL: https://issues.apache.org/jira/browse/HDFS-5234
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-5234.000.patch
>
>
> RpcFrameDecoder should be considered as a piece of the implementation details
> of the ONCRPC package, rather than a public API.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5235) Remove excessive copying due to XDR
Haohui Mai created HDFS-5235: Summary: Remove excessive copying due to XDR Key: HDFS-5235 URL: https://issues.apache.org/jira/browse/HDFS-5235 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Priority: Minor There are several places in the current implementation that copy the data before passing it into the XDR class. The new code should take advantage of HADOOP-9669 by passing in a ByteBuffer into XDR to remove the copies. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5231) Fix broken links in the document of HDFS Federation
[
https://issues.apache.org/jira/browse/HDFS-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772383#comment-13772383
]
Hadoop QA commented on HDFS-5231:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604109/HDFS-5231.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5005//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5005//console
This message is automatically generated.
> Fix broken links in the document of HDFS Federation
> ---
>
> Key: HDFS-5231
> URL: https://issues.apache.org/jira/browse/HDFS-5231
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-5231.000.patch
>
>
> The image links in the documents are broken.
> http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-hdfs/Federation.html
> It seems that these images are in YARN. These images need to be move into
> HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.04.patch Smaller patch by moving some changes to other Jiras. > Datanode must support all the volumes as individual storages > > > Key: HDFS-4988 > URL: https://issues.apache.org/jira/browse/HDFS-4988 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Suresh Srinivas >Assignee: Arpit Agarwal > Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, > HDFS-4988.03.patch, HDFS-4988.04.patch > > > Currently all the volumes on datanode is reported as a single storage. This > change proposes reporting them as individual storage. This requires: > # A unique storage ID for each storage > #* This needs to be generated during formatting > # There should be an option to allow existing disks to be reported as single > storage unit for backward compatibility. > # A functionality is also needed to split the existing all volumes as single > storage unit to to individual storage units. > # -Configuration must allow for each storage unit a storage type attribute. > (Now HDFS-5000)- > # Block reports must be sent on a per storage basis. In some cases (such > memory tier) block reports may need to be sent more frequently. That means > block reporting period must be on a per storage type basis. > My proposal is for new clusters to configure volumes by default as separate > storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5234) Move RpcFrameDecoder out of the public API
[ https://issues.apache.org/jira/browse/HDFS-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772410#comment-13772410 ] Brandon Li commented on HDFS-5234: -- +1. Thank you, Haohui. > Move RpcFrameDecoder out of the public API > -- > > Key: HDFS-5234 > URL: https://issues.apache.org/jira/browse/HDFS-5234 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5234.000.patch > > > RpcFrameDecoder should be considered as a piece of the implementation details > of the ONCRPC package, rather than a public API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5234) Move RpcFrameDecoder out of the public API
[ https://issues.apache.org/jira/browse/HDFS-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5234: - Attachment: HDFS-5234.000.patch > Move RpcFrameDecoder out of the public API > -- > > Key: HDFS-5234 > URL: https://issues.apache.org/jira/browse/HDFS-5234 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5234.000.patch > > > RpcFrameDecoder should be considered as a piece of the implementation details > of the ONCRPC package, rather than a public API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5234) Move RpcFrameDecoder out of the public API
Haohui Mai created HDFS-5234: Summary: Move RpcFrameDecoder out of the public API Key: HDFS-5234 URL: https://issues.apache.org/jira/browse/HDFS-5234 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Priority: Minor Attachments: HDFS-5234.000.patch RpcFrameDecoder should be considered as a piece of the implementation details of the ONCRPC package, rather than a public API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5234) Move RpcFrameDecoder out of the public API
[ https://issues.apache.org/jira/browse/HDFS-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5234: - Status: Patch Available (was: Open) > Move RpcFrameDecoder out of the public API > -- > > Key: HDFS-5234 > URL: https://issues.apache.org/jira/browse/HDFS-5234 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5234.000.patch > > > RpcFrameDecoder should be considered as a piece of the implementation details > of the ONCRPC package, rather than a public API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5230) Introduce RpcInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5230: - Summary: Introduce RpcInfo to decouple XDR classes from the RPC API (was: Introduce RPCInfo to decouple XDR classes from the RPC API) > Introduce RpcInfo to decouple XDR classes from the RPC API > -- > > Key: HDFS-5230 > URL: https://issues.apache.org/jira/browse/HDFS-5230 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5230.000.patch, HDFS-5230.001.patch > > > The XDR class is one fundamental aspect in the current implementation of NFS > server. While the client might potentially have a higher level APIs, it also > requires redundant copying since the upstream clients have insufficient > information. > This JIRA introduces a new class, RPCInfo, which (1) decouples XDR from the > APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly > to the client in order to open the opportunity for avoid copying. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5230) Introduce RpcInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5230: - Description: The XDR class is one fundamental aspect in the current implementation of NFS server. While the client might potentially have a higher level APIs, it also requires redundant copying since the upstream clients have insufficient information. This JIRA introduces a new class, RpcInfo, which (1) decouples XDR from the APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly to the client in order to open the opportunity for avoid copying. was: The XDR class is one fundamental aspect in the current implementation of NFS server. While the client might potentially have a higher level APIs, it also requires redundant copying since the upstream clients have insufficient information. This JIRA introduces a new class, RPCInfo, which (1) decouples XDR from the APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly to the client in order to open the opportunity for avoid copying. > Introduce RpcInfo to decouple XDR classes from the RPC API > -- > > Key: HDFS-5230 > URL: https://issues.apache.org/jira/browse/HDFS-5230 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5230.000.patch, HDFS-5230.001.patch > > > The XDR class is one fundamental aspect in the current implementation of NFS > server. While the client might potentially have a higher level APIs, it also > requires redundant copying since the upstream clients have insufficient > information. > This JIRA introduces a new class, RpcInfo, which (1) decouples XDR from the > APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly > to the client in order to open the opportunity for avoid copying. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5230) Introduce RPCInfo to decouple XDR classes from the RPC API
[
https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772167#comment-13772167
]
Hadoop QA commented on HDFS-5230:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604091/HDFS-5230.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5003//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5003//console
This message is automatically generated.
> Introduce RPCInfo to decouple XDR classes from the RPC API
> --
>
> Key: HDFS-5230
> URL: https://issues.apache.org/jira/browse/HDFS-5230
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-5230.000.patch
>
>
> The XDR class is one fundamental aspect in the current implementation of NFS
> server. While the client might potentially have a higher level APIs, it also
> requires redundant copying since the upstream clients have insufficient
> information.
> This JIRA introduces a new class, RPCInfo, which (1) decouples XDR from the
> APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly
> to the client in order to open the opportunity for avoid copying.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5227) Namenode.getAddress() should return namenode rpc-address
[
https://issues.apache.org/jira/browse/HDFS-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772293#comment-13772293
]
Chuan Liu commented on HDFS-5227:
-
bq. -1 core tests. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
I will take a look at the failures.
> Namenode.getAddress() should return namenode rpc-address
>
>
> Key: HDFS-5227
> URL: https://issues.apache.org/jira/browse/HDFS-5227
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.1-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
> Attachments: HDFS-5227-trunk.patch
>
>
> Currently, {{Namenode.getAddress(Configuration conf)}} will return default
> file system address as its result. The correct behavior should be returning
> config value of "dfs.namenode.rpc-address" if it presents in the
> configurations. Otherwise namenode will fail to start if the default file
> system is configured to another file system other than the one running in the
> cluster. We have a similar issue in 1.0 code base. The JIRA is HDFS-4320. The
> previous JIRA is closed and we cannot open it. Thus create a new one to track
> the issue in trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5232) Protocol changes to transmit StorageUuid
[ https://issues.apache.org/jira/browse/HDFS-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5232: Attachment: HDFS-5232.02.patch Thanks Nicholas! Updated patch. > Protocol changes to transmit StorageUuid > > > Key: HDFS-5232 > URL: https://issues.apache.org/jira/browse/HDFS-5232 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, hdfs-client, namenode >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-5232.01.patch, HDFS-5232.02.patch > > > Currently the StorageID is used to identify both the Datanode and the > storage. This works because we treat all storages attached to the Datanode as > a single storage unit. > We plan to replace the StorageID with two independent IDs: > # Datanode UUID - this will be a string that uniquely identifies each > Datanode in the cluster. For upgraded clusters, the Datanode UUID will be the > StorageID prior to the upgrade. i.e. we will reuse the prior Storage ID. > # Storage UUID - Each storage attached to the datanode will be identified by > a UUID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772350#comment-13772350 ] Arpit Agarwal commented on HDFS-4988: - New patch to follow once HDFS-5232 and HDFS-5233 are checked in. > Datanode must support all the volumes as individual storages > > > Key: HDFS-4988 > URL: https://issues.apache.org/jira/browse/HDFS-4988 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Suresh Srinivas >Assignee: Arpit Agarwal > Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, > HDFS-4988.03.patch > > > Currently all the volumes on datanode is reported as a single storage. This > change proposes reporting them as individual storage. This requires: > # A unique storage ID for each storage > #* This needs to be generated during formatting > # There should be an option to allow existing disks to be reported as single > storage unit for backward compatibility. > # A functionality is also needed to split the existing all volumes as single > storage unit to to individual storage units. > # -Configuration must allow for each storage unit a storage type attribute. > (Now HDFS-5000)- > # Block reports must be sent on a per storage basis. In some cases (such > memory tier) block reports may need to be sent more frequently. That means > block reporting period must be on a per storage type basis. > My proposal is for new clusters to configure volumes by default as separate > storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5232) Protocol changes to transmit StorageUuid
Arpit Agarwal created HDFS-5232: --- Summary: Protocol changes to transmit StorageUuid Key: HDFS-5232 URL: https://issues.apache.org/jira/browse/HDFS-5232 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client, namenode Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Arpit Agarwal Assignee: Arpit Agarwal Currently the StorageID is used to identify both the Datanode and the storage. This works because we treat all storages attached to the Datanode as a single storage unit. We plan to replace the StorageID with two independent IDs: # Datanode UUID - this will be a string that uniquely identifies each Datanode in the cluster. For upgraded clusters, the Datanode UUID will be the StorageID prior to the upgrade. # Storage UUID - Each storage attached to the datanode will be identified by a UUID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5233) Use Datanode UUID to identify Datanodes
[ https://issues.apache.org/jira/browse/HDFS-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5233: Attachment: HDFS-5233.01.patch > Use Datanode UUID to identify Datanodes > --- > > Key: HDFS-5233 > URL: https://issues.apache.org/jira/browse/HDFS-5233 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-5233.01.patch > > > StorageID currently identifies both a Datanode and a storage attached to the > Datanode. Once we start tracking multiple storages per datanode we would like > to deprecate StorageID in favor of a DatanodeUuid for the datanode and a > StorageUuid per storage. > This Jira track replacing StorageID with DatanodeUuid wherever it is used to > identify a Datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5233) Use Datanode UUID to identify Datanodes
Arpit Agarwal created HDFS-5233: --- Summary: Use Datanode UUID to identify Datanodes Key: HDFS-5233 URL: https://issues.apache.org/jira/browse/HDFS-5233 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Arpit Agarwal Assignee: Arpit Agarwal StorageID currently identifies both a Datanode and a storage attached to the Datanode. Once we start tracking multiple storages per datanode we would like to deprecate StorageID in favor of a DatanodeUuid for the datanode and a StorageUuid per storage. This Jira track replacing StorageID with DatanodeUuid wherever it is used to identify a Datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[
https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772160#comment-13772160
]
Andrew Wang commented on HDFS-5228:
---
Hey guys, thanks for finding this and posting a patch. I agree it's very
important to fix for 2.1.1, and almost certainly present in 2.1.0.
Couple patch notes:
* please remove the white-space only change in HdfsLocatedFileStatus
* please remove new {{@Deprecated}} annotations in DistributedFileSystem, let's
keep this fix small
* test name should be camel-cased as {{testListFiles}}
* should use {{#fixRelativePart()}} rather than in-lining the qualification.
Basically, besides the new test, I think all we need is (matching the style of
the rest of DFS):
{code}
Path absF = fixRelativePart(p);
{code}
and then renaming {{p}} to {{absF}} below.
I don't think this by itself is cause to revert HADOOP-9418. Basically all DFS
methods are supposed to pass Paths through {{fixRelativePart}} and then
{{getPathName}}, it just looks like we missed this one. It might be worth
auditing DFS to make sure we didn't miss anything else.
> The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw
> NPE
>
>
> Key: HDFS-5228
> URL: https://issues.apache.org/jira/browse/HDFS-5228
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h5228_20130919.patch, h5228_20130919_test.patch
>
>
> Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative
> path. Then, it will result a NullPointerException when calling hasNext()
> from the RemoteIterator.
> This bug was discovered by Arnaud:
> http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5227) Namenode.getAddress() should return namenode rpc-address
[
https://issues.apache.org/jira/browse/HDFS-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chuan Liu updated HDFS-5227:
Status: Open (was: Patch Available)
> Namenode.getAddress() should return namenode rpc-address
>
>
> Key: HDFS-5227
> URL: https://issues.apache.org/jira/browse/HDFS-5227
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.1-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
> Attachments: HDFS-5227-trunk.patch
>
>
> Currently, {{Namenode.getAddress(Configuration conf)}} will return default
> file system address as its result. The correct behavior should be returning
> config value of "dfs.namenode.rpc-address" if it presents in the
> configurations. Otherwise namenode will fail to start if the default file
> system is configured to another file system other than the one running in the
> cluster. We have a similar issue in 1.0 code base. The JIRA is HDFS-4320. The
> previous JIRA is closed and we cannot open it. Thus create a new one to track
> the issue in trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.03.patch New patch depends on HDFS-5232. > Datanode must support all the volumes as individual storages > > > Key: HDFS-4988 > URL: https://issues.apache.org/jira/browse/HDFS-4988 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Suresh Srinivas >Assignee: Arpit Agarwal > Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, > HDFS-4988.03.patch > > > Currently all the volumes on datanode is reported as a single storage. This > change proposes reporting them as individual storage. This requires: > # A unique storage ID for each storage > #* This needs to be generated during formatting > # There should be an option to allow existing disks to be reported as single > storage unit for backward compatibility. > # A functionality is also needed to split the existing all volumes as single > storage unit to to individual storage units. > # -Configuration must allow for each storage unit a storage type attribute. > (Now HDFS-5000)- > # Block reports must be sent on a per storage basis. In some cases (such > memory tier) block reports may need to be sent more frequently. That means > block reporting period must be on a per storage type basis. > My proposal is for new clusters to configure volumes by default as separate > storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[ https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772237#comment-13772237 ] Andrew Wang commented on HDFS-5228: --- We have extensive symlink tests checked in as part of HADOOP-8040, so it's unfair to say that it's not well tested. We've also been testing and fixing things upstream for the last few months; I think some number of bugs like this are unavoidable as symlinks was a pretty big change that required touching basically all the client API code. As to fixing this particular issue, it'd be great to get it done in a timely fashion for the 2.1.1 release. I'd be happy to rework Nicholas' v1 patch myself if that would help. Chris, thanks for going through DFS to check the other call sites, much appreciated. > The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw > NPE > > > Key: HDFS-5228 > URL: https://issues.apache.org/jira/browse/HDFS-5228 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.1.0-beta >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Blocker > Attachments: h5228_20130919.patch, h5228_20130919_test.patch > > > Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative > path. Then, it will result a NullPointerException when calling hasNext() > from the RemoteIterator. > This bug was discovered by Arnaud: > http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5231) Fix broken links in the document of HDFS Federation
[ https://issues.apache.org/jira/browse/HDFS-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5231: - Attachment: HDFS-5231.000.patch This patch updates the HTML. It also requires moving two files in order to work: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/federation-background.gif -> hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/federation-background.gif hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/federation.gif -> hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/federation.gif > Fix broken links in the document of HDFS Federation > --- > > Key: HDFS-5231 > URL: https://issues.apache.org/jira/browse/HDFS-5231 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5231.000.patch > > > The image links in the documents are broken. > http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-hdfs/Federation.html > It seems that these images are in YARN. These images need to be move into > HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5231) Fix broken links in the document of HDFS Federation
[ https://issues.apache.org/jira/browse/HDFS-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5231: - Status: Patch Available (was: Open) > Fix broken links in the document of HDFS Federation > --- > > Key: HDFS-5231 > URL: https://issues.apache.org/jira/browse/HDFS-5231 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5231.000.patch > > > The image links in the documents are broken. > http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-hdfs/Federation.html > It seems that these images are in YARN. These images need to be move into > HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4971) Move IO operations out of locking in OpenFileCtx
[
https://issues.apache.org/jira/browse/HDFS-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772284#comment-13772284
]
Brandon Li commented on HDFS-4971:
--
Thanks for the patch. Some quick comments:
1. in the cleanup(), it should check if dumpOut==null before trying to delete
the dump file.
2. in the dumper thread, it should check if dump is still enable before each
loop:
while (activeState && enabledDump) {
> Move IO operations out of locking in OpenFileCtx
>
>
> Key: HDFS-4971
> URL: https://issues.apache.org/jira/browse/HDFS-4971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-4971.000.patch, HDFS-4971.001.patch,
> HDFS-4971.002.patch, HDFS-4971.003.patch, HDFS-4971.004.patch,
> HDFS-4971.005.patch, HDFS-4971.006.patch
>
>
> Currently some IO operations (such as writing data to HDFS and dumping to
> local disk) in OpenFileCtx may hold a lock which can block processing
> incoming writing requests. This jira aims to optimize OpenFileCtx and move
> the IO operations out of the locking.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5230) Introduce RPCInfo to decouple XDR classes from the RPC API
[
https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772233#comment-13772233
]
Hadoop QA commented on HDFS-5230:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604099/HDFS-5230.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5004//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5004//console
This message is automatically generated.
> Introduce RPCInfo to decouple XDR classes from the RPC API
> --
>
> Key: HDFS-5230
> URL: https://issues.apache.org/jira/browse/HDFS-5230
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-5230.000.patch, HDFS-5230.001.patch
>
>
> The XDR class is one fundamental aspect in the current implementation of NFS
> server. While the client might potentially have a higher level APIs, it also
> requires redundant copying since the upstream clients have insufficient
> information.
> This JIRA introduces a new class, RPCInfo, which (1) decouples XDR from the
> APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly
> to the client in order to open the opportunity for avoid copying.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5232) Protocol changes to transmit StorageUuid
[ https://issues.apache.org/jira/browse/HDFS-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5232: Attachment: HDFS-5232.01.patch Retaining StorageID in protocol messages for wire compatibility and marking it as deprecated. > Protocol changes to transmit StorageUuid > > > Key: HDFS-5232 > URL: https://issues.apache.org/jira/browse/HDFS-5232 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, hdfs-client, namenode >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-5232.01.patch > > > Currently the StorageID is used to identify both the Datanode and the > storage. This works because we treat all storages attached to the Datanode as > a single storage unit. > We plan to replace the StorageID with two independent IDs: > # Datanode UUID - this will be a string that uniquely identifies each > Datanode in the cluster. For upgraded clusters, the Datanode UUID will be the > StorageID prior to the upgrade. > # Storage UUID - Each storage attached to the datanode will be identified by > a UUID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5232) Protocol changes to transmit StorageUuid
[ https://issues.apache.org/jira/browse/HDFS-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5232: Description: Currently the StorageID is used to identify both the Datanode and the storage. This works because we treat all storages attached to the Datanode as a single storage unit. We plan to replace the StorageID with two independent IDs: # Datanode UUID - this will be a string that uniquely identifies each Datanode in the cluster. For upgraded clusters, the Datanode UUID will be the StorageID prior to the upgrade. i.e. we will reuse the prior Storage ID. # Storage UUID - Each storage attached to the datanode will be identified by a UUID. was: Currently the StorageID is used to identify both the Datanode and the storage. This works because we treat all storages attached to the Datanode as a single storage unit. We plan to replace the StorageID with two independent IDs: # Datanode UUID - this will be a string that uniquely identifies each Datanode in the cluster. For upgraded clusters, the Datanode UUID will be the StorageID prior to the upgrade. # Storage UUID - Each storage attached to the datanode will be identified by a UUID. > Protocol changes to transmit StorageUuid > > > Key: HDFS-5232 > URL: https://issues.apache.org/jira/browse/HDFS-5232 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, hdfs-client, namenode >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-5232.01.patch > > > Currently the StorageID is used to identify both the Datanode and the > storage. This works because we treat all storages attached to the Datanode as > a single storage unit. > We plan to replace the StorageID with two independent IDs: > # Datanode UUID - this will be a string that uniquely identifies each > Datanode in the cluster. For upgraded clusters, the Datanode UUID will be the > StorageID prior to the upgrade. i.e. we will reuse the prior Storage ID. > # Storage UUID - Each storage attached to the datanode will be identified by > a UUID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5232) Protocol changes to transmit StorageUuid
[ https://issues.apache.org/jira/browse/HDFS-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772254#comment-13772254 ] Tsz Wo (Nicholas), SZE commented on HDFS-5232: -- - GetAdditionalDatanodeRequestProto.existingStorageIDs was committed only to the HDFS-2832 branch. We may rename it directly without deprecation. - Even for other cases like DatanodeStorageProto.storageID, we could rename it without deprecation since renaming a variable does not change the wire byte format. It does break programs compiled with the generated code but it is not something we need to avoid. > Protocol changes to transmit StorageUuid > > > Key: HDFS-5232 > URL: https://issues.apache.org/jira/browse/HDFS-5232 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, hdfs-client, namenode >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-5232.01.patch > > > Currently the StorageID is used to identify both the Datanode and the > storage. This works because we treat all storages attached to the Datanode as > a single storage unit. > We plan to replace the StorageID with two independent IDs: > # Datanode UUID - this will be a string that uniquely identifies each > Datanode in the cluster. For upgraded clusters, the Datanode UUID will be the > StorageID prior to the upgrade. i.e. we will reuse the prior Storage ID. > # Storage UUID - Each storage attached to the datanode will be identified by > a UUID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[
https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772185#comment-13772185
]
Chris Nauroth commented on HDFS-5228:
-
Thanks, Andrew. Your explanation makes sense, and I confirmed that everything
else in {{DistributedFileSystem}} uses a combination of {{fixRelativePart}} and
{{getPathName}}. This is the only call site that has the problem. I agree
that HADOOP-9418 doesn't need to be reverted.
> The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw
> NPE
>
>
> Key: HDFS-5228
> URL: https://issues.apache.org/jira/browse/HDFS-5228
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h5228_20130919.patch, h5228_20130919_test.patch
>
>
> Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative
> path. Then, it will result a NullPointerException when calling hasNext()
> from the RemoteIterator.
> This bug was discovered by Arnaud:
> http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5230) Introduce RPCInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5230: - Attachment: HDFS-5230.001.patch > Introduce RPCInfo to decouple XDR classes from the RPC API > -- > > Key: HDFS-5230 > URL: https://issues.apache.org/jira/browse/HDFS-5230 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5230.000.patch, HDFS-5230.001.patch > > > The XDR class is one fundamental aspect in the current implementation of NFS > server. While the client might potentially have a higher level APIs, it also > requires redundant copying since the upstream clients have insufficient > information. > This JIRA introduces a new class, RPCInfo, which (1) decouples XDR from the > APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly > to the client in order to open the opportunity for avoid copying. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5231) Fix broken links in the document of HDFS Federation
Haohui Mai created HDFS-5231: Summary: Fix broken links in the document of HDFS Federation Key: HDFS-5231 URL: https://issues.apache.org/jira/browse/HDFS-5231 Project: Hadoop HDFS Issue Type: Bug Reporter: Haohui Mai Assignee: Haohui Mai Priority: Minor The image links in the documents are broken. http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-hdfs/Federation.html It seems that these images are in YARN. These images need to be move into HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[ https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772191#comment-13772191 ] Tsz Wo (Nicholas), SZE commented on HDFS-5228: -- > I don't think this by itself is cause to revert HADOOP-9418. Basically all > DFS methods are supposed to pass Paths through fixRelativePart and then > getPathName, it just looks like we missed this one. It might be worth > auditing DFS to make sure we didn't miss anything else. Hi Andrew, I am sorry to hear that your HADOOP-9418 patch causes a serious bug in DistributFileSystem, which is a very important client API. How exactly to audit DFS do you suggest so that it could avoid further bugs? It does seem that HADOOP-9418 has not been well tested. Do you agree? I do think reverting HADOOP-9418 may make more sense. > The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw > NPE > > > Key: HDFS-5228 > URL: https://issues.apache.org/jira/browse/HDFS-5228 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.1.0-beta >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Blocker > Attachments: h5228_20130919.patch, h5228_20130919_test.patch > > > Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative > path. Then, it will result a NullPointerException when calling hasNext() > from the RemoteIterator. > This bug was discovered by Arnaud: > http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[ https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-5228: Priority: Blocker (was: Major) Target Version/s: 2.1.1-beta Affects Version/s: 2.1.0-beta > The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw > NPE > > > Key: HDFS-5228 > URL: https://issues.apache.org/jira/browse/HDFS-5228 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.1.0-beta >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Blocker > Attachments: h5228_20130919.patch, h5228_20130919_test.patch > > > Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative > path. Then, it will result a NullPointerException when calling hasNext() > from the RemoteIterator. > This bug was discovered by Arnaud: > http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5208) Only clear network location cache on specific nodes if invalid NetworkTopology happens
[ https://issues.apache.org/jira/browse/HDFS-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772100#comment-13772100 ] Colin Patrick McCabe commented on HDFS-5208: Sounds good to me. > Only clear network location cache on specific nodes if invalid > NetworkTopology happens > -- > > Key: HDFS-5208 > URL: https://issues.apache.org/jira/browse/HDFS-5208 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Junping Du >Assignee: Junping Du > Attachments: HDFS-5208-v1.patch > > > After HDFS-4521, once a DN is registered with invalid networktopology, all > cached rack info in DNSToSwitchMapping will be cleared. We should only clear > cache on specific nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5230) Introduce RPCInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5230: - Attachment: HDFS-5230.000.patch > Introduce RPCInfo to decouple XDR classes from the RPC API > -- > > Key: HDFS-5230 > URL: https://issues.apache.org/jira/browse/HDFS-5230 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5230.000.patch > > > The XDR class is one fundamental aspect in the current implementation of NFS > server. While the client might potentially have a higher level APIs, it also > requires redundant copying since the upstream clients have insufficient > information. > This JIRA introduces a new class, RPCInfo, which (1) decouples XDR from the > APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly > to the client in order to open the opportunity for avoid copying. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5230) Introduce RPCInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5230: - Status: Patch Available (was: Open) > Introduce RPCInfo to decouple XDR classes from the RPC API > -- > > Key: HDFS-5230 > URL: https://issues.apache.org/jira/browse/HDFS-5230 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai > > The XDR class is one fundamental aspect in the current implementation of NFS > server. While the client might potentially have a higher level APIs, it also > requires redundant copying since the upstream clients have insufficient > information. > This JIRA introduces a new class, RPCInfo, which (1) decouples XDR from the > APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly > to the client in order to open the opportunity for avoid copying. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5230) Introduce RPCInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5230: - Attachment: HDFS-5230.000.patch > Introduce RPCInfo to decouple XDR classes from the RPC API > -- > > Key: HDFS-5230 > URL: https://issues.apache.org/jira/browse/HDFS-5230 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai > > The XDR class is one fundamental aspect in the current implementation of NFS > server. While the client might potentially have a higher level APIs, it also > requires redundant copying since the upstream clients have insufficient > information. > This JIRA introduces a new class, RPCInfo, which (1) decouples XDR from the > APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly > to the client in order to open the opportunity for avoid copying. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5230) Introduce RPCInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5230: - Attachment: (was: HDFS-5230.000.patch) > Introduce RPCInfo to decouple XDR classes from the RPC API > -- > > Key: HDFS-5230 > URL: https://issues.apache.org/jira/browse/HDFS-5230 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai > > The XDR class is one fundamental aspect in the current implementation of NFS > server. While the client might potentially have a higher level APIs, it also > requires redundant copying since the upstream clients have insufficient > information. > This JIRA introduces a new class, RPCInfo, which (1) decouples XDR from the > APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly > to the client in order to open the opportunity for avoid copying. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5230) Introduce RPCInfo to decouple XDR classes from the RPC API
Haohui Mai created HDFS-5230: Summary: Introduce RPCInfo to decouple XDR classes from the RPC API Key: HDFS-5230 URL: https://issues.apache.org/jira/browse/HDFS-5230 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai The XDR class is one fundamental aspect in the current implementation of NFS server. While the client might potentially have a higher level APIs, it also requires redundant copying since the upstream clients have insufficient information. This JIRA introduces a new class, RPCInfo, which (1) decouples XDR from the APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly to the client in order to open the opportunity for avoid copying. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5216) NumDecomDeadDataNodes not returning correct number of dead decommissioned nodes
[ https://issues.apache.org/jira/browse/HDFS-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772057#comment-13772057 ] Trevor Lorimer commented on HDFS-5216: -- I have checked this on my cluster, where I added a datanode to the exclude file which decommissions a node, if the datanode is live it will be added to the Live Decommissioned count, if the datanode is stopped it will be added to the Dead Decommissioned count, which makes sense. There is however a bug in the code I submitted where the Dead count ends up being the same as the Live count, the following line need to be changed: getBlockManager().getDatanodeManager().fetchDatanodes(dead, null, true); should be: getBlockManager().getDatanodeManager().fetchDatanodes(null, dead, true); > NumDecomDeadDataNodes not returning correct number of dead decommissioned > nodes > > > Key: HDFS-5216 > URL: https://issues.apache.org/jira/browse/HDFS-5216 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta >Reporter: Trevor Lorimer > > For HDFS-4860 I essentially copied the process in > NamenodejspHelper.generateHealthReport(), so it would be in sync with the > original dfsHealth.jsp. > However looking at this now there may be a bug? in > getNumDecomDeadDataNodes(), where: > getBlockManager().getDatanodeManager().fetchDatanodes(dead, null, true); > > Where the parameter true indicates that decommissioned nodes should be > removed from the list. > If the flag is true fetchDatanodes calls removeDecomNodeFromList, which will > remove a node if an existing datanode does not appear in both include or > exclude lists and it has been decommissioned. > If I am looking to return the Number of Dead Decommissioned Nodes, should I > change the remove decommissioned nodes flag to False? i.e.: > getBlockManager().getDatanodeManager().fetchDatanodes(null, dead, false); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5219) Add configuration keys for retry policy in WebHDFSFileSystem
[ https://issues.apache.org/jira/browse/HDFS-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771881#comment-13771881 ] Hudson commented on HDFS-5219: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1527 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1527/]) HDFS-5219. Add configuration keys for retry policy in WebHDFSFileSystem. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524498) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java > Add configuration keys for retry policy in WebHDFSFileSystem > > > Key: HDFS-5219 > URL: https://issues.apache.org/jira/browse/HDFS-5219 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.1.1-beta > > Attachments: HDFS-5219.000.patch, HDFS-5219.001.patch, > HDFS-5219.002.patch, HDFS-5219.003.patch, HDFS-5219.004.patch, > HDFS-5219.005.patch > > > Currently the WebHDFS client reuses the configuration of DFSClient to > construct its RetryPolicy. This is problematic because: > 1. It makes the WebHDFS client depends on the DFSClient. > 2. The default values of the RetryPolicy of DFSClient might be ill fit for > the WebHDFS client, which transfers all data using HTTP. > This JIRA proposes to introduce new configuration keys for WebHDFS to resolve > this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5031) BlockScanner scans the block multiple times and on restart scans everything
[ https://issues.apache.org/jira/browse/HDFS-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771879#comment-13771879 ] Hudson commented on HDFS-5031: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1527 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1527/]) HDFS-5031. BlockScanner scans the block multiple times. (Vinay via Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524553) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RollingLogs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RollingLogsImpl.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeBlockScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java > BlockScanner scans the block multiple times and on restart scans everything > --- > > Key: HDFS-5031 > URL: https://issues.apache.org/jira/browse/HDFS-5031 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Vinay >Assignee: Vinay > Fix For: 2.3.0 > > Attachments: HDFS-5031.patch, HDFS-5031.patch, HDFS-5031.patch > > > BlockScanner scans the block twice, also on restart of datanode scans > everything. > Steps: > 1. Write blocks with interval of more than 5 seconds. write new block on > completion of scan for written block. > Each time datanode scans new block, it also scans, previous block which is > already scanned. > Now after restart, datanode scans all blocks again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) Support failover and retry in WebHdfsFileSystem for NN HA
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771882#comment-13771882
]
Hudson commented on HDFS-5122:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1527 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1527/])
Move HDFS-5122 from Release 2.1.1-beta to Release 2.3.0 in CHANGES.txt (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524581)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
HDFS-5122. Support failover and retry in WebHdfsFileSystem for NN HA.
Contributed by Haohui Mai. (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524562)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/WebHdfsTestUtil.java
> Support failover and retry in WebHdfsFileSystem for NN HA
> -
>
> Key: HDFS-5122
> URL: https://issues.apache.org/jira/browse/HDFS-5122
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, webhdfs
>Affects Versions: 2.1.0-beta
>Reporter: Arpit Gupta
>Assignee: Haohui Mai
> Fix For: 2.3.0
>
> Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
> HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
>
>
> Bug reported by [~arpitgupta]:
> If the dfs.nameservices is set to arpit,
> {code}
> hdfs dfs -ls webhdfs://arpit/tmp
> {code}
> does not work. You have to provide the exact active namenode hostname. On an
> HA cluster using dfs client one should not need to provide the active nn
> hostname.
> To fix this, we try to
> 1) let WebHdfsFileSystem support logical NN service name
> 2) add failover_and_retry functionality in WebHdfsFileSystem for NN HA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5031) BlockScanner scans the block multiple times and on restart scans everything
[ https://issues.apache.org/jira/browse/HDFS-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771869#comment-13771869 ] Hudson commented on HDFS-5031: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1553 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1553/]) HDFS-5031. BlockScanner scans the block multiple times. (Vinay via Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524553) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RollingLogs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RollingLogsImpl.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeBlockScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java > BlockScanner scans the block multiple times and on restart scans everything > --- > > Key: HDFS-5031 > URL: https://issues.apache.org/jira/browse/HDFS-5031 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Vinay >Assignee: Vinay > Fix For: 2.3.0 > > Attachments: HDFS-5031.patch, HDFS-5031.patch, HDFS-5031.patch > > > BlockScanner scans the block twice, also on restart of datanode scans > everything. > Steps: > 1. Write blocks with interval of more than 5 seconds. write new block on > completion of scan for written block. > Each time datanode scans new block, it also scans, previous block which is > already scanned. > Now after restart, datanode scans all blocks again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) Support failover and retry in WebHdfsFileSystem for NN HA
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771872#comment-13771872
]
Hudson commented on HDFS-5122:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1553 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1553/])
Move HDFS-5122 from Release 2.1.1-beta to Release 2.3.0 in CHANGES.txt (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524581)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
HDFS-5122. Support failover and retry in WebHdfsFileSystem for NN HA.
Contributed by Haohui Mai. (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524562)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/WebHdfsTestUtil.java
> Support failover and retry in WebHdfsFileSystem for NN HA
> -
>
> Key: HDFS-5122
> URL: https://issues.apache.org/jira/browse/HDFS-5122
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, webhdfs
>Affects Versions: 2.1.0-beta
>Reporter: Arpit Gupta
>Assignee: Haohui Mai
> Fix For: 2.3.0
>
> Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
> HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
>
>
> Bug reported by [~arpitgupta]:
> If the dfs.nameservices is set to arpit,
> {code}
> hdfs dfs -ls webhdfs://arpit/tmp
> {code}
> does not work. You have to provide the exact active namenode hostname. On an
> HA cluster using dfs client one should not need to provide the active nn
> hostname.
> To fix this, we try to
> 1) let WebHdfsFileSystem support logical NN service name
> 2) add failover_and_retry functionality in WebHdfsFileSystem for NN HA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5219) Add configuration keys for retry policy in WebHDFSFileSystem
[ https://issues.apache.org/jira/browse/HDFS-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771871#comment-13771871 ] Hudson commented on HDFS-5219: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1553 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1553/]) HDFS-5219. Add configuration keys for retry policy in WebHDFSFileSystem. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524498) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java > Add configuration keys for retry policy in WebHDFSFileSystem > > > Key: HDFS-5219 > URL: https://issues.apache.org/jira/browse/HDFS-5219 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.1.1-beta > > Attachments: HDFS-5219.000.patch, HDFS-5219.001.patch, > HDFS-5219.002.patch, HDFS-5219.003.patch, HDFS-5219.004.patch, > HDFS-5219.005.patch > > > Currently the WebHDFS client reuses the configuration of DFSClient to > construct its RetryPolicy. This is problematic because: > 1. It makes the WebHDFS client depends on the DFSClient. > 2. The default values of the RetryPolicy of DFSClient might be ill fit for > the WebHDFS client, which transfers all data using HTTP. > This JIRA proposes to introduce new configuration keys for WebHDFS to resolve > this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5219) Add configuration keys for retry policy in WebHDFSFileSystem
[ https://issues.apache.org/jira/browse/HDFS-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771794#comment-13771794 ] Hudson commented on HDFS-5219: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #337 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/337/]) HDFS-5219. Add configuration keys for retry policy in WebHDFSFileSystem. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524498) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java > Add configuration keys for retry policy in WebHDFSFileSystem > > > Key: HDFS-5219 > URL: https://issues.apache.org/jira/browse/HDFS-5219 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.1.1-beta > > Attachments: HDFS-5219.000.patch, HDFS-5219.001.patch, > HDFS-5219.002.patch, HDFS-5219.003.patch, HDFS-5219.004.patch, > HDFS-5219.005.patch > > > Currently the WebHDFS client reuses the configuration of DFSClient to > construct its RetryPolicy. This is problematic because: > 1. It makes the WebHDFS client depends on the DFSClient. > 2. The default values of the RetryPolicy of DFSClient might be ill fit for > the WebHDFS client, which transfers all data using HTTP. > This JIRA proposes to introduce new configuration keys for WebHDFS to resolve > this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) Support failover and retry in WebHdfsFileSystem for NN HA
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771795#comment-13771795
]
Hudson commented on HDFS-5122:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #337 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/337/])
Move HDFS-5122 from Release 2.1.1-beta to Release 2.3.0 in CHANGES.txt (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524581)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
HDFS-5122. Support failover and retry in WebHdfsFileSystem for NN HA.
Contributed by Haohui Mai. (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524562)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/WebHdfsTestUtil.java
> Support failover and retry in WebHdfsFileSystem for NN HA
> -
>
> Key: HDFS-5122
> URL: https://issues.apache.org/jira/browse/HDFS-5122
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, webhdfs
>Affects Versions: 2.1.0-beta
>Reporter: Arpit Gupta
>Assignee: Haohui Mai
> Fix For: 2.3.0
>
> Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
> HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
>
>
> Bug reported by [~arpitgupta]:
> If the dfs.nameservices is set to arpit,
> {code}
> hdfs dfs -ls webhdfs://arpit/tmp
> {code}
> does not work. You have to provide the exact active namenode hostname. On an
> HA cluster using dfs client one should not need to provide the active nn
> hostname.
> To fix this, we try to
> 1) let WebHdfsFileSystem support logical NN service name
> 2) add failover_and_retry functionality in WebHdfsFileSystem for NN HA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5031) BlockScanner scans the block multiple times and on restart scans everything
[ https://issues.apache.org/jira/browse/HDFS-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771792#comment-13771792 ] Hudson commented on HDFS-5031: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #337 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/337/]) HDFS-5031. BlockScanner scans the block multiple times. (Vinay via Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524553) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RollingLogs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RollingLogsImpl.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeBlockScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java > BlockScanner scans the block multiple times and on restart scans everything > --- > > Key: HDFS-5031 > URL: https://issues.apache.org/jira/browse/HDFS-5031 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Vinay >Assignee: Vinay > Fix For: 2.3.0 > > Attachments: HDFS-5031.patch, HDFS-5031.patch, HDFS-5031.patch > > > BlockScanner scans the block twice, also on restart of datanode scans > everything. > Steps: > 1. Write blocks with interval of more than 5 seconds. write new block on > completion of scan for written block. > Each time datanode scans new block, it also scans, previous block which is > already scanned. > Now after restart, datanode scans all blocks again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771699#comment-13771699
]
Junping Du commented on HDFS-5225:
--
Hi Tsuyoshi, I think this fix may also work as this is the only unsynchronized
block that accessing blockInfoSet. However, why don't you merge the
synchronized block with above code block which is synchronized also. It would
be nice if you can have a unit test to reproduce the bug and verify the patch
can fix it.
> datanode keeps logging the same 'is no longer in the dataset' message over
> and over again
> -
>
> Key: HDFS-5225
> URL: https://issues.apache.org/jira/browse/HDFS-5225
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.1.1-beta
>Reporter: Roman Shaposhnik
> Attachments: HDFS-5225.1.patch
>
>
> I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
> configuration: 4 nodes fully distributed cluster with security on.
> All of a sudden my DN ate up all of the space in /var/log logging the
> following message repeatedly:
> {noformat}
> 2013-09-18 20:51:12,046 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
> in the dataset
> {noformat}
> It wouldn't answer to a jstack and jstack -F ended up being useless.
> Here's what I was able to find in the NameNode logs regarding this block ID:
> {noformat}
> fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
> 2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
> BP-1884637155-10.224.158.152-1379524544853
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
> 2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.224.158.152:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.34.74.206:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.83.107.80:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,899 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
> newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
> 10.34.74.206:1004, 10.224.158.152:1004],
> clientName=DFSClient_NONMAPREDUCE_-450304083_1)
> 2013-09-18 18:03:17,904 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
> successfully to
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
> 2013-09-18 18:03:18,540 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
> newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
> 10.34.74.206:1004, 10.224.158.152:1004],
> clientName=DFSClient_NONMAPREDUCE_-450304083_1)
> 2013-09-18 18:03:18,548 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
> successfully to
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
> 2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
> blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
> 2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
> blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_1366,
> blk_1073742188_1368, b
[jira] [Created] (HDFS-5229) Add a storage type attribute to file
Tsz Wo (Nicholas), SZE created HDFS-5229: Summary: Add a storage type attribute to file Key: HDFS-5229 URL: https://issues.apache.org/jira/browse/HDFS-5229 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE When creating a file, user could specify the required storage type. The type information need to be stored so that it could be used for block placement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5222) Move block schedule information from DatanodeDescriptor to DatanodeStorageInfo
[ https://issues.apache.org/jira/browse/HDFS-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-5222: - Attachment: h5222_20130819.patch h5222_20130819.patch: moves the related code to DatanodeStorageInfo. > Move block schedule information from DatanodeDescriptor to DatanodeStorageInfo > -- > > Key: HDFS-5222 > URL: https://issues.apache.org/jira/browse/HDFS-5222 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h5222_20130819.patch > > > In HDFS-4990, the block placement target type was changed from > DatanodeDescriptor to DatanodeStorageInfo. The block schedule information, > such as the number of blocks scheduled for replication (i.e. > getBlocksScheduled()), should be moved from DatanodeDescriptor to > DatanodeStorageInfo. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
