[jira] [Created] (HDFS-5665) Remove the unnecessary writeLock while initializing CacheManager in FsNameSystem Ctor
Uma Maheswara Rao G created HDFS-5665:
-
Summary: Remove the unnecessary writeLock while initializing
CacheManager in FsNameSystem Ctor
Key: HDFS-5665
URL: https://issues.apache.org/jira/browse/HDFS-5665
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.2.0, 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
I just saw the below piece of code in Fsnamesystem ctor.
{code}
writeLock();
try {
this.cacheManager = new CacheManager(this, conf, blockManager);
} finally {
writeUnlock();
}
{code}
It seems unnecessary to keep writeLock here. I am not sure if there is a clear
reason to keep the lock.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5665) Remove the unnecessary writeLock while initializing CacheManager in FsNameSystem Ctor
[
https://issues.apache.org/jira/browse/HDFS-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847317#comment-13847317
]
Uma Maheswara Rao G commented on HDFS-5665:
---
Feel free to close this JIRA if you have a valid reason.
> Remove the unnecessary writeLock while initializing CacheManager in
> FsNameSystem Ctor
> -
>
> Key: HDFS-5665
> URL: https://issues.apache.org/jira/browse/HDFS-5665
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>
> I just saw the below piece of code in Fsnamesystem ctor.
> {code}
> writeLock();
> try {
> this.cacheManager = new CacheManager(this, conf, blockManager);
> } finally {
> writeUnlock();
> }
> {code}
> It seems unnecessary to keep writeLock here. I am not sure if there is a
> clear reason to keep the lock.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5632) Add Snapshot feature to INodeDirectory
[
https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847385#comment-13847385
]
Tsz Wo (Nicholas), SZE commented on HDFS-5632:
--
Thanks Jing for picking this up. Some minor comments on the patch:
- In INodeDirectory.recordModification, add snapshot feature only if
!shouldRecordInSrcSnapshot(latest).
- DirectoryWithSnapshotFeature.ChildrenDiff does not need to be public.
- DirectoryDiff.getChildrenList can be private.
- We still replace INodeDirectory with INodeDirectorySnapshottable. So the
following comment is not accurate.
{code}
+ /**
+ * Replace the given child with a new child. Note that we no longer need to
+ * replace an normal INodeDirectory or INodeFile into an
+ * INodeDirectoryWithSnapshot or INodeFileUnderConstruction. The only cases
+ * for child replacement is for reference nodes.
+ */
public void replaceChild(INode oldChild, final INode newChild,
final INodeMap inodeMap) {
{code}
- In INodeReference.destroyAndCollectBlocks code below, the referred node must
be with snapshot so that we don't need "if (dir.isWithSnapshot())" but we
should check precondition in
DirectoryWithSnapshotFeature.destroyDstSubtree(..). BTW, the comment needs to
be updated. It still refers "INodeDirectoryWithSnapshot".
{code}
// similarly, if referred is a directory, it must be an
// INodeDirectoryWithSnapshot
- INodeDirectoryWithSnapshot sdir =
- (INodeDirectoryWithSnapshot) referred;
- try {
-INodeDirectoryWithSnapshot.destroyDstSubtree(sdir, snapshot, prior,
-collectedBlocks, removedINodes);
- } catch (QuotaExceededException e) {
-LOG.error("should not exceed quota while snapshot deletion", e);
+ INodeDirectory dir = referred.asDirectory();
+ if (dir.isWithSnapshot()) {
+try {
+ DirectoryWithSnapshotFeature.destroyDstSubtree(dir, snapshot,
+ prior, collectedBlocks, removedINodes);
+} catch (QuotaExceededException e) {
+ LOG.error("should not exceed quota while snapshot deletion", e);
+}
}
{code}
- In DirectoryWithSnapshotFeature.cleanDeletedINode code below, we probably
should getDirectoryWithSnapshotFeature() first but not calling
isWithSnapshot(). Similarly, for destroyDstSubtree.
{code}
+if (dir.isWithSnapshot()) {
+ // delete files/dirs created after prior. Note that these
+ // files/dirs, along with inode, were deleted right after post.
+ DirectoryDiff priorDiff =
dir.getDirectoryWithSnapshotFeature().getDiffs()
+ .getDiff(prior);
{code}
> Add Snapshot feature to INodeDirectory
> --
>
> Key: HDFS-5632
> URL: https://issues.apache.org/jira/browse/HDFS-5632
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-5632.000.patch, HDFS-5632.001.patch,
> HDFS-5632.002.patch
>
>
> We will add snapshot feature to INodeDirectory and remove
> INodeDirectoryWithSnapshot in this jira.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-2832) Enable support for heterogeneous storages in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847392#comment-13847392 ] Hudson commented on HDFS-2832: -- FAILURE: Integrated in Hadoop-Yarn-trunk #420 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/420/]) HDFS-2832: Update binary file editsStored for TestOfflineEditsViewer (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550364) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored svn merge --reintegrate https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2832 for merging Heterogeneous Storage feature branch (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550363) * /hadoop/common/trunk * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockListAsLongs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/UnregisteredNodeException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor
[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred
[ https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847398#comment-13847398 ] Hudson commented on HDFS-5637: -- FAILURE: Integrated in Hadoop-Yarn-trunk #420 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/420/]) HDFS-5637. Try to refeatchToken while local read InvalidToken occurred. (Liang Xie via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550335) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java > try to refeatchToken while local read InvalidToken occurred > --- > > Key: HDFS-5637 > URL: https://issues.apache.org/jira/browse/HDFS-5637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, security >Affects Versions: 2.0.5-alpha, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Fix For: 2.4.0 > > Attachments: HDFS-5637-v2.txt, HDFS-5637.txt > > > we observed several warning logs like below from region server nodes: > 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to > connect to /10.2.201.110:11402 for block, add to deadNodes and continue. > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > at sun.reflect
[jira] [Commented] (HDFS-5652) refactoring/uniforming invalid block token exception handling in DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847402#comment-13847402 ] Hudson commented on HDFS-5652: -- FAILURE: Integrated in Hadoop-Yarn-trunk #420 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/420/]) HDFS-5652. Refactor invalid block token exception handling in DFSInputStream. (Liang Xie via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550620) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java > refactoring/uniforming invalid block token exception handling in > DFSInputStream > --- > > Key: HDFS-5652 > URL: https://issues.apache.org/jira/browse/HDFS-5652 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie >Priority: Minor > Fix For: 2.4.0 > > Attachments: HDFS-5652.txt > > > See comments from Junping and Colin's from HDFS-5637 -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[
https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847404#comment-13847404
]
Hudson commented on HDFS-4201:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #420 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/420/])
HDFS-4201. NPE in BPServiceActor#sendHeartBeat (jxiang via cmccabe) (cmccabe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550269)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
> NPE in BPServiceActor#sendHeartBeat
> ---
>
> Key: HDFS-4201
> URL: https://issues.apache.org/jira/browse/HDFS-4201
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Eli Collins
>Assignee: Jimmy Xiang
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: trunk-4201.patch, trunk-4201_v2.patch,
> trunk-4201_v3.patch
>
>
> Saw the following NPE in a log.
> Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not
> {{bpRegistration}}) due to a configuration or local directory failure.
> {code}
> 2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of
> 30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec;
> heartBeatInterval=3000
> 2012-09-25 04:33:20,782 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService
> for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id
> DS-1031100678-11.164.162.251-5010-1341933415989) service to
> svsrs00127/11.164.162.226:8020
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
> at java.lang.Thread.run(Thread.java:722)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847397#comment-13847397 ] Hudson commented on HDFS-5647: -- FAILURE: Integrated in Hadoop-Yarn-trunk #420 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/420/]) HDFS-5647. Merge INodeDirectory.Feature and INodeFile.Feature. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550469) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileUnderConstructionFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java > Merge INodeDirectory.Feature and INodeFile.Feature > -- > > Key: HDFS-5647 > URL: https://issues.apache.org/jira/browse/HDFS-5647 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0 > > Attachments: HDFS-5647.000.patch, HDFS-5647.001.patch, > HDFS-5647.002.patch, HDFS-5647.003.patch > > > HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features > introduced in HDFS-5284. The current code separates the INode feature of > INodeFile and INodeDirectory into two different class hierarchies. This > hinders the implementation of ACL since ACL is a concept that applies to both > INodeFile and INodeDirectory. > This jira proposes to merge the two class hierarchies (i.e., > INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation > of ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5023) TestSnapshotPathINodes.testAllowSnapshot is failing with jdk7
[ https://issues.apache.org/jira/browse/HDFS-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847399#comment-13847399 ] Hudson commented on HDFS-5023: -- FAILURE: Integrated in Hadoop-Yarn-trunk #420 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/420/]) HDFS-5023. TestSnapshotPathINodes.testAllowSnapshot is failing with jdk7 (Mit Desai via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550261) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSnapshotPathINodes.java > TestSnapshotPathINodes.testAllowSnapshot is failing with jdk7 > - > > Key: HDFS-5023 > URL: https://issues.apache.org/jira/browse/HDFS-5023 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots, test >Affects Versions: 3.0.0, 2.4.0 >Reporter: Ravi Prakash >Assignee: Mit Desai > Labels: java7, test > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5023.patch, HDFS-5023.patch, HDFS-5023.patch, > TEST-org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes.xml, > org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes-output.txt > > > The assertion on line 91 is failing. I am using Fedora 19 + JDK7. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847485#comment-13847485 ] Hudson commented on HDFS-5647: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1611 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1611/]) HDFS-5647. Merge INodeDirectory.Feature and INodeFile.Feature. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550469) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileUnderConstructionFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java > Merge INodeDirectory.Feature and INodeFile.Feature > -- > > Key: HDFS-5647 > URL: https://issues.apache.org/jira/browse/HDFS-5647 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0 > > Attachments: HDFS-5647.000.patch, HDFS-5647.001.patch, > HDFS-5647.002.patch, HDFS-5647.003.patch > > > HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features > introduced in HDFS-5284. The current code separates the INode feature of > INodeFile and INodeDirectory into two different class hierarchies. This > hinders the implementation of ACL since ACL is a concept that applies to both > INodeFile and INodeDirectory. > This jira proposes to merge the two class hierarchies (i.e., > INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation > of ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5652) refactoring/uniforming invalid block token exception handling in DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847488#comment-13847488 ] Hudson commented on HDFS-5652: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1611 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1611/]) HDFS-5652. Refactor invalid block token exception handling in DFSInputStream. (Liang Xie via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550620) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java > refactoring/uniforming invalid block token exception handling in > DFSInputStream > --- > > Key: HDFS-5652 > URL: https://issues.apache.org/jira/browse/HDFS-5652 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie >Priority: Minor > Fix For: 2.4.0 > > Attachments: HDFS-5652.txt > > > See comments from Junping and Colin's from HDFS-5637 -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5023) TestSnapshotPathINodes.testAllowSnapshot is failing with jdk7
[ https://issues.apache.org/jira/browse/HDFS-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847545#comment-13847545 ] Hudson commented on HDFS-5023: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1637 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1637/]) HDFS-5023. TestSnapshotPathINodes.testAllowSnapshot is failing with jdk7 (Mit Desai via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550261) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSnapshotPathINodes.java > TestSnapshotPathINodes.testAllowSnapshot is failing with jdk7 > - > > Key: HDFS-5023 > URL: https://issues.apache.org/jira/browse/HDFS-5023 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots, test >Affects Versions: 3.0.0, 2.4.0 >Reporter: Ravi Prakash >Assignee: Mit Desai > Labels: java7, test > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5023.patch, HDFS-5023.patch, HDFS-5023.patch, > TEST-org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes.xml, > org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes-output.txt > > > The assertion on line 91 is failing. I am using Fedora 19 + JDK7. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-2832) Enable support for heterogeneous storages in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847538#comment-13847538 ] Hudson commented on HDFS-2832: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1637 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1637/]) HDFS-2832: Update binary file editsStored for TestOfflineEditsViewer (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550364) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored svn merge --reintegrate https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2832 for merging Heterogeneous Storage feature branch (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550363) * /hadoop/common/trunk * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockListAsLongs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/UnregisteredNodeException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/Datano
[jira] [Commented] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847543#comment-13847543 ] Hudson commented on HDFS-5647: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1637 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1637/]) HDFS-5647. Merge INodeDirectory.Feature and INodeFile.Feature. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550469) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileUnderConstructionFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java > Merge INodeDirectory.Feature and INodeFile.Feature > -- > > Key: HDFS-5647 > URL: https://issues.apache.org/jira/browse/HDFS-5647 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0 > > Attachments: HDFS-5647.000.patch, HDFS-5647.001.patch, > HDFS-5647.002.patch, HDFS-5647.003.patch > > > HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features > introduced in HDFS-5284. The current code separates the INode feature of > INodeFile and INodeDirectory into two different class hierarchies. This > hinders the implementation of ACL since ACL is a concept that applies to both > INodeFile and INodeDirectory. > This jira proposes to merge the two class hierarchies (i.e., > INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation > of ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred
[ https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847544#comment-13847544 ] Hudson commented on HDFS-5637: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1637 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1637/]) HDFS-5637. Try to refeatchToken while local read InvalidToken occurred. (Liang Xie via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550335) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java > try to refeatchToken while local read InvalidToken occurred > --- > > Key: HDFS-5637 > URL: https://issues.apache.org/jira/browse/HDFS-5637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, security >Affects Versions: 2.0.5-alpha, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Fix For: 2.4.0 > > Attachments: HDFS-5637-v2.txt, HDFS-5637.txt > > > we observed several warning logs like below from region server nodes: > 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to > connect to /10.2.201.110:11402 for block, add to deadNodes and continue. > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > at
[jira] [Commented] (HDFS-5652) refactoring/uniforming invalid block token exception handling in DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847548#comment-13847548 ] Hudson commented on HDFS-5652: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1637 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1637/]) HDFS-5652. Refactor invalid block token exception handling in DFSInputStream. (Liang Xie via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550620) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java > refactoring/uniforming invalid block token exception handling in > DFSInputStream > --- > > Key: HDFS-5652 > URL: https://issues.apache.org/jira/browse/HDFS-5652 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie >Priority: Minor > Fix For: 2.4.0 > > Attachments: HDFS-5652.txt > > > See comments from Junping and Colin's from HDFS-5637 -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[
https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847550#comment-13847550
]
Hudson commented on HDFS-4201:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1637 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1637/])
HDFS-4201. NPE in BPServiceActor#sendHeartBeat (jxiang via cmccabe) (cmccabe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550269)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
> NPE in BPServiceActor#sendHeartBeat
> ---
>
> Key: HDFS-4201
> URL: https://issues.apache.org/jira/browse/HDFS-4201
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Eli Collins
>Assignee: Jimmy Xiang
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: trunk-4201.patch, trunk-4201_v2.patch,
> trunk-4201_v3.patch
>
>
> Saw the following NPE in a log.
> Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not
> {{bpRegistration}}) due to a configuration or local directory failure.
> {code}
> 2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of
> 30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec;
> heartBeatInterval=3000
> 2012-09-25 04:33:20,782 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService
> for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id
> DS-1031100678-11.164.162.251-5010-1341933415989) service to
> svsrs00127/11.164.162.226:8020
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
> at java.lang.Thread.run(Thread.java:722)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5484) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
[
https://issues.apache.org/jira/browse/HDFS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847588#comment-13847588
]
Eric Sirianni commented on HDFS-5484:
-
This fix was basically nullified by the following change made via HDFS-5542
{code}
+ DatanodeStorageInfo updateStorage(DatanodeStorage s) {
synchronized (storageMap) {
DatanodeStorageInfo storage = storageMap.get(s.getStorageID());
if (storage == null) {
@@ -670,8 +658,6 @@
" for DN " + getXferAddr());
storage = new DatanodeStorageInfo(this, s);
storageMap.put(s.getStorageID(), storage);
- } else {
-storage.setState(s.getState());
}
return storage;
}
{code}
Is there a reason that 'else' was removed? By no longer updating the state in
the {{BlockReport}} processing path, we effectively get the bogus state & type
that is set via the first heartbeat (see the fix for HDFS-5455):
{code}
+ if (storage == null) {
+// This is seen during cluster initialization when the heartbeat
+// is received before the initial block reports from each storage.
+storage = updateStorage(new DatanodeStorage(report.getStorageID()));
{code}
Even reverting the change and reintroducing the 'else' leaves the state & type
temporarily inaccurate until the first block report. Wouldn't a better fix be
to simply include the full {{DatanodeStorage}} object in the {{StorageReport}}
(as opposed to only the Storage ID)?
As a matter of bookkeeping, should I reopen this JIRA, or would you prefer a
new one be created?
> StorageType and State in DatanodeStorageInfo in NameNode is not accurate
>
>
> Key: HDFS-5484
> URL: https://issues.apache.org/jira/browse/HDFS-5484
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: Heterogeneous Storage (HDFS-2832)
>Reporter: Eric Sirianni
> Fix For: Heterogeneous Storage (HDFS-2832)
>
> Attachments: HDFS-5484-HDFS-2832--2.patch, HDFS-5484-HDFS-2832.patch
>
>
> The fields in DatanodeStorageInfo are updated from two distinct paths:
> # block reports
> # storage reports (via heartbeats)
> The {{state}} and {{storageType}} fields are updated via the Block Report.
> However, as seen in the code blow, these fields are populated from a "dummy"
> {{DatanodeStorage}} object constructed in the DataNode:
> {code}
> BPServiceActor.blockReport() {
> //...
> // Dummy DatanodeStorage object just for sending the block report.
> DatanodeStorage dnStorage = new DatanodeStorage(storageID);
> //...
> }
> {code}
> The net effect is that the {{state}} and {{storageType}} fields are always
> the default of {{NORMAL}} and {{DISK}} in the NameNode.
> The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
> {code}
> public Map getBlockReports(String bpid);
> {code}
> to:
> {code}
> public Map getBlockReports(String bpid);
> {code}
> thereby allowing {{BPServiceActor}} to send the "real" {{DatanodeStorage}}
> object with the block report.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5663) make the retry time and interval value configurable in openInfo()
[ https://issues.apache.org/jira/browse/HDFS-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-5663: Attachment: HDFS-5663.txt Attached is a trivial patch for it. > make the retry time and interval value configurable in openInfo() > - > > Key: HDFS-5663 > URL: https://issues.apache.org/jira/browse/HDFS-5663 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5663.txt > > > The original idea raised from Michael here: > https://issues.apache.org/jira/browse/HDFS-1605?focusedCommentId=13846972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13846972 > It would be better to have a lower interval value especially for online > service, like HBase. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Work started] (HDFS-5660) When SSL is enabled , the Namenode WEBUI redirects to Infosecport, which could be 0
[ https://issues.apache.org/jira/browse/HDFS-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-5660 started by Benoy Antony. > When SSL is enabled , the Namenode WEBUI redirects to Infosecport, which > could be 0 > --- > > Key: HDFS-5660 > URL: https://issues.apache.org/jira/browse/HDFS-5660 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5660.patch > > > (case 1) When SSL is enabled by setting "hadoop.ssl.enabled", SSL will be > enabled on the regular port (infoport) on Datanode. > (case 2) When SSL on HDFS is enabled by setting "dfs.https.enable", SSL will > be enabled on a separate port (infoSecurePort) on Datanode. > if SSL is enabled , the Namenode always redirects to infoSecurePort. > "infoSecurePort" will be 0 in case 1 above. > This breaks the file browsing via web. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Work started] (HDFS-5661) Browsing FileSystem via web ui, should use datanode's hostname instead of ip address
[ https://issues.apache.org/jira/browse/HDFS-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-5661 started by Benoy Antony. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address > > > Key: HDFS-5661 > URL: https://issues.apache.org/jira/browse/HDFS-5661 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5661.patch > > > If authentication is enabled on the web ui, then a cookie is used to keep > track of the authentication information. There is normally a domain > associated with the cookie. Since ip address doesn't have any domain , the > cookie will not be sent by the browser while making http calls with ip > address as the destination server. > This will break browsing files system via web ui , if authentication is > enabled. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5663) make the retry time and interval value configurable in openInfo()
[ https://issues.apache.org/jira/browse/HDFS-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-5663: Status: Patch Available (was: Open) > make the retry time and interval value configurable in openInfo() > - > > Key: HDFS-5663 > URL: https://issues.apache.org/jira/browse/HDFS-5663 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.2.0, 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5663.txt > > > The original idea raised from Michael here: > https://issues.apache.org/jira/browse/HDFS-1605?focusedCommentId=13846972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13846972 > It would be better to have a lower interval value especially for online > service, like HBase. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5365) Fix libhdfs compile error on FreeBSD9
[ https://issues.apache.org/jira/browse/HDFS-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847649#comment-13847649 ] Chris Nauroth commented on HDFS-5365: - Hi Radim, Unfortunately, this can't be merged to branch-2.2. AFAIK, there are not any additional 2.2.x releases on the schedule, and the plan is to proceed with 2.3.x forward: https://wiki.apache.org/hadoop/Roadmap Therefore, if we committed code to branch-2.2 now, then it wouldn't actually go into a release. I have confirmed that your patch is in branch-2.3 for inclusion in the 2.3.0 release, currently targeted for mid-December. If the plan changes and we decide to add another 2.2.x release on the schedule, then you could nominate this patch for inclusion and we'd commit it at that time. It seems unlikely at this point though. > Fix libhdfs compile error on FreeBSD9 > - > > Key: HDFS-5365 > URL: https://issues.apache.org/jira/browse/HDFS-5365 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Radim Kolar >Assignee: Radim Kolar > Labels: build > Fix For: 3.0.0, 2.3.0 > > Attachments: hdfs-bsd.txt > > > native library do not compiles on freebsd because: > * dlopen is in libc > * limits.h include is missing -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5663) make the retry time and interval value configurable in openInfo()
[ https://issues.apache.org/jira/browse/HDFS-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847661#comment-13847661 ] stack commented on HDFS-5663: - +1 > make the retry time and interval value configurable in openInfo() > - > > Key: HDFS-5663 > URL: https://issues.apache.org/jira/browse/HDFS-5663 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5663.txt > > > The original idea raised from Michael here: > https://issues.apache.org/jira/browse/HDFS-1605?focusedCommentId=13846972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13846972 > It would be better to have a lower interval value especially for online > service, like HBase. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5496) Make replication queue initialization asynchronous
[
https://issues.apache.org/jira/browse/HDFS-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847744#comment-13847744
]
Jing Zhao commented on HDFS-5496:
-
Thanks for the further explanation, Vinay. Now I see what you mean.
{code}
blockManager.clearQueues();
blockManager.processAllPendingDNMessages();
{code}
In my previous comment, I thought you also planned to remove the clearQueues()
call..
So if we clear all the queues, and continue with the ongoing initialization,
the blocks that were processed before the startActiveService call will not be
re-processed? I.e., we finally will generate the replication queues for only
part of the blocks.
> Make replication queue initialization asynchronous
> --
>
> Key: HDFS-5496
> URL: https://issues.apache.org/jira/browse/HDFS-5496
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Kihwal Lee
>Assignee: Vinay
> Attachments: HDFS-5496.patch, HDFS-5496.patch
>
>
> Today, initialization of replication queues blocks safe mode exit and certain
> HA state transitions. For a big name space, this can take hundreds of seconds
> with the FSNamesystem write lock held. During this time, important requests
> (e.g. initial block reports, heartbeat, etc) are blocked.
> The effect of delaying the initialization would be not starting replication
> right away, but I think the benefit outweighs. If we make it asynchronous,
> the work per iteration should be limited, so that the lock duration is
> capped.
> If full/incremental block reports and any other requests that modifies block
> state properly performs replication checks while the blocks are scanned and
> the queues populated in background, every block will be processed. (Some may
> be done twice) The replication monitor should run even before all blocks are
> processed.
> This will allow namenode to exit safe mode and start serving immediately even
> with a big name space. It will also reduce the HA failover latency.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5660) When SSL is enabled , the Namenode WEBUI redirects to Infosecport, which could be 0
[ https://issues.apache.org/jira/browse/HDFS-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847776#comment-13847776 ] Benoy Antony commented on HDFS-5660: I can get HDFS SSL work using dfs.https.enable and 3 additional configs ( NN port , DN port and keystore config.) With this scheme, two ports on NN and each data node will be opened. These additional ports will not be used in our case. I am doing all these steps just so that infosecurePort is set to 1006 which is essentially what this patch does. > When SSL is enabled , the Namenode WEBUI redirects to Infosecport, which > could be 0 > --- > > Key: HDFS-5660 > URL: https://issues.apache.org/jira/browse/HDFS-5660 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5660.patch > > > (case 1) When SSL is enabled by setting "hadoop.ssl.enabled", SSL will be > enabled on the regular port (infoport) on Datanode. > (case 2) When SSL on HDFS is enabled by setting "dfs.https.enable", SSL will > be enabled on a separate port (infoSecurePort) on Datanode. > if SSL is enabled , the Namenode always redirects to infoSecurePort. > "infoSecurePort" will be 0 in case 1 above. > This breaks the file browsing via web. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5661) Browsing FileSystem via web ui, should use datanode's hostname instead of ip address
[ https://issues.apache.org/jira/browse/HDFS-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1384#comment-1384 ] Haohui Mai commented on HDFS-5661: -- The namenode and the datanode have different origins therefore the browser will not attach the cookies when making a request to the datanode. Redirecting using the domain name will not help. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address > > > Key: HDFS-5661 > URL: https://issues.apache.org/jira/browse/HDFS-5661 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5661.patch > > > If authentication is enabled on the web ui, then a cookie is used to keep > track of the authentication information. There is normally a domain > associated with the cookie. Since ip address doesn't have any domain , the > cookie will not be sent by the browser while making http calls with ip > address as the destination server. > This will break browsing files system via web ui , if authentication is > enabled. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5660) When SSL is enabled , the Namenode WEBUI redirects to Infosecport, which could be 0
[ https://issues.apache.org/jira/browse/HDFS-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847779#comment-13847779 ] Haohui Mai commented on HDFS-5660: -- This is the expected behavior. Again, the bug no longer exists in 2.2. I suggest closing this bug as invalid. > When SSL is enabled , the Namenode WEBUI redirects to Infosecport, which > could be 0 > --- > > Key: HDFS-5660 > URL: https://issues.apache.org/jira/browse/HDFS-5660 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5660.patch > > > (case 1) When SSL is enabled by setting "hadoop.ssl.enabled", SSL will be > enabled on the regular port (infoport) on Datanode. > (case 2) When SSL on HDFS is enabled by setting "dfs.https.enable", SSL will > be enabled on a separate port (infoSecurePort) on Datanode. > if SSL is enabled , the Namenode always redirects to infoSecurePort. > "infoSecurePort" will be 0 in case 1 above. > This breaks the file browsing via web. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Assigned] (HDFS-5659) dfsadmin -report doesn't output cache information properly
[
https://issues.apache.org/jira/browse/HDFS-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reassigned HDFS-5659:
-
Assignee: Andrew Wang
> dfsadmin -report doesn't output cache information properly
> --
>
> Key: HDFS-5659
> URL: https://issues.apache.org/jira/browse/HDFS-5659
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: caching
>Affects Versions: 3.0.0
>Reporter: Akira AJISAKA
>Assignee: Andrew Wang
>
> I tried to cache a file by "hdfs cacheadmin -addDirective".
> I thought the file was cached because "CacheUsed" at jmx was more than 0.
> {code}
> {
> "name" :
> "Hadoop:service=DataNode,name=FSDatasetState-DS-1043926324-172.28.0.102-50010-1385087929296",
> "modelerType" :
> "org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl",
> "Remaining" : 5604772597760,
> "StorageInfo" : "FSDataset{dirpath='[/hadoop/data1/dfs/data/current,
> /hadoop/data2/dfs/data/current, /hadoop/data3/dfs/data/current]'}",
> "Capacity" : 5905374474240,
> "DfsUsed" : 11628544,
> "CacheCapacity" : 1073741824,
> "CacheUsed" : 360448,
> "NumFailedVolumes" : 0,
> "NumBlocksCached" : 1,
> "NumBlocksFailedToCache" : 0,
> "NumBlocksFailedToUncache" : 0
> },
> {code}
> But "dfsadmin -report" didn't output the same value as jmx.
> {code}
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5663) make the retry time and interval value configurable in openInfo()
[
https://issues.apache.org/jira/browse/HDFS-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847790#comment-13847790
]
Hadoop QA commented on HDFS-5663:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618620/HDFS-5663.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestStartup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5712//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5712//console
This message is automatically generated.
> make the retry time and interval value configurable in openInfo()
> -
>
> Key: HDFS-5663
> URL: https://issues.apache.org/jira/browse/HDFS-5663
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: HDFS-5663.txt
>
>
> The original idea raised from Michael here:
> https://issues.apache.org/jira/browse/HDFS-1605?focusedCommentId=13846972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13846972
> It would be better to have a lower interval value especially for online
> service, like HBase.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5661) Browsing FileSystem via web ui, should use datanode's hostname instead of ip address
[ https://issues.apache.org/jira/browse/HDFS-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847789#comment-13847789 ] Benoy Antony commented on HDFS-5661: Browser visits namenode.domainname.com:50070 . "hadoopauth" cookie is dropped with domain as "domainname.com". User clicks "browse File system" on the webui and browser gets redirected to ADataNode.domainname.com:1006 . Then the browser will send cookies to ADataNode.domainname.com . Without this patch, when user clicks "browse File system" on the webui and gets redirected to datanodeipaddress:1006 and hence no cookies will be sent. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address > > > Key: HDFS-5661 > URL: https://issues.apache.org/jira/browse/HDFS-5661 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5661.patch > > > If authentication is enabled on the web ui, then a cookie is used to keep > track of the authentication information. There is normally a domain > associated with the cookie. Since ip address doesn't have any domain , the > cookie will not be sent by the browser while making http calls with ip > address as the destination server. > This will break browsing files system via web ui , if authentication is > enabled. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[
https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847815#comment-13847815
]
Colin Patrick McCabe commented on HDFS-5634:
bq. I examined this code more carefully, and I found that it was actually using
LIFO at the moment.
Err, sorry, I mis-spoke. It is FIFO. Unfortunately, {{ConcurrentLinkedDeque}}
is not in JDK6 (although it is in JDK7), so it will be hard to test with LIFO
here.
> allow BlockReaderLocal to switch between checksumming and not
> -
>
> Key: HDFS-5634
> URL: https://issues.apache.org/jira/browse/HDFS-5634
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch,
> HDFS-5634.003.patch, HDFS-5634.004.patch
>
>
> BlockReaderLocal should be able to switch between checksumming and
> non-checksumming, so that when we get notifications that something is mlocked
> (see HDFS-5182), we can avoid checksumming when reading from that block.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5484) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
[
https://issues.apache.org/jira/browse/HDFS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847823#comment-13847823
]
Arpit Agarwal commented on HDFS-5484:
-
Let's create a new Jira.
I am fine with adding the {{DatanodeStorage}} object to {{StorageReportProto}}.
It needs to be a new optional field and we cannot remove the existing
{{StorageUuid}} for protocol compatibility.
> StorageType and State in DatanodeStorageInfo in NameNode is not accurate
>
>
> Key: HDFS-5484
> URL: https://issues.apache.org/jira/browse/HDFS-5484
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: Heterogeneous Storage (HDFS-2832)
>Reporter: Eric Sirianni
> Fix For: Heterogeneous Storage (HDFS-2832)
>
> Attachments: HDFS-5484-HDFS-2832--2.patch, HDFS-5484-HDFS-2832.patch
>
>
> The fields in DatanodeStorageInfo are updated from two distinct paths:
> # block reports
> # storage reports (via heartbeats)
> The {{state}} and {{storageType}} fields are updated via the Block Report.
> However, as seen in the code blow, these fields are populated from a "dummy"
> {{DatanodeStorage}} object constructed in the DataNode:
> {code}
> BPServiceActor.blockReport() {
> //...
> // Dummy DatanodeStorage object just for sending the block report.
> DatanodeStorage dnStorage = new DatanodeStorage(storageID);
> //...
> }
> {code}
> The net effect is that the {{state}} and {{storageType}} fields are always
> the default of {{NORMAL}} and {{DISK}} in the NameNode.
> The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
> {code}
> public Map getBlockReports(String bpid);
> {code}
> to:
> {code}
> public Map getBlockReports(String bpid);
> {code}
> thereby allowing {{BPServiceActor}} to send the "real" {{DatanodeStorage}}
> object with the block report.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5661) Browsing FileSystem via web ui, should use datanode's hostname instead of ip address
[ https://issues.apache.org/jira/browse/HDFS-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847826#comment-13847826 ] Haohui Mai commented on HDFS-5661: -- I'm curious how you manage to pass the cookie to the datanode. Even with your patch, the cookies should not be passed from the namenode and the datanode. The browser is not supposed to shared the cookies. An origin is defined by the scheme, host, and port of a URL[1], where two hosts with the same hostname but different ports are considered different origins. The browsers implement the same-origin policy, where the cookies are isolated in different origins [2]. [1] http://www.w3.org/Security/wiki/Same_Origin_Policy [2] https://code.google.com/p/browsersec/wiki/Part2#Same-origin_policy > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address > > > Key: HDFS-5661 > URL: https://issues.apache.org/jira/browse/HDFS-5661 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5661.patch > > > If authentication is enabled on the web ui, then a cookie is used to keep > track of the authentication information. There is normally a domain > associated with the cookie. Since ip address doesn't have any domain , the > cookie will not be sent by the browser while making http calls with ip > address as the destination server. > This will break browsing files system via web ui , if authentication is > enabled. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5661) Browsing FileSystem via web ui, should use datanode's hostname instead of ip address
[ https://issues.apache.org/jira/browse/HDFS-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847852#comment-13847852 ] Jing Zhao commented on HDFS-5661: - For the redirect, I think we are using DelegationToken which is included in the redirect URL? Thus we do not need to worry about hostname/ip address here I guess. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address > > > Key: HDFS-5661 > URL: https://issues.apache.org/jira/browse/HDFS-5661 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5661.patch > > > If authentication is enabled on the web ui, then a cookie is used to keep > track of the authentication information. There is normally a domain > associated with the cookie. Since ip address doesn't have any domain , the > cookie will not be sent by the browser while making http calls with ip > address as the destination server. > This will break browsing files system via web ui , if authentication is > enabled. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HDFS-5666) TestBPOfferService#/testBPInitErrorHandling fails intermittently
Colin Patrick McCabe created HDFS-5666:
--
Summary: TestBPOfferService#/testBPInitErrorHandling fails
intermittently
Key: HDFS-5666
URL: https://issues.apache.org/jira/browse/HDFS-5666
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 2.4.0
Reporter: Colin Patrick McCabe
Priority: Minor
Intermittent failure on this test:
{code}
Regression
org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testBPInitErrorHandling
Failing for the past 1 build (Since #5698 )
Took 0.16 sec.
Error Message
expected:<1> but was:<2>
Stacktrace
java.lang.AssertionError: expected:<1> but was:<2>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
{code}
see
https://builds.apache.org/job/PreCommit-HDFS-Build/5698//testReport/org.apache.hadoop.hdfs.server.datanode/TestBPOfferService/testBPInitErrorHandling/
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[
https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847854#comment-13847854
]
Andrew Wang commented on HDFS-5634:
---
Thanks for being patient with me, I think I finally grok what's going on.
bq.
Alright. It seems odd for datanodeID and block to ever be null since they're
used to key the FISCache, but okay. Still, we're currently not setting the
caching strategy in DFSInputStream#getBlockReader when pulling something out of
the FISCache, should we?
bq. We always buffer at least a single chunk, even if readahead is turned off.
The mechanics of checksumming require this.
In {{fillDataBuf(boolean)}}, it looks like we try to fill
{{maxReadaheadLength}} modulo the slop. If this is zero, what happens? I think
we need to take the checksum chunk size into account here somewhere.
It'd also be good to have test coverage for all these different read paths and
parameter combinations. It looks like TestParallel* already cover the
ByteBuffer and array variants of read with and without checksums, but we also
need to permute on different readahead sizes (e.g. 0, chunk-1, chunk+1).
Unfortunately even though the array and BB versions are almost the same, they
don't share any code, so we should probably still permute on that axis.
With the new getter, we could use that instead of passing around
{{canSkipChecksum}} everywhere, but I'm unsure if mlock toggling has
implications here. Your call.
There's still BlockReaderFactory#newBlockReader using verifyChecksum rather
than skipChecksum. Do you want to change this one too? There are only 6 usages
from what I see.
bq.
Yea, let's just skip this idea for now. We really should use a profiler for
guidance before trying optimizations anyway. That'd be interesting to do later.
bq.
I was thinking user documentation in hdfs-default.xml for
"dfs.client.read.shortcircuit.buffer.size" and "dfs.datanode.readahead.bytes"
but I see you filed a follow-on for that. It also feels inconsistent that the
datanode defaults to 4MB readahead while the client looks like it defaults to
0, but maybe there's some reasoning there.
> allow BlockReaderLocal to switch between checksumming and not
> -
>
> Key: HDFS-5634
> URL: https://issues.apache.org/jira/browse/HDFS-5634
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch,
> HDFS-5634.003.patch, HDFS-5634.004.patch
>
>
> BlockReaderLocal should be able to switch between checksumming and
> non-checksumming, so that when we get notifications that something is mlocked
> (see HDFS-5182), we can avoid checksumming when reading from that block.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Attachment: HDFS-5454.02.patch
Updated patch.
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Priority: Minor
> Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5350) Name Node should report fsimage transfer time as a metric
[ https://issues.apache.org/jira/browse/HDFS-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847869#comment-13847869 ] Andrew Wang commented on HDFS-5350: --- Hey Jimmy, thanks for the patch. It looks pretty good, just a few review comments: * GetImageServlet is used to transfer both images and edits. Since these look like they should be image-only metrics, we need to move the timing out of serveFile (which is used for both) into the right if statement. I think this is also more clear, since now all the timing will be in the same function. * Could you add a new test to TestNameNodeMetrics showing that this gets updated as expected? > Name Node should report fsimage transfer time as a metric > - > > Key: HDFS-5350 > URL: https://issues.apache.org/jira/browse/HDFS-5350 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Rob Weltman >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 3.0.0 > > Attachments: trunk-5350.patch > > > If the (Secondary) Name Node reported fsimage transfer times (perhaps the > last ten of them), monitoring tools could detect slowdowns that might > jeopardize cluster stability. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[
https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847873#comment-13847873
]
Colin Patrick McCabe commented on HDFS-5634:
bq. Alright. It seems odd for datanodeID and block to ever be null since
they're used to key the FISCache, but okay. Still, we're currently not setting
the caching strategy in DFSInputStream#getBlockReader when pulling something
out of the FISCache, should we?
Definitely. Fixed.
bq. In fillDataBuf(boolean), it looks like we try to fill maxReadaheadLength
modulo the slop. If this is zero, what happens? I think we need to take the
checksum chunk size into account here somewhere.
This is a bug. Basically it's a corner case where we need to fudge the user's
no-readahead request into a short-readahead request when doing a checksummed
read. Will fix...
bq. It'd also be good to have test coverage for all these different read paths
and parameter combinations. It looks like TestParallel* already cover the
ByteBuffer and array variants of read with and without checksums, but we also
need to permute on different readahead sizes (e.g. 0, chunk-1, chunk+1).
Unfortunately even though the array and BB versions are almost the same, they
don't share any code, so we should probably still permute on that axis.
I think the main thing is to have coverage for the no-readahead + checksum
case, which we don't really now. Having a test that did chunk-1 readahead, to
verify that the round-up functionality worked, would also be a good idea.
bq. With the new getter, we could use that instead of passing around
canSkipChecksum everywhere, but I'm unsure if mlock toggling has implications
here. Your call.
Accessing the atomic boolean does have a performance overhead that I wanted to
avoid. Also, as you said, it might change, which could create problems.
bq. I was thinking user documentation in hdfs-default.xml for
"dfs.client.read.shortcircuit.buffer.size" and "dfs.datanode.readahead.bytes"
but I see you filed a follow-on for that. It also feels inconsistent that the
datanode defaults to 4MB readahead while the client looks like it defaults to
0, but maybe there's some reasoning there.
Actually, the client defaults to "null" readahead which usually means the
datanode gets to decide. I guess it's a bit of a confusing construct, but here
it is:
{code}
Long readahead = (conf.get(DFS_CLIENT_CACHE_READAHEAD) == null) ?
null : conf.getLong(DFS_CLIENT_CACHE_READAHEAD, 0);
{code}
In {{BlockReaderLocal}} I am defaulting to {{dfs.datanode.readahead.bytes}}
when the client's value is null. If we wanted to be totally consistent with
how remote reads work, we'd have to get the readahead default from the DataNode
as part of the RPC. That seems a little bit like overkill, though.
bq. There's still BlockReaderFactory#newBlockReader using verifyChecksum rather
than skipChecksum. Do you want to change this one too? There are only 6 usages
from what I see.
OK.
> allow BlockReaderLocal to switch between checksumming and not
> -
>
> Key: HDFS-5634
> URL: https://issues.apache.org/jira/browse/HDFS-5634
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch,
> HDFS-5634.003.patch, HDFS-5634.004.patch
>
>
> BlockReaderLocal should be able to switch between checksumming and
> non-checksumming, so that when we get notifications that something is mlocked
> (see HDFS-5182), we can avoid checksumming when reading from that block.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5656) add some configuration keys to hdfs-default.xml
[
https://issues.apache.org/jira/browse/HDFS-5656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847874#comment-13847874
]
Colin Patrick McCabe commented on HDFS-5656:
also: dfs.datanode.readahead.bytes, dfs.client.read.shortcircuit.buffer.size
> add some configuration keys to hdfs-default.xml
> ---
>
> Key: HDFS-5656
> URL: https://issues.apache.org/jira/browse/HDFS-5656
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Colin Patrick McCabe
>Priority: Minor
>
> Some configuration keys like {{dfs.client.read.shortcircuit}} are not present
> in {{hdfs-default.xml}} as they should be.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5656) add some configuration keys to hdfs-default.xml
[
https://issues.apache.org/jira/browse/HDFS-5656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847875#comment-13847875
]
Colin Patrick McCabe commented on HDFS-5656:
bq. I'll add that "dfs.client.cache.readahead" doesn't have a default value in
DFSConfigKeys too.
Its default value is null, meaning that the datanode decides what readahead is
(when doing remote reads)
> add some configuration keys to hdfs-default.xml
> ---
>
> Key: HDFS-5656
> URL: https://issues.apache.org/jira/browse/HDFS-5656
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Colin Patrick McCabe
>Priority: Minor
>
> Some configuration keys like {{dfs.client.read.shortcircuit}} are not present
> in {{hdfs-default.xml}} as they should be.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5665) Remove the unnecessary writeLock while initializing CacheManager in FsNameSystem Ctor
[
https://issues.apache.org/jira/browse/HDFS-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847897#comment-13847897
]
Colin Patrick McCabe commented on HDFS-5665:
I agree-- I think it would be fine to initialize this without holding the
writeLock.
> Remove the unnecessary writeLock while initializing CacheManager in
> FsNameSystem Ctor
> -
>
> Key: HDFS-5665
> URL: https://issues.apache.org/jira/browse/HDFS-5665
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>
> I just saw the below piece of code in Fsnamesystem ctor.
> {code}
> writeLock();
> try {
> this.cacheManager = new CacheManager(this, conf, blockManager);
> } finally {
> writeUnlock();
> }
> {code}
> It seems unnecessary to keep writeLock here. I am not sure if there is a
> clear reason to keep the lock.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5664) try to relieve the BlockReaderLocal read() synchronized hotspot
[
https://issues.apache.org/jira/browse/HDFS-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847906#comment-13847906
]
Colin Patrick McCabe commented on HDFS-5664:
A single {{DFSInputStream}} is designed to be used by a single thread only. We
don't allow multiple reads from multiple threads on the same stream to go
forward at the same time. So I don't see how making {{BlockReader}} (or a
subclass like {{BlockReaderLocal}}) concurrent would help at all, since there
would still be a big "synchronized" on all the {{DFSInputStream#read}} methods
which use the {{BlockReader}}.
If multiple threads want to read the same file at the same time, they can open
multiple distinct streams for it. At that point, they're not sharing the same
{{BlockReader}}, so whether or not BRL is synchronized doesn't matter.
> try to relieve the BlockReaderLocal read() synchronized hotspot
> ---
>
> Key: HDFS-5664
> URL: https://issues.apache.org/jira/browse/HDFS-5664
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Liang Xie
>Assignee: Liang Xie
>
> Current the BlockReaderLocal's read has a synchronized modifier:
> {code}
> public synchronized int read(byte[] buf, int off, int len) throws IOException
> {
> {code}
> In a HBase physical read heavy cluster, we observed some hotspots from
> dfsclient path, the detail strace trace could be found from:
> https://issues.apache.org/jira/browse/HDFS-1605?focusedCommentId=13843241&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13843241
> I haven't looked into the detail yet, put some raw ideas here firstly:
> 1) replace synchronized with try lock with timeout pattern, so could
> fail-fast, 2) fallback to non-ssr mode if get a local reader lock failed.
> There're two suitable scenario at least to remove this hotspot:
> 1) Local physical read heavy, e.g. HBase block cache miss ratio is high
> 2) slow/bad disk.
> It would be helpful to achive a lower 99th percentile HBase read latency
> somehow.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Created] (HDFS-5667) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
Eric Sirianni created HDFS-5667:
---
Summary: StorageType and State in DatanodeStorageInfo in NameNode
is not accurate
Key: HDFS-5667
URL: https://issues.apache.org/jira/browse/HDFS-5667
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Eric Sirianni
Fix For: Heterogeneous Storage (HDFS-2832)
The fields in DatanodeStorageInfo are updated from two distinct paths:
# block reports
# storage reports (via heartbeats)
The {{state}} and {{storageType}} fields are updated via the Block Report.
However, as seen in the code blow, these fields are populated from a "dummy"
{{DatanodeStorage}} object constructed in the DataNode:
{code}
BPServiceActor.blockReport() {
//...
// Dummy DatanodeStorage object just for sending the block report.
DatanodeStorage dnStorage = new DatanodeStorage(storageID);
//...
}
{code}
The net effect is that the {{state}} and {{storageType}} fields are always the
default of {{NORMAL}} and {{DISK}} in the NameNode.
The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
{code}
public Map getBlockReports(String bpid);
{code}
to:
{code}
public Map getBlockReports(String bpid);
{code}
thereby allowing {{BPServiceActor}} to send the "real" {{DatanodeStorage}}
object with the block report.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5667) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
[
https://issues.apache.org/jira/browse/HDFS-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Sirianni updated HDFS-5667:
Description:
The fix for HDFS-5484 was accidentally regressed by the following change made
via HDFS-5542
{code}
+ DatanodeStorageInfo updateStorage(DatanodeStorage s) {
synchronized (storageMap) {
DatanodeStorageInfo storage = storageMap.get(s.getStorageID());
if (storage == null) {
@@ -670,8 +658,6 @@
" for DN " + getXferAddr());
storage = new DatanodeStorageInfo(this, s);
storageMap.put(s.getStorageID(), storage);
- } else {
-storage.setState(s.getState());
}
return storage;
}
{code}
By removing the 'else' and no longer updating the state in the BlockReport
processing path, we effectively get the bogus state & type that is set via the
first heartbeat (see the fix for HDFS-5455):
{code}
+ if (storage == null) {
+// This is seen during cluster initialization when the heartbeat
+// is received before the initial block reports from each storage.
+storage = updateStorage(new DatanodeStorage(report.getStorageID()));
{code}
Even reverting the change and reintroducing the 'else' leaves the state & type
temporarily inaccurate until the first block report.
As discussed with [~arpitagarwal], a better fix would be to simply include the
full DatanodeStorage object in the StorageReport (as opposed to only the
Storage ID). This requires adding the {{DatanodeStorage}} object to
{{StorageReportProto}}. It needs to be a new optional field and we cannot
remove the existing {{StorageUuid}} for protocol compatibility.
was:
The fields in DatanodeStorageInfo are updated from two distinct paths:
# block reports
# storage reports (via heartbeats)
The {{state}} and {{storageType}} fields are updated via the Block Report.
However, as seen in the code blow, these fields are populated from a "dummy"
{{DatanodeStorage}} object constructed in the DataNode:
{code}
BPServiceActor.blockReport() {
//...
// Dummy DatanodeStorage object just for sending the block report.
DatanodeStorage dnStorage = new DatanodeStorage(storageID);
//...
}
{code}
The net effect is that the {{state}} and {{storageType}} fields are always the
default of {{NORMAL}} and {{DISK}} in the NameNode.
The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
{code}
public Map getBlockReports(String bpid);
{code}
to:
{code}
public Map getBlockReports(String bpid);
{code}
thereby allowing {{BPServiceActor}} to send the "real" {{DatanodeStorage}}
object with the block report.
> StorageType and State in DatanodeStorageInfo in NameNode is not accurate
>
>
> Key: HDFS-5667
> URL: https://issues.apache.org/jira/browse/HDFS-5667
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: Heterogeneous Storage (HDFS-2832)
>Reporter: Eric Sirianni
> Fix For: Heterogeneous Storage (HDFS-2832)
>
>
> The fix for HDFS-5484 was accidentally regressed by the following change made
> via HDFS-5542
> {code}
> + DatanodeStorageInfo updateStorage(DatanodeStorage s) {
> synchronized (storageMap) {
>DatanodeStorageInfo storage = storageMap.get(s.getStorageID());
>if (storage == null) {
> @@ -670,8 +658,6 @@
> " for DN " + getXferAddr());
> storage = new DatanodeStorageInfo(this, s);
> storageMap.put(s.getStorageID(), storage);
> - } else {
> -storage.setState(s.getState());
>}
>return storage;
> }
> {code}
> By removing the 'else' and no longer updating the state in the BlockReport
> processing path, we effectively get the bogus state & type that is set via
> the first heartbeat (see the fix for HDFS-5455):
> {code}
> + if (storage == null) {
> +// This is seen during cluster initialization when the heartbeat
> +// is received before the initial block reports from each storage.
> +storage = updateStorage(new DatanodeStorage(report.getStorageID()));
> {code}
> Even reverting the change and reintroducing the 'else' leaves the state &
> type temporarily inaccurate until the first block report.
> As discussed with [~arpitagarwal], a better fix would be to simply include
> the full DatanodeStorage object in the StorageReport (as opposed to only the
> Storage ID). This requires adding the {{DatanodeStorage}} object to
> {{StorageReportProto}}. It needs to be a new optional field and we cannot
> remove the existing {{StorageUuid}} for protocol compatibility.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5667) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
[
https://issues.apache.org/jira/browse/HDFS-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Sirianni updated HDFS-5667:
Description:
The fix for HDFS-5484 was accidentally regressed by the following change made
via HDFS-5542
{code}
+ DatanodeStorageInfo updateStorage(DatanodeStorage s) {
synchronized (storageMap) {
DatanodeStorageInfo storage = storageMap.get(s.getStorageID());
if (storage == null) {
@@ -670,8 +658,6 @@
" for DN " + getXferAddr());
storage = new DatanodeStorageInfo(this, s);
storageMap.put(s.getStorageID(), storage);
- } else {
-storage.setState(s.getState());
}
return storage;
}
{code}
By removing the 'else' and no longer updating the state in the BlockReport
processing path, we effectively get the bogus state & type that is set via the
first heartbeat (see the fix for HDFS-5455):
{code}
+ if (storage == null) {
+// This is seen during cluster initialization when the heartbeat
+// is received before the initial block reports from each storage.
+storage = updateStorage(new DatanodeStorage(report.getStorageID()));
{code}
Even reverting the change and reintroducing the 'else' leaves the state & type
temporarily inaccurate until the first block report.
As discussed with [~arpitagarwal], a better fix would be to simply include the
full {{DatanodeStorage}} object in the {{StorageReport}} (as opposed to only
the Storage ID). This requires adding the {{DatanodeStorage}} object to
{{StorageReportProto}}. It needs to be a new optional field and we cannot
remove the existing {{StorageUuid}} for protocol compatibility.
was:
The fix for HDFS-5484 was accidentally regressed by the following change made
via HDFS-5542
{code}
+ DatanodeStorageInfo updateStorage(DatanodeStorage s) {
synchronized (storageMap) {
DatanodeStorageInfo storage = storageMap.get(s.getStorageID());
if (storage == null) {
@@ -670,8 +658,6 @@
" for DN " + getXferAddr());
storage = new DatanodeStorageInfo(this, s);
storageMap.put(s.getStorageID(), storage);
- } else {
-storage.setState(s.getState());
}
return storage;
}
{code}
By removing the 'else' and no longer updating the state in the BlockReport
processing path, we effectively get the bogus state & type that is set via the
first heartbeat (see the fix for HDFS-5455):
{code}
+ if (storage == null) {
+// This is seen during cluster initialization when the heartbeat
+// is received before the initial block reports from each storage.
+storage = updateStorage(new DatanodeStorage(report.getStorageID()));
{code}
Even reverting the change and reintroducing the 'else' leaves the state & type
temporarily inaccurate until the first block report.
As discussed with [~arpitagarwal], a better fix would be to simply include the
full DatanodeStorage object in the StorageReport (as opposed to only the
Storage ID). This requires adding the {{DatanodeStorage}} object to
{{StorageReportProto}}. It needs to be a new optional field and we cannot
remove the existing {{StorageUuid}} for protocol compatibility.
> StorageType and State in DatanodeStorageInfo in NameNode is not accurate
>
>
> Key: HDFS-5667
> URL: https://issues.apache.org/jira/browse/HDFS-5667
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: Heterogeneous Storage (HDFS-2832)
>Reporter: Eric Sirianni
> Fix For: Heterogeneous Storage (HDFS-2832)
>
>
> The fix for HDFS-5484 was accidentally regressed by the following change made
> via HDFS-5542
> {code}
> + DatanodeStorageInfo updateStorage(DatanodeStorage s) {
> synchronized (storageMap) {
>DatanodeStorageInfo storage = storageMap.get(s.getStorageID());
>if (storage == null) {
> @@ -670,8 +658,6 @@
> " for DN " + getXferAddr());
> storage = new DatanodeStorageInfo(this, s);
> storageMap.put(s.getStorageID(), storage);
> - } else {
> -storage.setState(s.getState());
>}
>return storage;
> }
> {code}
> By removing the 'else' and no longer updating the state in the BlockReport
> processing path, we effectively get the bogus state & type that is set via
> the first heartbeat (see the fix for HDFS-5455):
> {code}
> + if (storage == null) {
> +// This is seen during cluster initialization when the heartbeat
> +// is received before the initial block reports from each storage.
> +storage = updateStorage(new DatanodeStorage(report.getStorageID()));
> {code}
> Even reverting the change and reintroducing the 'else' leaves the state &
> type tem
[jira] [Created] (HDFS-5668) TestBPOfferService.testBPInitErrorHandling fails intermittently
Jimmy Xiang created HDFS-5668:
-
Summary: TestBPOfferService.testBPInitErrorHandling fails
intermittently
Key: HDFS-5668
URL: https://issues.apache.org/jira/browse/HDFS-5668
Project: Hadoop HDFS
Issue Type: Task
Components: test
Affects Versions: 3.0.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
The new test introduced in HDFS-4201 is a little flaky. I got failures locally
occasionally. It could be related to how we did the mockup.
{noformat}
Exception in thread "DataNode:
[file:/home/.../hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/testBPInitErrorHandling/data]
heartbeating to 0.0.0.0/0.0.0.0:0"
org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
SimulatedFSDataset$$EnhancerByMockitoWithCGLIB$$5cb7c720 cannot be returned by
getStorageId()
getStorageId() should return String
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.toString(BPOfferService.java:178)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.toString(BPServiceActor.java:133)
at java.lang.String.valueOf(String.java:2854)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:723)
2013-12-13 13:42:03,119 DEBUG datanode.DataNode
(BPServiceActor.java:sendHeartBeat(468)) - Sending heartbeat from service
actor: Block pool fake bpid (storage id null) service to 0.0.0.0/0.0.0.0:1
at java.lang.Thread.run(Thread.java:722)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5634: --- Attachment: HDFS-5634.005.patch > allow BlockReaderLocal to switch between checksumming and not > - > > Key: HDFS-5634 > URL: https://issues.apache.org/jira/browse/HDFS-5634 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch, > HDFS-5634.003.patch, HDFS-5634.004.patch, HDFS-5634.005.patch > > > BlockReaderLocal should be able to switch between checksumming and > non-checksumming, so that when we get notifications that something is mlocked > (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5632) Add Snapshot feature to INodeDirectory
[ https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5632: Attachment: HDFS-5632.003.patch Thanks for the review, Nicholas! Update the patch to address the comments. bq. but we should check precondition in DirectoryWithSnapshotFeature.destroyDstSubtree(..). The destroyDstSubtree may be invoked recursively, and in some cases the given directory may be not with snapshot. Thus I put the Precondition check in INodeReference#destroyAndCollectBlocks. > Add Snapshot feature to INodeDirectory > -- > > Key: HDFS-5632 > URL: https://issues.apache.org/jira/browse/HDFS-5632 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-5632.000.patch, HDFS-5632.001.patch, > HDFS-5632.002.patch, HDFS-5632.003.patch > > > We will add snapshot feature to INodeDirectory and remove > INodeDirectoryWithSnapshot in this jira. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847945#comment-13847945 ] Colin Patrick McCabe commented on HDFS-5634: I addressed the stuff discussed above. I also addressed the case where there are no checksums for the block. This can lead to some weird conditions like bytesPerChecksum being 0, so we want to avoid dividing by this number before we check it. This patch also sets the skipChecksum boolean when it detects one of these blocks, which should speed things up. This is an optimization that the old code didn't do. BlockReaderLocalTest is now an abstract class rather than an interface, so that we can have defaults. I added a bunch of new tests, including one where we set readahead just less than the block size. > allow BlockReaderLocal to switch between checksumming and not > - > > Key: HDFS-5634 > URL: https://issues.apache.org/jira/browse/HDFS-5634 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch, > HDFS-5634.003.patch, HDFS-5634.004.patch, HDFS-5634.005.patch > > > BlockReaderLocal should be able to switch between checksumming and > non-checksumming, so that when we get notifications that something is mlocked > (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5434) Write resiliency for replica count 1
[
https://issues.apache.org/jira/browse/HDFS-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847967#comment-13847967
]
Arpit Agarwal commented on HDFS-5434:
-
Eric, what is the advantage of retaining only one replica? Unless it is
temporary data that can be easily regenerated, having one replica guarantees
data loss in quick order. e.g. on a 5000-disk cluster with an AFR of 4% you'd
expect at least one disk failure every other day. This setting sounds somewhat
dangerous setting as it could give users a false sense of reliability.
> Write resiliency for replica count 1
>
>
> Key: HDFS-5434
> URL: https://issues.apache.org/jira/browse/HDFS-5434
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.2.0
>Reporter: Buddy
>Priority: Minor
>
> If a file has a replica count of one, the HDFS client is exposed to write
> failures if the data node fails during a write. With a pipeline of size of
> one, no recovery is possible if the sole data node dies.
> A simple fix is to force a minimum pipeline size of 2, while leaving the
> replication count as 1. The implementation for this is fairly non-invasive.
> Although the replica count is one, the block will be written to two data
> nodes instead of one. If one of the data nodes fails during the write, normal
> pipeline recovery will ensure that the write succeeds to the surviving data
> node.
> The existing code in the name node will prune the extra replica when it
> receives the block received reports for the finalized block from both data
> nodes. This results in the intended replica count of one for the block.
> This behavior should be controlled by a configuration option such as
> {{dfs.namenode.minPipelineSize}}.
> This behavior can be implemented in {{FSNameSystem.getAdditionalBlock()}} by
> ensuring that the pipeline size passed to
> {{BlockPlacementPolicy.chooseTarget()}} in the replication parameter is:
> {code}
> max(replication, ${dfs.namenode.minPipelineSize})
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847985#comment-13847985
]
Hadoop QA commented on HDFS-5454:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618671/HDFS-5454.02.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5713//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5713//console
This message is automatically generated.
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Priority: Minor
> Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5661) Browsing FileSystem via web ui, should use datanode's hostname instead of ip address
[ https://issues.apache.org/jira/browse/HDFS-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848011#comment-13848011 ] Benoy Antony commented on HDFS-5661: Only Namenode will authenticate a client using HDFS Delegation Token ( which is issued by the Namenode.) The client still needs to authenticate to the datanode and cannot use the Delegation Token for this purpose. The configuration related to the cookie domain is hadoop.http.authentication.cookie.domain and more details on the authentication of http consoles is here : https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/HttpAuthentication.html When browsing the filesystem via original webui, the DelegationToken is used by the DataNode to contact the Namenode . > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address > > > Key: HDFS-5661 > URL: https://issues.apache.org/jira/browse/HDFS-5661 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5661.patch > > > If authentication is enabled on the web ui, then a cookie is used to keep > track of the authentication information. There is normally a domain > associated with the cookie. Since ip address doesn't have any domain , the > cookie will not be sent by the browser while making http calls with ip > address as the destination server. > This will break browsing files system via web ui , if authentication is > enabled. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5431) support cachepool-based limit management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5431: --- Description: We should support cachepool-based limit management in path-based caching. (was: We should support cachepool-based quota management in path-based caching.) > support cachepool-based limit management in path-based caching > -- > > Key: HDFS-5431 > URL: https://issues.apache.org/jira/browse/HDFS-5431 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch > > > We should support cachepool-based limit management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5431) support cachepool-based limit management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5431: --- Assignee: Andrew Wang (was: Colin Patrick McCabe) > support cachepool-based limit management in path-based caching > -- > > Key: HDFS-5431 > URL: https://issues.apache.org/jira/browse/HDFS-5431 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch > > > We should support cachepool-based quota management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848041#comment-13848041 ] Colin Patrick McCabe commented on HDFS-5636: OK, I misinterpreted the proposal here. I thought you wanted a single TTL which would be applied to all directives in the pool. Instead, you want a max (relative) TTL that can be set on anything added to the pool, at the time of addition. Are you proposing to silently add these TTLs when they're not present in the request? Or hard error when we try to add something with a TTL that's "too high"? The latter seems simpler. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5434) Write resiliency for replica count 1
[
https://issues.apache.org/jira/browse/HDFS-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848046#comment-13848046
]
Colin Patrick McCabe commented on HDFS-5434:
I sort of assumed when I read this JIRA that Eric was considering situations
where the admin "knows" that the storage is reliable despite having only one
replica. For example, if there is RAID and we trust it, or if the backend
storage system which the DataNode is using is doing replication underneath
HDFS. This is definitely a little bit outside the normal Hadoop use-case, but
it's something we can support without too much difficulty with a client-side
config.
> Write resiliency for replica count 1
>
>
> Key: HDFS-5434
> URL: https://issues.apache.org/jira/browse/HDFS-5434
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.2.0
>Reporter: Buddy
>Priority: Minor
>
> If a file has a replica count of one, the HDFS client is exposed to write
> failures if the data node fails during a write. With a pipeline of size of
> one, no recovery is possible if the sole data node dies.
> A simple fix is to force a minimum pipeline size of 2, while leaving the
> replication count as 1. The implementation for this is fairly non-invasive.
> Although the replica count is one, the block will be written to two data
> nodes instead of one. If one of the data nodes fails during the write, normal
> pipeline recovery will ensure that the write succeeds to the surviving data
> node.
> The existing code in the name node will prune the extra replica when it
> receives the block received reports for the finalized block from both data
> nodes. This results in the intended replica count of one for the block.
> This behavior should be controlled by a configuration option such as
> {{dfs.namenode.minPipelineSize}}.
> This behavior can be implemented in {{FSNameSystem.getAdditionalBlock()}} by
> ensuring that the pipeline size passed to
> {{BlockPlacementPolicy.chooseTarget()}} in the replication parameter is:
> {code}
> max(replication, ${dfs.namenode.minPipelineSize})
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5661) Browsing FileSystem via web ui, should use datanode's hostname instead of ip address
[ https://issues.apache.org/jira/browse/HDFS-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848048#comment-13848048 ] Haohui Mai commented on HDFS-5661: -- The only way to access the data on a secure DN is to present a valid delegation token. The HTTP auth tokens do not contain the DT, presenting the HTTP auth tokens to the DN does not grant you the access, thus it makes no sense to pass them around. Regardless of what UI you're using, the NN fetches the DT on the behalf of the client, and the client presents this DT to authenticate with DN. This should the only way you can access the data. If you happen to get the data in your approach, this is a security hole and please file a jira to track it. Again, I'll encourage you to check out the new web UI. It accesses the data through WebHDFS which is much more robust. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address > > > Key: HDFS-5661 > URL: https://issues.apache.org/jira/browse/HDFS-5661 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5661.patch > > > If authentication is enabled on the web ui, then a cookie is used to keep > track of the authentication information. There is normally a domain > associated with the cookie. Since ip address doesn't have any domain , the > cookie will not be sent by the browser while making http calls with ip > address as the destination server. > This will break browsing files system via web ui , if authentication is > enabled. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5434) Write resiliency for replica count 1
[
https://issues.apache.org/jira/browse/HDFS-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848057#comment-13848057
]
Arpit Agarwal commented on HDFS-5434:
-
Thanks for the explanation, I didn't think of that.
If the target storage is reliable then what do we gain by adding an extra
replica in the pipeline? Just want to understand the use case.
> Write resiliency for replica count 1
>
>
> Key: HDFS-5434
> URL: https://issues.apache.org/jira/browse/HDFS-5434
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.2.0
>Reporter: Buddy
>Priority: Minor
>
> If a file has a replica count of one, the HDFS client is exposed to write
> failures if the data node fails during a write. With a pipeline of size of
> one, no recovery is possible if the sole data node dies.
> A simple fix is to force a minimum pipeline size of 2, while leaving the
> replication count as 1. The implementation for this is fairly non-invasive.
> Although the replica count is one, the block will be written to two data
> nodes instead of one. If one of the data nodes fails during the write, normal
> pipeline recovery will ensure that the write succeeds to the surviving data
> node.
> The existing code in the name node will prune the extra replica when it
> receives the block received reports for the finalized block from both data
> nodes. This results in the intended replica count of one for the block.
> This behavior should be controlled by a configuration option such as
> {{dfs.namenode.minPipelineSize}}.
> This behavior can be implemented in {{FSNameSystem.getAdditionalBlock()}} by
> ensuring that the pipeline size passed to
> {{BlockPlacementPolicy.chooseTarget()}} in the replication parameter is:
> {code}
> max(replication, ${dfs.namenode.minPipelineSize})
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5651) remove dfs.namenode.caching.enabled
[ https://issues.apache.org/jira/browse/HDFS-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5651: --- Status: Patch Available (was: Open) > remove dfs.namenode.caching.enabled > --- > > Key: HDFS-5651 > URL: https://issues.apache.org/jira/browse/HDFS-5651 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5651.001.patch > > > We can remove dfs.namenode.caching.enabled and simply always enable caching, > similar to how we do with snapshots and other features. The main overhead is > the size of the cachedBlocks GSet. However, we can simply make the size of > this GSet configurable, and people who don't want caching can set it to a > very small value. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5651) remove dfs.namenode.caching.enabled
[ https://issues.apache.org/jira/browse/HDFS-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848063#comment-13848063 ] Colin Patrick McCabe commented on HDFS-5651: It's worth noting that DNs don't send cache reports unless their configured mlockable capacity is greater than 0. So there's no overhead from that either. > remove dfs.namenode.caching.enabled > --- > > Key: HDFS-5651 > URL: https://issues.apache.org/jira/browse/HDFS-5651 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5651.001.patch > > > We can remove dfs.namenode.caching.enabled and simply always enable caching, > similar to how we do with snapshots and other features. The main overhead is > the size of the cachedBlocks GSet. However, we can simply make the size of > this GSet configurable, and people who don't want caching can set it to a > very small value. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5651) remove dfs.namenode.caching.enabled
[ https://issues.apache.org/jira/browse/HDFS-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5651: --- Attachment: HDFS-5651.001.patch > remove dfs.namenode.caching.enabled > --- > > Key: HDFS-5651 > URL: https://issues.apache.org/jira/browse/HDFS-5651 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5651.001.patch > > > We can remove dfs.namenode.caching.enabled and simply always enable caching, > similar to how we do with snapshots and other features. The main overhead is > the size of the cachedBlocks GSet. However, we can simply make the size of > this GSet configurable, and people who don't want caching can set it to a > very small value. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5651) remove dfs.namenode.caching.enabled
[ https://issues.apache.org/jira/browse/HDFS-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5651: --- Attachment: HDFS-5651.002.patch also update defaults and docs. > remove dfs.namenode.caching.enabled > --- > > Key: HDFS-5651 > URL: https://issues.apache.org/jira/browse/HDFS-5651 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5651.001.patch, HDFS-5651.002.patch > > > We can remove dfs.namenode.caching.enabled and simply always enable caching, > similar to how we do with snapshots and other features. The main overhead is > the size of the cachedBlocks GSet. However, we can simply make the size of > this GSet configurable, and people who don't want caching can set it to a > very small value. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5434) Write resiliency for replica count 1
[
https://issues.apache.org/jira/browse/HDFS-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848069#comment-13848069
]
Colin Patrick McCabe commented on HDFS-5434:
bq. If the target storage is reliable then what do we gain by adding an extra
replica in the pipeline? Just want to understand the use case.
Resiliency against transient network errors.
> Write resiliency for replica count 1
>
>
> Key: HDFS-5434
> URL: https://issues.apache.org/jira/browse/HDFS-5434
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.2.0
>Reporter: Buddy
>Priority: Minor
>
> If a file has a replica count of one, the HDFS client is exposed to write
> failures if the data node fails during a write. With a pipeline of size of
> one, no recovery is possible if the sole data node dies.
> A simple fix is to force a minimum pipeline size of 2, while leaving the
> replication count as 1. The implementation for this is fairly non-invasive.
> Although the replica count is one, the block will be written to two data
> nodes instead of one. If one of the data nodes fails during the write, normal
> pipeline recovery will ensure that the write succeeds to the surviving data
> node.
> The existing code in the name node will prune the extra replica when it
> receives the block received reports for the finalized block from both data
> nodes. This results in the intended replica count of one for the block.
> This behavior should be controlled by a configuration option such as
> {{dfs.namenode.minPipelineSize}}.
> This behavior can be implemented in {{FSNameSystem.getAdditionalBlock()}} by
> ensuring that the pipeline size passed to
> {{BlockPlacementPolicy.chooseTarget()}} in the replication parameter is:
> {code}
> max(replication, ${dfs.namenode.minPipelineSize})
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5661) Browsing FileSystem via web ui, should use datanode's hostname instead of ip address
[ https://issues.apache.org/jira/browse/HDFS-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848076#comment-13848076 ] Benoy Antony commented on HDFS-5661: --the client presents this DT to authenticate with DN. This s not correct. How can DN validate and read DT ? The DT is issued by NN. client/agent can authenticate using DT only with NN. Both NN/DN use https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/HttpAuthentication.html to authenticate a user's access to their http interfaces. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address > > > Key: HDFS-5661 > URL: https://issues.apache.org/jira/browse/HDFS-5661 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5661.patch > > > If authentication is enabled on the web ui, then a cookie is used to keep > track of the authentication information. There is normally a domain > associated with the cookie. Since ip address doesn't have any domain , the > cookie will not be sent by the browser while making http calls with ip > address as the destination server. > This will break browsing files system via web ui , if authentication is > enabled. > Browsing FileSystem via web ui, should use datanode's hostname instead of ip > address. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5661) Browsing FileSystem via web ui, should use datanode's hostname instead of ip address
[
https://issues.apache.org/jira/browse/HDFS-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848085#comment-13848085
]
Jing Zhao commented on HDFS-5661:
-
In AuthFilter#doFilter, we have the following code:
{code}
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain filterChain) throws IOException, ServletException {
final HttpServletRequest httpRequest =
toLowerCase((HttpServletRequest)request);
final String tokenString = httpRequest.getParameter(DelegationParam.NAME);
if (tokenString != null) {
//Token is present in the url, therefore token will be used for
//authentication, bypass kerberos authentication.
filterChain.doFilter(httpRequest, response);
return;
}
super.doFilter(httpRequest, response, filterChain);
}
{code}
In DatanodeJspHelper#generateDirectoryStructure, we have
{code}
String tokenString = req.getParameter(JspHelper.DELEGATION_PARAMETER_NAME);
UserGroupInformation ugi = JspHelper.getUGI(req, conf);
.
DFSClient dfs = getDFSClient(ugi, nnAddr, conf);
{code}
So I think here the whole process is:
1. NN generates DT and put the DT into the redirect URL
2. DN receives the redirect request, finds that there is DT in the request,
thus the corresponding SPNEGO filter will bypass the auth check
3. DN uses the DT and files a getFileInfo RPC call to NN
4. DN shows the result to web ui
> Browsing FileSystem via web ui, should use datanode's hostname instead of ip
> address
>
>
> Key: HDFS-5661
> URL: https://issues.apache.org/jira/browse/HDFS-5661
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-5661.patch
>
>
> If authentication is enabled on the web ui, then a cookie is used to keep
> track of the authentication information. There is normally a domain
> associated with the cookie. Since ip address doesn't have any domain , the
> cookie will not be sent by the browser while making http calls with ip
> address as the destination server.
> This will break browsing files system via web ui , if authentication is
> enabled.
> Browsing FileSystem via web ui, should use datanode's hostname instead of ip
> address.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848090#comment-13848090 ] Andrew Wang commented on HDFS-5636: --- I was thinking a hard error. It's better for users too since the default expiry is "never". > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[
https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848100#comment-13848100
]
Hadoop QA commented on HDFS-5634:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618694/HDFS-5634.005.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestCrcCorruption
org.apache.hadoop.hdfs.TestClientReportBadBlock
org.apache.hadoop.hdfs.TestDFSClientRetries
org.apache.hadoop.hdfs.server.namenode.TestNameNodeHttpServer
org.apache.hadoop.hdfs.server.namenode.TestCorruptFilesJsp
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
org.apache.hadoop.hdfs.server.namenode.TestFsck
org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
org.apache.hadoop.hdfs.TestDFSShell
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5714//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5714//console
This message is automatically generated.
> allow BlockReaderLocal to switch between checksumming and not
> -
>
> Key: HDFS-5634
> URL: https://issues.apache.org/jira/browse/HDFS-5634
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch,
> HDFS-5634.003.patch, HDFS-5634.004.patch, HDFS-5634.005.patch
>
>
> BlockReaderLocal should be able to switch between checksumming and
> non-checksumming, so that when we get notifications that something is mlocked
> (see HDFS-5182), we can avoid checksumming when reading from that block.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5632) Add Snapshot feature to INodeDirectory
[
https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848101#comment-13848101
]
Hadoop QA commented on HDFS-5632:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618695/HDFS-5632.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestSafeMode
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5715//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5715//console
This message is automatically generated.
> Add Snapshot feature to INodeDirectory
> --
>
> Key: HDFS-5632
> URL: https://issues.apache.org/jira/browse/HDFS-5632
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-5632.000.patch, HDFS-5632.001.patch,
> HDFS-5632.002.patch, HDFS-5632.003.patch
>
>
> We will add snapshot feature to INodeDirectory and remove
> INodeDirectoryWithSnapshot in this jira.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5618) NameNode: persist ACLs in fsimage.
[
https://issues.apache.org/jira/browse/HDFS-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-5618:
-
Attachment: HDFS-5618.000.patch
> NameNode: persist ACLs in fsimage.
> --
>
> Key: HDFS-5618
> URL: https://issues.apache.org/jira/browse/HDFS-5618
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS ACLs (HDFS-4685)
>Reporter: Chris Nauroth
>Assignee: Haohui Mai
> Attachments: HDFS-5618.000.patch
>
>
> Store ACLs in fsimage so that ACLs are retained across NameNode restarts.
> This requires encoding and saving the {{AclManager}} state as a new section
> of the fsimage, located after all existing sections (snapshot manager state,
> inodes, secret manager state, and cache manager state).
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Assigned] (HDFS-5619) NameNode: record ACL modifications to edit log.
[
https://issues.apache.org/jira/browse/HDFS-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai reassigned HDFS-5619:
Assignee: Haohui Mai
> NameNode: record ACL modifications to edit log.
> ---
>
> Key: HDFS-5619
> URL: https://issues.apache.org/jira/browse/HDFS-5619
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS ACLs (HDFS-4685)
>Reporter: Chris Nauroth
>Assignee: Haohui Mai
>
> Implement a new edit log opcode, {{OP_SET_ACL}}, which fully replaces the ACL
> of a specific inode. For ACL operations that perform partial modification of
> the ACL, the NameNode must merge the modifications with the existing ACL to
> produce the final resulting ACL and encode it into an {{OP_SET_ACL}}.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848117#comment-13848117 ] Colin Patrick McCabe commented on HDFS-5634: found a bug in the fastpath handling. will post an update soon > allow BlockReaderLocal to switch between checksumming and not > - > > Key: HDFS-5634 > URL: https://issues.apache.org/jira/browse/HDFS-5634 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch, > HDFS-5634.003.patch, HDFS-5634.004.patch, HDFS-5634.005.patch > > > BlockReaderLocal should be able to switch between checksumming and > non-checksumming, so that when we get notifications that something is mlocked > (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-5431:
--
Attachment: hdfs-5431-4.patch
Hi Colin, I made the suggested changes with the following notes:
* With the new EnumSet, I also added a version of add/modify directive without
a {{flags}} argument, since it's really annoying to put
{{EnumSet.noneOf(CacheFlag.class}} everywhere.
* I don't like putting {{bytesOverlimit}} in {{CachePoolEntry}} because it
feels inconsistent with {{CachePoolStats}}. Left it as is, unless you feel
strongly on this one.
* Tweaked the kick behavior somewhat. We now try to do read-after-write
consistency with the dirty bit. add/modify with force is essentially a read and
a write.
This also does complete the test refactor, so we can close that other JIRA out
after this one goes in.
> support cachepool-based limit management in path-based caching
> --
>
> Key: HDFS-5431
> URL: https://issues.apache.org/jira/browse/HDFS-5431
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 3.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch,
> hdfs-5431-4.patch
>
>
> We should support cachepool-based limit management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5406) Send incremental block reports for all storages in a single call
[
https://issues.apache.org/jira/browse/HDFS-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5406:
Attachment: h5406.04.patch
> Send incremental block reports for all storages in a single call
>
>
> Key: HDFS-5406
> URL: https://issues.apache.org/jira/browse/HDFS-5406
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: h5406.01.patch, h5406.02.patch, h5406.04.patch
>
>
> Per code review feedback from [~szetszwo] on HDFS-5390, we can combine all
> incremental block reports in a single {{blockReceivedAndDeleted}} call.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5651) remove dfs.namenode.caching.enabled
[
https://issues.apache.org/jira/browse/HDFS-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848155#comment-13848155
]
Hadoop QA commented on HDFS-5651:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618723/HDFS-5651.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5716//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5716//console
This message is automatically generated.
> remove dfs.namenode.caching.enabled
> ---
>
> Key: HDFS-5651
> URL: https://issues.apache.org/jira/browse/HDFS-5651
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5651.001.patch, HDFS-5651.002.patch
>
>
> We can remove dfs.namenode.caching.enabled and simply always enable caching,
> similar to how we do with snapshots and other features. The main overhead is
> the size of the cachedBlocks GSet. However, we can simply make the size of
> this GSet configurable, and people who don't want caching can set it to a
> very small value.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5632) Add Snapshot feature to INodeDirectory
[
https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848178#comment-13848178
]
Hadoop QA commented on HDFS-5632:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618695/HDFS-5632.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
-14 warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5717//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5717//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5717//console
This message is automatically generated.
> Add Snapshot feature to INodeDirectory
> --
>
> Key: HDFS-5632
> URL: https://issues.apache.org/jira/browse/HDFS-5632
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-5632.000.patch, HDFS-5632.001.patch,
> HDFS-5632.002.patch, HDFS-5632.003.patch
>
>
> We will add snapshot feature to INodeDirectory and remove
> INodeDirectoryWithSnapshot in this jira.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848212#comment-13848212
]
Hadoop QA commented on HDFS-5431:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618742/hdfs-5431-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
org.apache.hadoop.cli.TestCacheAdminCLI
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5718//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5718//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5718//console
This message is automatically generated.
> support cachepool-based limit management in path-based caching
> --
>
> Key: HDFS-5431
> URL: https://issues.apache.org/jira/browse/HDFS-5431
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 3.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch,
> hdfs-5431-4.patch
>
>
> We should support cachepool-based limit management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5484) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
[
https://issues.apache.org/jira/browse/HDFS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-5484:
-
Assignee: Eric Sirianni
> StorageType and State in DatanodeStorageInfo in NameNode is not accurate
>
>
> Key: HDFS-5484
> URL: https://issues.apache.org/jira/browse/HDFS-5484
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: Heterogeneous Storage (HDFS-2832)
>Reporter: Eric Sirianni
>Assignee: Eric Sirianni
> Fix For: Heterogeneous Storage (HDFS-2832)
>
> Attachments: HDFS-5484-HDFS-2832--2.patch, HDFS-5484-HDFS-2832.patch
>
>
> The fields in DatanodeStorageInfo are updated from two distinct paths:
> # block reports
> # storage reports (via heartbeats)
> The {{state}} and {{storageType}} fields are updated via the Block Report.
> However, as seen in the code blow, these fields are populated from a "dummy"
> {{DatanodeStorage}} object constructed in the DataNode:
> {code}
> BPServiceActor.blockReport() {
> //...
> // Dummy DatanodeStorage object just for sending the block report.
> DatanodeStorage dnStorage = new DatanodeStorage(storageID);
> //...
> }
> {code}
> The net effect is that the {{state}} and {{storageType}} fields are always
> the default of {{NORMAL}} and {{DISK}} in the NameNode.
> The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
> {code}
> public Map getBlockReports(String bpid);
> {code}
> to:
> {code}
> public Map getBlockReports(String bpid);
> {code}
> thereby allowing {{BPServiceActor}} to send the "real" {{DatanodeStorage}}
> object with the block report.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5566) HA namenode with QJM created from org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider should implement Closeable
[
https://issues.apache.org/jira/browse/HDFS-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848222#comment-13848222
]
Jimmy Xiang commented on HDFS-5566:
---
In NameNodeProxies#createProxy, it creates a proxy with interface
ClientProtocol which is not Closable, and a RetryInvocationHandler, for the HA
case. However, ConfiguredFailoverProxyProvider doesn't have field "h", the
InvocationHandler, which is the problem.
I think this is a valid bug we need to fix.
> HA namenode with QJM created from
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> should implement Closeable
> --
>
> Key: HDFS-5566
> URL: https://issues.apache.org/jira/browse/HDFS-5566
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: hadoop-2.2.0
> hbase-0.96
>Reporter: Henry Hung
>
> When using hbase-0.96 with hadoop-2.2.0, stopping master/regionserver node
> will result in {{Cannot close proxy - is not Closeable or does not provide
> closeable invocation}}.
> [Mail
> Archive|https://drive.google.com/file/d/0B22pkxoqCdvWSGFIaEpfR3lnT2M/edit?usp=sharing]
> My hadoop-2.2.0 configured as HA namenode with QJM, the configuration is like
> this:
> {code:xml}
>
> dfs.nameservices
> hadoopdev
>
>
> dfs.ha.namenodes.hadoopdev
> nn1,nn2
>
>
> dfs.namenode.rpc-address.hadoopdev.nn1
> fphd9.ctpilot1.com:9000
>
>
> dfs.namenode.http-address.hadoopdev.nn1
> fphd9.ctpilot1.com:50070
>
>
> dfs.namenode.rpc-address.hadoopdev.nn2
> fphd10.ctpilot1.com:9000
>
>
> dfs.namenode.http-address.hadoopdev.nn2
> fphd10.ctpilot1.com:50070
>
>
> dfs.namenode.shared.edits.dir
>
> qjournal://fphd8.ctpilot1.com:8485;fphd9.ctpilot1.com:8485;fphd10.ctpilot1.com:8485/hadoopdev
>
>
> dfs.client.failover.proxy.provider.hadoopdev
>
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
>
>
> dfs.ha.fencing.methods
> shell(/bin/true)
>
>
> dfs.journalnode.edits.dir
> /data/hadoop/hadoop-data-2/journal
>
>
> dfs.ha.automatic-failover.enabled
> true
>
>
> ha.zookeeper.quorum
> fphd1.ctpilot1.com:
>
> {code}
> I traced the code and found out that when stopping the hbase master node, it
> will try invoke method "close" on namenode, but the instance that created
> from {{org.apache.hadoop.hdfs.NameNodeProxies.createProxy}} with
> failoverProxyProviderClass
> {{org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider}}
> do not have the Closeable interface.
> If we use the Non-HA case, the created instance will be
> {{org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB}} that
> implement Closeable.
> TL;DR;
> With hbase connecting to hadoop HA namenode, when stopping the hbase master
> or regionserver, it couldn't find the {{close}} method to gracefully close
> namenode session.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Reopened] (HDFS-5566) HA namenode with QJM created from org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider should implement Closeable
[
https://issues.apache.org/jira/browse/HDFS-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang reopened HDFS-5566:
---
Assignee: Jimmy Xiang
> HA namenode with QJM created from
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> should implement Closeable
> --
>
> Key: HDFS-5566
> URL: https://issues.apache.org/jira/browse/HDFS-5566
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: hadoop-2.2.0
> hbase-0.96
>Reporter: Henry Hung
>Assignee: Jimmy Xiang
>
> When using hbase-0.96 with hadoop-2.2.0, stopping master/regionserver node
> will result in {{Cannot close proxy - is not Closeable or does not provide
> closeable invocation}}.
> [Mail
> Archive|https://drive.google.com/file/d/0B22pkxoqCdvWSGFIaEpfR3lnT2M/edit?usp=sharing]
> My hadoop-2.2.0 configured as HA namenode with QJM, the configuration is like
> this:
> {code:xml}
>
> dfs.nameservices
> hadoopdev
>
>
> dfs.ha.namenodes.hadoopdev
> nn1,nn2
>
>
> dfs.namenode.rpc-address.hadoopdev.nn1
> fphd9.ctpilot1.com:9000
>
>
> dfs.namenode.http-address.hadoopdev.nn1
> fphd9.ctpilot1.com:50070
>
>
> dfs.namenode.rpc-address.hadoopdev.nn2
> fphd10.ctpilot1.com:9000
>
>
> dfs.namenode.http-address.hadoopdev.nn2
> fphd10.ctpilot1.com:50070
>
>
> dfs.namenode.shared.edits.dir
>
> qjournal://fphd8.ctpilot1.com:8485;fphd9.ctpilot1.com:8485;fphd10.ctpilot1.com:8485/hadoopdev
>
>
> dfs.client.failover.proxy.provider.hadoopdev
>
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
>
>
> dfs.ha.fencing.methods
> shell(/bin/true)
>
>
> dfs.journalnode.edits.dir
> /data/hadoop/hadoop-data-2/journal
>
>
> dfs.ha.automatic-failover.enabled
> true
>
>
> ha.zookeeper.quorum
> fphd1.ctpilot1.com:
>
> {code}
> I traced the code and found out that when stopping the hbase master node, it
> will try invoke method "close" on namenode, but the instance that created
> from {{org.apache.hadoop.hdfs.NameNodeProxies.createProxy}} with
> failoverProxyProviderClass
> {{org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider}}
> do not have the Closeable interface.
> If we use the Non-HA case, the created instance will be
> {{org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB}} that
> implement Closeable.
> TL;DR;
> With hbase connecting to hadoop HA namenode, when stopping the hbase master
> or regionserver, it couldn't find the {{close}} method to gracefully close
> namenode session.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5406) Send incremental block reports for all storages in a single call
[
https://issues.apache.org/jira/browse/HDFS-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848226#comment-13848226
]
Hadoop QA commented on HDFS-5406:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618744/h5406.04.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5719//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5719//console
This message is automatically generated.
> Send incremental block reports for all storages in a single call
>
>
> Key: HDFS-5406
> URL: https://issues.apache.org/jira/browse/HDFS-5406
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: h5406.01.patch, h5406.02.patch, h5406.04.patch
>
>
> Per code review feedback from [~szetszwo] on HDFS-5390, we can combine all
> incremental block reports in a single {{blockReceivedAndDeleted}} call.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5632) Add Snapshot feature to INodeDirectory
[ https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-5632: - Hadoop Flags: Reviewed +1 patch looks good. > Add Snapshot feature to INodeDirectory > -- > > Key: HDFS-5632 > URL: https://issues.apache.org/jira/browse/HDFS-5632 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-5632.000.patch, HDFS-5632.001.patch, > HDFS-5632.002.patch, HDFS-5632.003.patch > > > We will add snapshot feature to INodeDirectory and remove > INodeDirectoryWithSnapshot in this jira. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
