[jira] [Commented] (HDFS-8113) NullPointerException in BlockInfoContiguous causes block report failure
[
https://issues.apache.org/jira/browse/HDFS-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496183#comment-14496183
]
Chengbing Liu commented on HDFS-8113:
-
Hi [~atm] and [~cmccabe], from the stacktrace we know that the
{{reportedState}} is RBW or RWR, and condition
{{storedBlock.getGenerationStamp() != reported.getGenerationStamp()}} is
satisfied. Since {{storedBlock}} is an entry in {{blocksMap}}, the file/block
should not have been deleted.
I did some tests using MiniDFSCluster. The result is as follows:
- If a file is deleted, then {{BlockInfo}} is removed from {{blocksMap}}.
- If a file is not deleted, then {{BlockInfo.bc}} is the file, which cannot be
null.
I'm wondering if it could happen that a block does not belong to any file, yet
it does exist? Could you kindly explain this? Thanks!
NullPointerException in BlockInfoContiguous causes block report failure
---
Key: HDFS-8113
URL: https://issues.apache.org/jira/browse/HDFS-8113
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.6.0
Reporter: Chengbing Liu
Assignee: Chengbing Liu
Attachments: HDFS-8113.patch
The following copy constructor can throw NullPointerException if {{bc}} is
null.
{code}
protected BlockInfoContiguous(BlockInfoContiguous from) {
this(from, from.bc.getBlockReplication());
this.bc = from.bc;
}
{code}
We have observed that some DataNodes keeps failing doing block reports with
NameNode. The stacktrace is as follows. Though we are not using the latest
version, the problem still exists.
{quote}
2015-03-08 19:28:13,442 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
RemoteException in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.(BlockInfo.java:80)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockToMarkCorrupt.(BlockManager.java:1696)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.checkReplicaCorrupt(BlockManager.java:2185)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReportedBlock(BlockManager.java:2047)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1950)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1823)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1750)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1069)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:152)
at
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:26382)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1623)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8148) NPE thrown at Namenode startup,.
[
https://issues.apache.org/jira/browse/HDFS-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496111#comment-14496111
]
nijel commented on HDFS-8148:
-
Hi archana and surendra
Looks like the patch on https://issues.apache.org/jira/browse/HDFS- fixed
the issue.
Please confirm
NPE thrown at Namenode startup,.
-
Key: HDFS-8148
URL: https://issues.apache.org/jira/browse/HDFS-8148
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Archana T
Assignee: surendra singh lilhore
Priority: Minor
At Namenode startup, NPE thrown when unsupported config parameter configured
in hdfs-site.xml
{code}
2015-04-15 10:43:59,880 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1219)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.close(FSNamesystem.java:1540)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.init(FSNamesystem.java:841)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:669)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8113) NullPointerException in BlockInfoContiguous causes block report failure
[
https://issues.apache.org/jira/browse/HDFS-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496334#comment-14496334
]
Chengbing Liu commented on HDFS-8113:
-
[~vinayrpet] Actually, whenever I start the problematic DataNode, NPE happens
in every block report. That doesn't seem to be a transient problem as you have
mentioned. Is it possible that the file is deleted without removal of the
blocks?
NullPointerException in BlockInfoContiguous causes block report failure
---
Key: HDFS-8113
URL: https://issues.apache.org/jira/browse/HDFS-8113
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.6.0
Reporter: Chengbing Liu
Assignee: Chengbing Liu
Attachments: HDFS-8113.patch
The following copy constructor can throw NullPointerException if {{bc}} is
null.
{code}
protected BlockInfoContiguous(BlockInfoContiguous from) {
this(from, from.bc.getBlockReplication());
this.bc = from.bc;
}
{code}
We have observed that some DataNodes keeps failing doing block reports with
NameNode. The stacktrace is as follows. Though we are not using the latest
version, the problem still exists.
{quote}
2015-03-08 19:28:13,442 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
RemoteException in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.(BlockInfo.java:80)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockToMarkCorrupt.(BlockManager.java:1696)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.checkReplicaCorrupt(BlockManager.java:2185)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReportedBlock(BlockManager.java:2047)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1950)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1823)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1750)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1069)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:152)
at
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:26382)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1623)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8117) More accurate verification in SimulatedFSDataset: replace DEFAULT_DATABYTE with patterned data
[
https://issues.apache.org/jira/browse/HDFS-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-8117:
--
Resolution: Fixed
Fix Version/s: (was: 3.0.0)
2.8.0
Status: Resolved (was: Patch Available)
LGTM, committed the branch-2 patch. Thanks Zhe!
More accurate verification in SimulatedFSDataset: replace DEFAULT_DATABYTE
with patterned data
--
Key: HDFS-8117
URL: https://issues.apache.org/jira/browse/HDFS-8117
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Fix For: 2.8.0
Attachments: HDFS-8117-branch2.patch, HDFS-8117.000.patch,
HDFS-8117.001.patch, HDFS-8117.002.patch, HDFS-8117.003.patch
Currently {{SimulatedFSDataset}} uses a single {{DEFAULT_DATABYTE}} to
simulate _all_ block content. This is not accurate because the return of this
byte just means the read request has hit an arbitrary position in an
arbitrary simulated block.
This JIRA aims to improve it with a more accurate verification. When position
{{p}} of a simulated block {{b}} is accessed, the returned byte is {{b}}'s
block ID plus {{p}}, moduled by the max value of a byte.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8055) NullPointerException when topology script is missing.
[ https://issues.apache.org/jira/browse/HDFS-8055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496387#comment-14496387 ] Hudson commented on HDFS-8055: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2114 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2114/]) HDFS-8055. NullPointerException when topology script is missing. Contributed by Anu Engineer. (cnauroth: rev fef596df038112cbbc86c4dc49314e274fca0190) * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/topology-broken-script.sh * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/topology-broken-script.cmd * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/topology-script.sh * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/topology-script.cmd NullPointerException when topology script is missing. - Key: HDFS-8055 URL: https://issues.apache.org/jira/browse/HDFS-8055 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.8.0 Attachments: hdfs-8055.001.patch, hdfs-8055.002.patch, hdfs-8055.003.patch We've received reports that the NameNode can get a NullPointerException when the topology script is missing. This issue tracks investigating whether or not we can improve the validation logic and give a more informative error message. Here is a sample stack trace : Getting NPE from HDFS: 2015-02-06 23:02:12,250 ERROR [pool-4-thread-1] util.HFileV1Detector: Got exception while reading trailer for file:hdfs://hqhd02nm01.pclc0.merkle.local:8020/hbase/.META./1028785192/info/1490a396aea448b693da563f76a28486^M org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException^M at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.sortLocatedBlocks(DatanodeManager.java:359)^M at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1789)^M at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:542)^M at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:362)^M at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)^M at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)^M at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)^M at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)^M at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)^M at java.security.AccessController.doPrivileged(Native Method)^M at javax.security.auth.Subject.doAs(Subject.java:415)^M at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)^M at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)^M ^M at org.apache.hadoop.ipc.Client.call(Client.java:1468)^M at org.apache.hadoop.ipc.Client.call(Client.java:1399)^M at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)^M at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)^M at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)^M at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)^M at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)^M at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)^M at java.lang.reflect.Method.invoke(Method.java:606)^M at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)^M at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)^M at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)^M at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)^M at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)^M at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)^M at
[jira] [Commented] (HDFS-8127) NameNode Failover during HA upgrade can cause DataNode to finalize upgrade
[
https://issues.apache.org/jira/browse/HDFS-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496386#comment-14496386
]
Hudson commented on HDFS-8127:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2114 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2114/])
HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize
upgrade. Contributed by Jing Zhao. (jing9: rev
fddd55279d0bdd08b3b40aba6fe2ded1d2e0d846)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDFSUpgradeWithHA.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandbyWithQJM.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
NameNode Failover during HA upgrade can cause DataNode to finalize upgrade
--
Key: HDFS-8127
URL: https://issues.apache.org/jira/browse/HDFS-8127
Project: Hadoop HDFS
Issue Type: Bug
Components: ha
Affects Versions: 2.4.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Blocker
Fix For: 2.7.1
Attachments: HDFS-8127.000.patch, HDFS-8127.001.patch
Currently for HA upgrade (enabled by HDFS-5138), we use {{-bootstrapStandby}}
to initialize the standby NameNode. The standby NameNode does not have the
{{previous}} directory thus it does not know that the cluster is in the
upgrade state. If NN failover happens, as response of block reports, the new
ANN will tell DNs to finalize the upgrade thus make it impossible to rollback
again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8113) NullPointerException in BlockInfoContiguous causes block report failure
[
https://issues.apache.org/jira/browse/HDFS-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496260#comment-14496260
]
Vinayakumar B commented on HDFS-8113:
-
There is one possibility.
Actual file delete and blocks removal will happen in separate writelocks in
FSNameSystem.
In the first lock, INode will be deleted and it will make {{blockInfo.bc}} to
{{null}}. As below in {{INodeFile#destroyAndCollectBlocks()}}
{code} for (BlockInfo blk : blks) {
collectedBlocks.addDeleteBlock(blk);
blk.setBlockCollection(null);
}{code}
Before actual removal of blocks from blocksMap in
{{FSNameSystem#removeBlocks}}, if blockreport comes, then it will find
{{blockInfo.bc}} as null.
NullPointerException in BlockInfoContiguous causes block report failure
---
Key: HDFS-8113
URL: https://issues.apache.org/jira/browse/HDFS-8113
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.6.0
Reporter: Chengbing Liu
Assignee: Chengbing Liu
Attachments: HDFS-8113.patch
The following copy constructor can throw NullPointerException if {{bc}} is
null.
{code}
protected BlockInfoContiguous(BlockInfoContiguous from) {
this(from, from.bc.getBlockReplication());
this.bc = from.bc;
}
{code}
We have observed that some DataNodes keeps failing doing block reports with
NameNode. The stacktrace is as follows. Though we are not using the latest
version, the problem still exists.
{quote}
2015-03-08 19:28:13,442 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
RemoteException in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.(BlockInfo.java:80)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockToMarkCorrupt.(BlockManager.java:1696)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.checkReplicaCorrupt(BlockManager.java:2185)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReportedBlock(BlockManager.java:2047)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1950)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1823)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1750)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1069)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:152)
at
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:26382)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1623)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8149) The footer of the Web UI Hadoop, 2014 is old
Akira AJISAKA created HDFS-8149: --- Summary: The footer of the Web UI Hadoop, 2014 is old Key: HDFS-8149 URL: https://issues.apache.org/jira/browse/HDFS-8149 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Akira AJISAKA Need to be updated to 2015. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6666) Abort NameNode and DataNode startup if security is enabled but block access token is not enabled.
[ https://issues.apache.org/jira/browse/HDFS-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496382#comment-14496382 ] Hudson commented on HDFS-: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2114 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2114/]) HDFS-. Abort NameNode and DataNode startup if security is enabled but block access token is not enabled. Contributed by Vijay Bhat. (cnauroth: rev d45aa7647b1fecf81860ec7b563085be2af99a0b) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferTestCase.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSecureNameNode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java Abort NameNode and DataNode startup if security is enabled but block access token is not enabled. - Key: HDFS- URL: https://issues.apache.org/jira/browse/HDFS- Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode, security Affects Versions: 2.7.1 Reporter: Chris Nauroth Assignee: Vijay Bhat Priority: Minor Fix For: 2.8.0 Attachments: HDFS-.001.patch, HDFS-.002.patch, HDFS-.003.patch, HDFS-.004.patch, HDFS-.005.patch Currently, if security is enabled by setting hadoop.security.authentication to kerberos, but HDFS block access tokens are disabled by setting dfs.block.access.token.enable to false (which is the default), then the NameNode logs an error and proceeds, and the DataNode proceeds without even logging an error. This jira proposes that this it's invalid to turn on security but not turn on block access tokens, and that it would be better to fail fast and abort the daemons during startup if this happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8055) NullPointerException when topology script is missing.
[ https://issues.apache.org/jira/browse/HDFS-8055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496308#comment-14496308 ] Hudson commented on HDFS-8055: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #165 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/165/]) HDFS-8055. NullPointerException when topology script is missing. Contributed by Anu Engineer. (cnauroth: rev fef596df038112cbbc86c4dc49314e274fca0190) * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/topology-broken-script.cmd * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/topology-script.sh * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/topology-script.cmd * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/topology-broken-script.sh NullPointerException when topology script is missing. - Key: HDFS-8055 URL: https://issues.apache.org/jira/browse/HDFS-8055 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.8.0 Attachments: hdfs-8055.001.patch, hdfs-8055.002.patch, hdfs-8055.003.patch We've received reports that the NameNode can get a NullPointerException when the topology script is missing. This issue tracks investigating whether or not we can improve the validation logic and give a more informative error message. Here is a sample stack trace : Getting NPE from HDFS: 2015-02-06 23:02:12,250 ERROR [pool-4-thread-1] util.HFileV1Detector: Got exception while reading trailer for file:hdfs://hqhd02nm01.pclc0.merkle.local:8020/hbase/.META./1028785192/info/1490a396aea448b693da563f76a28486^M org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException^M at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.sortLocatedBlocks(DatanodeManager.java:359)^M at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1789)^M at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:542)^M at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:362)^M at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)^M at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)^M at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)^M at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)^M at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)^M at java.security.AccessController.doPrivileged(Native Method)^M at javax.security.auth.Subject.doAs(Subject.java:415)^M at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)^M at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)^M ^M at org.apache.hadoop.ipc.Client.call(Client.java:1468)^M at org.apache.hadoop.ipc.Client.call(Client.java:1399)^M at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)^M at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)^M at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)^M at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)^M at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)^M at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)^M at java.lang.reflect.Method.invoke(Method.java:606)^M at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)^M at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)^M at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)^M at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)^M at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)^M at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)^M at
[jira] [Commented] (HDFS-6666) Abort NameNode and DataNode startup if security is enabled but block access token is not enabled.
[ https://issues.apache.org/jira/browse/HDFS-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496303#comment-14496303 ] Hudson commented on HDFS-: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #165 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/165/]) HDFS-. Abort NameNode and DataNode startup if security is enabled but block access token is not enabled. Contributed by Vijay Bhat. (cnauroth: rev d45aa7647b1fecf81860ec7b563085be2af99a0b) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSecureNameNode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferTestCase.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java Abort NameNode and DataNode startup if security is enabled but block access token is not enabled. - Key: HDFS- URL: https://issues.apache.org/jira/browse/HDFS- Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode, security Affects Versions: 2.7.1 Reporter: Chris Nauroth Assignee: Vijay Bhat Priority: Minor Fix For: 2.8.0 Attachments: HDFS-.001.patch, HDFS-.002.patch, HDFS-.003.patch, HDFS-.004.patch, HDFS-.005.patch Currently, if security is enabled by setting hadoop.security.authentication to kerberos, but HDFS block access tokens are disabled by setting dfs.block.access.token.enable to false (which is the default), then the NameNode logs an error and proceeds, and the DataNode proceeds without even logging an error. This jira proposes that this it's invalid to turn on security but not turn on block access tokens, and that it would be better to fail fast and abort the daemons during startup if this happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8127) NameNode Failover during HA upgrade can cause DataNode to finalize upgrade
[
https://issues.apache.org/jira/browse/HDFS-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496307#comment-14496307
]
Hudson commented on HDFS-8127:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #165 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/165/])
HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize
upgrade. Contributed by Jing Zhao. (jing9: rev
fddd55279d0bdd08b3b40aba6fe2ded1d2e0d846)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandbyWithQJM.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDFSUpgradeWithHA.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
NameNode Failover during HA upgrade can cause DataNode to finalize upgrade
--
Key: HDFS-8127
URL: https://issues.apache.org/jira/browse/HDFS-8127
Project: Hadoop HDFS
Issue Type: Bug
Components: ha
Affects Versions: 2.4.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Blocker
Fix For: 2.7.1
Attachments: HDFS-8127.000.patch, HDFS-8127.001.patch
Currently for HA upgrade (enabled by HDFS-5138), we use {{-bootstrapStandby}}
to initialize the standby NameNode. The standby NameNode does not have the
{{previous}} directory thus it does not know that the cluster is in the
upgrade state. If NN failover happens, as response of block reports, the new
ANN will tell DNs to finalize the upgrade thus make it impossible to rollback
again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8117) More accurate verification in SimulatedFSDataset: replace DEFAULT_DATABYTE with patterned data
[
https://issues.apache.org/jira/browse/HDFS-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496470#comment-14496470
]
Zhe Zhang commented on HDFS-8117:
-
Thanks Andrew for reviewing and taking care of the commit!
More accurate verification in SimulatedFSDataset: replace DEFAULT_DATABYTE
with patterned data
--
Key: HDFS-8117
URL: https://issues.apache.org/jira/browse/HDFS-8117
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Fix For: 2.8.0
Attachments: HDFS-8117-branch2.patch, HDFS-8117.000.patch,
HDFS-8117.001.patch, HDFS-8117.002.patch, HDFS-8117.003.patch
Currently {{SimulatedFSDataset}} uses a single {{DEFAULT_DATABYTE}} to
simulate _all_ block content. This is not accurate because the return of this
byte just means the read request has hit an arbitrary position in an
arbitrary simulated block.
This JIRA aims to improve it with a more accurate verification. When position
{{p}} of a simulated block {{b}} is accessed, the returned byte is {{b}}'s
block ID plus {{p}}, moduled by the max value of a byte.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8149) The footer of the Web UI Hadoop, 2014 is old
[ https://issues.apache.org/jira/browse/HDFS-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-8149: --- Attachment: HDFS-8149.patch The footer of the Web UI Hadoop, 2014 is old -- Key: HDFS-8149 URL: https://issues.apache.org/jira/browse/HDFS-8149 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Akira AJISAKA Assignee: Brahma Reddy Battula Attachments: HDFS-8149.patch Need to be updated to 2015. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-8149) The footer of the Web UI Hadoop, 2014 is old
[ https://issues.apache.org/jira/browse/HDFS-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reassigned HDFS-8149: -- Assignee: Brahma Reddy Battula The footer of the Web UI Hadoop, 2014 is old -- Key: HDFS-8149 URL: https://issues.apache.org/jira/browse/HDFS-8149 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Akira AJISAKA Assignee: Brahma Reddy Battula Need to be updated to 2015. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8113) NullPointerException in BlockInfoContiguous causes block report failure
[
https://issues.apache.org/jira/browse/HDFS-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496446#comment-14496446
]
Harsh J commented on HDFS-8113:
---
Stale block copies leftover in the DN can cause the condition - it
indeed goes away if you clear out the RBW directory in the DN.
Imagine this condition:
1. File is being written. Has replica on node X among others.
2. Replica write to node X in pipeline fails. Write carries on,
leaving stale block copy in RBW of node X.
3. File gets closed and deleted away soon/immediately after (but well
before a block report from X).
4. Block report now sends the RBW info but NN has no knowledge of the
block anymore.
I think modifying Colin's test this way should reproduce the issue:
1. start a mini dfs cluster with 2 datanodes
2. create a file with repl=2, but do not close it (flush it to ensure
on-disk RBW write)
3. take down one DN
4. close and delete the file
5. wait
6. bring back up the other DN, which will still have the RBW block
from the file which was deleted
--
Harsh J
NullPointerException in BlockInfoContiguous causes block report failure
---
Key: HDFS-8113
URL: https://issues.apache.org/jira/browse/HDFS-8113
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.6.0
Reporter: Chengbing Liu
Assignee: Chengbing Liu
Attachments: HDFS-8113.patch
The following copy constructor can throw NullPointerException if {{bc}} is
null.
{code}
protected BlockInfoContiguous(BlockInfoContiguous from) {
this(from, from.bc.getBlockReplication());
this.bc = from.bc;
}
{code}
We have observed that some DataNodes keeps failing doing block reports with
NameNode. The stacktrace is as follows. Though we are not using the latest
version, the problem still exists.
{quote}
2015-03-08 19:28:13,442 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
RemoteException in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.(BlockInfo.java:80)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockToMarkCorrupt.(BlockManager.java:1696)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.checkReplicaCorrupt(BlockManager.java:2185)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReportedBlock(BlockManager.java:2047)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1950)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1823)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1750)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1069)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:152)
at
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:26382)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1623)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8149) The footer of the Web UI Hadoop, 2014 is old
[ https://issues.apache.org/jira/browse/HDFS-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496479#comment-14496479 ] Brahma Reddy Battula commented on HDFS-8149: [~ajisakaa] thanks for reporting this issue..Attached patch..Kindly review.. The footer of the Web UI Hadoop, 2014 is old -- Key: HDFS-8149 URL: https://issues.apache.org/jira/browse/HDFS-8149 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Akira AJISAKA Assignee: Brahma Reddy Battula Attachments: HDFS-8149.patch Need to be updated to 2015. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8113) NullPointerException in BlockInfoContiguous causes block report failure
[
https://issues.apache.org/jira/browse/HDFS-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496505#comment-14496505
]
Aaron T. Myers commented on HDFS-8113:
--
That all makes sense to me as well.
[~chengbing.liu] - would you be up for adding a unit test to this patch as
Harsh and Colin have described?
NullPointerException in BlockInfoContiguous causes block report failure
---
Key: HDFS-8113
URL: https://issues.apache.org/jira/browse/HDFS-8113
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.6.0
Reporter: Chengbing Liu
Assignee: Chengbing Liu
Attachments: HDFS-8113.patch
The following copy constructor can throw NullPointerException if {{bc}} is
null.
{code}
protected BlockInfoContiguous(BlockInfoContiguous from) {
this(from, from.bc.getBlockReplication());
this.bc = from.bc;
}
{code}
We have observed that some DataNodes keeps failing doing block reports with
NameNode. The stacktrace is as follows. Though we are not using the latest
version, the problem still exists.
{quote}
2015-03-08 19:28:13,442 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
RemoteException in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.(BlockInfo.java:80)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockToMarkCorrupt.(BlockManager.java:1696)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.checkReplicaCorrupt(BlockManager.java:2185)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReportedBlock(BlockManager.java:2047)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1950)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1823)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1750)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1069)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:152)
at
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:26382)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1623)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8149) The footer of the Web UI Hadoop, 2014 is old
[ https://issues.apache.org/jira/browse/HDFS-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-8149: --- Status: Patch Available (was: Open) The footer of the Web UI Hadoop, 2014 is old -- Key: HDFS-8149 URL: https://issues.apache.org/jira/browse/HDFS-8149 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Akira AJISAKA Assignee: Brahma Reddy Battula Attachments: HDFS-8149.patch Need to be updated to 2015. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8150) Make getFileChecksum fail for blocks under construction
Kihwal Lee created HDFS-8150: Summary: Make getFileChecksum fail for blocks under construction Key: HDFS-8150 URL: https://issues.apache.org/jira/browse/HDFS-8150 Project: Hadoop HDFS Issue Type: Bug Reporter: Kihwal Lee Priority: Critical We have seen the cases of validating data copy using checksum then the content of target changing. It turns out the target wasn't closed successfully, so it was still under-construction. One hour later, a lease recovery kicked in and truncated the block. Although this can be prevented in many ways, if there is no valid use case for getting file checksum from under-construction blocks, can it be disabled? E.g. Datanode can throw an exception if the replica is not in the finalized state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8113) NullPointerException in BlockInfoContiguous causes block report failure
[
https://issues.apache.org/jira/browse/HDFS-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496485#comment-14496485
]
Colin Patrick McCabe commented on HDFS-8113:
Thanks for the explanation, guys. I wasn't aware of the invariant that
{{BlockInfoContiguous}} structures with {{bc == null}} were not in the
{{BlocksMap}}. I think we should remove this invariant, and instead simply
have the {{BlocksMap}} contain all the blocks. The memory savings from keeping
them out is trivial, since the number of blocks without associated inodes
should be very small. I think we can just check whether the INode field is
null when appropriate. That seems to be the direction that the patch here is
taking, and I think it makes sense.
NullPointerException in BlockInfoContiguous causes block report failure
---
Key: HDFS-8113
URL: https://issues.apache.org/jira/browse/HDFS-8113
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.6.0
Reporter: Chengbing Liu
Assignee: Chengbing Liu
Attachments: HDFS-8113.patch
The following copy constructor can throw NullPointerException if {{bc}} is
null.
{code}
protected BlockInfoContiguous(BlockInfoContiguous from) {
this(from, from.bc.getBlockReplication());
this.bc = from.bc;
}
{code}
We have observed that some DataNodes keeps failing doing block reports with
NameNode. The stacktrace is as follows. Though we are not using the latest
version, the problem still exists.
{quote}
2015-03-08 19:28:13,442 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
RemoteException in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.(BlockInfo.java:80)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockToMarkCorrupt.(BlockManager.java:1696)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.checkReplicaCorrupt(BlockManager.java:2185)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReportedBlock(BlockManager.java:2047)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1950)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1823)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1750)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1069)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:152)
at
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:26382)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1623)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8144) Split TestLazyPersistFiles into multiple tests
[ https://issues.apache.org/jira/browse/HDFS-8144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496566#comment-14496566 ] Hudson commented on HDFS-8144: -- SUCCESS: Integrated in Hadoop-trunk-Commit #7589 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7589/]) HDFS-8144. Split TestLazyPersistFiles into multiple tests. (Arpit Agarwal) (arp: rev 9e8309a1b2989d07d43e20940d9ac12b7b43482f) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistReplicaPlacement.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistReplicaRecovery.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyWriter.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java Split TestLazyPersistFiles into multiple tests -- Key: HDFS-8144 URL: https://issues.apache.org/jira/browse/HDFS-8144 Project: Hadoop HDFS Issue Type: Improvement Components: test Affects Versions: 2.7.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 2.8.0 Attachments: HDFS-8144.01.patch, HDFS-8144.02.patch TestLazyPersistFiles has grown too large and includes both NN and DN tests. We can split up related tests into smaller files to keep the test case manageable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp
[
https://issues.apache.org/jira/browse/HDFS-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8151:
Affects Version/s: 2.7.0
Always use snapshot path as source when invalid snapshot names are used for
diff based distcp
-
Key: HDFS-8151
URL: https://issues.apache.org/jira/browse/HDFS-8151
Project: Hadoop HDFS
Issue Type: Bug
Components: distcp
Affects Versions: 2.7.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Attachments: HDFS-8151.000.patch
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff
report computation on the target cluster fails
# there is modification happening in the target cluster thus
{{checkNoChange}} returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp
[
https://issues.apache.org/jira/browse/HDFS-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8151:
Status: Patch Available (was: Open)
Always use snapshot path as source when invalid snapshot names are used for
diff based distcp
-
Key: HDFS-8151
URL: https://issues.apache.org/jira/browse/HDFS-8151
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Attachments: HDFS-8151.000.patch
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff
report computation on the target cluster fails
# there is modification happening in the target cluster thus
{{checkNoChange}} returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp
[
https://issues.apache.org/jira/browse/HDFS-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8151:
Component/s: distcp
Always use snapshot path as source when invalid snapshot names are used for
diff based distcp
-
Key: HDFS-8151
URL: https://issues.apache.org/jira/browse/HDFS-8151
Project: Hadoop HDFS
Issue Type: Bug
Components: distcp
Affects Versions: 2.7.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Attachments: HDFS-8151.000.patch
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff
report computation on the target cluster fails
# there is modification happening in the target cluster thus
{{checkNoChange}} returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp
[
https://issues.apache.org/jira/browse/HDFS-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-8151:
--
Hadoop Flags: Reviewed
+1 patch looks good.
Always use snapshot path as source when invalid snapshot names are used for
diff based distcp
-
Key: HDFS-8151
URL: https://issues.apache.org/jira/browse/HDFS-8151
Project: Hadoop HDFS
Issue Type: Bug
Components: distcp
Affects Versions: 2.7.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Attachments: HDFS-8151.000.patch
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff
report computation on the target cluster fails
# there is modification happening in the target cluster thus
{{checkNoChange}} returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7891) A block placement policy with best rack failure tolerance
[ https://issues.apache.org/jira/browse/HDFS-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-7891: Status: Patch Available (was: Open) To trigger Jenkins.. A block placement policy with best rack failure tolerance - Key: HDFS-7891 URL: https://issues.apache.org/jira/browse/HDFS-7891 Project: Hadoop HDFS Issue Type: New Feature Components: namenode Reporter: Walter Su Assignee: Walter Su Priority: Minor Attachments: HDFS-7891.005.patch a block placement policy tries its best to place replicas to most racks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp
Jing Zhao created HDFS-8151:
---
Summary: Always use snapshot path as source when invalid snapshot
names are used for diff based distcp
Key: HDFS-8151
URL: https://issues.apache.org/jira/browse/HDFS-8151
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff report
computation on the target cluster fails
# there is modification happening in the target cluster thus {{checkNoChange}}
returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp
[
https://issues.apache.org/jira/browse/HDFS-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8151:
Attachment: HDFS-8151.000.patch
Patch to fix.
Always use snapshot path as source when invalid snapshot names are used for
diff based distcp
-
Key: HDFS-8151
URL: https://issues.apache.org/jira/browse/HDFS-8151
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Attachments: HDFS-8151.000.patch
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff
report computation on the target cluster fails
# there is modification happening in the target cluster thus
{{checkNoChange}} returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8144) Split TestLazyPersistFiles into multiple tests
[ https://issues.apache.org/jira/browse/HDFS-8144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8144: Resolution: Fixed Fix Version/s: 2.8.0 Target Version/s: (was: 2.8.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the reviews [~xyao]. I committed it to trunk and branch-2. Split TestLazyPersistFiles into multiple tests -- Key: HDFS-8144 URL: https://issues.apache.org/jira/browse/HDFS-8144 Project: Hadoop HDFS Issue Type: Improvement Components: test Affects Versions: 2.7.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 2.8.0 Attachments: HDFS-8144.01.patch, HDFS-8144.02.patch TestLazyPersistFiles has grown too large and includes both NN and DN tests. We can split up related tests into smaller files to keep the test case manageable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496627#comment-14496627
]
Zhe Zhang commented on HDFS-7949:
-
Thanks Rakesh for the update!
bq. Also, updated the hard-coded assert value.
What I meant in the previous review was to remove this hard-coded number
{{13107200}} and use something like the below instead:
{code}
long expected = BlockInfoStriped.spaceConsumed(xxx);
// use expected in assertion
{code}
I just noticed that {{spaceConsumed}} needs to be updated too. Please see the
latest patch under HDFS-8120. We should call
{{StripedBlockUtil#getInternalBlockLength}} to calculate size of parity blocks.
I think that patch will be committed soon. Maybe we should wait for that and
then move the static {{spaceConsumed}} method to there?
Thanks for the good work on this JIRA and sorry about the back and forth.
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7891) A block placement policy with best rack failure tolerance
[ https://issues.apache.org/jira/browse/HDFS-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-7891: Attachment: HDFS-7891.005.dup.patch Submitting a duplicate patch of 005 to trigger Jenkins A block placement policy with best rack failure tolerance - Key: HDFS-7891 URL: https://issues.apache.org/jira/browse/HDFS-7891 Project: Hadoop HDFS Issue Type: New Feature Components: namenode Reporter: Walter Su Assignee: Walter Su Priority: Minor Attachments: HDFS-7891.005.dup.patch, HDFS-7891.005.patch a block placement policy tries its best to place replicas to most racks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8120) Erasure coding: created util class to analyze striped block groups
[
https://issues.apache.org/jira/browse/HDFS-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8120:
Attachment: HDFS-8120.008.patch
Thanks Bo for the update! I compared 007 to 005 and the changes look good to
me.
I made some minor updates in 008 patch (added some comments, changed a test
parameter in {{TestDFSStripedOutputStream#TestFileMoreThanABlockGroup2}} to
make it different from {{TestFileMoreThanABlockGroup1}}.
[~jingzhao] Please let me know if the latest patch looks OK. W.r.t. HDFS-8145,
I think your solution is a good idea.
Erasure coding: created util class to analyze striped block groups
--
Key: HDFS-8120
URL: https://issues.apache.org/jira/browse/HDFS-8120
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8120.000.patch, HDFS-8120.001.patch,
HDFS-8120.002.patch, HDFS-8120.003.patch, HDFS-8120.004.patch,
HDFS-8120.005.patch, HDFS-8120.006.patch, HDFS-8120.007.patch,
HDFS-8120.008.patch
The patch adds logic of calculating size of individual blocks in a striped
block group.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7349) Support DFS command for the EC encoding
[
https://issues.apache.org/jira/browse/HDFS-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496601#comment-14496601
]
Zhe Zhang commented on HDFS-7349:
-
Thanks Vinay for the work! The latest set of commands look very good.
Support DFS command for the EC encoding
---
Key: HDFS-7349
URL: https://issues.apache.org/jira/browse/HDFS-7349
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Fix For: HDFS-7285
Attachments: HDFS-7349-001.patch, HDFS-7349-002.patch,
HDFS-7349-003.patch, HDFS-7349-004.patch, HDFS-7349-005.patch,
HDFS-7349-006.patch
Support implementation of the following commands
{noformat}Usage: hdfs erasurecode [generic options]
[-createZone [-s schemaName] path]
[-getZoneInfo path]
[-help [cmd ...]]
[-listSchemas]
[-usage [cmd ...]]{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-7952) On starting Standby with rollback option, lastPromisedEpoch gets updated and Active Namenode is shutting down.
[
https://issues.apache.org/jira/browse/HDFS-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao resolved HDFS-7952.
-
Resolution: Duplicate
Resolve as duplicate after HDFS-7934 got committed.
On starting Standby with rollback option, lastPromisedEpoch gets updated
and Active Namenode is shutting down.
Key: HDFS-7952
URL: https://issues.apache.org/jira/browse/HDFS-7952
Project: Hadoop HDFS
Issue Type: Bug
Reporter: J.Andreina
Assignee: J.Andreina
Priority: Critical
Step 1: Start NN1 as active , NN2 as standby .
Step 2: Perform hdfs dfsadmin -rollingUpgrade prepare
Step 3: Start NN2 active and NN1 as standby with rolling upgrade started
option.
Step 4: DN also restarted in upgrade mode and write files to hdfs
Step 5: Stop both Namenode and DN
Step 6: Restart NN2 as active and NN1 as standby with rolling upgrade
rollback option.
Issue:
=
On restarting NN1 as standby with rollback option , lastPromisedEpoch gets
updated and active NN2 is shutting down with following exception.
{noformat}
15/03/18 16:25:56 FATAL namenode.FSEditLog: Error: flush failed for required
journal (JournalAndStream(mgr=QJM to [XXX:8485, YYY:8485],
stream=QuorumOutputStream starting at txid 22))
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many
exceptions to achieve quorum size 2/2. 2 exceptions thrown:
XXX:8485: IPC's epoch 5 is less than the last promised epoch 6
at
org.apache.hadoop.hdfs.qjournal.server.Journal.checkRequest(Journal.java:418)
at
org.apache.hadoop.hdfs.qjournal.server.Journal.checkWriteRequest(Journal.java:446)
at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:341)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7891) A block placement policy with best rack failure tolerance
[
https://issues.apache.org/jira/browse/HDFS-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496882#comment-14496882
]
Zhe Zhang commented on HDFS-7891:
-
Thanks for the patch Walter! The patch looks good overall. Please find my
review below.
Logic:
# I think the current flow is functionally correct. But {{maxNodesPerRack}} no
longer denotes the maximum nodes to be allocated per rack, making the code less
readable. Right now {{chooseTargetInOrder}} treats {{maxNodesPerRack}} as the
number of replicas to allocate to each rack _in the first round_.
# If we simply let {{getMaxNodesPerRack()}} return the ceiling of
{{totalNumOfReplicas/numOfRacks}}, like {{(totalNumOfReplicas - 1) / numOfRacks
+ 1}}, then do not override {{chooseTargetInOrder()}}, can we get the right
distribution we want?
Nits:
# Override annotation missing in
{{BlockPlacementPolicyRackFaultTolarent#chooseTargetInOrder}}
# Could remove the empty constructor of
{{BlockPlacementPolicyRackFaultTolarent}}
# {{chooseOnce}| needs some Javadoc.
[~szetszwo]: Since this is targeted for trunk now, it would be great if you can
review the patch as well. Thanks!
A block placement policy with best rack failure tolerance
-
Key: HDFS-7891
URL: https://issues.apache.org/jira/browse/HDFS-7891
Project: Hadoop HDFS
Issue Type: New Feature
Components: namenode
Reporter: Walter Su
Assignee: Walter Su
Priority: Minor
Attachments: HDFS-7891.005.dup.patch, HDFS-7891.005.patch
a block placement policy tries its best to place replicas to most racks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8149) The footer of the Web UI Hadoop, 2014 is old
[ https://issues.apache.org/jira/browse/HDFS-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-8149: Resolution: Fixed Fix Version/s: 2.7.1 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk, branch-2, and branch-2.7. Thanks [~brahmareddy] for the contribution. The footer of the Web UI Hadoop, 2014 is old -- Key: HDFS-8149 URL: https://issues.apache.org/jira/browse/HDFS-8149 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Akira AJISAKA Assignee: Brahma Reddy Battula Fix For: 2.8.0, 2.7.1 Attachments: HDFS-8149.patch Need to be updated to 2015. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8152) Refactoring of lazy persist storage cases
[
https://issues.apache.org/jira/browse/HDFS-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496890#comment-14496890
]
Xiaoyu Yao commented on HDFS-8152:
--
Thanks [~arpitagarwal] for refactoring these tests. The patch looks good to me
with one minor issue: Can you reword the Precondition check message below.
Looks like the logic is to check against invalid values instead of setting them
both. Other than that, +1 pending Jenkins.
{code}
Preconditions.checkState(
ramDiskReplicaCapacity 0 || ramDiskStorageLimit == Long.MAX_VALUE,
Cannot specify both ramDiskReplicaCapacity and ramDiskStorageLimit);
{code}
Refactoring of lazy persist storage cases
-
Key: HDFS-8152
URL: https://issues.apache.org/jira/browse/HDFS-8152
Project: Hadoop HDFS
Issue Type: Improvement
Components: test
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Attachments: HDFS-8152.01.patch
Add a builder pattern to make it easier to test new configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8149) The footer of the Web UI Hadoop, 2014 is old
[ https://issues.apache.org/jira/browse/HDFS-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496905#comment-14496905 ] Hudson commented on HDFS-8149: -- FAILURE: Integrated in Hadoop-trunk-Commit #7593 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7593/]) HDFS-8149. The footer of the Web UI Hadoop, 2014 is old. Contributed by Brahma Reddy Battula. (aajisaka: rev de0f1700c150a819b38028c44ef1926507086e6c) * hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary/status.html * hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/explorer.html * hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/index.html * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/journal/index.html * hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html The footer of the Web UI Hadoop, 2014 is old -- Key: HDFS-8149 URL: https://issues.apache.org/jira/browse/HDFS-8149 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Akira AJISAKA Assignee: Brahma Reddy Battula Fix For: 2.8.0, 2.7.1 Attachments: HDFS-8149.patch Need to be updated to 2015. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp
[
https://issues.apache.org/jira/browse/HDFS-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8151:
Resolution: Fixed
Fix Version/s: 2.7.1
Status: Resolved (was: Patch Available)
Thanks for the review, Nicholas! I've committed this.
Always use snapshot path as source when invalid snapshot names are used for
diff based distcp
-
Key: HDFS-8151
URL: https://issues.apache.org/jira/browse/HDFS-8151
Project: Hadoop HDFS
Issue Type: Bug
Components: distcp
Affects Versions: 2.7.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Fix For: 2.7.1
Attachments: HDFS-8151.000.patch
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff
report computation on the target cluster fails
# there is modification happening in the target cluster thus
{{checkNoChange}} returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8120) Erasure coding: created util class to analyze striped block groups
[ https://issues.apache.org/jira/browse/HDFS-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-8120. - Resolution: Fixed Fix Version/s: HDFS-7285 Hadoop Flags: Reviewed I've committed this. Erasure coding: created util class to analyze striped block groups -- Key: HDFS-8120 URL: https://issues.apache.org/jira/browse/HDFS-8120 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Fix For: HDFS-7285 Attachments: HDFS-8120.000.patch, HDFS-8120.001.patch, HDFS-8120.002.patch, HDFS-8120.003.patch, HDFS-8120.004.patch, HDFS-8120.005.patch, HDFS-8120.006.patch, HDFS-8120.007.patch, HDFS-8120.008.patch The patch adds logic of calculating size of individual blocks in a striped block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8149) The footer of the Web UI Hadoop, 2014 is old
[
https://issues.apache.org/jira/browse/HDFS-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496812#comment-14496812
]
Hadoop QA commented on HDFS-8149:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725624/HDFS-8149.patch
against trunk revision fddd552.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/10280//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HDFS-Build/10280//console
This message is automatically generated.
The footer of the Web UI Hadoop, 2014 is old
--
Key: HDFS-8149
URL: https://issues.apache.org/jira/browse/HDFS-8149
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Akira AJISAKA
Assignee: Brahma Reddy Battula
Attachments: HDFS-8149.patch
Need to be updated to 2015.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8149) The footer of the Web UI Hadoop, 2014 is old
[ https://issues.apache.org/jira/browse/HDFS-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-8149: Target Version/s: 2.7.1 The footer of the Web UI Hadoop, 2014 is old -- Key: HDFS-8149 URL: https://issues.apache.org/jira/browse/HDFS-8149 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Akira AJISAKA Assignee: Brahma Reddy Battula Attachments: HDFS-8149.patch Need to be updated to 2015. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7934) Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN
[ https://issues.apache.org/jira/browse/HDFS-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7934: Summary: Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN (was: During Rolling upgrade rollback ,standby namenode startup fails.) Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN Key: HDFS-7934 URL: https://issues.apache.org/jira/browse/HDFS-7934 Project: Hadoop HDFS Issue Type: Bug Reporter: J.Andreina Assignee: J.Andreina Priority: Critical Attachments: HDFS-7934.1.patch, HDFS-7934.2.patch During Rolling upgrade rollback , standby namenode startup fails , while loading edits and when there is no local copy of edits created after upgrade ( which is already been removed by Active Namenode from journal manager and from Active's local). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8120) Erasure coding: created util class to analyze striped block groups
[ https://issues.apache.org/jira/browse/HDFS-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496817#comment-14496817 ] Zhe Zhang commented on HDFS-8120: - Thanks Jing for reviewing! Erasure coding: created util class to analyze striped block groups -- Key: HDFS-8120 URL: https://issues.apache.org/jira/browse/HDFS-8120 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Fix For: HDFS-7285 Attachments: HDFS-8120.000.patch, HDFS-8120.001.patch, HDFS-8120.002.patch, HDFS-8120.003.patch, HDFS-8120.004.patch, HDFS-8120.005.patch, HDFS-8120.006.patch, HDFS-8120.007.patch, HDFS-8120.008.patch The patch adds logic of calculating size of individual blocks in a striped block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7934) Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN
[ https://issues.apache.org/jira/browse/HDFS-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7934: Component/s: documentation Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN Key: HDFS-7934 URL: https://issues.apache.org/jira/browse/HDFS-7934 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: J.Andreina Assignee: J.Andreina Priority: Critical Fix For: 2.7.1 Attachments: HDFS-7934.1.patch, HDFS-7934.2.patch During Rolling upgrade rollback , standby namenode startup fails , while loading edits and when there is no local copy of edits created after upgrade ( which is already been removed by Active Namenode from journal manager and from Active's local). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7934) Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN
[ https://issues.apache.org/jira/browse/HDFS-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7934: Resolution: Fixed Fix Version/s: 2.7.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the fix, [~andreina]! The patch looks good to me. +1 I've already committed this to trunk, branch-2, and branch-2.7. Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN Key: HDFS-7934 URL: https://issues.apache.org/jira/browse/HDFS-7934 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: J.Andreina Assignee: J.Andreina Priority: Critical Fix For: 2.7.1 Attachments: HDFS-7934.1.patch, HDFS-7934.2.patch During Rolling upgrade rollback , standby namenode startup fails , while loading edits and when there is no local copy of edits created after upgrade ( which is already been removed by Active Namenode from journal manager and from Active's local). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7934) Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN
[ https://issues.apache.org/jira/browse/HDFS-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7934: Affects Version/s: 2.4.0 Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN Key: HDFS-7934 URL: https://issues.apache.org/jira/browse/HDFS-7934 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 2.4.0 Reporter: J.Andreina Assignee: J.Andreina Priority: Critical Fix For: 2.7.1 Attachments: HDFS-7934.1.patch, HDFS-7934.2.patch During Rolling upgrade rollback , standby namenode startup fails , while loading edits and when there is no local copy of edits created after upgrade ( which is already been removed by Active Namenode from journal manager and from Active's local). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7934) Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN
[ https://issues.apache.org/jira/browse/HDFS-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496863#comment-14496863 ] Hudson commented on HDFS-7934: -- FAILURE: Integrated in Hadoop-trunk-Commit #7592 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7592/]) HDFS-7934. Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN. Contributed by J. Andreina. (jing9: rev b172d03595d1591e7f542791224607d8c5fce3e2) * hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN Key: HDFS-7934 URL: https://issues.apache.org/jira/browse/HDFS-7934 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 2.4.0 Reporter: J.Andreina Assignee: J.Andreina Priority: Critical Fix For: 2.7.1 Attachments: HDFS-7934.1.patch, HDFS-7934.2.patch During Rolling upgrade rollback , standby namenode startup fails , while loading edits and when there is no local copy of edits created after upgrade ( which is already been removed by Active Namenode from journal manager and from Active's local). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8120) Erasure coding: created util class to analyze striped block groups
[ https://issues.apache.org/jira/browse/HDFS-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496787#comment-14496787 ] Jing Zhao commented on HDFS-8120: - +1 for the 008 patch. Thanks for the work, Zhe and Bo! Erasure coding: created util class to analyze striped block groups -- Key: HDFS-8120 URL: https://issues.apache.org/jira/browse/HDFS-8120 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: HDFS-8120.000.patch, HDFS-8120.001.patch, HDFS-8120.002.patch, HDFS-8120.003.patch, HDFS-8120.004.patch, HDFS-8120.005.patch, HDFS-8120.006.patch, HDFS-8120.007.patch, HDFS-8120.008.patch The patch adds logic of calculating size of individual blocks in a striped block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8152) Refactoring of lazy persist storage cases
[ https://issues.apache.org/jira/browse/HDFS-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8152: Attachment: HDFS-8152.01.patch Patch to simplify cluster initialization using builder. Refactoring of lazy persist storage cases - Key: HDFS-8152 URL: https://issues.apache.org/jira/browse/HDFS-8152 Project: Hadoop HDFS Issue Type: Improvement Components: test Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: HDFS-8152.01.patch Add a builder pattern to make it easier to test new configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8152) Refactoring of lazy persist storage cases
[ https://issues.apache.org/jira/browse/HDFS-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8152: Status: Patch Available (was: Open) Refactoring of lazy persist storage cases - Key: HDFS-8152 URL: https://issues.apache.org/jira/browse/HDFS-8152 Project: Hadoop HDFS Issue Type: Improvement Components: test Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: HDFS-8152.01.patch Add a builder pattern to make it easier to test new configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8149) The footer of the Web UI Hadoop, 2014 is old
[ https://issues.apache.org/jira/browse/HDFS-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496728#comment-14496728 ] Akira AJISAKA commented on HDFS-8149: - LGTM, +1 pending Jenkins. The footer of the Web UI Hadoop, 2014 is old -- Key: HDFS-8149 URL: https://issues.apache.org/jira/browse/HDFS-8149 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Akira AJISAKA Assignee: Brahma Reddy Battula Attachments: HDFS-8149.patch Need to be updated to 2015. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp
[
https://issues.apache.org/jira/browse/HDFS-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496748#comment-14496748
]
Hadoop QA commented on HDFS-8151:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725650/HDFS-8151.000.patch
against trunk revision 9e8309a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/10282//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HDFS-Build/10282//console
This message is automatically generated.
Always use snapshot path as source when invalid snapshot names are used for
diff based distcp
-
Key: HDFS-8151
URL: https://issues.apache.org/jira/browse/HDFS-8151
Project: Hadoop HDFS
Issue Type: Bug
Components: distcp
Affects Versions: 2.7.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Attachments: HDFS-8151.000.patch
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff
report computation on the target cluster fails
# there is modification happening in the target cluster thus
{{checkNoChange}} returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8152) Refactoring of lazy persist storage cases
Arpit Agarwal created HDFS-8152: --- Summary: Refactoring of lazy persist storage cases Key: HDFS-8152 URL: https://issues.apache.org/jira/browse/HDFS-8152 Project: Hadoop HDFS Issue Type: Improvement Components: test Reporter: Arpit Agarwal Assignee: Arpit Agarwal Add a builder pattern to make it easier to test new configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp
[
https://issues.apache.org/jira/browse/HDFS-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496783#comment-14496783
]
Hudson commented on HDFS-8151:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7591 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7591/])
HDFS-8151. Always use snapshot path as source when invalid snapshot names are
used for diff based distcp. Contributed by Jing Zhao. (jing9: rev
4c097e473bb1f18d1510deb61bae2bcb8c156f18)
*
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
*
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Always use snapshot path as source when invalid snapshot names are used for
diff based distcp
-
Key: HDFS-8151
URL: https://issues.apache.org/jira/browse/HDFS-8151
Project: Hadoop HDFS
Issue Type: Bug
Components: distcp
Affects Versions: 2.7.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Fix For: 2.7.1
Attachments: HDFS-8151.000.patch
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff
report computation on the target cluster fails
# there is modification happening in the target cluster thus
{{checkNoChange}} returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HDFS-8078: Status: Patch Available (was: Open) HDFS client gets errors trying to to connect to IPv6 DataNode - Key: HDFS-8078 URL: https://issues.apache.org/jira/browse/HDFS-8078 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Affects Versions: 2.6.0 Reporter: Nate Edel Assignee: Nate Edel Labels: ipv6 Attachments: HDFS-8078.1.patch 1st exception, on put: 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 2401:db00:1010:70ba:face:0:8:0:50010 at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) Appears to actually stem from code in DataNodeID which assumes it's safe to append together (ipaddr + : + port) -- which is OK for IPv4 and not OK for IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 Currently using InetAddress.getByName() to validate IPv6 (guava InetAddresses.forString has been flaky) but could also use our own parsing. (From logging this, it seems like a low-enough frequency call that the extra object creation shouldn't be problematic, and for me the slight risk of passing in bad input that is not actually an IPv4 or IPv6 address and thus calling an external DNS lookup is outweighed by getting the address normalized and avoiding rewriting parsing.) Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() --- 2nd exception (on datanode) 15/04/13 13:18:07 ERROR datanode.DataNode: dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: /2401:db00:11:d010:face:0:2f:0:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) at java.lang.Thread.run(Thread.java:745) Which also comes as client error -get: 2401 is not an IP string literal. This one has existing parsing logic which needs to shift to the last colon rather than the first. Should also be a tiny bit faster by using lastIndexOf rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8153) Error Message points to wrong parent directory in case of path component name length error
[
https://issues.apache.org/jira/browse/HDFS-8153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-8153:
---
Attachment: hdfs-8153.001.patch
With this patch applied the directory names are as expected.
$ ./bin/hdfs dfs -mkdir /user/hrt_qa
$ ./bin/hdfs dfs -mkdir /user/hrt_qa/FileNameLength
$ ./bin/hdfs dfs -mkdir /user/hrt_qa/FileNameLength/ReallyLongFileName_003_Fail
mkdir: The maximum path component name limit of ReallyLongFileName_003_Fail in
directory /user/hrt_qa/FileNameLength is exceeded: limit=20 length=27
$ ./bin/hdfs dfs -touchz /user/hrt_qa/FileNameLength/ReallyLongFileName_003_Fail
touchz: The maximum path component name limit of ReallyLongFileName_003_Fail in
directory /user/hrt_qa/FileNameLength is exceeded: limit=20 length=27
Error Message points to wrong parent directory in case of path component name
length error
--
Key: HDFS-8153
URL: https://issues.apache.org/jira/browse/HDFS-8153
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.5.2
Reporter: Anu Engineer
Assignee: Anu Engineer
Attachments: hdfs-8153.001.patch
If the name component length is greater than the permitted length, the error
message points to wrong parent directory for mkdir and touchz.
Here are examples where the parent directory name is in error message. In
this example dfs.namenode.fs-limits.max-component-length is set to 19.
{code}
hdfs dfs -mkdir /user/hrt_qa/FileNameLength/really_big_name_dir01
mkdir: The maximum path component name limit of really_big_name_dir01 in
directory /user/hrt_qa/ is exceeded: limit=19 length=21
{code}
The expected value for the directory was _/user/hrt_qa/FileNameLength_. The
same behavior is observed for touchz
{code}
hdfs dfs -touchz /user/hrt_qa/FileNameLength/really_big_name_0004
touchz: The maximum path component name limit of really_big_name_0004 in
directory /user/hrt_qa/ is exceeded: limit=19 length=20
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8152) Refactoring of lazy persist storage cases
[
https://issues.apache.org/jira/browse/HDFS-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497297#comment-14497297
]
Hadoop QA commented on HDFS-8152:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725665/HDFS-8152.01.patch
against trunk revision 4c097e4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/10283//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HDFS-Build/10283//console
This message is automatically generated.
Refactoring of lazy persist storage cases
-
Key: HDFS-8152
URL: https://issues.apache.org/jira/browse/HDFS-8152
Project: Hadoop HDFS
Issue Type: Improvement
Components: test
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Attachments: HDFS-8152.01.patch, HDFS-8152.02.patch
Add a builder pattern to make it easier to test new configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7891) A block placement policy with best rack failure tolerance
[
https://issues.apache.org/jira/browse/HDFS-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497096#comment-14497096
]
Hadoop QA commented on HDFS-7891:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725649/HDFS-7891.005.dup.patch
against trunk revision 9e8309a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestFileCreation
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/10281//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HDFS-Build/10281//console
This message is automatically generated.
A block placement policy with best rack failure tolerance
-
Key: HDFS-7891
URL: https://issues.apache.org/jira/browse/HDFS-7891
Project: Hadoop HDFS
Issue Type: New Feature
Components: namenode
Reporter: Walter Su
Assignee: Walter Su
Priority: Minor
Attachments: HDFS-7891.005.dup.patch, HDFS-7891.005.patch
a block placement policy tries its best to place replicas to most racks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp
[
https://issues.apache.org/jira/browse/HDFS-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8151:
Reporter: Sushmitha Sreenivasan (was: Jing Zhao)
Always use snapshot path as source when invalid snapshot names are used for
diff based distcp
-
Key: HDFS-8151
URL: https://issues.apache.org/jira/browse/HDFS-8151
Project: Hadoop HDFS
Issue Type: Bug
Components: distcp
Affects Versions: 2.7.0
Reporter: Sushmitha Sreenivasan
Assignee: Jing Zhao
Priority: Minor
Fix For: 2.7.1
Attachments: HDFS-8151.000.patch
This is a bug reported by [~ssreenivasan]:
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff
report computation on the target cluster fails
# there is modification happening in the target cluster thus
{{checkNoChange}} returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp
[
https://issues.apache.org/jira/browse/HDFS-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8151:
Description:
This is a bug reported by [~ssreenivasan]:
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff report
computation on the target cluster fails
# there is modification happening in the target cluster thus {{checkNoChange}}
returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
was:
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff report
computation on the target cluster fails
# there is modification happening in the target cluster thus {{checkNoChange}}
returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
Always use snapshot path as source when invalid snapshot names are used for
diff based distcp
-
Key: HDFS-8151
URL: https://issues.apache.org/jira/browse/HDFS-8151
Project: Hadoop HDFS
Issue Type: Bug
Components: distcp
Affects Versions: 2.7.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Fix For: 2.7.1
Attachments: HDFS-8151.000.patch
This is a bug reported by [~ssreenivasan]:
HDFS-8036 makes the diff-based distcp use snapshot path as the source. This
should also happen when
# invalid snapshot names are provided as distcp parameters thus the diff
report computation on the target cluster fails
# there is modification happening in the target cluster thus
{{checkNoChange}} returns false
In other cases like source and target FS are not DistributedFileSystem, we
should throw exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8036) Use snapshot path as source when using snapshot diff report in DistCp
[ https://issues.apache.org/jira/browse/HDFS-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8036: Reporter: Sushmitha Sreenivasan (was: Jing Zhao) Use snapshot path as source when using snapshot diff report in DistCp - Key: HDFS-8036 URL: https://issues.apache.org/jira/browse/HDFS-8036 Project: Hadoop HDFS Issue Type: Bug Components: distcp Affects Versions: 2.7.0 Reporter: Sushmitha Sreenivasan Assignee: Jing Zhao Fix For: 2.7.0 Attachments: HDFS-8036.000.patch, HDFS-8036.001.patch When using snapshot diff report for distcp (HDFS-7535), the semantic should be apply the diff to the target in order to sync the target with source@snapshot2. Therefore after syncing based on the snapshot diff report, we should append the name of snapshot2 to the original source path and use it as the new source name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7859) Erasure Coding: Persist EC schemas in NameNode
[ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497306#comment-14497306 ] Kai Zheng commented on HDFS-7859: - [~szetszwo], bq.Since we don't not yet support add/delete/update/rename schema operations, we don't need to persist anything in NN at this moment. We will support some of these schema operations down the road. We may persist schemas at that time. Sound good? Please note it's not true we don't need to persist anything in NN at this moment.. We had already persisted some hard-coded values that should be covered by a schema in the image. Without this, we will definitely need to revisit the image format change some time later. As I said above, it's flexible enough in the schema definition and if we persist the whole schema object in image, we would not likely need to change the image later. Please note this issue blocks many subsequent issues and I thought we still have enough time for them right before the merge happening. Erasure Coding: Persist EC schemas in NameNode -- Key: HDFS-7859 URL: https://issues.apache.org/jira/browse/HDFS-7859 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin Attachments: HDFS-7859.001.patch In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we persist EC schemas in NameNode centrally and reliably, so that EC zones can reference them by name efficiently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8152) Refactoring of lazy persist storage cases
[ https://issues.apache.org/jira/browse/HDFS-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497101#comment-14497101 ] Xiaoyu Yao commented on HDFS-8152: -- Thanks [~arpitagarwal] for the updates. +1 for V2 patch pending Jenkins. Refactoring of lazy persist storage cases - Key: HDFS-8152 URL: https://issues.apache.org/jira/browse/HDFS-8152 Project: Hadoop HDFS Issue Type: Improvement Components: test Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: HDFS-8152.01.patch, HDFS-8152.02.patch Add a builder pattern to make it easier to test new configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8155) Support OAuth2 authentication in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497105#comment-14497105 ] Kai Zheng commented on HDFS-8155: - Hello [~jakobhoman], Thanks for having this and the good thought. We're working on HADOOP-11817, where both JWT token and OAuth2 token are to be supported for Hadoop web thru a generic token representation and API by pluggable approach. We use [CloudFoundry|https://github.com/cloudfoundry/uaa] for the OAuth2 test. We'll post our initial patch in this week and I hope our work can meet with your need. We would be glad to help with the web HDFS case, would you mind our side working on this issue as well? We would definitely welcome your thoughts, ideas and reviews, considering your concrete OAuth2 token provider and cases. Thanks. Support OAuth2 authentication in WebHDFS Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HDFS-8078: Status: Open (was: Patch Available) HDFS client gets errors trying to to connect to IPv6 DataNode - Key: HDFS-8078 URL: https://issues.apache.org/jira/browse/HDFS-8078 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Affects Versions: 2.6.0 Reporter: Nate Edel Assignee: Nate Edel Labels: ipv6 Attachments: HDFS-8078.1.patch 1st exception, on put: 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 2401:db00:1010:70ba:face:0:8:0:50010 at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) Appears to actually stem from code in DataNodeID which assumes it's safe to append together (ipaddr + : + port) -- which is OK for IPv4 and not OK for IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 Currently using InetAddress.getByName() to validate IPv6 (guava InetAddresses.forString has been flaky) but could also use our own parsing. (From logging this, it seems like a low-enough frequency call that the extra object creation shouldn't be problematic, and for me the slight risk of passing in bad input that is not actually an IPv4 or IPv6 address and thus calling an external DNS lookup is outweighed by getting the address normalized and avoiding rewriting parsing.) Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() --- 2nd exception (on datanode) 15/04/13 13:18:07 ERROR datanode.DataNode: dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: /2401:db00:11:d010:face:0:2f:0:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) at java.lang.Thread.run(Thread.java:745) Which also comes as client error -get: 2401 is not an IP string literal. This one has existing parsing logic which needs to shift to the last colon rather than the first. Should also be a tiny bit faster by using lastIndexOf rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HDFS-8078: Attachment: HDFS-8078.1.patch Now with unit test, and the one bad import removed. HDFS client gets errors trying to to connect to IPv6 DataNode - Key: HDFS-8078 URL: https://issues.apache.org/jira/browse/HDFS-8078 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Affects Versions: 2.6.0 Reporter: Nate Edel Assignee: Nate Edel Labels: ipv6 Attachments: HDFS-8078.1.patch 1st exception, on put: 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 2401:db00:1010:70ba:face:0:8:0:50010 at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) Appears to actually stem from code in DataNodeID which assumes it's safe to append together (ipaddr + : + port) -- which is OK for IPv4 and not OK for IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 Currently using InetAddress.getByName() to validate IPv6 (guava InetAddresses.forString has been flaky) but could also use our own parsing. (From logging this, it seems like a low-enough frequency call that the extra object creation shouldn't be problematic, and for me the slight risk of passing in bad input that is not actually an IPv4 or IPv6 address and thus calling an external DNS lookup is outweighed by getting the address normalized and avoiding rewriting parsing.) Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() --- 2nd exception (on datanode) 15/04/13 13:18:07 ERROR datanode.DataNode: dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: /2401:db00:11:d010:face:0:2f:0:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) at java.lang.Thread.run(Thread.java:745) Which also comes as client error -get: 2401 is not an IP string literal. This one has existing parsing logic which needs to shift to the last colon rather than the first. Should also be a tiny bit faster by using lastIndexOf rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HDFS-8078: Attachment: (was: HDFS-8078.patch) HDFS client gets errors trying to to connect to IPv6 DataNode - Key: HDFS-8078 URL: https://issues.apache.org/jira/browse/HDFS-8078 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Affects Versions: 2.6.0 Reporter: Nate Edel Assignee: Nate Edel Labels: ipv6 Attachments: HDFS-8078.1.patch 1st exception, on put: 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 2401:db00:1010:70ba:face:0:8:0:50010 at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) Appears to actually stem from code in DataNodeID which assumes it's safe to append together (ipaddr + : + port) -- which is OK for IPv4 and not OK for IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 Currently using InetAddress.getByName() to validate IPv6 (guava InetAddresses.forString has been flaky) but could also use our own parsing. (From logging this, it seems like a low-enough frequency call that the extra object creation shouldn't be problematic, and for me the slight risk of passing in bad input that is not actually an IPv4 or IPv6 address and thus calling an external DNS lookup is outweighed by getting the address normalized and avoiding rewriting parsing.) Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() --- 2nd exception (on datanode) 15/04/13 13:18:07 ERROR datanode.DataNode: dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: /2401:db00:11:d010:face:0:2f:0:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) at java.lang.Thread.run(Thread.java:745) Which also comes as client error -get: 2401 is not an IP string literal. This one has existing parsing logic which needs to shift to the last colon rather than the first. Should also be a tiny bit faster by using lastIndexOf rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497288#comment-14497288 ] Kai Zheng commented on HDFS-7337: - Thanks [~vinayrpet] for the comments, suggestions and more options. Before to decide which way to go, I thought it would make sense to figure out first the following questions: * What possible erasure codes or codecs we would have, for now and the future? XOR, RS, HitchHiker, LRC, and even more, typical codes from broad industry experiences. * What kinds of schema parameters it would have for each possible erasure codec? Let's slow down and let me find some time for the further investigation. With such questions well answered, I thought it would not be hard to tell which way sounds better, creating schema in command line or thru a schema definition file. Configurable and pluggable Erasure Codec and schema --- Key: HDFS-7337 URL: https://issues.apache.org/jira/browse/HDFS-7337 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Kai Zheng Attachments: HDFS-7337-prototype-v1.patch, HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip, PluggableErasureCodec-v2.pdf, PluggableErasureCodec-v3.pdf, PluggableErasureCodec.pdf According to HDFS-7285 and the design, this considers to support multiple Erasure Codecs via pluggable approach. It allows to define and configure multiple codec schemas with different coding algorithms and parameters. The resultant codec schemas can be utilized and specified via command tool for different file folders. While design and implement such pluggable framework, it’s also to implement a concrete codec by default (Reed Solomon) to prove the framework is useful and workable. Separate JIRA could be opened for the RS codec implementation. Note HDFS-7353 will focus on the very low level codec API and implementation to make concrete vendor libraries transparent to the upper layer. This JIRA focuses on high level stuffs that interact with configuration, schema and etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8070) ShortCircuitShmManager goes into dead mode, stopping all operations
[
https://issues.apache.org/jira/browse/HDFS-8070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497287#comment-14497287
]
Colin Patrick McCabe commented on HDFS-8070:
Both Hadoop 2.7 and the Haodop 2 branch (which is what I assume you mean by
Hadoop 2.8) have HDFS-7915. So I think there should not be any compatibility
issues on that front.
Can you check whether the patch up at HADOOP-11802 solves your issue? At very
least, it should get you a more informative exception.
ShortCircuitShmManager goes into dead mode, stopping all operations
---
Key: HDFS-8070
URL: https://issues.apache.org/jira/browse/HDFS-8070
Project: Hadoop HDFS
Issue Type: Bug
Components: caching
Affects Versions: 2.8.0
Reporter: Gopal V
Assignee: Kihwal Lee
HDFS ShortCircuitShm layer keeps the task locked up during multi-threaded
split-generation.
I hit this immediately after I upgraded the data, so I wonder if the
ShortCircuitShim wire protocol has trouble when 2.8.0 DN talks to a 2.7.0
Client?
{code}
2015-04-06 00:04:30,780 INFO [ORC_GET_SPLITS #3] orc.OrcInputFormat: ORC
pushdown predicate: leaf-0 = (IS_NULL ss_sold_date_sk)
expr = (not leaf-0)
2015-04-06 00:04:30,781 ERROR [ShortCircuitCache_SlotReleaser]
shortcircuit.ShortCircuitCache: ShortCircuitCache(0x29e82045): failed to
release short-circuit shared memory slot Slot(slotIdx=2,
shm=DfsClientShm(a86ee34576d93c4964005d90b0d97c38)) by sending
ReleaseShortCircuitAccessRequestProto to /grid/0/cluster/hdfs/dn_socket.
Closing shared memory segment.
java.io.IOException: ERROR_INVALID: there is no shared memory segment
registered with shmId a86ee34576d93c4964005d90b0d97c38
at
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-04-06 00:04:30,781 INFO [ORC_GET_SPLITS #5] orc.OrcInputFormat: ORC
pushdown predicate: leaf-0 = (IS_NULL ss_sold_date_sk)
expr = (not leaf-0)
2015-04-06 00:04:30,781 WARN [ShortCircuitCache_SlotReleaser]
shortcircuit.DfsClientShmManager: EndpointShmManager(172.19.128.60:50010,
parent=ShortCircuitShmManager(5e763476)): error shutting down shm: got
IOException calling shutdown(SHUT_RDWR)
java.nio.channels.ClosedChannelException
at
org.apache.hadoop.util.CloseableReferenceCount.reference(CloseableReferenceCount.java:57)
at
org.apache.hadoop.net.unix.DomainSocket.shutdown(DomainSocket.java:387)
at
org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.shutdown(DfsClientShmManager.java:378)
at
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:223)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-04-06 00:04:30,783 INFO [ORC_GET_SPLITS #7] orc.OrcInputFormat: ORC
pushdown predicate: leaf-0 = (IS_NULL cs_sold_date_sk)
expr = (not leaf-0)
2015-04-06 00:04:30,785 ERROR [ShortCircuitCache_SlotReleaser]
shortcircuit.ShortCircuitCache: ShortCircuitCache(0x29e82045): failed to
release short-circuit shared memory slot Slot(slotIdx=4,
shm=DfsClientShm(a86ee34576d93c4964005d90b0d97c38)) by sending
ReleaseShortCircuitAccessRequestProto to /grid/0/cluster/hdfs/dn_socket.
Closing shared memory segment.
java.io.IOException: ERROR_INVALID: there is no shared memory segment
registered with shmId a86ee34576d93c4964005d90b0d97c38
at
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208)
at
[jira] [Commented] (HDFS-7859) Erasure Coding: Persist EC schemas in NameNode
[ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497447#comment-14497447 ] Tsz Wo Nicholas Sze commented on HDFS-7859: --- ... We had already persisted some hard-coded values that should be covered by a schema ... What do you mean? Could you give an example? ... Please note this issue blocks many subsequent issues and I thought we still have enough time for them right before the merge happening. What are the subsequent issues? Erasure Coding: Persist EC schemas in NameNode -- Key: HDFS-7859 URL: https://issues.apache.org/jira/browse/HDFS-7859 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin Attachments: HDFS-7859.001.patch In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we persist EC schemas in NameNode centrally and reliably, so that EC zones can reference them by name efficiently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8157) Writes to RAM DISK reserve locked memory for block files
Arpit Agarwal created HDFS-8157: --- Summary: Writes to RAM DISK reserve locked memory for block files Key: HDFS-8157 URL: https://issues.apache.org/jira/browse/HDFS-8157 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Arpit Agarwal Assignee: Arpit Agarwal Per discussion on HDFS-6919, the first step is that writes to RAM disk will reserve locked memory via the FsDatasetCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Define some system schemas in codes
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497472#comment-14497472 ] Kai Zheng commented on HDFS-8156: - Hi [~zhz], [~vinayrpet], would you take a look at this? Please check if this is good enough for our following-on issues. Thanks. Define some system schemas in codes --- Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch This is to define and add some system schemas in codes, and also resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8033:
Attachment: HDFS-8033.001.patch
This patch implements stateful read by overriding {{blockSeekTo}} and
{{readBuffer}} from {{DFSInputStream}}. Basically, {{blockSeekTo}} takes care
of transitioning to the _next block group_ while {{readBuffer}} handles
switching _among internal blocks_ in the current block group.
It updates the logic of handling the following states to fit the striping
layout:
# {{pos}}: The extended {{readBuffer}} calls {{super.readBuffer()}}, which
forwards {{pos}} as usual.
# {{blockEnd}}: This state controls when {{blockSeekTo}} is called. So it is
maintained to be the end of the block group.
# {{blockReader}}: Similar to how HDFS-7889 handles {{streamer}}, this patch
keeps replacing {{blockReader}} with one of the {{blockReaders}} that it
creates at every {{blockSeekTo}} call. Then {{super.readBuffer()}} just uses
the selected {{blockReader}}.
# {{currentNode}} and {{currentLocatedBlock}} are used to record bad nodes
and block locations. They will be handled when read failures are supported
(HDFS-7678)
Erasure coding: stateful (non-positional) read from files in striped layout
---
Key: HDFS-8033
URL: https://issues.apache.org/jira/browse/HDFS-8033
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497386#comment-14497386 ] Kai Zheng commented on HDFS-7866: - I agree this moves out, and will resolve some TODOs left for this in HDFS-8156. Thanks. Erasure coding: NameNode manages EC schemas --- Key: HDFS-7866 URL: https://issues.apache.org/jira/browse/HDFS-7866 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, HDFS-7866-v3.patch This is to extend NameNode to load, list and sync predefine EC schemas in authorized and controlled approach. The provided facilities will be used to implement DFSAdmin commands so admin can list available EC schemas, then could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7859) Erasure Coding: Persist EC schemas in NameNode
[ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497485#comment-14497485 ] Kai Zheng commented on HDFS-7859: - bq.What do you mean? Could you give an example? Well, my last said was bad and inaccurate. After double checking related codes, I saw only stripped blocks derived from the following hard-coded values are persisted in the image. So please ignore the saying. bq.What are the subsequent issues? We do have some and will sort them out later. I have opened HDFS-8156 to resolve some deps caused by HDFS-7866, originally planned to be done here. Erasure Coding: Persist EC schemas in NameNode -- Key: HDFS-7859 URL: https://issues.apache.org/jira/browse/HDFS-7859 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin Attachments: HDFS-7859.001.patch In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we persist EC schemas in NameNode centrally and reliably, so that EC zones can reference them by name efficiently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[
https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497484#comment-14497484
]
Hadoop QA commented on HDFS-8078:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725733/HDFS-8078.1.patch
against trunk revision 1b89a3e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHDFS
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/10285//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HDFS-Build/10285//console
This message is automatically generated.
HDFS client gets errors trying to to connect to IPv6 DataNode
-
Key: HDFS-8078
URL: https://issues.apache.org/jira/browse/HDFS-8078
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 2.6.0
Reporter: Nate Edel
Assignee: Nate Edel
Labels: ipv6
Attachments: HDFS-8078.1.patch
1st exception, on put:
15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception
java.lang.IllegalArgumentException: Does not contain a valid host:port
authority: 2401:db00:1010:70ba:face:0:8:0:50010
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153)
at
org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
Appears to actually stem from code in DataNodeID which assumes it's safe to
append together (ipaddr + : + port) -- which is OK for IPv4 and not OK for
IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which
requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010
Currently using InetAddress.getByName() to validate IPv6 (guava
InetAddresses.forString has been flaky) but could also use our own parsing.
(From logging this, it seems like a low-enough frequency call that the extra
object creation shouldn't be problematic, and for me the slight risk of
passing in bad input that is not actually an IPv4 or IPv6 address and thus
calling an external DNS lookup is outweighed by getting the address
normalized and avoiding rewriting parsing.)
Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress()
---
2nd exception (on datanode)
15/04/13 13:18:07 ERROR datanode.DataNode:
dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown
operation src: /2401:db00:20:7013:face:0:7:0:54152 dst:
/2401:db00:11:d010:face:0:2f:0:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226)
at java.lang.Thread.run(Thread.java:745)
Which also comes as client error -get: 2401 is not an IP string literal.
This one has existing parsing logic which needs to shift to the last colon
rather than the first. Should also be a tiny bit faster by using lastIndexOf
rather than split. Could alternatively use the techniques above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7369) Erasure coding: distribute recovery work for striped blocks to DataNode
[
https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497518#comment-14497518
]
Kai Zheng commented on HDFS-7369:
-
When I work on related following on issues, just noticed
{{BlockECRecoveryInfo#liveBlockIndices}} is used for good blocks in the block
group. I thought it would be better to have {{erasedBlockIndices}} instead,
since most time we would have only one block erased. Agree?
Erasure coding: distribute recovery work for striped blocks to DataNode
---
Key: HDFS-7369
URL: https://issues.apache.org/jira/browse/HDFS-7369
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Fix For: HDFS-7285
Attachments: HDFS-7369-000-part1.patch, HDFS-7369-000-part2.patch,
HDFS-7369-001.patch, HDFS-7369-002.patch, HDFS-7369-003.patch,
HDFS-7369-004.patch
This JIRA updates NameNode to handle background / offline recovery of erasure
coded blocks. It includes 2 parts:
# Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to
appropriate priority levels.
# Update {{ReplicationMonitor}} to distinguish block codec tasks and send a
new DataNode command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7369) Erasure coding: distribute recovery work for striped blocks to DataNode
[
https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497526#comment-14497526
]
Jing Zhao commented on HDFS-7369:
-
You need {{liveBlockIndices}} to reconstruct the ID of each healthy block seen
by the source DataNode.
Erasure coding: distribute recovery work for striped blocks to DataNode
---
Key: HDFS-7369
URL: https://issues.apache.org/jira/browse/HDFS-7369
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Fix For: HDFS-7285
Attachments: HDFS-7369-000-part1.patch, HDFS-7369-000-part2.patch,
HDFS-7369-001.patch, HDFS-7369-002.patch, HDFS-7369-003.patch,
HDFS-7369-004.patch
This JIRA updates NameNode to handle background / offline recovery of erasure
coded blocks. It includes 2 parts:
# Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to
appropriate priority levels.
# Update {{ReplicationMonitor}} to distinguish block codec tasks and send a
new DataNode command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8113) NullPointerException in BlockInfoContiguous causes block report failure
[
https://issues.apache.org/jira/browse/HDFS-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497606#comment-14497606
]
Chengbing Liu commented on HDFS-8113:
-
Hi [~qwertymaniac] and [~atm], this is one of the test sequences I did
yesterday, but was still not able to reproduce the issue. The problem is that
if you delete the file, the block will not be in {{blocksMap}}, then we won't
be able to reproduce it.
To reproduce this, we must make sure that the {{blockInfo}} is in {{blocksMap}}
and {{blockInfo.bc == null}}. I tried several test sequences with no luck.
I just tried cleaning the rbw directory and restarted the DataNode. However,
the problem still exists. Maybe you have ideas about this?
And [~cmccabe], are you suggesting the patch here is ok or we should
additionally check nullity for each {{storedBlock.getBlockCollection()}}?
NullPointerException in BlockInfoContiguous causes block report failure
---
Key: HDFS-8113
URL: https://issues.apache.org/jira/browse/HDFS-8113
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.6.0
Reporter: Chengbing Liu
Assignee: Chengbing Liu
Attachments: HDFS-8113.patch
The following copy constructor can throw NullPointerException if {{bc}} is
null.
{code}
protected BlockInfoContiguous(BlockInfoContiguous from) {
this(from, from.bc.getBlockReplication());
this.bc = from.bc;
}
{code}
We have observed that some DataNodes keeps failing doing block reports with
NameNode. The stacktrace is as follows. Though we are not using the latest
version, the problem still exists.
{quote}
2015-03-08 19:28:13,442 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
RemoteException in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.(BlockInfo.java:80)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockToMarkCorrupt.(BlockManager.java:1696)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.checkReplicaCorrupt(BlockManager.java:2185)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReportedBlock(BlockManager.java:2047)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1950)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1823)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1750)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1069)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:152)
at
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:26382)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1623)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8156) Define some system schemas in codes
Kai Zheng created HDFS-8156: --- Summary: Define some system schemas in codes Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng This is to define and add some system schemas in codes, and also resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7891) A block placement policy with best rack failure tolerance
[ https://issues.apache.org/jira/browse/HDFS-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497428#comment-14497428 ] Walter Su commented on HDFS-7891: - bq. If we simply let getMaxNodesPerRack() return the ceiling of totalNumOfReplicas/numOfRacks, like (totalNumOfReplicas - 1) / numOfRacks + 1, then do not override chooseTargetInOrder(), can we get the right distribution we want? If totalNumOfReplicas numOfRacks, it's much simpler, maxNodesPerRack==1. It's usually the case in production cluster. The tricky thing is, when totalNumOfReplicas numOfRacks. *Notation* R = total racks in cluster X = total expected replicas Q=Math.floor(X/R) T=X%R if XR, X racks have 1 replica. if XR T==0, R racks have Q replicas. if XR T!=0, X-T racks have Q replicas. T racks have Q+1 replicas. The tricky thing is, 1. if XR T!=0, X-T racks have Q replicas. T racks have Q+1 replicas. *We want local rack in the T racks group*. So we have to override chooseTargetInOrder() anyway. 2. if XR T!=0, and if _we simply let getMaxNodesPerRack() return the ceiling of totalNumOfReplicas/numOfRacks_, there is a chance that more than T racks have Q+1 replicas while some racks have none. A block placement policy with best rack failure tolerance - Key: HDFS-7891 URL: https://issues.apache.org/jira/browse/HDFS-7891 Project: Hadoop HDFS Issue Type: New Feature Components: namenode Reporter: Walter Su Assignee: Walter Su Priority: Minor Attachments: HDFS-7891.005.dup.patch, HDFS-7891.005.patch a block placement policy tries its best to place replicas to most racks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7863) Missing description of parameter fsd in javadoc
[
https://issues.apache.org/jira/browse/HDFS-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497471#comment-14497471
]
Brahma Reddy Battula commented on HDFS-7863:
Test case failures are unrelated this patch..
Missing description of parameter fsd in javadoc
Key: HDFS-7863
URL: https://issues.apache.org/jira/browse/HDFS-7863
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Yongjun Zhang
Assignee: Brahma Reddy Battula
Priority: Minor
Attachments: HDFS-7863-002.patch, HDFS-7863-003.patch, HDFS-7863.patch
HDFS-7573 did refactoring of delete() code. New parameter {{FSDirectory fsd}}
is added to resulted methods, but the javadoc is not updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497589#comment-14497589
]
Kai Zheng commented on HDFS-7949:
-
The patch looks good. Besides what's commented by Zhe above, ref. below codes,
the assertion isn't clear to me.
{code}
+ assertTrue(
+ Wrongly computed file size contains striped blocks, file status:
+ + fileStatus, fileStatus.contains(length\:13107200));
{code}
bq.can we use a null for the schema here? (when calling
{{createErasureCodingZone}})
Yes, null is allowed, and the system default schema will be used.
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7891) A block placement policy with best rack failure tolerance
[ https://issues.apache.org/jira/browse/HDFS-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497437#comment-14497437 ] Walter Su commented on HDFS-7891: - bq. We want local rack in the T racks group. Sorry, I need to correct myself. Actually it doesn't matter in normal case because of pipeline. It might be useful in specific like EC because every replica is an individual block. I still need to override chooseTargetInOrder() for 2nd reason. A block placement policy with best rack failure tolerance - Key: HDFS-7891 URL: https://issues.apache.org/jira/browse/HDFS-7891 Project: Hadoop HDFS Issue Type: New Feature Components: namenode Reporter: Walter Su Assignee: Walter Su Priority: Minor Attachments: HDFS-7891.005.dup.patch, HDFS-7891.005.patch a block placement policy tries its best to place replicas to most racks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Define some system schemas in codes
[
https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497463#comment-14497463
]
Kai Zheng commented on HDFS-8156:
-
In {{ErasureCodingZoneManager}}, it serializes and saves EC schema object into
Xattr, which dosen't follow our related discussion. We should only save schema
name into the Xattr associated with the EC Zone. This will fix the issue as
well.
Define some system schemas in codes
---
Key: HDFS-8156
URL: https://issues.apache.org/jira/browse/HDFS-8156
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
This is to define and add some system schemas in codes, and also resolve some
TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further
discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8136) Client gets and uses EC schema when reads and writes a stripping file
[
https://issues.apache.org/jira/browse/HDFS-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497501#comment-14497501
]
Kai Zheng commented on HDFS-8136:
-
{{DFSStripedInputStream}} extends {{DFSInputStream}} which contains
{{DFSClient}}. I thought that's good enough.
Client gets and uses EC schema when reads and writes a stripping file
-
Key: HDFS-8136
URL: https://issues.apache.org/jira/browse/HDFS-8136
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Zheng
Assignee: Kai Sasaki
Discussed with [~umamaheswararao] and [~vinayrpet], in client when reading
and writing a stripping file, it can invoke a separate call to NameNode to
request the EC schema associated with the EC zone where the file is in. Then
the schema can be used to guide the reading and writing. Currently it uses
hard-coded values.
Optionally, as an optimization consideration, client may cache schema info
per file or per zone or per schema name. We could add schema name in
{{HdfsFileStatus}} for that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Define some system schemas in codes
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497389#comment-14497389 ] Kai Zheng commented on HDFS-8156: - It will also help resolve the deps of HDFS-7866 for various subsequent issues, as HDFS-7866 is moved out. Define some system schemas in codes --- Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng This is to define and add some system schemas in codes, and also resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8136) Client gets and uses EC schema when reads and writes a stripping file
[
https://issues.apache.org/jira/browse/HDFS-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497482#comment-14497482
]
Kai Sasaki commented on HDFS-8136:
--
I have a question about how to obtain schema information from NN.
{{DFSClient}} is assumed to use {{DFSStripedInputStream}} after opening file.
Which is better to pass {{ECSchema}} from {{DFSClient}} or to restore
{{ECSchema}} inside {{DFSStripedInputStream}} itself? The reason why I have
this question is that {{DFSStripedInputStream}} has not reference to
{{ClientProtocol}} now. I think these requests to NN should be delegated to
{{DFSClient}} because it is already responsible to handle {{ClientProtocol}}.
Thank you.
Client gets and uses EC schema when reads and writes a stripping file
-
Key: HDFS-8136
URL: https://issues.apache.org/jira/browse/HDFS-8136
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Zheng
Assignee: Kai Sasaki
Discussed with [~umamaheswararao] and [~vinayrpet], in client when reading
and writing a stripping file, it can invoke a separate call to NameNode to
request the EC schema associated with the EC zone where the file is in. Then
the schema can be used to guide the reading and writing. Currently it uses
hard-coded values.
Optionally, as an optimization consideration, client may cache schema info
per file or per zone or per schema name. We could add schema name in
{{HdfsFileStatus}} for that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8158) when try to roll one key which not exist in kms ,will have nullpointer Exception
huangyitian created HDFS-8158: - Summary: when try to roll one key which not exist in kms ,will have nullpointer Exception Key: HDFS-8158 URL: https://issues.apache.org/jira/browse/HDFS-8158 Project: Hadoop HDFS Issue Type: Bug Components: encryption Affects Versions: 2.6.0 Reporter: huangyitian Assignee: J.Andreina Priority: Minor Test Step: 1.try to roll one key which is not existed in kms: ./hadoop key roll hyt Test reslt: will have a nullPointer Exception in Linux consol: vm-204:/opt/OpenSource/install/hadoop/namenode/bin # ./hadoop key roll hyt 15/04/16 11:58:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Rolling key version from KeyProvider: KMSClientProvider[http://9.91.8.204:16000/kms/v1/] for key name: hyt java.lang.NullPointerException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:485) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:443) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.rollNewVersionInternal(KMSClientProvider.java:649) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.rollNewVersion(KMSClientProvider.java:660) at org.apache.hadoop.crypto.key.KeyShell$RollCommand.execute(KeyShell.java:347) at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:515) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7993) Incorrect descriptions in fsck when nodes are decommissioned
[
https://issues.apache.org/jira/browse/HDFS-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-7993:
-
Attachment: HDFS-7993.4.patch
Thanks [~mingma] and [~vinayrpet] for the review and comments.
bq.Maybe we can change the description from repl to live repl? It will address
the confusion others might have.
Agree with this point. Have updated the patch accordingly.
bq.It will be useful to show stale block content replica
Updated the patch to provide information whether stale replica is due to
either Stale Datanode or Stale block content.
Please review the patch.
Incorrect descriptions in fsck when nodes are decommissioned
Key: HDFS-7993
URL: https://issues.apache.org/jira/browse/HDFS-7993
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Ming Ma
Assignee: J.Andreina
Attachments: HDFS-7993.1.patch, HDFS-7993.2.patch, HDFS-7993.3.patch,
HDFS-7993.4.patch
When you run fsck with -files or -racks, you will get something like
below if one of the replicas is decommissioned.
{noformat}
blk_x len=y repl=3 [dn1, dn2, dn3, dn4]
{noformat}
That is because in NamenodeFsck, the repl count comes from live replicas
count; while the actual nodes come from LocatedBlock which include
decommissioned nodes.
Another issue in NamenodeFsck is BlockPlacementPolicy's verifyBlockPlacement
verifies LocatedBlock that includes decommissioned nodes. However, it seems
better to exclude the decommissioned nodes in the verification; just like how
fsck excludes decommissioned nodes when it check for under replicated blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497364#comment-14497364 ] Kai Zheng commented on HDFS-7949: - Sorry for late on this. I will take a look at the patch today. Thanks for working on this! WebImageViewer need support file size calculation with striped blocks - Key: HDFS-7949 URL: https://issues.apache.org/jira/browse/HDFS-7949 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hui Zheng Assignee: Rakesh R Priority: Minor Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, HDFS-7949-003.patch, HDFS-7949-004.patch The file size calculation should be changed when the blocks of the file are striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8152) Refactoring of lazy persist storage cases
[
https://issues.apache.org/jira/browse/HDFS-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497382#comment-14497382
]
Hadoop QA commented on HDFS-8152:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725690/HDFS-8152.02.patch
against trunk revision 1b89a3e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/10284//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HDFS-Build/10284//console
This message is automatically generated.
Refactoring of lazy persist storage cases
-
Key: HDFS-8152
URL: https://issues.apache.org/jira/browse/HDFS-8152
Project: Hadoop HDFS
Issue Type: Improvement
Components: test
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Attachments: HDFS-8152.01.patch, HDFS-8152.02.patch
Add a builder pattern to make it easier to test new configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Define some system schemas in codes
[
https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497459#comment-14497459
]
Kai Zheng commented on HDFS-8156:
-
It will also fix the following test failure:
In {{TestErasureCodingZones}}
org.junit.ComparisonFailure: Default schema should be returned
Expected :RS-6-3
Actual :SYS-DEFAULT-RS-6-3
Define some system schemas in codes
---
Key: HDFS-8156
URL: https://issues.apache.org/jira/browse/HDFS-8156
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
This is to define and add some system schemas in codes, and also resolve some
TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further
discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8156) Define some system schemas in codes
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-8156: Attachment: HDFS-8156-v1.patch Uploaded a patch for the purposes and also fixing the related issues. Define some system schemas in codes --- Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch This is to define and add some system schemas in codes, and also resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7369) Erasure coding: distribute recovery work for striped blocks to DataNode
[
https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497563#comment-14497563
]
Kai Zheng commented on HDFS-7369:
-
Thanks Jing, I see. I thought healthy block IDs are constructed in NameNode
side before sending out.
Erasure coding: distribute recovery work for striped blocks to DataNode
---
Key: HDFS-7369
URL: https://issues.apache.org/jira/browse/HDFS-7369
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Fix For: HDFS-7285
Attachments: HDFS-7369-000-part1.patch, HDFS-7369-000-part2.patch,
HDFS-7369-001.patch, HDFS-7369-002.patch, HDFS-7369-003.patch,
HDFS-7369-004.patch
This JIRA updates NameNode to handle background / offline recovery of erasure
coded blocks. It includes 2 parts:
# Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to
appropriate priority levels.
# Update {{ReplicationMonitor}} to distinguish block codec tasks and send a
new DataNode command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8153) Error Message points to wrong parent directory in case of path component name length error
Anu Engineer created HDFS-8153:
--
Summary: Error Message points to wrong parent directory in case of
path component name length error
Key: HDFS-8153
URL: https://issues.apache.org/jira/browse/HDFS-8153
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.5.2
Reporter: Anu Engineer
Assignee: Anu Engineer
If the name component length is greater than the permitted length, the error
message points to wrong parent directory for mkdir and touchz.
Here are examples where the parent directory name is in error message. In this
example dfs.namenode.fs-limits.max-component-length is set to 19.
{code}
hdfs dfs -mkdir /user/hrt_qa/FileNameLength/really_big_name_dir01
mkdir: The maximum path component name limit of really_big_name_dir01 in
directory /user/hrt_qa/ is exceeded: limit=19 length=21
{code}
The expected value for the directory was /user/hrt_qa/FileNameLength. The same
behavior is observed for touchz
{code}
hdfs dfs -touchz /user/hrt_qa/FileNameLength/really_big_name_0004
touchz: The maximum path component name limit of really_big_name_0004 in
directory /user/hrt_qa/ is exceeded: limit=19 length=20
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8153) Error Message points to wrong parent directory in case of path component name length error
[
https://issues.apache.org/jira/browse/HDFS-8153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-8153:
---
Description:
If the name component length is greater than the permitted length, the error
message points to wrong parent directory for mkdir and touchz.
Here are examples where the parent directory name is in error message. In this
example dfs.namenode.fs-limits.max-component-length is set to 19.
{code}
hdfs dfs -mkdir /user/hrt_qa/FileNameLength/really_big_name_dir01
mkdir: The maximum path component name limit of really_big_name_dir01 in
directory /user/hrt_qa/ is exceeded: limit=19 length=21
{code}
The expected value for the directory was _/user/hrt_qa/FileNameLength_. The
same behavior is observed for touchz
{code}
hdfs dfs -touchz /user/hrt_qa/FileNameLength/really_big_name_0004
touchz: The maximum path component name limit of really_big_name_0004 in
directory /user/hrt_qa/ is exceeded: limit=19 length=20
{code}
was:
If the name component length is greater than the permitted length, the error
message points to wrong parent directory for mkdir and touchz.
Here are examples where the parent directory name is in error message. In this
example dfs.namenode.fs-limits.max-component-length is set to 19.
{code}
hdfs dfs -mkdir /user/hrt_qa/FileNameLength/really_big_name_dir01
mkdir: The maximum path component name limit of really_big_name_dir01 in
directory /user/hrt_qa/ is exceeded: limit=19 length=21
{code}
The expected value for the directory was /user/hrt_qa/FileNameLength. The same
behavior is observed for touchz
{code}
hdfs dfs -touchz /user/hrt_qa/FileNameLength/really_big_name_0004
touchz: The maximum path component name limit of really_big_name_0004 in
directory /user/hrt_qa/ is exceeded: limit=19 length=20
{code}
Error Message points to wrong parent directory in case of path component name
length error
--
Key: HDFS-8153
URL: https://issues.apache.org/jira/browse/HDFS-8153
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.5.2
Reporter: Anu Engineer
Assignee: Anu Engineer
If the name component length is greater than the permitted length, the error
message points to wrong parent directory for mkdir and touchz.
Here are examples where the parent directory name is in error message. In
this example dfs.namenode.fs-limits.max-component-length is set to 19.
{code}
hdfs dfs -mkdir /user/hrt_qa/FileNameLength/really_big_name_dir01
mkdir: The maximum path component name limit of really_big_name_dir01 in
directory /user/hrt_qa/ is exceeded: limit=19 length=21
{code}
The expected value for the directory was _/user/hrt_qa/FileNameLength_. The
same behavior is observed for touchz
{code}
hdfs dfs -touchz /user/hrt_qa/FileNameLength/really_big_name_0004
touchz: The maximum path component name limit of really_big_name_0004 in
directory /user/hrt_qa/ is exceeded: limit=19 length=20
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7966) New Data Transfer Protocol via HTTP/2
[ https://issues.apache.org/jira/browse/HDFS-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497003#comment-14497003 ] Haohui Mai commented on HDFS-7966: -- There is one proposal from Qianqian Shi in the GSoC system. I don't know the details of the process, but it looks like it is being reviewed. [~stack] can you please help on it? Thanks. New Data Transfer Protocol via HTTP/2 - Key: HDFS-7966 URL: https://issues.apache.org/jira/browse/HDFS-7966 Project: Hadoop HDFS Issue Type: New Feature Reporter: Haohui Mai Assignee: Qianqian Shi Labels: gsoc, gsoc2015, mentor The current Data Transfer Protocol (DTP) implements a rich set of features that span across multiple layers, including: * Connection pooling and authentication (session layer) * Encryption (presentation layer) * Data writing pipeline (application layer) All these features are HDFS-specific and defined by implementation. As a result it requires non-trivial amount of work to implement HDFS clients and servers. This jira explores to delegate the responsibilities of the session and presentation layers to the HTTP/2 protocol. Particularly, HTTP/2 handles connection multiplexing, QoS, authentication and encryption, reducing the scope of DTP to the application layer only. By leveraging the existing HTTP/2 library, it should simplify the implementation of both HDFS clients and servers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
